Note to self: testing test result processing with groovy in jenkins, use a jenkins job with the groovy in

In jenkins we have test jobs which produce junit test result .xml files from python tests.
We use jenkins junit plugin to harvest results.
We use some groovy to produce test report in with PostBuild "Execute a set of scripts"
The .groovy script is checked out from source control in the test job workspace
  "Groovy script files to evaluate"
  File Path: $WORKSPACE/testscripts/testreport.groovy

I sort of don't know what programming language I program in any more as it is a mix of bash python groovy go and more! Anyway being able to test changes is required. Testing in the past was done by fiddling with the test jobs using vi to edit the groovy and using dummy test results but that is a bit painful. Editing file in source-controlled area is troublematic too.

Making a separate job using actual test results and with the ,groovy editable in job configuration in PostBuild Groovy script makes changing and testing the test report script easier. A bit obvious maybe.

The same code works in Postbuild Groovy as in PostBuild "Execute a set of scripts" "Groovy script files to evaluate".

Test job looks something like:

http://jenkinsserver/job/zTest_testreport.groovy/

Execute shell:

# copy junit test results from a job with an interesting set of results 

          # touch the results every time so jenkins will harvest them

FWS=/var/lib/jenkins/jobs/testjob/workspace
if ! ls junitresult*.xml ; then
  rsync -avzhP $FWS/junitresult*.xml ./
  touch junitresult*.xml
else
  touch junitresult*.xml
fi
rm testreport.html
echo "tesht wershion" > VERSION.txt

Execute system groovy script:

// NOTE: the objects available here are not the same as in the PostBuild groovy

          // the manager object is not available
          // just showing example here how to set build description

import hudson.model.*
// groovy.lang.MissingPropertyException: No such property: manager for class: Script1
//def ws = manager.build.workspace.getRemote()
def currentBuild = Thread.currentThread().executable
def ws = currentBuild.workspace.getRemote()
String desc = new File(ws + "/VERSION.txt").text
currentBuild.setDescription(desc)

Post-build Actions

Publish JUnit test result report: junitresult*.xml

Groovy Postbuild (script as below here):

Archive the artifacts, Files to archive: testreport*.html

import hudson.model.*

def jobList3 = [ "test2", "test3" ]
def build = Thread.currentThread().executable
// THIS testreport.groovy script is invoked in jenkins Post-build Actions
//  after Publish JUnit testresult report
workspace = build.getEnvVars()["WORKSPACE"]
reportfilename = workspace + "/testreport.html"
rf = new File(reportfilename);
def testCount = "0"
def testPassed = "0"
def testFailed = "0"
def testSkipped = "0"
def buildDuration = "0"
def workspace = "unknown"
def buildName = "unknown"
def BUILD_STATUS = ""
def BUILD_URL = ""
def testResult = null
def testResult1 = null
def testResult2 = null
def testDuration = ""
def caseResult = null
def buildNumber = 0
def buildNumHash = ""
def buildTimeString = ""
def rooturl = ""
try {
    buildNumber = build.number
    buildNumHash = build.getDisplayName()
    buildTimeString = build.getTime().format("YYYY-MMM-dd HH:mm:ss")
    if(build.testResultAction) {
        testResult = build.testResultAction
        testCount = String.format("%d",(testResult.totalCount))
        testPassed = String.format("%d",(testResult.result.passCount))
        testFailed = String.format("%d",(testResult.result.failCount))
        testSkipped = String.format("%d",(testResult.result.skipCount))
        testDuration = String.format("%.2f",(testResult.result.duration ))
    }
    workspace = build.getEnvVars()["WORKSPACE"]
    buildName = build.getEnvVars()["JOB_NAME"]
    BUILD_STATUS = build.getEnvVars()["BUILD_STATUS"]
    BUILD_URL = build.getEnvVars()["BUILD_URL"]
    // if(build.testResultAction) {}
    testResult1 = hudson.tasks.junit.TestResult
    testResult2 = build.getAction(hudson.tasks.junit.TestResultAction.class)
    caseResult = hudson.tasks.junit.CaseResult
    rooturl = manager.hudson.rootUrl
} catch(Exception ex) {
    rf << "exception accessing build.testResultAction object.";
    //rf << ex;
}
def thisTestReportSummary = String.format("%s,%s,%s,%s",testCount,testPassed,testFailed,testSkipped)
def previousTestReport = ""
def previousTestReportSummary = ""
def previousTestReportLine = ""
if (buildName.endsWith("_Retry")) {
    previousJobName = buildName - "_Retry"
    previousJobWorkspace = workspace - "_Retry"
    previousTestReport = previousJobWorkspace + "/testreport.html"
    File pf = new File( previousTestReport )
    if( pf.exists() ) {
        // find original testreport.html, get original test total,pass,fail,skip
        pf.eachLine { line ->
            if (line.trim()) {
                if (line.contains("TEST RESULT: ")) {
                    previousTestReportLine = line
                    println previousTestReportLine
                    // TEST RESULT: 1 total, 1 pass, 0 fail, 0 skip.

                    l = line.replaceAll("[^0-9][^0-9]*"," ")
                    a = l.split()
                    pCount = Integer.parseInt(a[0])
                    pPassed = Integer.parseInt(a[1])
                    pFailed = Integer.parseInt(a[2])
                    pSkipped = Integer.parseInt(a[3])
                    previousTestReportSummary = String.format("%d,%d,%d,%d",pCount,pPassed,pFailed,pSkipped)
                    println previousTestReportSummary
                    testCount = String.format("%d",(Integer.parseInt(testCount) + pCount))
                    testPassed = String.format("%d",(Integer.parseInt(testPassed) + pPassed))
                    testFailed = String.format("%d",(Integer.parseInt(testFailed) + pFailed))
                    testSkipped = String.format("%d",(Integer.parseInt(testSkipped) + pSkipped))
                }
            }
        }
    }
}
//def newTestReportSummary = String.format("%d,%d,%d,%d",testCount,testPassed,testFailed,testSkipped)
def newTestReportSummary = String.format("%s,%s,%s,%s",testCount,testPassed,testFailed,testSkipped)
// in groovy the write RE-creates the file, rf << "whatever" is used to append.
rf.write ""
rf << "Summary test report

\n\
TEST RESULT: $testCount total, $testPassed pass, $testFailed fail, $testSkipped skip.
\n\
Workspace : $workspace
\n\
Project Name : $buildName $buildNumHash

\n\
"

if (previousTestReportLine != "") {
   rf << "
\n\
This test report summary (tot,pass,fail,skip) : $thisTestReportSummary
\n\
Previous test report summary (tot,pass,fail,skip) : $previousTestReportSummary
\n\
New test report summary (tot,pass,fail,skip) : $newTestReportSummary
\n\
Previous test report : $previousTestReport

\n\
"
}

if (build) {
    rf << """<!-- GENERAL INFO \n\
\n\
\n\
  \n\
  \n\
  \n\
  \n\
  \n\
  \n\
  \n\

\n\     \n\       \n\         \n\       \n\   \n\         \n\       \n\       \n\         \n\       \n\     \n\	BUILD ${build.result}
Build URL	${rooturl}${build.url}
Project:	${buildName}
Date of build:	${buildTimeString}
Build duration:	${build.durationString}
Test duration:	${testDuration}
Build description:	${build.description}

\n\

\n\
"""
}
if(!testResult) {
    rf << "
No test result
"
    rf << "

-->"    println "testreport.groovy no test result
"
    return ("No test result")
}
// it.JUnitTestResult, it is email-ext plugin object, is a reference to the ScriptContentBuildWrapper class.
//def junitResultList = it.JUnitTestResult
def junitResultList = [];
junitResultList.add(testResult.getResult())
if (junitResultList.size() > 0) {
    rf << "
test result from build.testResultAction"
} else {
    junitResultList.add(testResult2.getResult())
    if (junitResultList.size() > 0) {
        rf << "
test result from build.getAction"
    } else {
        rf << "
No results in 'testResult2'
\n"
        junitResultList.add(testResult1.getResult())
    }
}
//rf << "
DEBUG" + junitResultList.size() + " test items"
// API: http://hudson-ci.org/javadoc/hudson/tasks/junit/PackageResult.html
rf << "\n"
if (junitResultList.size() > 0) {
    rf << '
\n'
    rf << " \n"
    junitResultList.each { junitResult ->
        junitResult.getChildren().each { packageResult ->
            rf << " \n"
            packageResult.getChildren().each { suite ->
                suite.getChildren().each { test ->
                    def colour = "lightgreen"
                    def highlight1=""
                    def highlight2=""
                    def moremessage1=""
                    RESULT = test.getStatus().name() // FAILED or PASSED or SKIPPED (.name() not .value)
                    // hudson.tasks.junit.CaseResult.Status.FAILED
                    if (RESULT == "PASSED") {
                        moremessage1 += "
test.getSkippedMessage():" + test.getSkippedMessage() + ""
                        moremessage1 += "
test.getStatus().getMessage():" + test.getStatus().getMessage() + ""
                        moremessage1 += "
test.getTitle():" + test.getTitle() + ""
                        moremessage1 += "
test.getStdout():" + test.getStdout() + ""
                        moremessage1 += "
test.getStderr():" + test.getStderr() + ""
                        moremessage1 += "
test.getErrorDetails():" + test.getErrorDetails() + ""
                        moremessage1 += "
test.getErrorStackTrace():" + test.getErrorStackTrace() + ""
                        moremessage1 += "
test.getName():" + test.getName() + ""
                        moremessage1 += "
test.getSafeName():" + test.getSafeName() + ""
                        moremessage1 += "
test.getSimpleName():" + test.getSimpleName() + ""
                        moremessage1 += "
test.getFullName():" + test.getFullName() + ""
                        moremessage1 += "
test.getClassName():" + test.getClassName() + ""
                        moremessage1 += "
test.getDisplayName():" + test.getDisplayName() + ""
                        moremessage1 += "
test.getPackageName():" + test.getPackageName() + ""
                    }

                    if (RESULT == "FAILED" || RESULT == "REGRESSION") {
                        colour = "#ffcccc"
                        highlight1=""
                        highlight2=""
                        moremessage1 += "
test.getSkippedMessage():" + test.getSkippedMessage() + ""
                        moremessage1 += "
test.getStatus().getMessage():" + test.getStatus().getMessage() + ""
                        moremessage1 += "
test.getTitle():" + test.getTitle() + ""
                        moremessage1 += "
test.getStdout():" + test.getStdout() + ""
                        moremessage1 += "
test.getStderr():" + test.getStderr() + ""
                        moremessage1 += "
test.getErrorDetails():" + test.getErrorDetails() + ""
                        moremessage1 += "
test.getErrorStackTrace():" + test.getErrorStackTrace() + ""
                        moremessage1 += "
test.getName():" + test.getName() + ""
                        moremessage1 += "
test.getSafeName():" + test.getSafeName() + ""
                        moremessage1 += "
test.getSimpleName():" + test.getSimpleName() + ""
                        moremessage1 += "
test.getFullName():" + test.getFullName() + ""
                        moremessage1 += "
test.getClassName():" + test.getClassName() + ""
                        moremessage1 += "
test.getDisplayName():" + test.getDisplayName() + ""
                        moremessage1 += "
test.getPackageName():" + test.getPackageName() + ""
                    }
                    if (RESULT == "SKIPPED") {
                        colour = "#ffffb3"
                        // getSkippedMessage is in junit-plugin]$ less ./src/main/java/hudson/tasks/junit/CaseResult.java
                        // but not documented in http://hudson-ci.org/javadoc/hudson/tasks/junit/CaseResult.html#getSuiteResult()
                        moremessage1 += "
test.getSkippedMessage():" + test.getSkippedMessage() + ""
                        moremessage1 += "
test.getStatus().getMessage():" + test.getStatus().getMessage() + ""
                        moremessage1 += "
test.getTitle():" + test.getTitle() + ""
                        moremessage1 += "
test.getStdout():" + test.getStdout() + ""
                        moremessage1 += "
test.getStderr():" + test.getStderr() + ""
                        moremessage1 += "
test.getErrorDetails():" + test.getErrorDetails() + ""
                        moremessage1 += "
test.getErrorStackTrace():" + test.getErrorStackTrace() + ""
                        moremessage1 += "
test.getName():" + test.getName() + ""
                        moremessage1 += "
test.getSafeName():" + test.getSafeName() + ""
                        moremessage1 += "
test.getSimpleName():" + test.getSimpleName() + ""
                        moremessage1 += "
test.getFullName():" + test.getFullName() + ""
                        moremessage1 += "
test.getClassName():" + test.getClassName() + ""
                        moremessage1 += "
test.getDisplayName():" + test.getDisplayName() + ""
                        moremessage1 += "
test.getPackageName():" + test.getPackageName() + ""
                        // float getDuration() Gets the duration of the test, in seconds
                    }
       
                    rf << " \n"
                }
            }
        }
    }
    rf << '

${junitResultList.first().displayName}

TEST SUITE: ${packageResult.getName()} Failed: ${packageResult.getFailCount()} test(s), Passed: ${packageResult.getPassCount()} test(s), Skipped: ${packageResult.getSkipCount()} test(s), Total: ${packageResult.getPassCount()+packageResult.getFailCount()+packageResult.getSkipCount()} test(s)

${highlight1} ${RESULT}: ${test.getFullName()} ${moremessage1} ${highlight2}

\n'
}
rf << "testreport.groovy\n"
println "testreport.groovy"
// groovy doesn't implement the close.!?
//rf.close()

Some work on improving test reports test counts of late . . .
Stackoverflow.com proving somewhat helpful but found more questions I could answer in the jenkins PostBuild .groovy plugin area.

I asked a question and worked out the answer the following 2 days with the help of the new test testreport job.
https://stackoverflow.com/questions/59631365/in-groovy-script-in-jenkins-test-result-how-can-we-get-information-on-skipped-te/59668777#59668777

In groovy script in jenkins test result how can we get information on skipped tests to show if test was expectedFailure(pass or fail) or Skipped?

Ask Question

Asked 2 days ago

Active today

Viewed 21 times

0

The jenkins test result GUI shows skipped or expectedFail tests as skipped. The individual test view for skipped or expectedFail tests shows "Skipped Message" and "Standard Output"

e.g. "Skipped Message" can be:

• a custom skip message

      *  e.g. from python @unittest.skip("some reason") tag or 
      *  e.g. raise unittest.SkipTest("thing not found.")

• "expected test failure"
• "xfail-marked test passes unexpectedly"

http://doc.pytest.org/en/latest/skipping.html

We are using groovy script to generate test reports. We would like to include more information about skipped tests rather than just "skipped". How can we get info on the skipped test like in the "Skipped Message" in the GUI view?

The jenkins API is documented here:

https://javadoc.jenkins.io/plugin/junit/hudson/tasks/junit/TestResult.html

The answer is:

Call the undocumented test.getSkippedMessage() method. 

py.test writes skip message into junit xml. A tag like this inside the test result: 

That message can be a custom message from message in skip() call, or in case of xfail or xpass the message is set by pytest to "expected test failure" or "xfail-marked test passes unexpectedly".

jenkins junit plugin reads the junit xml. The message is available through the API here: class CaseResult getSkippedMessage() see https://github.com/jenkinsci/junit-plugin/blob/master/src/main/java/hudson/tasks/junit/CaseResult.java Although it is not documented here: http://hudson-ci.org/javadoc/hudson/tasks/junit/CaseResult.html

More questions in the jenkins PostBuild .groovy plugin area . . . .

https://stackoverflow.com/questions/15482846/can-i-make-skipped-tests-more-prominent-in-the-jenkins-dashboard/59632047?noredirect=1#comment105465764_59632047

https://stackoverflow.com/questions/50598794/how-to-get-jenkins-build-monitor-plugin-to-display-badges/59663098#59663098

https://stackoverflow.com/questions/38592457/why-does-py-test-show-my-xfail-tests-as-skipped/59666952#59666952