TestNG XML report

About TestNG

TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as annotations, flexible test configurations, support for data-driven testing, and so much more.

Common structure

In a basic TestNG report, you should find common tags depicting test suites and test cases.

TestNG report example

XML

<?xml version="1.0" encoding="UTF-8"?>
<testng-results skipped="1" failed="1" total="5" passed="3">
  <suite name="SampleTestSuite" duration-ms="1200" started-at="2024-12-10T10:00:00Z" finished-at="2024-12-10T10:00:01Z">
    <test name="SampleTest" duration-ms="1200" started-at="2024-12-10T10:00:00Z" finished-at="2024-12-10T10:00:01Z">
      <class name="com.example.tests.SampleTests">
        <test-method name="testMethod1" status="PASS" duration-ms="200" started-at="2024-12-10T10:00:00Z" finished-at="2024-12-10T10:00:00Z"/>
        <test-method name="testMethod2" status="FAIL" duration-ms="300" started-at="2024-12-10T10:00:00Z" finished-at="2024-12-10T10:00:00Z">
          <exception class="java.lang.AssertionError">
            <message><![CDATA[Expected value did not match actual value]]></message>
            <stack-trace><![CDATA[
              at com.example.tests.SampleTests.testMethod2(SampleTests.java:25)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            ]]></stack-trace>
          </exception>
        </test-method>
        <test-method name="testMethod3" status="SKIP" duration-ms="0" started-at="2024-12-10T10:00:01Z" finished-at="2024-12-10T10:00:01Z">
          <exception class="java.lang.Exception">
            <message><![CDATA[Skipped due to dependent test failure]]></message>
          </exception>
        </test-method>
        <test-method name="testMethod4" status="PASS" duration-ms="300" started-at="2024-12-10T10:00:01Z" finished-at="2024-12-10T10:00:01Z"/>
        <test-method name="testMethod5" status="PASS" duration-ms="400" started-at="2024-12-10T10:00:01Z" finished-at="2024-12-10T10:00:01Z"/>
      </class>
    </test>
  </suite>
</testng-results>

To visualize the structure more clearly, let’s take a look at the diagram below.

image (6)-20241210-083312.png — TestNG test report structure diagram

How AgileTest would parse a TestNG report

In AgileTest, all test cases which are imported via automation feature are designated as Generic tests.

Imported Test cases differ from each other by name attributes of <class> element and <test-method> element parsed from the XML file. The combination of these attributes is called Test definition in AgileTest. User should find it right under the Description area of Test case tickets. The Summary of each Test case issue will be the value of name attribute.

Supported status tags

Test run status tag	Description
`<pass>`, `<passed>` or no tag	PASSED → Test case yields the expected result.
`<failed>`, `<fail>`, or `<error>`	FAILED → Test case returns the actual result not as expected.
`<blocked>`	BLOCKED → Test case is considered blocked due to a specific reason.
`<skipped>`	SKIPPED → Test case is skipped in the test run.
`<retest>`	RETEST → Test case should be rerun again.
`<query>`	QUERY → The test case requires more details to have the final result.
`<todo>`, `<to do>`, or non-defined statuses	TODO → Test case is not yet run. It’s the default status of every Test case in AgileTest.

The priority of status tags in AgileTest is as follows:

failed > blocked > retest > skipped > query > passed (or no tag).

If multiple tags are added to a single <test> element, the tag with the highest priority will be used as the test status in Agile Test.

Test result with attachments

AgileTest supports 2 customized structure to integrate attachments into your test reports. Users could use either one to upload test result along with evidence files.

1. Attachment element with name attribute and content inside

Element structure

XML

<attachment name="filename"> base64 content </attachment>

Example

XML

<test-method>
  ....
  <attachments>
    <attachment name="test.png">iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAAAQ0lEQVR42u3PQREAAAgDINc/9Mzg14MGZNrOAxERERERERERERERERERERERERERERERERERERERERERERERERERuVh9TJWdbWjzsQAAAABJRU5ErkJggg==</attachment>
  </attachments>
</test-method>

2. Attachment element with separate name and content elements inside

Element structure

XML

<attachment> 
  <name>filename</name>
  <rawcontent> base64 content </rawcontent>
</attachment>

Example

XML

<test-method>
  ....
  <attachments>
    <attachment> 
      <name>test.png</name>
      <rawcontent>iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAAAQ0lEQVR42u3PQREAAAgDINc/9Mzg14MGZNrOAxERERERERERERERERERERERERERERERERERERERERERERERERERuVh9TJWdbWjzsQAAAABJRU5ErkJggg==</rawcontent>
    </attachment>
  </attachments>
</test-method>

Should you need any assistance or further AgileTest inquiries, please contact us here!