Robot framework XML report

This guide explains the basic structure of a Robot Framework XML report, helping you understand how AgileTest reads and displays report data. Additionally, you can learn how to customize the report to add attachment details.

1. Common structure of Robot Framework report

Below is the common structure of the Robot Framework XML report.

In the Robot Framework report, the structure is organized starting from the <robot> element. Inside the report, the <statistics> element provides a summary of test statistics; <errors> shows error counts and <suite> elements, each representing a test suite. Inside each suite, there are <testcase> elements, which contain individual test results, including their <status> and <message> elements.

In a basic Robot Framework report, you should find common tags depicting test suites and test cases.

<?xml version="1.0" encoding="UTF-8"?>
<robot generator="Robot Framework 6.1.1" generated="20241210 10:00:00">
  <suite name="SampleTestSuite" source="SampleTestSuite.robot">
    <status status="FAIL" starttime="20241210 09:58:00.000" endtime="20241210 10:00:00.000">
      <message>Test suite failed due to test case failures.</message>
    </status>
    <test name="Test Case 1">
      <status status="PASS" starttime="20241210 09:58:10.000" endtime="20241210 09:58:20.000"/>
    </test>
    <test name="Test Case 2">
      <status status="FAIL" starttime="20241210 09:58:30.000" endtime="20241210 09:58:40.000">
        <message>Error: Expected value '42', but got '41'.</message>
      </status>
    </test>
    <test name="Test Case 3">
      <status status="SKIP" starttime="20241210 09:59:00.000" endtime="20241210 09:59:10.000">
        <message>Test skipped due to a precondition failure.</message>
      </status>
    </test>
  </suite>
  <statistics>
    <total>
      <stat pass="1" fail="1" skip="1" total="3"/>
    </total>
    <tag>
      <stat name="Critical" pass="1" fail="1" skip="1"/>
    </tag>
  </statistics>
  <errors>
    <msg timestamp="20241210 09:58:50.000">
      Some setup steps failed, causing subsequent tests to fail or skip.
    </msg>
  </errors>
</robot>

Once you understand the structure of a Robot Framework report, the next step is to see how AgileTest interprets this data when the report is imported. AgileTest reads specific elements in the XML file to create test cases and determine their execution status.

2. How AgileTest would parse a Robot Framework report

Imported test cases are distinguished by the name attributes of the <suite> and <test> elements, following the format: name of <suite> element + '.' + name of <test> element.

This combination is referred to as the Test Definition in Agile Test and can be found just below the Description area of the test case ticket. The Summary of each test case issue is based on the value of the name attribute.

Each report file, after execution, contains Robot Framework status values automatically mapped to AgileTest Test Results. Therefore, you do not need extra setup to read test results.

Below are explanations for the supported status of the Robot Framework reports in AgileTest.

Prioritization rule for determining the test status in AgileTest:

• If multiple tags are added to a single <test> element, the tag with the highest priority will be used as the test status in AgileTest.

• The priority of status tags in AgileTest is as follows: FAILED > BLOCKED > RETEST > SKIPPED > QUERY > PASSED.

In the Robot Framework report, test case status is calculated based on the status properties of the <status> of the testtag. In Robot XML report, executed test cases are marked with <status> elements containing the status attribute inside blocks of <kw>.

<kw name="Open Browser" library="SeleniumLibrary">
  <arguments>
    <arg>https://example.com</arg>
    <arg>Chrome</arg>
  </arguments>
  <status status="PASS" starttime="20241220 10:01:05.000" endtime="20241220 10:01:10.000"/>
</kw>

The following table shows how Robot Framework status values are mapped to AgileTest execution statuses.

3. Test Result with attachments

By default, many reports do not include attachments. This section explains how to add attachments to your test reports.

AgileTest supports 2 customized structures for integrating attachments into your Robot Framework test reports. Users can choose either option to upload test results along with evidence files.

3.1 Attachment element with name attribute and content inside

To allow AgileTest to recognize attachments in the report, you need to follow a specific XML element structure.

<attachment name="filename"> base64 content </attachment>

For example, you can put this Element structure into your report like this:

<test>
  ....
  <attachments>
    <attachment name="test.png">iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAAAQ0lEQVR42u3PQREAAAgDINc/9Mzg14MGZNrOAxERERERERERERERERERERERERERERERERERERERERERERERERERuVh9TJWdbWjzsQAAAABJRU5ErkJggg==</attachment>
  </attachments>
</test>

3.2 Attachment element with separate name and content elements inside

To allow AgileTest to recognize attachments in the report, you can add this element structure:

<attachment> 
  <name>filename</name>
  <rawcontent> base64 content </rawcontent>
</attachment>

For example, your report will look like:

<test>
  ....
  <attachments>
    <attachment> 
      <name>test.png</name>
      <rawcontent>iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAYAAAAeP4ixAAAAQ0lEQVR42u3PQREAAAgDINc/9Mzg14MGZNrOAxERERERERERERERERERERERERERERERERERERERERERERERERERuVh9TJWdbWjzsQAAAABJRU5ErkJggg==</rawcontent>
    </attachment>
  </attachments>
</test>

Should you need any assistance or further AgileTest inquiries, please contact us here!