Cucumber & Behave JSON report

This guide explains the basic structure of a Cucumber & Behave JSON report, helping you understand how AgileTest reads and displays report data. Additionally, you can learn how to customize the report to add attachment details.

1. Common structure of Cucumber & Behave report

Below is the common structure of the Cucumber and Behave JSON report.

Since both frameworks are similar in many ways, their report also hold much resemblance in their structures. Both types of report contain basic blocks, namely Feature, Scenario, Given, When, and Then.

Cucumber report example

CODE

[
  {
    "uri": "features/sample.feature",
    "id": "sample-feature",
    "keyword": "Feature",
    "name": "Sample Feature",
    "description": "This is a sample feature file.",
    "line": 1,
    "elements": [
      {
        "id": "sample-feature;sample-scenario",
        "keyword": "Scenario",
        "name": "Sample Scenario",
        "description": "",
        "line": 3,
        "type": "scenario",
        "steps": [
          {
            "keyword": "Given ",
            "name": "I have a Cucumber test",
            "line": 4,
            "match": {
              "location": "sample_steps.rb:10"
            },
            "result": {
              "status": "passed",
              "duration": 1000000
            }
          },
          {
            "keyword": "When ",
            "name": "I run the test",
            "line": 5,
            "match": {
              "location": "sample_steps.rb:15"
            },
            "result": {
              "status": "passed",
              "duration": 2000000
            }
          },
          {
            "keyword": "Then ",
            "name": "I should see a report",
            "line": 6,
            "match": {
              "location": "sample_steps.rb:20"
            },
            "result": {
              "status": "failed",
              "error_message": "Expected report to be visible",
              "duration": 1500000
            }
          }
        ]
      }
    ]
  }
]

Behave report example

CODE

[
  {
    "keyword": "Feature",
    "name": "Sample Feature",
    "description": "This is a sample feature.",
    "status": "passed",
    "elements": [
      {
        "keyword": "Scenario",
        "name": "Sample Scenario",
        "description": "",
        "tags": ["@tag1"],
        "steps": [
          {
            "keyword": "Given ",
            "step_type": "given",
            "name": "I have a Behave test",
            "status": "passed",
            "duration": 0.001
          },
          {
            "keyword": "When ",
            "step_type": "when",
            "name": "I run the test",
            "status": "passed",
            "duration": 0.002
          },
          {
            "keyword": "Then ",
            "step_type": "then",
            "name": "I should see a report",
            "status": "failed",
            "error_message": "Expected to see a report but did not.",
            "duration": 0.001
          }
        ]
      }
    ]
  }
]

Once you understand the structure of a Cucumber or Behave report, the next step is to see how AgileTest interprets this data when the report is imported. AgileTest reads specific elements in the JSON file to create test cases and determine their execution status.

2. How AgileTest would parse a Cucumber or Behave report

For Cucumber and Behave, users must include the issue key in the report before uploading it to AgileTest. Otherwise, the import will fail.

Since Agile Test doesn’t create new tests in case of importing Gherkin reports, only test case result and attachments will be parsed and attached to pre-existing tests in your instance.

Each report file, after execution, contains xUnit status values automatically mapped to AgileTest Test Results. Therefore, you do not need extra setup to read test results.

Below are explanations for the supported status of the xUnit reports in AgileTest.

Prioritization rule for determining the test status in AgileTest:

If multiple tags are added to a single <test> element, the tag with the highest priority will be used as the test status in AgileTest.
The priority of status tags in AgileTest is as follows: FAILED > BLOCKED > RETEST > SKIPPED > QUERY > PASSED.

Each step has a result status. Test Case status is calculated based on the total combination of status properties of the step tag. The following table shows how status values are mapped to AgileTest execution statuses.

Test case status	Description
`pass`, `passed`	PASSED → Test case yields the expected result.
`fail`, `failed`, `failure`or `error`	FAILED → Test case returns the actual result, not as expected.
`blocked`	BLOCKED → Test case is considered blocked due to a specific reason.
`skipped`	SKIPPED→ Test case is skipped in the test run.
`retest`	RETEST → Test case should be rerun again.
`query`	QUERY → The test case requires more details to have the final result.
`todo`, `to do`, or non-defined statuses	TODO → Test case is not yet run. It’s the default status of every Test case in AgileTest.

3. Test Result with attachments

By default, many reports do not include attachments. This section explains how to add attachments to your test reports.

To allow AgileTest to recognize attachments in the report, you need to follow a specific JSON element structure.

CODE

 "embeddings": [
              {
                "data": "<BASE64_ENCODED_FILE>",
                "mime_type": "<mime type>"
              }
            ]

For example, you can put this Element structure into your report like this:

CODE

[
  {
    "elements": [
      {
        ...
        "steps": [
          {
            ...
            "embeddings": [
              {
                "data": "SGVsbG8gd29ybGQ=",
                "mime_type": "text/plain"
              }
            ]
          }
        ]
      }
    ]
  }
]

Should you need any assistance or further AgileTest inquiries, please contact us here!