Test Results

In Controller > Test Results you can view the test results for all your models and scripts to make sure that your tests are running correctly. In addition, you can:

• Determine which of your tests have passed, failed, or are in progress.
• View the execution steps, logs, and screenshots for the selected test run.
• See which execution environment was used for each test run.
• See which system under test (SUT) was used for each test run.
• Delete test results that are no longer required.
• Abort tests that are in progress.

Test Results showing all Run Statuses

Filtering

By default, you see all your test results on the Test Results page. You can limit the results you see by applying filters. DAI provides several types of filters, which are described below.

The top of the Test Results page provides the following options:

• Search by ID: a search field where you can enter the exact ID of a test result you want to see. The results for each test execution have a unique ID generated by the Eggplant DAI database.
• Filters: a button you can click to show or hide the filter selection bar. The Filters button also shows the number of filters applied, if any.
• No filters applied or Clear all button depending on whether any filters are applied.

You might also see the filter selectors, which display in a bar below the Filters button when they are showing. Click the Filters button to show or hide the filter selection bar. For example, screen below shows the Suites filter in use with the selected suite name. You also see the Filters button showing the number 1 in purple, indicating one filter is applied.

Test Results filtered by Status

The filter types are as follows:

By ID

Narrow the results shown to a specfic test result by entering the exact test result ID in the Search by ID field at the top of the Test Results page. The results for each test execution have a unique ID generated by the Eggplant DAI database.

ノート

You only see the following filter options when the filter selection bar is showing. Click the Filters button to see the filter selection bar if it is not showing.

By Run Status

Narrow the results shown by the run status. The run status options are as follows:

• Passed: successful tests are highlighted in green.
• Failed: unsuccessful tests are highlighted in red.
• In Progress: incomplete tests are highlighted in yellow.
• Cancelled: tests that were stopped before they could be completed.
• Error: tests that resulted in error as well as tests that were interrupted due to loss of network connectivity or the agent process ending abruptly.

ノート

The All Run Statuses filter option was removed in DAI 7.2. All of the run statuses display when no filters are applied.

By Date range

Narrow the results shown to a specific timeframe that matters most to you. Click Date Range to see the date filter:

• After: filters the test results to only those that exist after the selected date.
• Before: filters the test results to only those that exist before the selected date.
• Between: allows you to specify a range by selecting two dates.
• None: removes the date filter.

By Model

Narrow the results shown to a specific model by clicking the Models filter button and selecting the model you want from the list. For more information about models, see DAI Build a Model.

Enable Show submodels results at the bottom of the Model filter list to show the test results for the main model and submodels, or disable it to show only the main model results.

ノート

When you filter results By Model, you select the name of a model and then see the results for only that model. If you enable Show submodel results, you will not see the model's submodel results unless the submodel name matches the name of the model you filtered on.

By Suite

Narrow the results shown to a specific suite by clicking the Suites filter button and selecting the suite you want from the list. Suites contain the SenseTalk snippets (or scripts) that you use in your DAI models or test cases.

ノート

Script-based test results generated before upgrading to DAI 6.4 will not appear when filtering by suite.

By Test Case

Narrow the results shown to a specific test case by clicking the Test case filter button and selecting the test case you want from the list. For more information about test cases, see Managing Test Cases.

ノート

If there are multiple test cases with the same name, the drop-down selector groups them into one selection, but it will be able to filter the results accordingly.

By Feature

Narrow the results shown to a specific Gherkin feature by clicking the Feature filter button. Gherkin features are test steps included in script-based test configs.

By Scenario

Narrow the results shown to a specific scenario of a Gherkin feature by clicking the Scenario filter button. A scenario is a test case defined in a Gherkin feature.

By Test Case Tag

Narrow the results shown to a specific test case tag by clicking the Test case tag filter button.

By Requirement Group

Narrow the results shown to a specific requirment group by clicking the Requirement group filter button and selecting the requirement group you want from the list. For more information about requirement group, see Requirements Traceability.

By Requirement

Narrow the results shown to a specific requirment ID by clicking the Requirement filter button and selecting the requirement ID you want from the list. The Requirement filter will remain disabled until a requirement group is selected. For more information about requirement id, see Requirements Traceability.

ノート

You must purchase a license specifically for the Requirements Traceability feature in order to use Requirement group and Requirement filters. To obtain this license, please contact our Customer Support.

When you receive the license from Customer Support, navigate to System > Licenses and enter it as described in Adding Licenses on the DAI License page.

By Test Config / Live run

Narrow the results shown to a specific test configuration ("test config") or filter the results to show only test configs or only live runs by clicking the Test config / Live run button and selecting the options from the list. For more information about test configs, see Working with Test Configurations.

Test config / live run combo box

By Test Config Tag

Narrow the results shown to a specific test configuration ("test config") tag. For more information about test configuration tags, see Working with Test Configurations.

By Execution Environment

Narrow the results shown to a specific execution environment. For more information about execution environments, see Environments.

By SUT

Narrow the results shown to a specific system under test (SUT). For more information about SUTs, see Managing SUTs in Eggplant DAI.

ノート

When you change the name of an existing SUT, DAI retains the ID of that SUT. If you filter by the new SUT name, you may see the original SUT name in the filtered results.

Clear All Filters

You can clear all filters at once by clicking the Clear all button, which is to the right of the Filters button at the top of the Test Results page. Once you click the Clear all button, you see No filters applied in its place.

ノート

If no filters are applied, you do not see the Clear all button. You see no filters applied in its place.

Results

The list of test results provides summary information and execution metrics for the results that match your criteria:

• Status & ID: The status and ID show in the same column.

  • status: The outcome of the test run. You can filter the results by Cancelled, Error, Failed, In Progress, or Passed statuses. Click the status to view the execution details for the test.
  • ID: The unique ID of the test result.
  ノート

  The Status & ID column sorts by run status, not ID. Prior to DAI 7.2, the Status and ID were separate columns.

• Test case / Feature & Scenario / Exploratory / Replay: Type of test execution, either: Test case or Exploratory, Replay (a test case name for directed test cases), or a Script name in the case of a script-based run. Exploratory or Replay display as a label. For test cases, each test case row will display associated test case tags.

• Requirement: The Requirement tag that represents the requirement assigned to a specific test case.

  ノート

  If a requirement tag is unassigned from a test case, clicking on the requirement tag displayed in the historical test result will navigate to the Requirement page with the test case filtered. However, the requirement will not appear in the list since it has been removed from the test case.

  ノート

  You must purchase a license specifically for the Requirements Traceability feature in order to see Requirement column. To obtain this license, please contact our Customer Support.

  When you receive the license from Customer Support, navigate to System > Licenses and enter it as described in Adding Licenses on the DAI License page.

• Model / Suite: Model or suite used in the test run.

• Test config / Live run: Test configuration to which the test belongs, or Live run. If the test config contains any tags, they display underneath.

ノート

Prior to DAI 7.2, this column's name was Run mode.

• Start date: Date and time when the test started running.
• Duration: The duration of the test runs, formatted in hours, minutes, and seconds.

ノート

If you export test results to a csv file, the Duration column is formatted in seconds in the file.

• Environment: The name of the execution environment used for the test run, or None if no execution environment was used.
• SUT: The system under test (SUT) used in the test run.
• Failed at: The last failed action for each failed test run.

If a test is in In Progress status, a stop button () appears in the row. You can stop a test run or test config runs using the button.

ノート

When you abort a test run that is part of a test configuration schedule, all runs that are part of that configuration schedule are aborted.

• Actions menu : Provides the option to delete test runs. You can remove unwanted results, such as failed test results that remain in an errored state.

• Delete: If the test result was for a Live Run, and not part of a test configuration run. This option deletes just the selected test run.

• Batch Delete: If the test result was part of a test configuration run. This option will delete the results for all tests in that run of the test configuration. For more information, see Batch Delete below.

ノート

A test result is considered part of a test configuration if it has a value in its row under the Test Configuration column.

For more information, see Batch Delete below.

• View > (Details): to view the execution steps, log entries and screenshots for the selected test run.

Sorting Results

You can sort results by clicking on a column header on the Test Results page. The columns you can sort show the sort button . Clicking the sort button the first time, sorts the results in ascending order. Clicking it again, sorts it in descending order. Then it works as toggle sorting in ascending or descending order as you click it.

• If a column is sorted in ascending order, you see an up arrow .
• if a column is sorted in descending order, you see a down arrow .

Exporting Results

For reference or analyzing test run outcomes, you can export your test results as a comma-separated value (.csv) file.

Click the Export button in the top right corner of the Test Results page to export the list of results currently showing on the page. This means the export contains results limited by any applied filters, or all of the results if no filters are applied.

Batch Deletion from Test Results

Sometimes, test results are generated from a Test Configuration run. Therefore, they have to be deleted in a batch. A test result is considered a part of Test Configuration if it has a value in its row under the Test Configuration column.

To delete the test result, click the Actions menu and click Batch Delete.

The Batch Delete option will not be available if the test result has In Progress status.

You see a confirmation dialog box informing you that the test result belongs to a Test Configuration run. Click View Batch on the dialog box. This opens the Results page showing all the individual test runs for the Test Configuration run.

At the top right corner of the screen is a red button labeled as Delete Run. Clicking it displays the confirmation dialog box. Click Confirm to proceed with the deletion.

This action deletes the instance of the selected test configuration run and all of its test results.

After the deletion, you see the Test Results page again.

Scheduling DAI to Delete Test Results

You can configure DAI to delete test results, including logs and screenshots, for you on a scheduled basis by setting the following two environment variables in your DAI Server configuration file.

• DELETE_RESULTS_AFTER_DAYS

• DELETE_RESULTS_TRIGGER_TIME

The DAI Server configuration file does not include these variables by default. You will need to manually add them to your file to use this feature. See Enabling Scheduled Test Artifact Deletion for explanations of these variables and instructions for modifying your DAI Server configuration file.

Viewing a Result from the Runner page

You can click Test result within a test run dot on the Runner page to redirect to the Test result page and check the result. Once you are on the Test result page, you can use the Quick run button to quickly rerun a test. Note that the Quick run button is only available when you redirect to the Test result page from the Runner page.

The following quick run options are available:

• Failed Tests → Runs selected steps with the Failed status pre-selected.

• Errored Tests → Runs selected steps with the Error status pre-selected.

• Failed & Errored Tests → Runs selected steps with both Failed and Error statuses pre-selected.

Note that the Quick run button is disabled when a test configuration is running.