Working with Test Configurations

A test configuration is a collection of tests that can be run on demand or be scheduled to run on a regular basis in Eggplant DAI. A test configuration ("test config") is essentially a blueprint of tests that can be run one after the other. You can run directed test cases and model-based exploratory tests in a test config. See About Test Cases for more information about test cases.

Go to Controller > Test Config, to create, edit, or view your defined test configs.

The following columns are displayed to provide at-a-glance details about each test configuration:

• Name: Display name of the test configuration.
• Model/Suite: Name of the model or suite depending on the type of test configuration you choose to run.
• Description: Description of the test configuration. Provide a meaningful description to help distinguish it from other test configurations.
• Tags: Tags defined for the selected test configuration that you're viewing.
• SUT: Associated SUT, or criteria for choosing a SUT for the test configuration.
• Updated: Date and time when the test configuration was last updated.
• Steps: Number of test steps included in the test configuration. A test step is either a test case from the model or a script based test case, or an exploratory test with stop criteria, which could be defined by duration, iterations, or a target coverage percentage.

Click the Actions menu to Edit, Delete, or Duplicate a test configuration.

Filtering Test Configurations

You can filter test configurations in the following ways:

• By name: Type part of a test configuration name in the Search by name search bar. DAI filters the test configurations you see as you type.
• By model: If you want to see the test configurations associated with a model, click the arrow on the Models button and select a model from the suggested list or start typing a model name in the Search... field. DAI suggests models as you type.
• By suite: If you want to see the list of test configurations associated with a suite, click the arrow on the Suites button and and select a suite from the suggested list or start typing a suite name in the Search... field. DAI suggests suites as you type.

Creating Test Configurations

Go to Controller > Test Config to create a test configuration ("test config").

Step by Step: Creating a Test Configuration

Creating a test config consists of three steps, with a page for each step. The steps and pages are summarized below, followed by detailed instructions:

• Step 1: General: the page where you provide the general information about your test config, including a name, description, test type, test config tags, execution settings, and SUT.
• Step 2: Steps/actions: the page where you configure the exploratory test or test case actions you want your test config to run.
• Step 3: Advanced Settings: the page where you can configure if you want to automatically rerun failed test steps, stop a run when a step fails, and set up email notifications to advise of completed test configuration runs.

A progress tracker at the top of the page shows where you are in the creation process as shown in the image below. The highlighted number in purple shows the step you are on in the process. Once you finish entering information on a page and advance to the next one, the progress tracker shows a green check for completed the step.

The pages include Cancel and Back buttons so you can cancel the test config creation or return to a previous page when you want.

note

Providing a name and test type enables additional settings.

Step 1: Configure the General information for the Test Configuration

1. Go to Controller > Test Config. The Test Configs page opens.

2. On the right side of the top menu, click + New Test Config. The Create new Test Config page opens with the first step in the process, the General page shown above, where you provide the Basic Settings for your test config.

3. Provide a Name and Description for the test configuration. This is the name and description that show in the list of test configs. Also note the Clear all button you can use to clear the Description field.

4. Select a Test type (Model based or Script based) for the test configuration. After you specify the test type, more settings become visible. Some of the options you see depend on the test type selected. For both test types, you see the Test configuration tags and Execution settings selectors.

   • If you selected the Model based test type, you see the Model selector.

     

   • If you selected the Script based test type, you see the Suites selector so you can select the suite that contains the scripts/snippets you want your test config to run.

     

5. Select the model or suite from the selector as appropriate for your test type.

   note

   Note that selecting your model or suite enables the Next button at the bottom of the page. You can proceed to the Steps/actions page at this point with the default values for the remaining settings on this page.

6. Add or select Test configuration tags by entering a name for the tag and pressing Enter. Tags provide another way for you to identify and filter by your test config. You can add as many test tags as you need. You can also clear the list of selected tags by pressing the Clear all button.

   

7. In the Execution settings section of the page, select the type of execution environment (Local or Cloud) that you want your test config to use.

8. Select a SUT option for this test config that is appropriate for your execution environment as follows:

   For Local execution environments, choose from the following SUTs:

   • None (no SUT) if the SUT is defined in your code or not required.

   • SUT by name to select a predefined SUT.

     

   • SUT by criteria tag to select a SUT tag from the drop-down list. The SUT tag is associated with a SUT that is available on the SUTs page (Controller > SUTs). If you want to create a SUT or SUT tag, navigate to the SUTs page.

     

   For Cloud execution environments, choose from the following SUTs:

   • The same options listed above for Local execution environments.

   • Eggplant hosted cloud: Select this to execute tests on Cloud SUTs where Eggplant will provision the Cloud Execution Environment (including the Agent and Eggplant Functional) for you. For more information, see Connecting to DAI Cloud SUTs. (This option is only available for cloud SUTs.)

9. When you finish configuring the Basic Settings, click Next. You advance to the second part of the test configuration creation process: the Step 2: Test steps/actions page.

Step 2: Configure the Steps/actions for the Test Configuration

The options you see on the Steps/actions page depend on the test type you selected. The top of the page shows the test type you selected (Model based or Script based) and either the model or suite as appropriate for the test type. Follow the appropriate steps below for your test type:

• Configuring steps/actions for Model based test types
• Configuring steps/actions for Script based test types

Configuring steps/actions for Model based test types

For Model based test types, you see the Available exploratory and test cases list.

1. Click to select any of the test cases from the Available exploratory and test cases list. If the Available exploratory and test cases list is too long, you can filter it by typing test case names. What you see added to the Steps added list on the right side of the page depends on the type of test you select:

   

   • For exploratory tests, you see exploratory test options where you can specify the number of Iterations, Coverage, and Time (hrs) as shown below.

     

     note

     For exploratory steps, a stop criterion is always set by duration, iterations, or target coverage. After the test runs out of steps, Eggplant DAI will examine the execution of each step and indicate whether the test configuration is a success or failure.

   • For test cases, you see the test case.

     

2. Repeat step 1 above until you have added all the tests you want your test config to run or click Add All to add all the test cases in Available exploratory and test cases list. Note that clicking Add All when filters are applied to the Available exploratory and test cases adds only the test cases in the filtered list.

   note

   Clicking Add all while the Available exploratory and test cases list is filtered only adds test cases in the filtered list.

3. Click Next to advance to Step 3: Advanced Settings page and follow the instructions for that page.

Configuring steps/actions for Script based test types

For Script based test types, you see the Available test cases and features selector that contains two tabs. The Test case tab lists the available scripted test cases and the Features tab lists the available Gherkin feature files included in the suite used to create the script-based test config. Each list has a Search... bar so you can search for test cases or features by their names.

1. Select a scripted test case or feature from the list or search for it by name in the search bar and then select it. You see the test case or feature added to the Steps added list on the right.

2. Repeat step 1 above until you have added all of the steps you want your test config to run.

3. Click Next to advance to the Step 3: Advanced settings page and continue to the following section for information about configuring it.

Global Parameters

Global parameters are parameters that are shared across all test cases within a script-based test config. Once they are defined, you can use them as your custom parameter values. Global parameters are only available in script-based test configs.

To Add global parameter of a script-based tests config:

1. Go to Steps / actions tab.

2. Click the Global parameters button to open global parameters' panel.

3. Once global parameters' panel is visible, click the Add global parameter button.

4. Enter a unique Global parameter name. You cannot specify a global parameter name that already exists in the Global parameters' list.

5. Enter a Default value to override during the test run.

6. Click the Add button to add the new global parameter to the global parameters' list.

To remove the added global parameter from the global parameters' list, click the trash button.

note

Once you add a global parameter, you cannot edit its name. However, you can still change the default value directly from the global parameters' list.

To set a global parameter as a custom parameter value on the Edit parameters' panel:

1. Click the Edit parameters button on a test case in the Steps added list to open the Edit parameters panel.

2. Click the Global parameters chip on the Custom parameter tab to copy the global parameter name.

3. Paste the copied value into the Custom parameter value's input box.

note

Clicking chips to copy parameters might not work as expected on an insecure website (for example, a website with a URL that begins with http instead of https).

If you want to manually enter the global parameter as a custom parameter value, you must enter in it in the format of [[global_parameter_name]].

Remember to click Save before you leave the Edit parameters and Global parameters panels. If you do not save your changes, the newly configured settings will not take effect when you are configuring the test config.

Step 3: Configure Advanced settings for the Test Configuration

When you reach the Advanced settings page, you have the following options:

• If you want to create your test config, click the Create test config button. See Finish Creating a Test Config for more information.

• If you want to enable automatic reruns of failed or errored test steps for your test config, click the corresponding slider next to Automatically rerun failed test steps or Automatically rerun errored test steps. See Enabling Reruns for an explanation of how to configure DAI to rerun your test config.

• If you want to stop your test config run when a step fails, disable Run all test config steps to completion by clicking the slider next to it. See Stop On Failure for an explanation of how to configure DAI to run all test config steps to completion or stop when a step fails.

• If you want to enable email notifications and advise of completed test configuration runs, click Manage notifications ->. The notification settings page opens:

  

  Select the users you want to receive notifications in the Recipients box and select the status(es) you want the recipients to be notified about.

• If you want to set a custom script to run when executing this test config, select the desired custom script from the dropdown list. If a default custom script is already set, you do not need to make any changes unless you wish to override it.

  

  A message below the dropdown indicates the currently set default custom script. This message will disappear once you make a selection.

Important

This custom script setting is only editable for DAI Administrator users.

See Finish Creating a Test Config below to finish creating your test config.

Finish Creating a Test Config

When you finish creating your test config, click the Create test config button in the lower-right corner. DAI saves your new test config and the Test Configs page opens. The new test config will be in the list but you may need to scroll or search for it to see it.

Configure Custom Parameters for Steps Added

The Custom Parameters feature allows you to overwrite the default value of a test case's parameter with a Custom value. DAI then uses your Custom value when a test config runs the test case. You can customize as many of your test case parameters as you want.

To configure a custom test case parameter value for any test step:

1. Click the icon on the step. A slide panel that displays all test case parameters will open from the right.

2. Add a value in the Custom value column to overwrite the default value of the test case parameter.

3. Click Save.

This configuration allows you to diversify the same test case with different parameter values without replicating the test case with different default values.

Enabling Reruns

The rerun feature allows you to save time and effort by automatically rerunning failed or/and errored test steps up to 50 times without executing the entire test configuration. A test step can be a model based test case, a script based test case, or an exploratory test.

To activate the rerun functionality for failed or errored test steps:

1. Enable reruns by sliding the Automatically rerun failed test steps or/and Automatically rerun errored test steps toggle to the right. (A green check indicates the option is enabled.)

2. Configure the rerun parameters with the following controls:

   • Number of Runs: Specify the maximum rerun count for failed and errored test steps. You can adjust this value using the text box or slider, with a maximum of 50 reruns.
   • Wait Duration: Define the interval between each rerun, optimizing the rerun process. Choose a time interval from the dropdown list, measured in seconds. The default is set to zero, implying no wait interval.

This streamlined process allows you to effectively manage reruns, enhancing your testing workflow.

note

• The rerun count and wait duration are the shared settings between automatically rerun failed and errored test steps.
• The chosen rerun count and wait duration will be saved even if you deactivate the rerun feature. This way, you can easily re-enable it at a later time without reconfiguring.
• Access rerun statistics by viewing the total reruns completed under the Controller menu on the Runner page. Expand individual test results to find this information.

Stop On Failure

DAI provides the option to stop a test config run when a test step failure occurs. This option allows you to save resources, such as SUTs allocated for the test, and license usage. Rather than continuing to run the remaining steps of a test which may already be compromised, stopping a test config when a failure occurs can eliminate doubt about the cause of any issues subsequently encountered if the test were allowed to continue. Note that a test step can be a model-based test case, a script-based test case, or an exploratory test.

By default, DAI runs test configs to completion. This is controlled by the Run all test config steps to completion option on the test config Advanced Settings tab. To configure DAI to stop test config runs when a failure occurs, you disable this option as shown in the following sample screen.

To configure DAI to stop test config runs when a failed test step or error occurs:

1. Disable the Run all test config steps to completion option by sliding its toggle to the left. (Disabling Run all test config steps to completion enables the stopping of test config runs.)

note

• By default, even when a test step failure occurs, the test config runs to completion.
• You can access the failing step by viewing the test configuration logs under the Controller menu on the Runner page. Expand individual test results to find this information.

Running a Test Configuration

You can run a test configuration from Controller > Runner by clicking against the test you want to run.

Go to Controller > Runner to configure test schedules with specific frequencies to be run at regular intervals. You can only have one schedule per test configuration.

tip

Remember to always associate suites to a model before running tests.

note

When you abort a test run that is part of a test configuration schedule, all runs that are part of that configuration schedule are aborted.

Eggplant Functional Stop and Restart Settings for Test Configuration Runs

When running a test configuration, you can configure the length of time to run Eggplant Functional (EPF) before restarting or the grace periods for poll-time stopping and restarting with the following environment variables:

Environment Variables	Default	Description
RESTART_EPF_AFTER_SECONDS	1800	The period from when EPF launches until the next EPF restart. This variable takes effect after the current step finishes and before moving to the next. It only applies to test configurations run with the DAI Run Agent. During live runs, EPF runs continuously without restarting. Default: 30 minutes.
EPF_QUIT_POLL_TIME_SECONDS	20	The grace period for EPF to quit during the restart process.
EPF_RESTART_POLL_TIME_SECONDS	120	The grace period for EPF to start up after having quit. EPF_RESTART_POLL_TIME_SECONDS does not include EPF_QUIT_POLL_TIME_SECONDS.

Viewing, Editing and Notification Settings of an Existing Test Configuration

To view or edit a test configuration:

1. Go to Controller > Test Config.

2. Find the test configuration you want to update or view, then from the Actions menu (), select Edit.

3. From here you can view or update the test configuration details.

4. You can also manage the notification settings of the test configuration by clicking the Manage Notifications link.

Notification Settings of an Existing Test Configuration

Important

Test Configuration notifications must be enabled globally before you can set up notifications for a specific Test Configuration within DAI. You need to be an administrator to do this. See Manage DAI Settings for details.

By default, the notifications for all Test Configurations are disabled. To enable notifications, follow these steps:

1. Go to Controller > Test Config.
2. Find the test configuration you want, then from the Actions menu, select Edit.
3. Click the Advanced settings tab to display it.
4. Click Manage Notifications () in the Notifications area on the tab.
5. From here, you can enable or disable the Send email notifications and assign verified users to be Recipients who receive the notifications.
6. You can also manage the Notification type you wish to receive by selecting from the various options as shown below:
   • Rules for Recipients:
     • If notifications are enabled, you must pick at least one recipient before you are allowed to save.
     • If notifications are disabled, then you can leave the field blank and save.
   • Rules for Notification type:
     • If notifications are enabled and you deselect all checkboxes, notifications are disabled and toggle to the disabled status.
     • If notifications are disabled and all checkboxes are deselected, when you toggle Send email notifications, notifications are enabled and all checkboxes will be enabled by default.

Removing a Test Configuration

To remove an existing test configuration:

1. Go to Controller > Test Config.
2. Find the test configuration that you want to delete, then from the Actions menu, select Delete.
3. From here you can view or update the test configuration details.

Duplicating an Existing Test Configuration

To duplicate an existing test configuration:

1. Go to Controller > Test Config.
2. Find the test configuration that you want to duplicate, then from the Actions menu, select Duplicate.
3. From here, you will be required to enter a new name for the Test Config.
4. Click Create once a valid name is entered.
5. From here, you will be able to edit your newly-duplicated Test Config.

Tips

• Make sure to avoid entering the same name as the Test Config that is to be duplicated.
• The new name must follow these rules:
  1. 1 to 100 characters (maximum length)
  2. Special characters are acceptable except for the following: , \\ ; : " ' { }.