Measuring Coverage with Test Advisor

Test Advisor helps you determine which mobile device systems under test (SUTs) you should use to run your tests in the Eggplant Manager test management system. Using filtering criteria including market share, user location, and operating system, you can calculate and optimize your test coverage.

Note: You must have an Eggplant Automation Cloud license to use Test Advisor.

Filtering SUT Options

Go to Tools > Test Advisor from the main menu in Eggplant Manager. When you enter filtering criteria, Test Advisor returns a list of mobile devices to give you the coverage you want for your tests.

Select the country or countries where your users are from the Where are your users? drop-down menu.
Enter the Date Range for your test.
Choose the Market Percent Threshold devices must have to be included in your test.
Choose Smartphones or Tablets (or both) for the Device Type.
Choose iOS or Android (or both) for the Device Platforms.
Click Calculate.

Set Data Bindings

This feature allows you to set logical equivalents between metadata on your mobile SUTs for use in Test Advisor. You can create data mappings for Manufacturer, Model, OS, and Version. Think of it as providing a list of synonyms for Test Advisor to reference. If you have an iPhone running iOS 10.3.2, the Version can be entered as iOS 10.3, 10.3.2, or anything you define as equivalent to 10.3.2, as in the following example.

Step-by-Step: Create Data Bindings

From the main Test Advisor screen, click Set Data Bindings.
Select what type of mapping you want to set from the Category drop-down menu.
Enter the values you want Test Advisor to treat as equivalent in the text box below the Matches box. Click Add Match.
If you want to create mapping in more than one category, click Add Data Mapping and repeat steps 1 through 3.
To return to the main Test Advisor screen, click Calculate Coverage.

Reading the Results

Test Advisor returns a list of SUTs that fit the criteria you selected AND that provide the amount of coverage you want.

Target coverage results in Test Advisor. Click image to enlarge.

The Target Coverage percentage is set to 30% by default. Change this value using the blue slider. Test Advisor lists your coverage target, the percent coverage of the selected devices, and the percent coverage of the devices you own.

The results table beneath the slider shows a combination of devices that fit the Market Percent Threshold, Device Type, and Device Platform you selected. Test Advisor lists as many devices of the selected types and platforms as needed to reach the target coverage you set. Note that Test Advisor lists all possible SUTs that meet those criteria, not just those that you own.

Important: If Test Advisor doesn't display a leaf next to a SUT that you own in the model table, try setting Data Bindings to allow for variations in the entry of information about that device.

Test Advisor adds up the market share for multiple devices until it reaches your target coverage percentage, then displays those devices with the checkbox selected. Note that for some sets of criteria, there can be dozens of possible SUT combinations that meet your coverage needs. You can select or clear checkboxes to alter the device combinations as needed.

This topic was last updated on 2月 01, 2019, at 11:13:23 午前.