Using AI to Find Elements by Description *Preview*
Eggplant Functional (EPF) enables you to interact with the system under test (SUT) using natural language descriptions and the power of artificial intelligence (AI). This functionality is called Find by Description.
Find By Description lets you script interactions with elements on your system under test (SUT) by providing descriptions that are used by a multimodal large language model (MLLM), which we will refer to as an AI model (LLM), to search and analyze the SUT screen. This search type can be used instead of or in conjunction with image-based searches, optical character recognition (OCR) text searches, and hard-coded coordinates Since Find By Description leverages state of the art AI, you'll often find that it's much less labor intensive to write and maintain tests, letting you focus on what matters: making sure your application is working properly.
EPF version 26.2.1 introduces Find by Description as a preview feature. At this time it is available to all users of Eggplant Functional who want to try it out. To provide feedback on this functionality, contact Eggplant Support. Please note that use of this feature may incur additional costs in the form of tokens through your AI provider. For more information on tokens, see AI Model (LLM) Token Usage.
Prerequisites
To use Find by Description, you must have both a working internet connection, and access to a supported AI large language model (LLM). Access to the AI model must be configured in EPF to use the feature. For more information, see the Find by Description Prerequisites.
Setting the AI Model (LLM) Provider and Adding an API Key
To establish communication between EPF and your AI model, first set the AI provider in EPF, and then authenticate the connection with an API key. This can be done either in the Eggplant Preferences, or through a command line interface.
Setting up via the Eggplant Functional Preferences
Navigate to the Eggplant Functional Account Preferences panel by navigating to Eggplant > Preferences > Account (Eggplant > Settings > Account on Mac) and fill out the AI Account section by selecting your AI provider, entering your API key, and defining your region if needed (not always required).
How to Use Find by Description
Find by Description is implemented as a description parameter that is used in conjunction with existing SenseTalk commands and functions. For example, it can be used with both the click command and the waitfor function.
Each time a command or function is run that uses a description parameter, the AI model (LLM) specified during setup scans the screen of the SUT for the related element described, and then Eggplant performs the associated action, whether that is clicking the element, or registering the element as having been found for a waitfor function.
Example of How to Use Find by Description:
click description: "Chrome icon" --> The LLM searches the SUT for the Chrome icon, and Eggplant Functional clicks that icon.
waitFor description: "Login button" --> Look for an element on the screen that confirms that the web page has loaded before proceeding with the test.
How to Write Descriptions
Like working with any LLM, sometimes the best way to figure out prompt phrasing is through trial and error. The descriptions you use may be different depending on the LLM you are working with, and the language you are using to prompt it.
Start with clear, simple language. You can add more detail later as needed. For instance, using the description "history icon" may not be sufficient if the LLM you're using doesn't immediately recognize the icon in the UI of your program as being history-related, and saying "blue history icon" may be too specific if your interface changes color scheme later. A better description might be "clock icon" if the history icon looks like an analog clock, or "clock icon in the menu bar" to provide more context for the search.
Keep in mind that the LLM may not be familiar with terms that seem natural to you, such as internal company nomenclature that you use on a daily basis. For elements that you typically refer to using such internal nomenclature, use a more generic description, as you might describe the element to someone who isn't familiar with your company, products, or the software you're testing.
To craft effective descriptions:
- Start simple, add detail only if needed, and avoid overly specific details
- Describe visual characteristics, avoiding internal company nomenclature for specific elements
- Include location or context when helpful
- Avoid relying on color alone
Dynamic Descriptions
SenseTalk sees and treats the description parameter text as a string literal. Because of this, you can use SenseTalk variables and concatenation to dynamically compose descriptions in your script. Using these key elements of the language enables flexible and data-driven testing.
Examples of Dynamic Descriptions:
Determine which icon is clicked based on a value stored in a variable, like so:
click description: iconName && "icon" --> if the iconName variable contained the value "Chrome", this would result in the description "Chrome icon"
Or, create a description that varies based on the user account the test is being run against:
click description: customerName & "'s user profile in the menu bar" --> If the customerName variable contained "Elizabeth" this would result in the description "Elizabeth's user profile in the menu bar"
Reusing Descriptions
If you have tests that will refer to the same element repeatedly, you can use SenseTalk variables to store the description, which will allow you to edit the description for that element in a single location, reducing the effort required to maintain your tests.
Examples of Reusing Descriptions:
In this example, there are two "Account" buttons displayed on the screen at the same time. The description therefore specifies a region of the screen in which to search. This description is stored in a variable, LRAcctButton, which is then later referenced as part of a click command:
put "the account button in the lower right" into "LRAcctButton"
click description: LRAcctButton
To be able to reference the variable across scripts, use a global variable declared in a handler. The handler could be another script, or the same script:
global button
put "LRAcctButton" into button
click description: LRAcctButton
AI Model (LLM) Token Usage
AI model (LLM) usage is measured in “tokens”. Any data sent between Eggplant Functional and the model in the form of text or images incurs additional token usage costs.
The cost of tokens depends on your AI provider, model, and geography. With a local model, token usage is “free” (the only cost being electricity used by your machine). Typically newer and more resource-intensive models are priced higher than smaller and older models.
For more information about token cost, please refer to the documentation for your AI provider.
Frequently Asked Questions
Q: Is the description parameter case sensitive?
No. SenseTalk is not case sensitive, and the description parameter is no exception. The text string passed to the description parameter is also not case sensitive, so DeScRiPtIoN: "tHe oKAY bUtTon oN tHe PoPuP wInDoW” is the same as Description: “the okay button on the popup window”.
Q: What spoken languages are supported for writing descriptions?
The supported spoken languages are a function of what languages the underlying model supports. For example, Anthropic Sonnet 4.5 through Amazon Bedrock includes multilingual support.
Keep in mind that models can be “better” or “worse” at a particular language. For instance using a description such as “the Okay button” may work in English, but the same prompt translated into Spanish, "el botón de Aceptar", may not.
Q: I wrote a really great description, but Eggplant didn’t find what it was looking for. Why?
The performance and capabilities of “Find by Description” depend highly on the AI model (LLM) it is working with.
AI models open up a new world of capabilities, but due to the underlying technology, the way that the models arrive at the output they produce cannot be predicted ahead of time or discerned after the fact. Because of this, it’s not possible for Keysight, the LLM providers, or the model developers to fully explain why a specific match did or did not occur.
If your description isn’t matching, we recommend modifying the description and trying again. With time, you can learn to work with your AI model and write great descriptions that it will understand.
If you are seeing an error message, it could be that your API authorization isn't configured properly, or that you are having network connection issues. Check your configuration and network connectivity, and then try again.
If you are still struggling to get your description to work, you can always use the traditional image- and OCR-based searches.
Q: I care deeply about verifying formatting or specific content and don't want to leave it up to the AI model to determine whether or not an element is correct. How should I script for this?
In some cases, you may care deeply about specific details of a UI element, such as its precise location or color. In these cases, the flexibility of Find by Description may actually be unwanted. To verify such a specific detail, you can always use the fine-grained control of Eggplant's traditional captured image or OCR searches. In fact, you can freely intermix Find by Description with image- and OCR-based testing as much as you'd like.
As a rule of thumb, we suggest you use Find by Description where the purpose of the particular action you are scripting is simply to do something, and the details of the UI appearance are not as important. For example, in many cases you may just want to click the "Login" button and don't need to verify whether it's blue or red. In that majority of cases, you could use Find by Description. Then, if you do care that the login button is a specific color, write a separate test which enforces exactly that characteristic. That way, if the button does change color, most of your tests will adjust automatically (via the reasoning of the AI model) but you'll have one failing test which alerts you to the change in color.