Image and OCR Searches
These commands and functions search for elements on the system under test (SUT) using one of Eggplant Functional (EPF)'s search methods:
- Find by Description: Search for elements on the SUT screen by providing a description of the element to an LLM (Large Language Model).
- Image-Based Testing: Capture images of elements on the SUT screen, which are then referenced from SenseTalk scripts in order to search for and interact with those elements.
- OCR: Use optical character recognition to search for or read text on the screen of the SUT.
You can further control the search behavior of the commands and functions below by using global properties:
EveryImageLocation Function
You can use AI searches with this function.
For more information, see Find by Description.
Behavior: Searches the SUT for all occurrences of the elements described (Find by Description), images specified (captured image search), or text strings (OCR).
Returns a list of the hot spot coordinates for every occurrence of the elements or images, or a list for the center of the text string when using OCR. Coordinates are listed in the order in which they are found, where elements, images, and text closest to the top left corner of the SUT are found first. If no element, image, or text is found, EveryImageLocation returns an empty list.
Parameters: One or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches).
Syntax:
EveryImageLocation( imageOrTextReference )
Returns: The rectangle coordinates of every instance of the described element(s) (Find by Description), specified captured image(s), or text (OCR). For example:
[[288,325],[288,356],[288,387],[288,448]]
Example: Using a Find by Description Search:
Use an AI-powered Find by Description search to locate every instance of the user account icon on the SUT screen, by providing a description property:
log everyimagelocation(description:"the Eggplant Functional icon") -- Logs a list of hot spot coordinates for every occurrence of the element found.
Example: Using a Captured Image Search:
Including the ImageName property is required when passing additional image search properties as in this example:
log EveryImageLocation(ImageName:"CheckBoxes",scale:[1,.5]) -- Logs a list of hot spot coordinates for every occurrence of the image found at both the original size and half scale. Note the use of the ImageName property when including additional properties for the image search.
Example: Using an OCR Text Search:
To find text using OCR, provide a text property.
log everyimagelocation(text:"Eggplant") -- Logs a list of hot spot coordinates for every occurrence of the text found.
Example: Using a Repeat Loop with everyImageLocation:
This example uses everyImageLocation to click each found instance of a captured image, using a repeat loop:
set the RemoteWorkInterval to .1 -- Shortens the RemoteWorkInterval (default=.7) to decrease the pause between actions that Eggplant Functional sends to your SUT. This change allows the series of clicks to execute much faster.
repeat with EveryImageLocation("RadioButton") -- Iterates through each occurrence of the image found with everyImageLocation().
click it -- Using the it variable, clicks the current occurrence the repeat loop is operating on.
end repeat
set the RemoteWorkInterval to .7
Or, to do the same thing, but using alternate syntax:
repeat with each item of EveryImageLocation("RadioButton")
click it -- Using the it variable, clicks the current occurrence the repeat loop is operating on.
end repeat
Example: Referring to a Specific Occurrence of a Found Text String Using OCR:
This example clicks the penultimate (second to last) item in the list of occurrences of the text (using OCR):
click the penultimate item of EveryImageLocation(text:"AddressField")
When you call the EveryImageLocation function, you can immediately call the result to return ImageInfo for every image represented in EveryImageLocation:
put EveryImageLocation("GreenSaveButton","BlueSaveButton","RedSaveButton") -- Returns a list of hot spot coordinates for every occurrence of the image.
put the result -- Returns imageInfo for every image represented in EveryImageLocation.
EveryImageRectangle Function
You can use AI searches with this function.
For more information, see Find by Description.
Behavior: Searches the SUT for all occurrences of the elements described (Find by Description), images specified (captured image search), or text strings (OCR).
Returns a list of the rectangle coordinates for every occurrence of the elements, images, or text. Coordinates are listed in the order in which they are found, where elements, images, and text closest to the top left corner of the SUT are found first. For Find by Description and OCR searches, the ImageRectangle is automatically fitted around the element or text string as it appears on the SUT. If no element, image, or text is found, EveryImageRectangle returns an empty list.
Parameters: One or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches).
Syntax:
EveryImageRectangle( imageOrTextReference )
Returns: The rectangle coordinates of every instance of the described element(s) (Find by Description), provided captured image(s), or text (OCR).
Example: Using a Find by Description Search:
Use an AI-powered Find by Description search to locate every instance of a trash can icon on the SUT screen and print the associated rectangle coordinates in the Run window:
put everyimagerectangle(description:"trash can icon") -- Locates all of the trash can icons on the screen of the SUT and returns a list of recangle coordinates for every occurance
Example: Using Captured Image Searches:
This example returns a list of all instances of two different captured images:
put EveryImageRectangle("TotalAmountLabel", "ItemizedAmountLabel") -- Returns a list of rectangle coordinates for every occurrence of the images.
This example searches for a specific captured image, within a specified portion of the screen:
assert that the number of items in EveryImageRectangle(ImageName:"CheckBoxes",searchRectangle:["SelectOptionsUpperLeft","SelectOptionsLowerRight"]) is 4 -- Asserts whether the number of occurrences of the image is equal to the expected number, 4. Example output: AssertionFailed (assertion: the number of items in EveryImageRectangle(ImageName:"ImageName") is 4; actually: 5 IS NOT equal to 4)
Example: Using everyImageRectangle to Identify a Status Label:
Set StatusRectangles to EveryImageRectangle(text:"StatusLabel") -- Stores the return of EveryImageRectangle in a variable called StatusRectangles.
Log "There are"&&the number of items of StatusRectangles&&"statuses to check." -- Uses concatenation to join two strings with the number of occurrences in the list stored in StatusRectangles, and logs the output.
repeat with each Rectangle of StatusRectangles -- Iterates on each item in the list using the variable name Rectangle.
Set StatusIndicatorLocation to the TopRight of Rectangle plus (15,0) -- Makes a coordinate adjustment 15 pixels to the right of the top right corner of the current occurrence and stores it in variable StatusIndicatorLocation.
If ColorAtLocation(StatusIndicatorLocation) is "59,170,43" then Log "The status is green." -- Uses the ColorAtLocation() function to find the RGB value at StatusIndicatorLocation, then compares it to the expected RGB value, 59,170,43.
end repeat
When you call the EveryImageRectangle function, you can immediately call the result to return ImageInfo for every image represented in EveryImageRectangle:
put EveryImageRectangle("GreenSaveButton","BlueSaveButton","RedSaveButton") -- Returns a list of rectangle coordinates for every occurrence of these images.
put the result -- Returns imageInfo for every image represented in EveryImageRectangle.
ImageFound Function
You can use AI searches with this function.
For more information, see Find by Description.
Behavior: Searches the SUT for the elements described (Find by Description), images specified (captured image search), or text strings (OCR). Returns true if the element, image, or text is found, and false if it is not found. A maximum wait time parameter allows time for an element/image/text to be found before false is returned.
Parameters: One or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches).
Syntax:
ImageFound( imageOrTextReference )
Returns: True or False.
Example: Using a Find by Description Search:
Use an AI-powered Find by Description search to determine if a web page has loaded, by looking for an element that is expected to appear on that page:
If imagefound(description:"Sign in button") then log "the page has loaded" -- Upon a successful search, this logs "The page has loaded."
Examples: Using a Captured Image Search:
Uses a captured image to locate an element, and then either click it, or close the application using a keyboard shortcut:
if ImageFound(ImageName:"OKButton") then -- Checks whether the image is found
click FoundImageLocation() -- If ImageFound returns true, uses the FoundImageLocation function to click the stored location where the image was found by the ImageFound function
else
TypeText AltKey,F4 -- If ImageFound returns false, executes a keystroke to close the application
end if
Including the ImageName property is required when passing additional image search properties:
Repeat until ImageFound(ImageName:"NextButton",WaitFor:0) -- This repeat loop iterates until the ImageFound function returns true.
Typetext PageDown -- Sends a Page Down keyboard action to scroll down the page.
Wait 2 -- Waits 2 seconds to allow the application to stop moving after the scrolling action.
if the repeatindex is greater than 5 then throw "Image Not Found:", "NextButton not found when scrolling." -- This conditional statement checks to see whether the repeat loop has iterated more than 5 times, and if so, throws an exception to stop the execution.
end repeat
Example: Using an OCR Text Search:
To find text using OCR, provide a text property. Setting a search area using SearchRectangle() is useful when searching for text.
if imagefound(text:"Hello!", SearchRectangle:("Image1","Image2"))
then
click foundimagelocation()
log "clicked it!"
else
logerror "didn't find the word"
end if
A maximum time of 0 causes EPF to perform an immediate search, without refreshing the Viewer window or repositioning the mouse in a further attempt to find the image. This is the fastest way to search for an image that may not be present.
ImageLocation Function
You can use AI searches with this function.
For more information, see Find by Description.
Behavior: Searches the SUT for the elements described (Find by Description), images specified (captured image search), or text strings (OCR). Returns the hot spot coordinates of the found element or captured image, or the center of the found text (OCR) rectangle, of the first occurrence found. If no element, image, or text is found, an exception is thrown.
Parameters: One or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches).
Syntax:
ImageLocation( imageOrTextReference )
Returns: Coordinates of the hot spot location for the element, image, or the center of the found text (OCR) rectangle. For example: [338, 335]
Example: Using a Find by Description Search:
Use an AI-powered Find by Description search to determine the location of the first User Account icon on the SUT screen (the first being the closest to the top left corner of the screen):
Log "The first location the AI is finding for the User Account icon is: " & imageLocation(description: "user account icon") -- Upon a successful search, logs something like "The first location the AI is finding for the User Account icon is: [53,607]"
Example: Using an OCR Text Search:
Uses an OCR search to locate a section header:
set SectionHeaderLocation to ImageLocation(text:"Billing Address") -- Stores the return of ImageLocation in a variable called SectionHeaderLocation. Because it is a text (OCR) search, the return is the coordinates of the center of the "Billing Address" rectangle.
Example: Using a Coordinate Location Based on a Captured Image Search:
Clicks in a specific coordinate location, calculated based on the location of a found image:
click ImageLocation("NameLabel") + [50,-10] -- Clicks 50 pixels to the right and 10 pixels up from the hot spot location of the image.
Example: Using a Captured Image Search to Set a SearchRectangle:
Taps an "OK" button within a specific area of the screen, which is determined using the found location of an image:
tap {ImageName:"OKButton",searchRectangle:["PopupHeader",ImageLocation("LowerRightAnchor") - [400,500]]} -- Sets the lower right corner of the searchRectangle, adjusted 400 pixels up and 500 pixels to the left of the hot spot of the LowerRightAnchor image.
Example: Using the Stored Location of a Found Captured Image:
By storing the location of the scroll bar down button and clicking the stored location, Eggplant Functional has to search for the image only once, which increases the execution speed of clicking the button 10 times:
set ScrollDownButtonLocation to ImageLocation("ScrollDownButton")
repeat 10 times
click ScrollDownButtonLocation
end repeat
ImageRectangle Function
You can use AI searches with this function.
For more information, see Find by Description.
Behavior: Searches the SUT for elements described (Find by Description), images specified (captured image search), or text strings (OCR).
Returns the rectangle coordinates of the first occurrence of the first element, image, or text found. For captured image searches, the ImageRectangle honors the size and shape of the captured image found, unless the ClipRectangle image search property is in use. For Find by Description and OCR searches, the ImageRectangle is automatically fitted around the element or text string as it appears on the SUT. If no image or text (OCR) is found, an exception is thrown.
Parameters: One or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches).
Syntax:
ImageRectangle( imageOrTextReference )
Returns: The rectangle coordinates of the found element, image, or text. For example: [348,451,460,554]
Example: Using a Find by Description Search:
Use an AI-powered Find by Description search to locate the "Contact Us" form on a website, and record the image rectangle coordinates for the form:
log "The Contact Us form is located at: " & imagerectangle(description:"Contact Us form")
Examples: Using Captured Image Searches:
Search for a captured image and print the image rectangle coordinates in the Run window:
put ImageRectangle("CompanyLogo") -- Prints the return of ImageRectangle in the Run window only
This example uses concatenation to join a string and the return value of ImageRectangle with a space between them. Note the use of the ImageName property when including additional properties for the image search.
log "The coordinates of the rectangle are" && ImageRectangle(ImageName:"PhoneTypeLabel",searchRectangle:["PhoneContactHeader","SectionDivider"])
click the topLeft of ImageRectangle(ImageName:"EnableSetting") -- The ImageRectangle Function can be combined with commands that control the SUT. Note the use of the topLeft function to select a specific corner of the ImageRectangle.
Example: Using an OCR Text Search:
set NameFieldRectangle to ImageRectangle(Text:"Name") -- Stores the rectangle coordinates of the found text (OCR) into a variable.
set NameFieldValue to ReadText(Right of NameFieldRectangle, Top of NameFieldRectangle, Width of the RemoteScreenSize, Bottom of NameFieldRectangle) -- Uses ReadText to read the area directly to the right of the NameFieldLocation, all the way to the right edge of the SUT screen.
WaitFor Command, WaitFor Function
You can use AI searches with this command/function.
For more information, see Find by Description.
Behavior: Halts the script execution until the element described (Find by Description), image specified (captured image search), or text string (OCR) is found on the SUT. If no element, image, or text (OCR) is found within the specified wait time, an exception is thrown. The actual wait time can be higher than the specified wait time because the wait time adjusts automatically to factors such as SUT resolution, Discrepancy settings, network latency, and image collection size.
Parameters: A wait time, one or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches).
Command Syntax:
WaitFor time , imageOrTextReference1 {, imageOrTextReference2 ...}Function Syntax:
WaitFor( time , imageOrTextReference1 {, imageOrTextReference2 ...} )
Returns: The length of time spent waiting until one of the specified elements was found. When WaitFor is called as a command, the returned value is available on the next line of the script using the result.
Example: Using a Find by Description Search:
Use an AI-powered Find by Description search to locate the "Contact Us" form on a website, and record the image rectangle coordinates for the form:
Waitfor 4, description: "Contact Us form"
Examples: Using Captured Image Searches:
This example waits for one of two captured images to be found. Note that EPF searches for images in the order specified:
WaitFor 6.5, "OKButton", "ExitButton" -- Waits up to 6.5 seconds for either image to appear.
Wait for an installation to complete, verifying that the installation message is displayed using captured images:
WaitFor 2 minutes, {ImageName:"InstallationCompleteMessage",searchRectangle:["InstallProgressDialogUpperLeft","InstallProgressDialogLowerRight"]}
Put the result into elapsedTime -- record the actual time that passed until the image was found
Examples: Using an OCR Text Search:
Wait for an installation to complete, by verifying that the installation message, "Installation Complete," is found using a text search (OCR):
Waitfor 4, text: "Installation Complete"
The ImageSearch Global Property
You can also use the ImageSearchTime Global Property to affect the search time for a section of Find by Description, captured image, or text (OCR) searches.
The wait command is distinct from the waitFor and waitForAll commands in that the script always waits the specific amount of time you set, and then the script proceeds to the next step. The waitFor and waitForAll commands wait only until the specified element is detected on the SUT screen; if the element isn't detected within the wait time, an exception is thrown.
Waitfor as a Property
WaitFor can also be used as a property of any Find by Description, captured image, or text (OCR) search command. Like the WaitFor command, the WaitFor property is the maximum time Eggplant Functional waits for the given image to appear on the SUT.
Example: Using a Captured Image Search:
Click {ImageName: "Done_button", waitFor: 2 minutes}
Related
Wait: Use this command to insert a wait of a specific amount of time.WaitForNotFound: Use this command to wait for an element to disappear from the screen.WaitForAll: Use this command to wait for a number of elements to all be found simultaneously.
WaitForNotFound Command, WaitForNotFound Function
You can use AI searches with this command/function.
For more information, see Find by Description.
Behavior: Halts the script execution until none of the specified elements are found on the SUT. This command searches the SUT for one or more screen elements (using Find by Description, captured images, or OCR text searches). If any of the given elements are found on the SUT, the search is repeated until none are found. If the specified wait time elapses without the element(s) being gone from the SUT, an exception is thrown.
WaitForNotFound is functionally the opposite of the WaitFor command. Where WaitFor waits a length of time until something is visible on the screen, WaitForNotFound waits a length of time until something is not visible on the screen.
Parameters: A wait time, one or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches). The default unit of wait time is seconds.
Command Syntax:
WaitForNotFound time , imageOrTextReference1 {, imageOrTextReference2 ...}Function Syntax:
WaitForNotFound( time , imageOrTextReference1 {, imageOrTextReference2 ...} )
Returns: The length of time spent waiting until none of the specified elements were found. When WaitForNotFound is called as a command, the returned value is available on the next line of the script using the result.
Example: Using a Find by Description Search:
Use an AI-powered Find by Description search to confirm that a loading bar is no longer being displayed:
WaitforNotFound 10, description: "Loading bar"
Examples: Using Captured Image Searches:
This example waits for neither of two captured images to be found. Note that EPF searches for images in the order specified:
WaitForNotFound 6.5, "OKButton", "ExitButton" -- Waits up to 6.5 seconds until neither image is found.
Wait for an installation to complete, verifying that the installation message is displayed using captured images:
WaitForNotFound 2 minutes, {ImageName:"InstallationCompleteMessage",searchRectangle:["InstallProgressDialogUpperLeft","InstallProgressDialogLowerRight"]} -- Waits until the specified image is gone
Put the result into timeToDismiss -- Records the actual time taken until the image was gone
Examples: Using an OCR Text Search:
Wait until the installation program "Installing..." status text is gone, indicating that the installation is complete:
WaitforNotFound 2 minutes, text: "Installing..."
Related
WaitFor: Use this command to wait for an element to be displayed.
WaitForAll Command
You can use AI searches with this command.
For more information, see Find by Description.
Behavior: Halts the script execution until all of the specified elements are found on the SUT screen at the same time. If not all of the elements are found simultaneously within the maximum wait time, an exception is thrown.
Parameters: A wait time, one or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), or one or more Text Properties (for OCR searches). The default unit of wait time is seconds.
Syntax:
WaitForAll time , imageOrTextReference1 {, imageOrTextReference2 ...}
Examples: Using Find by Description Searches and Captured Images Together:
Confirm that a web page has loaded. In this example, multiple AI-powered Find by Description searches are used to confirm that multiple elements of a webpage have loaded:
WaitforAll 5, (description: "account icon"),(description: "sign in button"),(description: "like button")
This example uses a mix of AI-powered Find by Description searches and captured images to do the same confirmation:
WaitforAll 5, (image: "AccountIcon"),(description: "sign in button")
Example: Using an OCR Text Search:
In this instance, WaitForAll is waiting up to 15 seconds for a single text string, "Aretha Franklin".
WaitForAll 15, text:"Aretha Franklin"
Example: Using Captured Image Searches:
If you need to wait for two images, the names can be passed as strings, like this:
WaitForAll 15, "OKButton", "ExitButton"
Example: Searching for Images and Text (OCR) Simultaneously:
In order to use WaitForAll to search for images and text strings simultaneously, or multiple text strings simultaneously, you must pass separate property lists for each, as shown in the below examples.
WaitForAll 30, {text:"Updates Available"}, {image:"OKButton"}
WaitForAll 15, {text:"Diana Ross"}, {text:"Baby Love"}
Example:
WaitForAll 5, {image:"Welcome"}, {text:"Susan",CaseSensitive:"yes"}
If an enclosed list of described elements (Find by Description), captured images, or text strings (OCR) is passed as a WaitForAll parameter, Eggplant Functional considers that parameter "found" when any one of the items on the list is found. Here is an example using captured image searches:
WaitForAll 20, "crust", "sauce", ["peppers", "olives"]
The command above has three parameters: "crust", "sauce", and a list of two other items (enclosed in square brackets). It can be thought to mean:
WaitFor20, "crust" -- and simultaneously...
WaitFor 20, "sauce" -- and simultaneously...
WaitFor 20, "peppers", "olives" -- ... and simultaneously wait for either of these items
The wait command is distinct from the waitFor and waitForAll commands in that the script always waits the specific amount of time you set, and then the script proceeds to the next step. The waitFor and waitForAll commands wait only until the specified element is detected on the SUT screen; if the element isn't detected within the wait time, an exception is thrown.
Related
Wait: Use this command to insert a wait of a specific amount of time.
WaitForColor Command
Behavior: Halts the next line of script execution until the specified color is found within the specified rectangle on the SUT screen. If the color is not found within the maximum wait time, an exception is thrown.
Parameters: A wait time, a color, and a rectangle that you've defined. The waitForColor command accepts colors in any format supported by Eggplant Functional, as well as any rectangle formatting supported in Eggplant Functional.
Syntax:
WaitForColor time , color , rectangle
Example:
WaitForColor 5, "blue", [22,576,86,640]
SetSearchRectangle Command (Deprecated)
The SetSearchRectangle command has been deprecated by the SearchRectangle Eggplant Functional global property. Setting the SearchRectangle Eggplant Functional global property is the recommended way to set a search rectangle. Use the SetSearchrectangle command shown here as an alternative in certain situations.
Behavior: Takes two pairs of screen coordinates as parameters; these points define two diagonal corners of the search area.
Parameters: Takes two sets of coordinates or two images, defining the top left corner and bottom right corner of an area of the screen.
The top left corner of the SUTscreen is always 0,0.
Syntax:
SetSearchRectangle( rectangle )
Example:
SetSearchRectangle [[0,0],[100,100]]
Example:
SetSearchRectangle ["TLImage","BRImage"]