Version: 26.2

Mouse Events and Control

This section describes the Eggplant Functional (EPF) commands and functions that execute mouse events or return information about the mouse cursor on the SUT. Many of these commands and functions, including Click, DoubleClick, Drag, Drop, DragandDrop, MouseButtonDown, and MouseLocation(), work against mobile SUTs as well as desktop SUTs, which is helpful when creating scripts that automate both mobile and desktop platforms. SenseTalk also includes a set of mobile commands and functions, a subset of which (Tap, DoubleTap, Press, etc.) are compatible with desktop SUTs.

`Click` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

note

This definition explains the use of the Click command for visual scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: Click Command.

Behavior: Clicks the SUT mouse at the location indicated by the parameter specified.
For AI-powered Find by Description searches, the click location is determined by the AI model, at the first location deemed to match the description provided. The other search types behave as follows: the click event occurs at the hot spot location for the first found image, at the center of the first text match found using optical character recognition (OCR), or at the coordinate location provided.
If more than one element description, image, or text string is passed, EPF searches for images in the order in which they are listed.

tip

The Tap and Click commands perform the same event and can be used against both desktop and mobile SUTs.

Parameters: Optional. One or more descriptions (for Find by Description), Image References (for captured image searches), Text Properties (for OCR searches), or a single coordinate location.

Example: Using a Find by Description Search:

Provide a description of an application to click, in order to launch it using a Find by Description AI search:

click description: "the Outlook icon in the start bar" -- launches Outlook

Example: Using a Captured Image Search with a Repeat Loop:

This example iterates through a list of images using a repeat loop, clicking each one in turn:

repeat with each item of ["Option1","Option2","Option3"] -- Repeats through a list of images
  click it -- Clicks the hot spot of the current image in the list
end repeat

Example: Using an OCR Text Search:

Search for and click a text string using OCR:

click{text:"Tweet",waitFor:10,searchRectangle:["AnchorLeft","AnchorRight"]} -- Clicks the center of string "Tweet", waiting up to ~10 seconds for it to appear with the specified searchRectangle

Example: Using Captured Images To Select and Copy Files:

Use code like this to select a section of items, such as files in File Explorer/Finder, and then copy the files:

Click "FirstEntry" -- Clicks the hot spot of image "FirstEntry"
KeyDown ShiftKey -- Holds down the shift key
Click "LastEntry" -- Clicks the hot spot of image "LastEntry"
KeyUp ShiftKey -- Releases the shift key
RightClick -- Right clicks at the same location as the previous mouse event
Click "CopyMenuItem"

Example: Searching for Images to Confirm a Page Has Loaded:

Sometimes searching for an image twice is helpful when automating UIs that shift as their content loads. This means searching once before the click, and then again as part of the click command:

WaitFor 10, "NewTweet" -- Waits for the image NewTweet to appear
Click{Image:"NewTweet",HotSpot:[80,0]} -- Searches for NewTweet again and sends the click 80 pixels to the right of the image's upper left corner. Note the addition of the "image:" property when including the HotSpot image property

Example: Using an OCR Text Search:

Click text found using OCR:

if ImageFound(text:"Cancel") then click FoundImageLocation() -- Clicks the location of the center of the found string "Cancel"

Example: Using a Specified Location:

Click at a specified location:

Click the RemoteScreenSize*.5 -- Clicks the center point of the screen

Example: Performing a Triple Click:

Use code like this to perform a triple click event:

on TripleClick imageToClick -- Declares a custom command handler named "TripleClick" with parameter "imageToClick"
  //The next 3 lines change the timing between Eggplant Functional commands so that 3 quick clicks can be sent in succession
  put the RemoteWorkInterval into defaultRWI -- Stores the current Remote Work Interval value so that it can be restored later
  set the RemoteWorkInterval to .01 -- Sets the RemoteWorkInterval to .01 seconds, allowing events to be sent to the SUT much more rapidly

  //These 3 lines click the image once and then click twice more in the same location, at the interval specified by the RemoteWorkInterval
  Click imagetoClick -- Sends a click at the hot spot location of the image passed in as a parameter

  //The subsequent 2 clicks occur at the same location as the previous click
  Click
  Click
  set the RemoteWorkInterval to defaultRWI -- Restores the RemoteWorkInterval to the original value
end TripleClick

`DoubleClick` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

note

This definition explains the use of the DoubleClick command for visual scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: DoubleClick Command.

Behavior: Double-clicks the SUT mouse at the location indicated by the parameter specified.
For AI-powered Find by Description searches, the click location is determined by the AI model, at the first location deemed to match the description. The other search types behave as follows: the double-click event occurs at the hot spot location for the first found image, at the center of the first text match found using OCR, or at the coordinate location specified.
If more than one element description, image, or text string is passed, EPF searches for images in the order in which they are listed.

tip

The DoubleTap and DoubleClick commands perform the same event and can be used against both desktop and mobile SUTs.

Parameters: Optional. One or more descriptions (for Find by Description searches), Image References (for captured image searches), Text Properties (for OCR searches), or a single coordinate location.

Example: Using a Find by Description Search:

Provide a description of where to double click, using a Find by Description AI search:

doubleclick description: "the map" -- zooms in on a map displayed on the SUT

Example: Using a Captured Image Search:

Use code like this to scrape values from the SUT using the clipboard:

doubleclick "ConfirmationNumber" -- DoubleClicks the hot spot of image ConfirmatioNumber in order to highlight text on the SUT
typetext controlKey, "c" -- Uses key strokes to perform the copy event on the SUT in order to put the highlighted text into the SUT clipboard
put the remoteClipboard into ConfNum -- Retrieves the content of the SUT's clipboard and stores it in a variable

Example: Performing a Triple Click:

This example is similar to the TripleClick example above, but uses a slightly different approach with DoubleClick, as well as alternate syntax:

to handle TripleClick -- Declares a custom command handler named "TripleClick"
  put the MouseDoubleClickDelay into clickTime -- Stores the MouseDoubleClickDelay in variable clickTime for later use
  set the RemoteWorkInterval to clickTime -- Sets the RemoteWorkInterval, the interval between events sent to the SUT, to the value stored in clickTime
  DoubleClick param(1) -- DoubleClicks the first parameter value (element description, image name, coordinate pair, etc.) passed to the TripleClick handler
  Click -- Clicks the same location as the previous DoubleClick
  set the RemoteWorkInterval to .7 -- Sets the RemoteWorkInterval to .7 second
end TripleClick

`RightClick` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

note

This code definition explains the use of the RightClick command for image-based scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: RightClick Command.

Behavior: Right-clicks the SUT mouse at the location indicated by the parameter specified.
For AI-powered Find by Description searches, the click location is determined by the AI model, at the first location deemed to match the description provided. The other search types behave as follows: the right-click event occurs at the hot spot location for the first found image, at the center of the first text match found using OCR, or at the coordinate location provided.
If more than one element description, image, or text string is passed, EPF searches for images in the order in which they are listed.

Parameters: Optional. One or more descriptions (for Find by Description), one or more Image References (for captured image searches), one or more Text Properties (for OCR searches), or a single coordinate location.

Example: Using a Find by Description Search:

Use a right-click action to bring up an application menu and choose an option, by providing descriptions of where the actions should be performed using Find by Description AI searches:

rightClick description: "chrome in the start bar" -- Opens the Chrome menu from the start bar
click description: "new window option in the menu" -- Chooses the option to open a new window from the menu

Example: Using a Captured Image Search:

Right-click the hot spot of an image named "Login", using the searchType:smooth image property for the image search. Note the addition of the imageName: property when including an image property.

RightClick {imageName: "Login", searchType: "smooth"}

Example: Using an OCR Text Search:

Right click using an OCR search:

rightClick text:"Firefox" -- Right clicks at the center of string "Firefox"

Example: Using Captured Images to Select Files and Copy Them:

Use code like this to select multiple items, such as files in File Explorer/Finder, and then copy the files:

Click "FirstEntry" -- Clicks the hot spot of image "FirstEntry"
KeyDown ControlKey -- Holds down the control key
Click "LastEntry" -- Clicks the hot spot of image "LastEntry"
KeyUp AllKeys -- Releases any and all keys that might be down, including the control key
RightClick -- Right clicks at the same location as the previous mouse event
Click "CopyMenuItem" -- Clicks the hot spot of image "CopyMenuItem"

`MouseButtonDown`, `MouseButtonUp` Commands

tip

You can use AI searches with these commands.
For more information, see Find by Description.

Behavior: Presses (MouseButtonDown) or releases (MouseButtonUp) the mouse button indicated by the parameter. For standard 3-button mice, button 1 is the left button, button 2 is the middle button, and button 3 is the right button. Buttons 4 and 5 control scroll-wheel behavior.

tip

The Swipe, Drag, and Drop commands are useful in many of the same situations as MouseButtonDown 1 / MouseButtonUp 1 in many cases as they all leverage the left mouse button event.
When scrolling on desktop, the ScrollWheelDown/ScrollWheelUp commands are generally easier to use than MouseButtonDown 4 / MouseButtonUp 5.

Parameter: A single mouse button number, 1-8.

Example:

MouseButtonDown 3 -- Holds down the mouse right click on most mice
wait 1
MouseButtonUp 3

Example:

Use code like this to grab an App icon and drag it:

MoveTo "FirefoxIcon" -- Moves the mouse cursor to the hot spot location of FirefoxIcon
MouseButtonDown 1 -- Holds down the left mouse button
MoveTo the remoteScreenSize*(3/4) -- Moves the mouse cursor to the lower right quadrant of the SUT, dragging the Firefox icon along with it
MouseButtonUp 1 -- Releases the left mouse button

Example:

Use code like this to paste content into a mobile SUT:

setremoteClipboard "This is my test message." -- Puts the specified string into the clipboard of the SUT
moveto "MemoEditor" -- Moves the mouse cursor to the hot spot location of the AddressBar image
mousebuttondown 1 -- Holds down the left mouse button
wait 2 -- Waits a sufficient amount of time for the paste dialog to appear
mousebuttonup 1 -- Releases the left mouse button
tap "PasteDialog"

Example:

set the remoteWorkInterval to .1 -- Decreases the remoteWorkInterval so that the mousebuttondown and mousebuttonup events occur .1 seconds apart
repeat until imagefound(image:"NewAddress",waitFor:0) -- Repeats until the image of interest shows on the SUT

//The next two commands use the mouse scroll wheel to scroll down the page one increment at a time
  MouseButtonDown 5
  MouseButtonUp 5
  Wait 1 -- Waits 1 second to allow the UI to settle before searching for "NewAddress"
end repeat
set the remoteWorkInterval to .7

`ScrollWheelDown`, `ScrollWheelUp` Commands

Behavior: Scrolls the mouse wheel up or down. The actual amount of scrolling per increment varies by mouse driver, platform, and application.

tip

An easy way to experiment with the mouse wheel of a SUT is to run a ScrollWheelUp or ScrollWheelDown command with a parameter of 1 in the Ad Hoc Do Box. At times, the direction (up/down) that the scroll wheel needs to turn to scroll the desired direction is counter-intuitive, so if the direction you tried first does not work, try the other direction as well.
In most cases, a Moveto command is required before the ScrollWheelDown/ScrollWheelUp commands, in order to position the cursor over the window or pane you want to scroll.

Parameter: A single integer representing mouse wheel increments (audible clicks on some mice).

Example:

ScrollWheelDown 8

Example:

repeat until imagefound(image:"NewAddress",waitFor:0) -- Repeats until the specified image shows on the SUT
  ScrollWheelDown 4 -- Scrolls 4 increments
  wait 1 -- Waits 1 second to allow the UI to settle before searching for "NewAddress"
end repeat
click "NewAddress"

Example:

Moveto {image:"CustomerPanelHeader",hotspot:[200,150]} -- Moves the cursor to a location 200 pixels to the right and 150 pixels down from the upper left corner of the CustomerPanelHeader image.
ScrollWheelUp 10 -- Scrolls 10 increments.

Example:

Moveto{image:"CustomerPanelHeader",hotspot:[200,150]} -- Moves the cursor to a location 200 pixels to the right and 150 pixels down from the upper left corner of the CustomerPanelHeader image.
ScrollWheelUp 10 -- Scrolls 10 increments.

`MoveTo` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

note

This code definition explains the use of the MoveTo command for visual testing. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: MoveTo Command.

Behavior: Moves the SUT's mouse cursor to the location indicated by the parameter provided.
For AI-powered Find by Description searches, the click location is determined by the AI model, at the first location deemed to match the description provided. The other search types behave as follows: the MoveTo event occurs at the hot spot location for the first found image, at the center of the first text match found using OCR, or at the coordinate location provided.
If more than one element description, image, or text string is passed, EPF searches for images in the order in which they are listed.

tip

Sometimes when working with hover-over elements (menus or tooltips), changing the MouseMoveSpeed global property is necessary so that the hover-over event registers properly on the SUT. Using the default value of 0, the cursor jumps between locations instead of moving smoothly between them.
When transitioning between linked hover-over elements, such as in a series of fly-out menus, changing the MouseMoveMode to avoid the default diagonal movement of the cursor might be necessary.

Parameters: One or more descriptions (for Find by Description), one or more Image References (for captured image searches), one or more Text Properties (for OCR searches), or a single coordinate location.

Example Using a Find by Description Search:

Move the mouse to hover over an element on the SUT screen by providing a description of where to move to, which uses a Find by Description AI search:

moveto description: "settings icon in the menu bar"

Example: Using a Captured Image Search:

Use MoveTo with captured images:

moveto "Link" -- Moves the mouse cursor over the "Link" image
waitfor 5, "Tooltip" -- Waits for the expected "tooltip" image to appear

Example: Using an Exact Location:

Use MoveTo by providing an exact location:

moveto the remotescreensize times .5 -- Moves the mouse cursor to the center of the SUT screen
repeat until imagefound(image:"ContactUsLink",waitFor:0) -- Repeats until the "ContactUsLink" image is visible on the SUT
  if the repeatIndex > 3 then Throw "Image not found.","Contact Us link not found." -- Throws an exception if the repeat loop repeats more than 3 times
  scrollwheeldown 1 -- Scrolls one increment
  wait 1 -- Waits 1 second to allow the UI to settle after the scroll
end repeat

Example: Working with a Horizontal Fly-Out Menu:

Use code like this to use MoveTo to work with a horizontal fly-out menu:

Click "ObjectsMenu"
set the MouseMoveMode to 1 -- Changes the movement behavior of the mouse cursor so that it moves first in the horizontal direction and then in the vertical
set the MouseMoveSpeed to 5 -- Changes the movement speed of the mouse cursor so that it moves smoothly between locations instead of jumping
Moveto "Circles" -- Moves the mouse cursor to the hot spot location of image "Circles"
Moveto text:"Radius" -- Moves the mouse cursor to the center of string "Radius"
Moveto "Value"
set the MouseMoveMode to 0 -- Returns the mouse movement behavior back to the default
set the MouseMoveSpeed to 0 -- Returns the mouse movement speed back to the default

`MoveToEach` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

Behavior: Moves the SUT’s mouse to each of the locations indicated by the parameters, in the order they are listed.

Parameters: One or more descriptions (for Find by Description searches), one or more Image References (for captured image searches), one or more Text Properties (for OCR searches), or coordinate locations.

Example: Using Multiple Find by Description Searches:

Move the SUT's mouse to multiple locations by providing a description of each, which uses a Find by Description AI search to find the different locations:

moveToEach (description: "the first youtube video in the list"), (description: "the second youtube video in the list")

Example: Using Captured Image Searches Alongside Coordinates:

Move the SUT's mouse to multiple locations, defined using two images and a set of coordinates:

MoveToEach "Edit", (780, 91), "Copy"

tip

Sometimes when working with hover-over elements (menus or tooltips), changing the MouseMoveSpeed is necessary so that the hover-over event registers properly on the SUT. Using the default value of 0, the cursor jumps between locations instead of moving smoothly between them.
When transitioning between linked hover-over elements, such as in a series of fly-out menus, changing the MouseMoveMode to avoid the default diagonal movement of the cursor might be necessary.

Example: Working with a Horizontal Fly-Out Menu:

Use code like this to use MoveToEach to work with a horizontal fly-out menu:

Click "ObjectsMenu"
setOptions {MouseMoveMode:2,MouseMoveSpeed:10} -- Changes the movement behavior and movement speed of the mouse cursor.
MovetoEach "Circles", {text:"Radius"}, "Value" -- Moves the mouse cursor to the hot spot or center of each image or text string, respectively, in the specified order.
setOptions {MouseMoveMode:0,MouseMoveSpeed:0} -- Returns the MouseMoveMode and the MouseMoveSpeed to their default values of 0.

`Drag` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

note

This code definition explains the use of the Drag command for visual scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: Drag Command.

Behavior: Presses and holds the SUT’s left mouse button at the location indicated by the parameter specified.
For AI-powered Find by Description searches, the click location is determined by the AI model, at the first location deemed to match the description provided. The other search types behave as follows: the Drag event occurs at the hot spot location for the first found image, at the center of the first text match found using OCR, or at the coordinate location specified.
If more than one element description, image, or text string is passed, EPF searches for images in the order in which they are listed.

tip

Drag, Drop, and DragandDrop work on mobile SUTs and provide the option to choose where to start and release the left mouse button event, while the swipe commands allow you to choose where to start the left mouse button event only.

Example: Using a Find by Description Search:

This example increases the volume by dragging the slider 100px, using a Find by Description AI search to locate the control slider:

drag description: "the vertical bar on the left side of the volume control slider" -- Uses a more specific description to be sure the drag begins at the right location
drop foundimagelocation()+(100,100) -- References the location where the volume control slider was found and where the drag began, and dropping 100px to the right of that location

Example: Using a Captured Image Search to Remove an App from the Dock:

This example reduces the drag speed, and then drags an icon found using a captured image to a desired location, in this case, removing it from the dock:

set the MouseDragSpeed to 5 -- Slows down the speed of the drag action
Drag "FirefoxIcon" -- Begins a drag action on the "FirefoxIcon" image
Drop the remoteScreenSize*(3/4) -- Ends the drag action in the lower right quadrant of the SUT screen
set the MouseDragSpeed to 10 -- Returns the speed of dragging to the default value

Example: Using the Stored Location of a Successful Captured Image Search:

This example uses a variable to store the location of a found image, and then drag from that location to another location determined based on the foundImageLocation.

put imageLocation("Node") into ElementLoc -- Stores the location of the "Node" image in a variable called "ElementLoc".
Drag ElementLoc -- Begins a drag action at the location stored in variable ElementLoc.
Drag ElementLoc - [400,300] -- Ends the drag action 400 pixels to the left and 300 pixels up from the location in ElementLoc.

Example: Using Specified Coordinates:

This example uses coordinate locations based on the screen size to perform both the drag and drop actions:

put the remoteScreenSize into ScreenSize -- Stores the resolution of the SUT in a variable.
Drag ScreenSize*[.7,.5] -- Begins the drag 70% across and 50% down the SUT screen.
Drop ScreenSize*[.7,.2] -- Ends the drag 70% across and 20% down the SUT screen.

`Drop` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

note

This code definition explains the use of the Drop command for visual scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: Drop Command.

Behavior: Moves to the location specified and releases the left mouse button.

tip

Example: Using a Find by Description Search:

This example increases the volume by dragging the slider 100px, using a Find by Description AI search to locate the control slider:

Drag description: "the volume control slider"
Drop foundImageLocation()+(100,0) -- References the location where the volume control slider was found and where the drag began, and dropping 100px to the right of that location

Example: Using Captured Image Searches:

Drag "Apple" -- Begins a drag action on the "Apple" image
Moveto "Polish" -- Continues the drag to the "Polish" image
Drop text:"Bucket" -- Ends the drag at the string "Bucket"

Example: Using Specified Locations:

set ScreenSize to remoteScreenSize() -- Stores the resolution of the SUT in a variable.
Drag ScreenSize times[7/10,1/2] -- Begins the drag 7/10 across and 1/2 down the SUT screen.
Drop ScreenSize times[7/10,1/5] -- Ends the drag 7/10 across and 1/5 down the SUT screen.

`DragAndDrop` Command

tip

You can use AI searches with this command.
For more information, see Find by Description.

Behavior: Presses and holds the SUT’s left mouse button in the first location, moves the mouse to subsequent locations, and releases the mouse button in the final location. All locations are specified as parameters to the command.
For AI-powered Find by Description searches, the event location is determined by the AI model, at the first location deemed to match the description. The other search types behave as follows: the event is performed at the hot spot location for the first found image, at the center of the first text match found using OCR, or at the coordinate location provided.

tip

Parameters: Two or more descriptions (for Find by Description searches), two or more Image References (for captured image searches), two or more Text Properties (for OCR searches), or coordinate pairs.

Example: Using a Find by Description Search:

Using AI-powered Find by Description searches to indicate the drag and drop locations:

DragandDrop (description: "the folder called Test Folder"), (description: "the Recycle Bin icon") -- Drags a folder on the desktop into the Recycle Bin

Example: Using Both Captured Image and OCR Text Searches:

Using captured images to indicate the drag, move, and drop action locations:

DragandDrop "Apple", "Polish", text:"Bucket" -- Begins the drag on the "Apple" image, continues the drag to the "Polish" image, and ends the drag on the string "Bucket"

Example: Using a Specified Location:

set ScreenSize to remoteScreenSize() -- Stores the size of the SUT in a variable.
DragandDrop ScreenSize*[.7,.5],ScreenSize*[.7,.2] -- Begins the drag 70% across and 50% down the SUT screen and ends it 70% across and 20% down.

`MouseLocation` Function

Behavior: Returns the coordinates of the current mouse location. This function works with mobile SUTs as well, though the cursor is generally not visible.

Example:

log MouseLocation() -- Logs the current location of the mouse cursor

Example:

Click "Submit"
put the mouseLocation into StartingPoint -- Stores the current mouse cursor location, based on where the previous action left it, in a variable
Click "Update"
Click StartingPoint -- Clicks the location stored in StartingPoint

Click Command​

DoubleClick Command​

RightClick Command​

MouseButtonDown, MouseButtonUp Commands​

ScrollWheelDown, ScrollWheelUp Commands​

MoveTo Command​

MoveToEach Command​

Drag Command​

Drop Command​

DragAndDrop Command​

MouseLocation Function​

`Click` Command

`DoubleClick` Command

`RightClick` Command

`MouseButtonDown`, `MouseButtonUp` Commands

`ScrollWheelDown`, `ScrollWheelUp` Commands

`MoveTo` Command

`MoveToEach` Command

`Drag` Command

`Drop` Command

`DragAndDrop` Command

`MouseLocation` Function