Mouse Events and Control
This section describes the Eggplant Functional commands and functions that execute mouse events or return information about the mouse cursor on the SUT. Many of these commands and functions, including Click, DoubleClick, Drag, Drop, DragandDrop, MouseButtonDown, and MouseLocation(), work against mobile SUTs as well as desktop SUTs, which is helpful when creating scripts that automate both mobile and desktop platforms. SenseTalk also includes a set of mobile commands and functions, a subset of which (Tap, DoubleTap, Press, etc.) are compatible with desktop SUTs.
Click Command
This code definition explains the use of the Click command for image-based scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: Click Command.
Behavior: Clicks the SUT mouse on the hot spot (Image) or center (OCR) of the first location found. If more than one image or text string is passed, Eggplant Functional searches for images in the order in which they are listed.
The Tap and Click commands perform the same event and can be used against both desktop and mobile SUTs.
Parameters: One or more Image References or Text Properties (for OCR), or a single coordinate location.
Example:
repeat with each item of ["Option1","Option2","Option3"] //Repeats through a list of images
click it//Clicks the hot spot of the current image in the list
end repeat
Example:
click{text:"Tweet",waitFor:10,searchRectangle:["AnchorLeft","AnchorRight"]} //Clicks the center of string "Tweet", waiting up to ~10 seconds for it to appear with the specified searchRectangle
Example:
Use code like this to select a section of items, such as files in File Explorer/Finder, and then copy the files
Click "FirstEntry" //Clicks the hot spot of image "FirstEntry"
KeyDown ShiftKey //Holds down the shift key
Click "LastEntry" //Clicks the hot spot of image "LastEntry"
KeyUp ShiftKey //Releases the shift key
RightClick //Right clicks at the same location as the previous mouse event
Click "CopyMenuItem"
Example:
Sometimes searching for an image twice, once before the click, is helpful when automating UIs that shift as their content loads
WaitFor 10, "NewTweet" //Waits for the image NewTweet to appear
Click{Image:"NewTweet",HotSpot:[80,0]} //Searches for NewTweet again and sends the click 80 pixels to the right of the image's upper left corner. Note the addition of the "image:" property when including the HotSpot image property
Example:
if ImageFound(text:"Cancel") then click FoundImageLocation() //Clicks the location of the center of the found string "Cancel"
Example:
Click the RemoteScreenSize*.5 //Clicks the center point of the screen
Example:
//Use code like this to perform a triple click event
on TripleClick imageToClick //Declares a custom command handler named "TripleClick" with parameter "imageToClick"
//The next 3 lines change the timing between Eggplant Functional commands so that 3 quick clicks can be sent in succession
put the RemoteWorkInterval into defaultRWI //Stores the current Remote Work Interval value so that it can be restored later
set the RemoteWorkInterval to .01 //Sets the RemoteWorkInterval to .01 seconds, allowing events to be sent to the SUT much more rapidly
//These 3 lines click the image once and then click twice more in the same location, at the interval specified by the RemoteWorkInterval
Click imagetoClick //Sends a click at the hot spot location of the image passed in as a parameter
//The subsequent 2 clicks occur at the same location as the previous click
Click
Click
set the RemoteWorkInterval to defaultRWI //Restores the RemoteWorkInterval to the original value
end TripleClick
DoubleClick Command
This code definition explains the use of the DoubleClick command for image-based scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: DoubleClick Command.
Behavior: Double-clicks the SUT mouse on the hot spot (Image) or center (OCR) of the first location found. If more than one image or text string is passed, Eggplant Functional searches for images in the order in which they are listed.
The DoubleTap and DoubleClick commands perform the same event and can be used against both desktop and mobile SUTs.
Parameters: One or more Image References or Text Properties (for OCR), or a single coordinate location.
Example:
//Use code like this to scrape values from the SUT using the clipboard
doubleclick "ConfirmationNumber" //DoubleClicks the hot spot of image ConfirmatioNumber in order to highlight text on the SUT
typetext controlKey, "c" //Uses key strokes to perform the copy event on the SUT in order to put the highlighted text into the SUT clipboard
put the remoteClipboard into ConfNum //Retrieves the content of the SUT's clipboard and stores it in a variable
Example:
//This example is similar to the TripleClick example above, but uses a slightly different approach with DoubleClick as well as alternate syntax
to handle TripleClick //Declares a custom command handler named "TripleClick"
put the MouseDoubleClickDelay into clickTime //Stores the MouseDoubleClickDelay in variable clickTime for later use
set the RemoteWorkInterval to clickTime //Sets the RemoteWorkInterval, the interval between events sent to the SUT, to the value stored in clickTime
DoubleClick param(1) //DoubleClicks the first parameter value (image name, coordinate pair, etc.) passed to the TripleClick handler
Click//Clicks the same location as the previous DoubleClick
set the RemoteWorkInterval to .7 //Sets the RemoteWorkInterval to .7 second
end TripleClick
RightClick Command
This code definition explains the use of the RightClick command for image-based scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: RightClick Command.
Behavior: Right-clicks the SUT mouse on the hot spot (Image) or center (OCR) of the first location found. If more than one image or text string is passed, Eggplant Functional searches for images in the order in which they are listed.
Parameters: One or more Image References or Text Properties (for OCR), or a single coordinate location.
Example:
RightClick {imageName: "Login", searchType: "smooth"} //Right-clicks the hot spot of image "Login", using the searchType:smooth image property for the image search. Note the addition of the "imageName:" property when including an image property.
Example:
rightClick text:"Firefox" //Right clicks at the center of string "Firefox"
moveTo text:"Send to" //Moves the mouse cursor to the center of string "Send to"
Example:
//Use code like this to select multiple items, such as files in File Explorer/Finder, and then copy the files
Click "FirstEntry" //Clicks the hot spot of image "FirstEntry"
KeyDown ControlKey //Holds down the control key
Click "LastEntry" //Clicks the hot spot of image "LastEntry"
KeyUp AllKeys //Releases any and all keys that might be down, including the control key
RightClick //Right clicks at the same location as the previous mouse event
Click "CopyMenuItem"
MouseButtonDown, MouseButtonUp Commands
Behavior: Presses (MouseButtonDown) or releases (MouseButtonUp) the mouse button indicated by the parameter.
For standard 3-button mice, button 1 is the left button, button 2 is the middle button, and button 3 is the right button. Buttons 4 and 5 control scroll-wheel behavior.
The Swipe, Drag, and Drop commands are useful in many of the same situations as MouseButtonDown/Up 1 in many cases as they all leverage the left mouse button event.
When scrolling on desktop, the ScrollWheelDown/Up commands are generally easier to use than MouseButtonDown/Up 4/5.
Parameter: A single mouse button number, 1-8.
Example:
MouseButtonDown 3//Holds down the mouse right click on most mice
wait 1
MouseButtonUp 3
Example:
MoveTo "FirefoxIcon" //Moves the mouse cursor to the hot spot location of FirefoxIcon
MouseButtonDown 1 //Holds down the left mouse button
MoveTo the remoteScreenSize*(3/4) //Moves the mouse cursor to the lower right quadrant of the SUT, dragging the Firefox icon along with it
MouseButtonUp 1 //Releases the left mouse button
Example:
//Use code like this to paste content into a mobile SUT
setremoteClipboard "This is my test message." //Puts the specified string into the clipboard of the SUT
moveto "MemoEditor" //Moves the mouse cursor to the hot spot location of the AddressBar image
mousebuttondown 1 //Holds down the left mouse button
wait 2 //Waits a sufficient amount of time for the paste dialog to appear
mousebuttonup 1 //Releases the left mouse button
tap "PasteDialog"
Example:
set the remoteWorkInterval to .1 //Decreases the remoteWorkInterval so that the mousebuttondown and mousebuttonup events occur .1 seconds apart
repeat until imagefound(image:"NewAddress",waitFor:0) //Repeats until the image of interest shows on the SUT
//The next two commands use the mouse scroll wheel to scroll down the page one increment at a time
MouseButtonDown 5
MouseButtonUp 5
Wait 1 //Waits 1 second to allow the UI to settle before searching for "NewAddress"
end repeat
set the remoteWorkInterval to .7
ScrollWheelDown, ScrollWheelUp Commands
Behavior: Scrolls the mouse wheel up or down. The actual amount of scrolling per increment varies by mouse driver, platform, and application.
An easy way to experiment with the mouse wheel of a SUT is to run a ScrollWheelUp or ScrollWheelDown command with a parameter of 1 in the Ad Hoc Do Box. At times, the direction (up/down) that the scroll wheel needs to turn to scroll the desired direction is counter-intuitive, so if the direction you tried first does not work, try the other direction as well.
In most cases, a Moveto command is required before the ScrollWheelDown/Up commands, in order to position the cursor over the window or pane you want to scroll.
Parameter: A single integer representing mouse wheel increments (audible clicks on some mice.).
Example:
ScrollWheelDown 8
Example:
repeat until imagefound(image:"NewAddress",waitFor:0) //Repeats until the specified image shows on the SUT
Moveto remotescreensize()*(1/2) //Moves the mouse cursor to the center of the SUT screen
repeat until imagefound(image:"NewAddress",waitFor:0) //Repeats until the specified image shows on the SUT
ScrollWheelDown 4 //Scrolls 4 increments
wait 1 //Waits 1 second to allow the UI to settle before searching for "NewAddress"
end repeat
click "NewAddress"
Example:
Moveto {image:"CustomerPanelHeader",hotspot:[200,150]} //Moves the cursor to a location 200 pixels to the right and 150 pixels down from the upper left corner of the CustomerPanelHeader image.
ScrollWheelUp 10 //Scrolls 10 increments.
Example:
Moveto{image:"CustomerPanelHeader",hotspot:[200,150]} //Moves the cursor to a location 200 pixels to the right and 150 pixels down from the upper left corner of the CustomerPanelHeader image.
ScrollWheelUp 10 //Scrolls 10 increments.
MoveTo Command
This code definition explains the use of the MoveTo command for image-based scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: MoveTo Command.
Behavior: Moves the SUT’s mouse cursor to the hot spot of the first image found, or in the location indicated by a coordinate pair.
Sometimes when working with hover-over elements (menus or tooltips), changing the MouseMoveSpeed global property is necessary so that the hover-over event registers properly on the SUT. Using the default value of 0, the cursor jumps between locations instead of moving smoothly between them.
When transitioning between linked hover-over elements, such as in a series of fly-out menus, changing the MouseMoveMode to avoid the default diagonal movement of the cursor might be necessary.
Parameters: One or more Image References or Text Properties (for OCR), or a single coordinate location.
Example:
moveto "Link" //Moves the mouse cursor over the Link image
waitfor 5, "Tooltip" //Waits for the expected tooltip to appear
Example:
moveto the remotescreensize times .5 //Moves the mouse cursor to the center of the SUT screen
repeat until imagefound(image:"ContactUsLink",waitFor:0) //Repeats until the "ContactUsLink" image is visible on the SUT
if the repeatIndex > 3 then Throw "Image not found.","Contact Us link not found." //Throws an exception if the repeat loop repeats more than 3 times
scrollwheeldown 1 //Scrolls one increment
wait 1 //Waits 1 second to allow the UI to settle after the scroll
end repeat
Example:
Use code like this to work with a horizontal fly-out menu:
Click "ObjectsMenu"
set the MouseMoveMode to 1 //Changes the movement behavior of the mouse cursor so that it moves first in the horizontal direction and then in the vertical
set the MouseMoveSpeed to 5 //Changes the movement speed of the mouse cursor so that it moves smoothly between locations instead of jumping
Moveto "Circles" //Moves the mouse cursor to the hot spot location of image "Circles"
Moveto text:"Radius" //Moves the mouse cursor to the center of string "Radius"
Moveto "Value"
set the MouseMoveMode to 0 //Returns the mouse movement behavior back to the default
set the MouseMoveSpeed to 0 //Returns the mouse movement speed back to the default
MoveToEach Command
Behavior: Moves the SUT’s mouse to the locations indicated by the parameters, in the order they are listed.
Parameters: One or more Image References or Text Properties (for OCR).
Example:
MoveToEach "Edit", (780, 91), "Copy"
Sometimes when working with hover-over elements (menus or tooltips), changing the MouseMoveSpeed is necessary so that the hover-over event registers properly on the SUT. Using the default value of 0, the cursor jumps between locations instead of moving smoothly between them.
When transitioning between linked hover-over elements, such as in a series of fly-out menus, changing the MouseMoveMode to avoid the default diagonal movement of the cursor might be necessary.
Example:
Use code like this to work with a horizontal fly-out menu:
Click "ObjectsMenu"
setOptions {MouseMoveMode:2,MouseMoveSpeed:10} //Changes the movement behavior and movement speed of the mouse cursor.
MovetoEach "Circles", {text:"Radius"}, "Value" //Moves the mouse cursor to the hot spot or center of each image or text string, respectively, in the specified order.
setOptions {MouseMoveMode:0,MouseMoveSpeed:0} //Returns the MouseMoveMode and the MouseMoveSpeed to their default values of 0.
Drag Command
This code definition explains the use of the Drag command for image-based scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: Drag Command.
Behavior: Presses and holds the SUT’s left mouse button in the hot spot of the first image found, or in the location indicated by a coordinate pair.
Drag and Drop and DragandDrop work on mobile SUTs and provide the option to choose where to start and release the left mouse button event, while the Swipe commands allow you to choose where to start the left mouse button event only.
Parameters: One or more Image References or Text Properties (for OCR), or a single coordinate location.
Example:
set the MouseDragSpeed to 5 //Slows down the speed of the drag action
Drag "FirefoxIcon" //Begins a drag action on the "FirefoxIcon" image
Drop the remoteScreenSize*(3/4) //Ends the drag action in the lower right quadrant of the SUT screen
set the MouseDragSpeed to 10 //Returns the speed of dragging to the default value
Example:
put imageLocation("Node") into ElementLoc //Stores the location of the "Node" image in a variable.
Drag ElementLoc //Begins a drag action at the location stored in variable ElementLoc.
Drag ElementLoc - [400,300] //Ends the drag action 400 pixels to the left and 300 pixels up from the location in ElementLoc.
Example:
put the remoteScreenSize into ScreenSize //Stores the resolution of the SUT in a variable.
Drag ScreenSize*[.7,.5] //Begins the drag 70% across and 50% down the SUT screen.
Drop ScreenSize*[.7,.2] //Ends the drag 70% across and 20% down the SUT screen.
Drop Command
This code definition explains the use of the Drop command for image-based scripting. For information about using this command for WebDriver connections, see WebDriver Mouse and Keyboard Events: Drop Command.
Behavior: Moves to the given location and releases the left mouse button.
Drag and Drop and DragandDrop work on mobile SUTs and provide the option to choose where to start and release the left mouse button event, while the Swipe commands allow you to choose where to start the left mouse button event only.
Parameters: One or more Image References or Text Properties (for OCR), or a single coordinate location.
Example:
Drag "Apple" //Begins a drag action on the "Apple" image
Moveto "Polish" //Continues the drag to the "Polish" image
Drop text:"Bucket" //Ends the drag at the string "Bucket"
Example:
set ScreenSize to remoteScreenSize() //Stores the resolution of the SUT in a variable.
Drag ScreenSize times[7/10,1/2] //Begins the drag 7/10 across and 1/2 down the SUT screen.
Drop ScreenSize times[7/10,1/5] //Ends the drag 7/10 across and 1/5 down the SUT screen.
DragAndDrop Command
Behavior: Presses and holds the SUT’s left mouse button in the first location, moves the mouse to subsequent locations, and releases the mouse button in the final location.
Drag and Drop and DragandDrop work on mobile SUTs and provide the option to choose where to start and release the left mouse button event, while the Swipe commands allow you to choose where to start the left mouse button event only.
Parameters: Two or more locations, identified by Image References, Text Properties (for OCR), or coordinate pairs.
Example:
DragandDrop "Apple", "Polish", text:"Bucket" //Begins the drag on the "Apple" image, continues the drag to the "Polish" image, and ends the drag on the string "Bucket"
Example:
set ScreenSize to remoteScreenSize() //Stores the resolution of the SUT in a variable.
DragandDrop ScreenSize*[.7,.5],ScreenSize*[.7,.2] //Begins the drag 70% across and 50% down the SUT screen and ends it 70% across and 20% down.
MouseLocation Function
Behavior: Returns the coordinates of the current mouse location. This function works with mobile SUTs as well, though the cursor is generally not visible.
Example:
log MouseLocation() //Logs the current location of the mouse cursor
Example:
Click "Submit"
put the mouseLocation into StartingPoint //Stores the current mouse cursor location, based on where the previous action left it, in a variable
Click "Update"
Click StartingPoint //Clicks the location stored in StartingPoint