Organizing Your Testing
Have a Plan
When you start developing a test script, you should already be familiar with the application you are testing, and know exactly what you intend to test. Simply put, you should have a test plan.
If you have conducted manual tests of the software in the past, you might have developed manual test scripts—written sequences of steps that a person can perform to verify that particular parts of the system are functioning correctly. Because Eggplant Functional interacts with software just like a person does, these manual test scripts are an ideal starting point for developing automated test scripts.
If you do not have any formal tests already, you might be wondering where to start. Some people choose to model the entire application, and map out a strategy to systematically exercise every part of it. Others choose to focus on areas of a system that present the highest risk, defining the highest risk as the greatest potential cost of failure or the most likely to contain bugs.
Whatever approach you take, the most productive method is to start small, get some simple tests working, and then build larger and more ambitious tests as you progress. Remember that a small amount of automation is already better than none; and every small, manageable step brings you closer to your ultimate goal.
Think in Terms of End-User Functionality
Eggplant Functional interacts with the system under test (SUT) like a "virtual user", making it ideal for testing user-level functionality. Think in terms of the high-level tasks that a user performs with your software: What are the major functions that must be working for end users to get value from your product? How can you test these functions? Depending on how much time you have, go down a level and repeat this process.
Kent Beck presents this concept of testing very well in his book Extreme Programming Explained (Addison Wesley Longman, 2000). Many people want to relate testing coverage to a percentage of the application. Beck makes the point that you should not set goals that you cannot measure. When people say they test 80% of their application, what exactly are they counting? Would 100% testing mean that all valid and invalid data cases were tested, or that every permutation of system options and application options was tested? A more meaningful goal is to test all of the application’s essential functions.
Be Aware of the Visible Interface
Automated testing with Eggplant Functional requires that you be keenly aware of how a real user would go through the steps that you want to test. You always have to keep in mind what the user would see and how the user would interact with what is actually visible on screen.
For example, suppose you are testing these actions:
- Double-click an icon to launch a web browser.
- Type a URL in the address field.
- Press Return to load the page.
If these actions are executed too quickly, the browser might not be open when the text is typed. A human user would know to wait for the browser to open before typing a URL, so your script needs to know it, too. A WaitFor "Address field" command would do the trick.
Most operating systems let you customize the interface, at least to some extent. This ability is very user-friendly, but not so friendly to image-based testing. For example, on Mac, you can select the color that is used to indicate the currently chosen menu item. If you capture images of the menu items highlighted blue and then change the highlight color to gray, Eggplant Functional continues to look for blue and probably fails.
It is best if your SUTs are computers that you control so that they are not customized by other users between test cycles. If this is not the case, try to establish standard visual settings for all of the SUTs you use. You might even create scripts that check your settings and change back any that are different from the standards.
Note: For most operating systems, the default settings are the best to adopt. This can make new SUTs ready to use right out of the box; and even SUTs that have been around the block are likely to use many of the default settings.
For Mac, find the following settings in System Preferences > General:
- Appearance: Blue or graphite, (in Appearance)
- Use LCS font smoothing when available (in Appearance)
- Show scrollbars (in Appearance)
- Color (In Displays)
- Solid desktop color (in Desktop & Screen Saver)
For Microsoft Windows systems, find the following settings in the Display Properties control panel:
- "Use the following method to smooth edges of screen fonts": Off
- All settings in the Taskbar and Start Menu Properties control panel
- All settings in the Folder Options control panel.
Regardless of the operating system you are testing, standardize the color depth at which all of your SUTs of that type run. Some older computers cannot display 32-bit color ("millions of colors") and even if they are able to display 24-bit color, the colors that they send through VNC can be very different from those displayed on a 32-bit system.
Some of these color differences might fall within your search type tolerances, but others can cause a script that was generated on a 32-bit system to fail on a 24-bit system, and vice versa. If you have SUTs with different maximum color depths, you need to decide whether you are going to use the lower color depth for all of them, or capture an additional set of images if they are not matching up.
Finally, keep in mind that solid desktop backgrounds are easier to use than pictures or patterns. If you capture an image with part of a picture in the background, Eggplant Functional might not be able to recognize it at another place on the desktop.
Record the System State
One of the benefits of testing on multiple SUTs is the ability to observe how an application responds to different system set-ups. Here are some considerations:
- What operating system is running on the SUT?
- How much memory does it have?
- Which System preferences differ between SUTs?
By documenting these and similar data, you can address each configuration with tests to validate its impact on the application.
Organizing Your Images
This article offers practical advice for organizing images and reducing your own image search time.
An Image Collection is a folder of related images that are all acceptable versions of a basic image. For example, an Image Collection might contain a "normal" image of a button, a selected version, and a disabled version. It helps to have a theme, such as separate Image Collections for each of your applications windows or panels.
Any image folder in your suite can be used as an Image Collection, but keep in mind that Eggplant Functional looks for every image in the Image Collection, so a very large folder could slow down your script execution and cause Eggplant Functional to find matches you are not interested in.
Naming conventions make it a quite a bit easier to identify your images, and much easier to identify your co-workers’ images. The table below shows one QA team’s image-naming conventions; it might be a helpful starting point for your own guidelines.
|Menu item||mi_ prefix|
|List box||lb_ prefix|
|Text box||tb_ prefix|
|Group box||gb_ prefix|
|Pull-down menu||pd_ prefix|
|Selected pull-down menu item||pds_prefix|
|Tab menu image||tm_ prefix|
|Dialog title||t_ prefix|
|Screen capture||sc_ image|
Use a Test Management Tool
Test Management tools help your team to stay organized and provide a single point of control for your tests. This is critical to keep in mind as your testing infrastructure grows and you automate more of your tests.
Eggplant provides a test management tool called Eggplant Manager that allows you to construct tests from a variety of existing Eggplant Functional tests, schedule the execution of those tests, and then collect and analyze results. This is the easiest and best way to execute multiple Eggplant Functional tests or instances of a test simultaneously.
To read more about Eggplant Manager, see the Eggplant Manager Documentation.
Eggplant also provides the ability to integrate into any Continuous Integration (CI) software or Application Lifecycle Management (ALM) tool with Eggplant Integrations. For specifics on these integrations, see the Eggplant Integrations Documentation.
There is nothing more frustrating than being deep into your testing when suddenly, due to inactivity on the screen, your screen saver comes on causing your test to fail. Luckily, there are ways to keep this from happening.
Mac and Windows SUTs
Mouse Jiggler is a free piece of software you can use to keep Mac and Windows SUTs awake. Its only function is to emulate mouse input and jiggle the mouse pointer back and forth. This feature can be useful for testing purposes when monitoring a long operation that requires little or no activity on the screen so that you don't have to change your screen saver settings. You can find Mouse Jiggler for Windows >here, and for Mac here.
Some mobile devices have a setting you can choose so that the device doesn't go to sleep when the user isn't using it. The procedure for setting this option is different depending on the device; follow the appropriate steps below.
- Go to Settings on the device.
- Tap Display & Brightness.
- Tap Auto-Lock.
- Tap Never.
Some Android devices allow the display to never turn off, while others don't. This is a device-specific option. If your Android SUT doesn't have this option, you can use the SenseTalk PressHomeButton command to wake the screen up.