Reading Test Results
Viewing Results in Eggplant Functional
The Results pane in the Suite Editor contains records of the tests you have run in the current suite. For each script, there is a record of all of the tests performed, the complete results log for each test, and images captured during script execution.
Eggplant Functional records a lot of different forms of run data, such as suite statistics, run history, and log files. These different kinds of data and possible ways to use them are described below.
A LogFile.txt file is a plain text file that contains the detailed log for a script run, with a line for each Log entry. Log entries include the following tab-separated fields:
- Date and Time. The exact time of this log entry.
- Event Name. The name of the command or function executed, or event that occurred. In the last line, this value is Success or Failure.
- Image Name. The name of the image used for this event; or the text that was typed.
- Location. The screen location where image hot spot was found; or other information.
- Line Number. The line number within a handler.
- Handler Name. The name of the handler which produced this log entry.
- Script Name. The path name of the script containing the handler.
A RunHistory.csv file is a comma-separated text file that summarizes all of the executions of a script, with one line for each execution. The first line of this file identifies each of the fields:
- Date Run. The date and time of the run.
- Status. The status of the run.
- Time. The elapsed time (duration) of the run.
- # of Errors. The number of errors that occurred during the run.
- # of Warnings. The number of warnings that were logged during the run.
- # of Exceptions. The number of exceptions that occurred during the run.
- Log File. The full path to the LogFile.txt file for the run.
- Return Value. Any value returned by the script that was run.
- Error Message. The final error message for the run, if it was a failure.
This format can be read by many programs, including spreadsheets such as Excel. The second item on each line (Status) is Success or Failure, so simply reading the last line of this file can be a simpler way to find out whether the latest run of a script succeeded, rather than digging into the LogFile.txt file in the individual run folder.
The SuiteStatistics.csv file is another comma-separated-value file, this time with one line for each script that has generated statistics. The fields on each line are:
- Script. The name of the script.
- Last Status. Either Success or Failure for the most recent run of this script.
- Runs. The total number of runs of this script.
- Fails. The number of runs of this script which resulted in failure.
- First Run. The date and time of the first recorded run of this script.
- Last Run. The date and time of the most recent run of this script.
- Avg Time(Success). The average length of time taken to run the script when it was successful. (Failed runs are not included in this time.)
The CSV file format is very easy to import into a spreadsheet program, such as Excel. This section describes some of the other ways to handle your results.
- RunWithNewResults and the Result. See Running from a Master Script to observe a way to use the RunWithNewResults command and the Result() function to write results to a log.
- The ScriptResults(). This function returns the RunHistory.csv file for any script. The script must be in an open suite, or you must provide the full path name.
- SendMail. Used in conjunction with RunWithNewResults() or the scriptResults() function, the SendMail command can report the results of a test run by email.
- JSONFormat() and JSONValue(). These functions can be used to convert text data between JSON and SenseTalk formats, for communicating results with other processes.
- XMLRPCFormat() and XMLRPCValue(). These functions can be used to convert text data between XML-RPC and SenseTalk formats, for communicating results with other processes.
Using SenseTalk for Reporting
Additional SenseTalk functionality helpful for reporting results:
- The Merge() function. This can be used to create sophisticated formatted documents using templates, such as reports in HTML or RTF format.
- File-reading and writing. The file-reading and writing capabilities of SenseTalk can be used to write results to files on any file system that is mounted on the Eggplant Functional computer, including network mounted drives.
- URL access capabilities. You can use these to access remote files through a web interface.
- The Open Socket, Read From Socket and Write To Socket commands. These commands allow a script to communicate directly with another running process on the network (if it has a socket interface). For more information see Socket, Process, and Stream Input and Output.
- The Shell() function. The Shell() function enables a script to run any Unix command, including passing test result values off to other processes.
- The DoAppleScript command. This command can be used to interact with any AppleScript-able programs on the Eggplant Functional computer or other Mac OS X computers on the network. Some of the uses include populating an Excel spreadsheet or generating a Word document. There is also a DoAppleScript function. See the DoAppleScript command for more information.
Viewing Results in Eggplant Manager
Eggplant Manager is the easiest and best way to collect and analyze Eggplant Functional test results.
This test management tool is a browser based application that provides a single point of control for Eggplant Functional suites. It provides the ability to define tests, as well as schedule execution of multiple tests simultaneously, as well as a central location for the collection and analysis of results.