Reading Test Results

Results from Eggplant Functional (EPF) test runs can be viewed directly in Eggplant Functional, or in our test management tool, called Eggplant Manager, which also provides the ability to analyze test results over multiple test runs.

Viewing Results in Eggplant Functional

The Results pane in the Suite Editor contains records of the tests you have run in the current suite. For each script, there is a record of all of the tests performed, the complete results log for each test, and images captured during script execution.

Test Results

Eggplant Functional records a number of different types of run data such as suite statistics, run history, and log files. These different data types and their uses are described below.

Log File

A Log file file is a plain text file that contains the detailed log for a script run, with a line for each Log entry. Log entries include the following tab-separated values:

• Date and Time. The exact time of this log entry.
• Event Name. The name of the command or function executed, or event that occurred. In the last line, this value is Success or Failure.
• Image Name. The name of the image used for this event; or the text that was typed.
• Location. The screen location where image hot spot was found; or other information.
• Line Number. The line number within a handler.
• Handler Name. The name of the handler which produced this log entry.
• Script Name. The path name of the script containing the handler.

To access the Log file (LogFile.txt), click the arrow in the Log column associated with the script run of interest in the Results pane.

Run History

A RunHistory.csv file is a comma-separated text file that summarizes all of the executions of a script, with one line for each execution. The first line of this file identifies each of the fields:

• Date Run. The date and time of the run.
• Status. The status of the run.
• Time. The elapsed time (duration) of the run.
• # of Errors. The number of errors that occurred during the run.
• # of Warnings. The number of warnings that were logged during the run.
• # of Exceptions. The number of exceptions that occurred during the run.
• Log File. The full path to the LogFile.txt file for the run.
• Return Value. Any value returned by the script that was run.
• Error Message. The final error message for the run, if it was a failure.

This format can be read by many programs, including spreadsheets such as Excel. The second item on each line (Status) is Success or Failure, so simply reading the last line of this file can be a simpler way to find out whether the latest run of a script succeeded, rather than digging into the LogFile.txt file in the individual run folder.

Suite Statistics

The SuiteStatistics.csv file is another comma-separated-value file, this time with one line for each script that has generated statistics. The fields on each line are:

• Script. The name of the script.
• Last Status. Either Success or Failure for the most recent run of this script.
• Runs. The total number of runs of this script.
• Fails. The number of runs of this script which resulted in failure.
• First Run. The date and time of the first recorded run of this script.
• Last Run. The date and time of the most recent run of this script.
• Avg Time(Success). The average length of time taken to run the script when it was successful. (Failed runs are not included in this time.)

Reporting Results

The CSV file format is very easy to import into a spreadsheet program, such as Excel. This section describes some of the other ways to handle your results.

• RunWithNewResults and The Result. See Running from a Primary Script to observe a way to use the RunWithNewResults command and the Result function to write results to a log.
• The ScriptResults function. This function returns the RunHistory.csv file for any script. The script must be in an open suite, or you must provide the full path name.
• SendMail. Used in conjunction with RunWithNewResults or the scriptResults function, the SendMail command can report the results of a test run by email.
• JSONFormat and JSONValue functions. These functions can be used to convert text data between JSON and SenseTalk formats, for communicating results with other processes.
• XMLRPCFormat and XMLRPCValue functions. These functions can be used to convert text data between XML-RPC and SenseTalk formats, for communicating results with other processes.

Using SenseTalk for Reporting

Additional SenseTalk functionality helpful for reporting results:

• The Merge function. This can be used to create sophisticated formatted documents using templates, such as reports in HTML or RTF format.
• File-reading and writing. The file-reading and writing capabilities of SenseTalk can be used to write results to files on any file system that is mounted on the Eggplant Functional computer, including network mounted drives. For more information, see File and Folder Interaction.
• URL access capabilities. You can use these to access remote files through a web interface.
• The Open Socket, Read From Socket and Write To Socket commands. These commands allow a script to communicate directly with another running process on the network (if it has a socket interface). For more information see Socket, Process, and Stream Input and Output.
• The Shell function. The Shell function enables a script to run any Unix command, including passing test result values off to other processes.
• The DoAppleScript command. This command can be used to interact with any AppleScript-able programs on the Eggplant Functional computer or other Mac OS X computers on the network. Some of the uses include populating an Excel spreadsheet or generating a Word document. There is also a DoAppleScript function.

Viewing Results in Eggplant Manager

Eggplant Manager is the easiest and best way to collect and analyze Eggplant Functional test results.

This test management tool is a browser based application that provides a single point of control for Eggplant Functional suites. It provides the ability to define tests, as well as schedule execution of multiple tests simultaneously, as well as a central location for the collection and analysis of results.

To read more about viewing results in Eggplant Manager, see our documentation on the Eggplant Manager Dashboard and Viewing Test Results.