Testing Set-Top Boxes and Gaming Consoles with Eggplant Functional
Set-top boxes and gaming consoles are content delivery devices that use proprietary and modified operating systems, which can make automation and testing difficult. With Eggplant Functional, you use Virtual Network Computing (VNC) or Remote Desktop Protocol (RDP) to automate systems under test (SUTs). However, it typically is not possible to host a VNC or RDP server on a set-top box. This article explains how to use video capture devices combined with Eggplant Functional to test these unique devices.
If you're testing the 4th generation Apple TV, use the methods outlined in Automating Apple TV with iOS Gateway.
How Eggplant Functional Works with Set-Top Boxes and Gaming Consoles
During testing, the VNC/RDP server sends a real-time replica of the SUT environment to the Eggplant Functional client so that you can perform image searches. The VNC/RDP server also accepts keyboard and mouse events sent from the Eggplant Functional client to control the SUT. However, not all operating systems are compatible with a VNC or RDP server, and there is not always a compatible server available on the market.
The two major categories of devices are set-top boxes (Roku, cable TV boxes, etc.) and gaming consoles (XBox, Playstation, etc.). Set-top boxes are used for accessing over-the-top (OTT) media content and usually can run various applications from different providers. Gaming consoles are used primarily for playing video games, but many modern gaming consoles also function as OTT devices.
Hosting a VNC server directly on a set-top box or gaming console is not usually possible, unless you heavily augment the device’s operating system. This process can be highly invasive and disruptive to the typical behavior of the SUT, or not feasible at all. In addition, set-top boxes and gaming consoles are usually controlled by Infrared Remote (IR), Bluetooth, or other types of remote controls rather than by keyboard and mouse events, which means the VNC and RDP methods of SUT control won't work.
There are various consumer-level hardware products and well-supported open-source software solutions available that allow Eggplant Functional to access the device's display in order to perform image and OCR searches while simultaneously controlling the device. Depending on the specific device you are testing and what your testing use case is, a particular approach might be most appropriate.
Before deciding on an automation approach, consider your use cases with your particular device. Are you validating the device's operating system? Are you validating OTT streaming content? Are you validating video game content and game-play? Some approaches for connecting to the device from Eggplant Functionalare suited to particular use cases but are not useful for others.
The content below describes approaches for connecting to a set-top box or gaming console from Eggplant Functional.
Accessing Set-Top Box or Gaming Console Displays
There are two primary appliances that allow access to the display of the device for the purpose of validating the user interface via image and OCR searches:
- Video capture device
- KVM-over-IP device
Video capture devices are affordable, consumer-grade products. They are the best option for testing gaming consoles where game-play is one of the target testing use cases. However, video capture devices require a dedicated "middle-man" desktop or laptop machine in order to display the video capture device output. See Using Video Capture Devices to learn more about this approach.
KVM-over-IP devices are generally considerably more expensive than video capture devices but do not require a dedicated middle-man machine to host the display of the set-top box. For use cases that do not include game-play, a KVM-over-IP device can simplify environment setup. See Using a KVM-over-IP Device for more information. Note: when testing set-top boxes and gaming consoles with KVM-over-IP devices, only the display component of the KVM is used as generally set-top boxes and gaming consoles do not have keyboard and mouse inputs for the KVM to leverage.
Using Video Capture Devices with Eggplant Functional
Video capture devices allow you to output the display of a set-top box or gaming console to a video capture application installed on a standard desktop or laptop system. The typical use of these video capture devices is to allow the recording or streaming of the display of the set-top box or gaming console, which means the use of a video capture device during testing also enables the recording or remote monitoring of the device under test. However, the primary purpose of the video capture device in the Eggplant Functional environment is to allow image- and OCR-based validation of the SUT.
The middle-man desktop system hosts a VNC server or supports RDP connections, which you then use to connect from the Eggplant Functional controller machine. Eggplant Functional then performs its image and OCR searches against the display of the set-top box or gaming console as it appears within the video capture application running on the desktop machine.
Video capture device environment diagram
Recommended Video Capture Devices
We recommend the AverMedia ExtremeCap U3 Game Capture device. The RECentral application for the ExtremeCap U3 is available for Windows and Mac.
Video Capture Device Setup
Follow the installation and setup manual that came with your video capture device, and be mindful of hardware and operating system specifications for the video capture device. We recommend against using a virtual machine (VM) to host the video capture device because the USB sharing capabilities of the VM application might be insufficient for hosting the video capture device. We recommend connecting both the Eggplant Functional controller machine and the middle-man machine to the network via Ethernet, to allow the best performance possible when automating.
After you set up the video capture device and the associated video capture application, and connect the video capture device to your set-top box or gaming console, establish a connection to the desktop system running the video capture application from Eggplant Functional, so that the video capture application is visible through the Viewer window, as shown below.
AverMedia RECentral application
See Connecting to Windows, Mac, and Linux SUTs for more information on how to connect to the desktop system.
AverMedia RECentral Application Settings
The AverMedia RECentral application includes a number of settings that affect the way the video capture device displays the output from the set-top box or gaming console. The default RECentral settings should be appropriate for use with Eggplant Functional. It's important to choose RECentral settings early, before a significant amount of image capturing and OCR tuning occurs, as changing the settings could affect the success of image and OCR searches.
Clicking Ready on the RECentral application allows the display from the set-top box or gaming console to appear, as shown below:
_
_AverMedia RECentral application, displaying a set-top box stream
The AverMedia RECentral application window can be resized, which affects the displayed size of the set-top box or gaming console on the desktop machine. The simplest approach to handling this feature is to use a consistent resolution on the desktop system and then maximize the RECentral application window before script creation and test execution so that the displayed stream of the device is a consistent size.
AverMedia RECentral application (maximized)
For Eggplant Functional to be able to validate the display of your set-top box or gaming console during testing, the RECentral application must remain open and the foremost window on the middle-man machine. We recommend closing all other applications on the middle-man that do not play a role in the automated test, to avoid disruption and other complications during automation.
High Definition Content Protection (HDCP) Restrictions
In most cases, you will use an HDMI cable to connect the set-top box or gaming console to the video capture device. Modern HDMI cable standards include encryption known as HDCP. Depending on the specific set-top box/gaming console and video capture device you are using, HDCP might block viewing video or game content through the video capture application for the purposes of preventing the recording of copyrighted material. Below shows an example of HDCP blocking streaming video content on an OTT device.
However, when using AverMedia ExtremeCap U3, RECentral V.1.3.0.116 Driver 1.0.64.87 (Firmware V61), and Roku and XBox One, we experienced no issues with HDCP blocking viewing of video and game content..
If you encounter HDCP blocking and have an automated testing requirement that includes being able to view video or gaming content, you can consider bypassing HDCP for testing purposes only.
The PlayStation 4 has an option to disable HDCP. For other devices and situations, consider procuring an HDMI splitter that bypasses HDCP. The HDMI splitter sits between the set-top box or gaming console and the video capture device in the environment diagram above. Note that not all HDMI splitters bypass HDCP, and this characteristic is not usually advertised, so order your HDMI splitter carefully. How to Geek has an article that more fully explains HDCP and the methods to bypass it.
Controlling Set-Top Boxes and Gaming Consoles with Eggplant Functional
In addition to getting access to the display of the set-top box or gaming console, you need a method for controlling the set-top box or gaming console during automation. Different devices are controlled in different ways, and some devices are controlled in more than one way. Consider the approaches available for your specific device.
Many set-top boxes and gaming consoles can be controlled by a mobile app. While automating the mobile app with Eggplant Functional is possible, it's not necessarily the simplest approach as it requires constantly switching back and forth between the mobile device and the set-top box or gaming console display during testing.
If your device is a gaming console and your use cases include the requirement to play video games, consider using a free, third-party utility called GIMX, which allows keyboard and mouse control of the gaming console. Generally the use of GIMX is combined with the video capture device approach described above. To learn more about GIMX, see Automating Video Game Consoles with GIMX.
Many set-top boxes and even some gaming console OTT capabilities are controlled via infrared signals. Linux Infrared Remote Control (LIRC) (which is available for Windows as WinLIRC) can be controlled by Eggplant Functional to then control your set-top box or portions of your gaming console. See Testing Set-Top Boxes with LIRC to learn how to use LIRC to control test devices from Eggplant Functional.
The use of LIRC can be combined with either the KVM-over-IP or video capture device approaches discussed above.
Testing Smart TVs with Eggplant Functional
Smart TVs have embedded hardware and software that function very much like a set-top box. Unfortunately, unlike standalone set-top boxes, the large majority of smart TVs don’t have display out ports to allow you to connect the smart TV to a video capture device or KVM-over-IP device, which means the UI of the smart TV is not accessible for Eggplant Functional to perform image and OCR searches.
Some smart TV manufacturers, such as Samsung, provide developer tools including emulator apps or options for displaying the screen of the smart TV on a desktop machine or mobile device, which can then be used in conjunction with Eggplant Functional for automation.