Android: Platform Testing

Test Development Workflow

Compatibility Test Suite (CTS)

The CTS( download ) runs on a desktop machine and executes test cases directly on attached devices.
The CTS is an automated testing harness that includes two major software components:

• The CTS tradefed test harness runs on your desktop machine and manages test execution.
• Individual test cases are executed on the Device Under Test (DUT). The test cases are written in Java as JUnit tests and packaged as Android .apk files to run on the actual device target.

The Compatibility Test Suite Verifier (CTS Verifier, download ) is a supplement to the CTS.
The CTS Verifier is a tool for manual testing and includes the following software components:

• The CTS verifier app that is executed on the DUT and collects the results.
• The executable(s) or script(s) that are executed on the desktop machine to provide data or additional control for some test cases in the CTS Verifier app.

Setting up CTS

Physical environment

CTS tests require a Wi-Fi network that supports IPv6, can treat the Device Under Test (DUT) as an isolated client, and has an internet connection.

Desktop machine setup

Before running the CTS, insall recent versions of both Android Debug Bridge (adb) and Android Asset Packaging Tool (AAPT):

• platform-tools
Download SDK Platform-Tools for Linux.

unzip platform-tools_r28.0.1-linux.zip

• aapt

apt-get install aapt

• Java Development Kit (JDK)

$ sudo apt-get install openjdk-8-jre
$ sudo apt-get install openjdk-8-jdk

• CTS files
• Download and open the CTS packages
Android 9 is the release of the development milestone code-named P. The source code for the following tests, including tests for instant apps, can be synced with the 'android-cts-9.0_r6' tag in the open-source tree.
• Android 9.0 R6 Compatibility Test Suite (CTS) - ARM
• Android 9.0 R6 CTS Verifier - ARM
• Android 9.0 R6 CTS for Instant Apps - ARM
• Download and open the latest version of the CTS Media Files.
• CTS Media 1.4
• Device detection
Make sure the following is ready:
• DUT is USB debug-enabled
• adb
• Make sure that you are in the plugdev group.
Use id to see what groups you are in. Use sudo usermod -aG plugdev $LOGNAME to add yourself to the plugdev group.
• Connect to your device
Verify that your device is connected by running the:

adb devices

Issue any adb command with the -d flag to target your device.

Android device setup

First API level build property

Certain CTS requirements depend on the build a device was originally shipped with.
The device manufacturers may define the build-time property:

 ro.product.first_api_level

OEMs can add PRODUCT_PROPERTY_OVERRIDES into their device.mk file to set this property:

#ro.product.first_api_level indicates the first api level that the device has been commercially launched on.

PRODUCT_PROPERTY_OVERRIDES +=\
ro.product.first_api_level=21

For devices launched with Android 9 :

PRODUCT_PROPERTY_OVERRIDES +=\
ro.product.first_api_level=28

Running CTS tests

Trade Federation (tradefed or TF for short) is a continuous test framework designed for running tests on Android devices. It's a Java application which runs on a host computer, and communicates to one or more Android devices using ddmlib (the library behind DDMS) over adb.

TF users will fulfill any of three primary roles: Developer, Integrator, and Test Runner:

• Developers
Developers spend the majority of their time creating TF modules that are written in Java.
• Integrators
Integrators spend the majority of their time creating XML test configurations, or command files (which are written in a simple shell-like language).
• Test Runners
Test Runners spend the majority of their time executing tests and generally making sure that test results are being generated

Getting started with CTS tradefed

1. Connect at least one device.
2. Press the home button to set the device to the home screen at the start of CTS.
3. Device must be kept in a stationary position (to avoid triggering sensor activity)
4. Do not press any keys on the device while the CTS is running.
5. Launch the CTS console by running the cts-tradefed script from the folder where the CTS package has been unzipped

$ ./android-cts/tools/cts-tradefed

6. Start the default test plan (contains all test packages)

cts-tf> run cts --plan CTS

7. Alternately, run the CTS plan listed

cts-tf> run cts --plan

8. View test progress and results reported on the console.

For more commands, Using the CTS v2 console.

Vendor Test Suite (VTS) & Infrastructure

Overview

Video and Codelab Tutorials

Systems Testing With VTS

The Vendor Test Suite (VTS) automates HAL and OS kernel testing.

Establishing a test environment

• Install Python development kit

sudo apt-get install python-dev

• Install Protocol Buffer tools (for Python)

sudo apt-get install python-protobuf
sudo apt-get install protobuf-compiler

• Install Python virtual environment-related tools

sudo apt-get install python-virtualenv
sudo apt-get install python-pip

Debugging Native Android Platform Code