Test runner

Running your test suite in an automated workflow helps increase certainty when merging.

Use Unity - Test runner to run your Unity tests.

Optional to include test coverage, use Unity - Code Coverage

Basic setup

By default, the test runner will run both playmode and editmode tests.

Create or edit the file called .github/workflows/main.yml and add a job to it.

Personal license

Personal licenses require a one-time manual activation step.

Make sure you acquire and activate your license file and add it as a secret.

Then, define the test step as follows:

- uses: game-ci/unity-test-runner@v3
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    projectPath: path/to/your/project
    githubToken: ${{ secrets.GITHUB_TOKEN }}

Professional license

Make sure you have set up these variables in the activation step.

• UNITY_EMAIL (should contain the email address for your Unity account)
• UNITY_PASSWORD (the password that you use to login to Unity)
• UNITY_SERIAL (the serial provided by Unity)

Define the test step as follows:

- uses: game-ci/unity-test-runner@v3
  env:
    UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
    UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
    UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
  with:
    projectPath: path/to/your/project
    githubToken: ${{ secrets.GITHUB_TOKEN }}

License Server

If you host your own Unity license server you can provide its url using unityLicensingServer. A floating license will be acquired before the tests run, and returned after.

Example of use:

- uses: game-ci/unity-test-runner@v3
  with:
    projectPath: path/to/your/project
    unityLicensingServer: [url to your license server]

That is all you need to test your project.

Testing Unity packages

The test runner offers basic support for testing Unity packages, with a few caveats.

Additional setup

After performing Basic setup, three extra steps are required for testing Unity packages.

Setting packageMode

To indicate that the test runner is being run for a Unity package rather than a Unity project, set the packageMode configuration option to true.

- uses: game-ci/unity-test-runner@v3
  with:
    packageMode: true

Passing a unityVersion

Make sure that you explicitly pass the unityVersion argument, as it's necessary for the test runner to test Unity packages.

- uses: game-ci/unity-test-runner@v3
  with:
    unityVersion: [your desired Unity version]

See the GitHub Issue for more details on why this is necessary.

Choosing a package directory

Make sure that the directory of the package being tested is not the same as the repository's root directory. This condition will cause the test runner to hang and fail.

Currently, the only known workarounds for this failure are these two options:

1. Restructure your repository to place the package being tested in a subdirectory of the repository's root directory.
2. Somehow temporarily move the package being tested to a subdirectory of the repository's root directory within the GitHub Workflow where you're using the test runner.

Whichever workaround you choose, make sure that the projectPath corresponds to wherever the package being tested is located at the time the test runner is called in your GitHub Workflow.

See the GitHub Issue for more details and potential workarounds.

Caveats

• As stated above, the action will fail if the directory of the package being tested is the same as the root directory of the repository (this issue is being tracked here).
• As stated above, excluding unityVersion or passing a unityVersion configuration of auto when testing a Unity package will cause the test runner to fail. The Unity version must be manually set in the configuration options (this issue is being tracked here).
• The test runner can only test packages on Linux runners - Windows runners are currently not supported (this issue is being tracked here)
• Packages with dependencies outside of the Unity Registry have not been tested an may or may not work with the test runner (this issue is being tracked here).
• There is currently no caching set up for the testing of Unity packages (this issue is being tracked here).
• If using the customImage parameter to use a custom Docker image to test the package, that image must have jq installed, or else the test runner will error. It is used to add the package being tested to a temporary Unity project's package manifest at runtime.

Viewing test results

The test results can be viewed from a GitHub Status Check.

To get this functionality, simply provide the GitHub Token in order to view the tests results from a check run.

- uses: game-ci/unity-test-runner@v3
  with:
    githubToken: ${{ secrets.GITHUB_TOKEN }}

If you choose not to provide the githubToken, you may still upload the artifacts in order to access them.

Storing test results

To be able to access the test results, they need to be uploaded as artifacts.

To do this, it is recommended to use the official Github Actions upload artifact action.

By default, Test Runner outputs its results to a folder named artifacts.

- uses: actions/upload-artifact@v2
  if: always()
  with:
    name: Test results
    path: artifacts

Test results can now be downloaded as Artifacts in the Actions tab.

Specifying artifacts folder

You can specify a different artifactsPath in the test runner and reference this path using the id of the test step.

- uses: game-ci/unity-test-runner@v3
  id: myTestStep

- uses: actions/upload-artifact@v2
  if: always()
  with:
    name: Test results
    path: ${{ steps.myTestStep.outputs.artifactsPath }}

Getting coverage results

The results for the test code coverage are stored in the folder CodeCoverage and the path can be read by the step's outputs.coveragePath. These coverage options can be configured as part of the test and by default will create an XML and HTML web page reports.

- uses: actions/upload-artifact@v2
  if: always()
  with:
    name: Coverage results
    path: ${{ steps.myTestStep.outputs.coveragePath }}

The coverage results will be generated for whatever tests were selected to run. If both editmode and playmode are selected by using all, then the results will have the combined coverage of both test modes.

Note Coverage results are generated with root permission so to move or edit the output in later steps you may need to use elevated permissions such as sudo.

Caching

In order to make test runs (and builds) faster, you can cache Library files from previous runs.

To do so, simply add Github Actions' official cache action before any unity steps.

- uses: actions/cache@v2
  with:
    path: path/to/your/project/Library
    key: Library-MyProjectName-TargetPlatform
    restore-keys: |
      Library-MyProjectName-
      Library-

This simple addition could speed up your test runs by more than 50%.

Note that caching in this manner only applies to testing Unity Projects, not Unity Packages (see Caveats).

Configuration options

Below options can be specified under with: for the unity-test-runner action.

unityVersion

Version of Unity to use for testing the project. Use "auto" to get from your ProjectSettings/ProjectVersion.txt. ⚠️ If testing a Unity Package, this field is required and cannot be set to "auto".

required: false default: auto

customImage

Specific docker image that should be used for testing the project. If packageMode is true, this image must have jq installed.

- uses: game-ci/unity-test-runner@v3
  with:
    customImage: 'unityci/editor:2020.1.14f1-base-0'

required: false default: ""

projectPath

Specify the path to your Unity project or package to be tested. The path should be relative to the root of your project.

required: false default: <your project root>

customParameters

Custom parameters to configure the test runner.

For example, you may refer to the Unity Test Framework command line arguments for options that could help with configuring your tests.

Parameters must start with a hyphen (-) and may be followed by a value (without hyphen).

Parameters without a value will be considered booleans (with a value of true).

- uses: game-ci/unity-test-runner@v3
  with:
    customParameters: -profile SomeProfile -someBoolean -someValue exampleValue

required: false default: ""

testMode

The type of tests to be run by the test runner.

Options are: All, PlayMode, EditMode, and Standalone.

Note: All only runs PlayMode and EditMode. You must explicitly specify Standalone for it to run.

required: false default: All

artifactsPath

Path where the test results should be stored.

In this folder a folder will be created for every test mode.

required: false default: artifacts

coverageOptions

Options for configuring code coverage, this configures the -coverageOptions parameter described unity's CodeCoverage documentation: Using Code Coverage in batchmode.

By default, this parameter is configured to generate an HTML report with additional metrics. This should be fine for most projects, but the webpage report can be quite large for projects with hundreds or thousands of large files.

You can add arguments for assemblyFilters to specify an assembly to filter files that results are generated for. This is commonly used to ignore test files or external libraries. See the Using Code Coverage in batchmode. For example, you can add the argument assemblyFilters:+my.assembly.* to only include files in the assemblies that match +my.assembly.*, the * is a wildcard. This will require making an assembly definition to manage scripts, see Unity's documentation on Assembly definitions for how to create and manage assembly definitions.

To get coverage when testing a package, make sure you pass assemblies from the package you want covered to the assemblyFilters option.

required: false default:generateAdditionalMetrics;generateHtmlReport;generateBadgeReport

useHostNetwork

Initializes Docker using the host network.

This is useful if Unity needs to access a local server that was started as part of your workflow.

Options are: true, false

required: false default: false

sshAgent

SSH Agent path to forward to the container.

This is useful if your manifest has a dependency on a private GitHub repo.

required: false default: ``

sshPublicKeysDirectoryPath

SSH directory path to mount in the container on ~/.ssh. Must be used with sshAgent.

It is recommended to have only public keys in this directory, and rely on sshAgent to manage private keys.

This is useful if your manifest has a dependency on multiple private GitHub repos and you need to use multiple SSH deploy keys.

required: false default: ``

gitPrivateToken

GitHub Private Access Token (PAT) to pull from GitHub.

This is useful if your manifest has a dependency on a private GitHub repo.

required: false default: ``

githubToken

Token to authorize access to the GitHub REST API. If provided, a check run will be created with the test results.

It is recommended to use githubToken: ${{ secrets.GITHUB_TOKEN }}, but creating the check from a fork of your repo may require using a Personal Access Token.

Reference the GitHub Checks API docs for details on creating CI tests with the Checks API.

required: false default: ``

checkName

Name for the check run that is created when a github token is provided.

It may be useful to customize the check name if, for example, you have a job matrix with multiple unity versions.

required: false default: Test Results

packageMode

Whether the tests are being run for a Unity package instead of a Unity project.

If true, the action can only be run on Linux runners, and any custom docker image passed to this action must have jq installed.

NOTE: may not work properly for packages with dependencies outside of the Unity Registry.

See Testing Unity packages for more information.

required: false default: false

Complete example

A complete workflow that tests all modes separately could look like this:

name: Test project

on: [push, pull_request]

jobs:
  testAllModes:
    name: Test in ${{ matrix.testMode }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        projectPath:
          - test-project
        testMode:
          - playmode
          - editmode
          - standalone
    steps:
      - uses: actions/checkout@v2
        with:
          lfs: true
      - uses: actions/cache@v2
        with:
          path: ${{ matrix.projectPath }}/Library
          key: Library-${{ matrix.projectPath }}
          restore-keys: |
            Library-
      - uses: game-ci/unity-test-runner@v3
        id: tests
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          projectPath: ${{ matrix.projectPath }}
          testMode: ${{ matrix.testMode }}
          artifactsPath: ${{ matrix.testMode }}-artifacts
          githubToken: ${{ secrets.GITHUB_TOKEN }}
          checkName: ${{ matrix.testMode }} Test Results
          coverageOptions: 'generateAdditionalMetrics;generateHtmlReport;generateBadgeReport;assemblyFilters:+my.assembly.*'
      - uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Test results for ${{ matrix.testMode }}
          path: ${{ steps.tests.outputs.artifactsPath }}
      - uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Coverage results for ${{ matrix.testMode }}
          path: ${{ steps.tests.outputs.coveragePath }}

Complete Example (Testing a Package)

A complete workflow that tests all modes separately (for a package rather than a project) could look like this:

name: Test package

on: [push, pull_request]

jobs:
  testAllModes:
    name: Test in ${{ matrix.testMode }}
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        projectPath:
          - test-package
        unityVersion: '2020.3.0f1' # some version must be included for package testing
        testMode:
          - playmode
          - editmode
    steps:
      - uses: actions/checkout@v2
        with:
          lfs: true
      - uses: game-ci/unity-test-runner@v3
        id: tests
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          packageMode: true
          projectPath: ${{ matrix.projectPath }}
          unityVersion: ${{ matrix.unityVersion }}
          testMode: ${{ matrix.testMode }}
          artifactsPath: ${{ matrix.testMode }}-artifacts
          githubToken: ${{ secrets.GITHUB_TOKEN }}
          checkName: ${{ matrix.testMode }} Test Results
          coverageOptions: 'generateAdditionalMetrics;generateHtmlReport;generateBadgeReport;assemblyFilters:+my.assembly.*'
      - uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Test results for ${{ matrix.testMode }}
          path: ${{ steps.tests.outputs.artifactsPath }}
      - uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Coverage results for ${{ matrix.testMode }}
          path: ${{ steps.tests.outputs.coveragePath }}