How we improved testing Ubuntu on WSL – and how you can too!

As the popularity of Windows Subsystem for Linux increases, the Ubuntu development team is committed to delivering a first class experience for Linux developers on Windows. To achieve this we’ve made improvements to our automated testing workflows via the creation of WSL-specific GitHub actions. In this post, Ubuntu WSL engineer Eduard Gómez Escandell talks us through the motivation for this approach and how you can implement these actions for your own CI workflows.  

The importance of testing can never be overstated. When you develop a product, the last thing you want is to spend valuable time playing whack-a-mole with bugs or having your inbox flooded by issues asking for help. This importance is not lost on Ubuntu developers, where all packages must pass their test suite before having patches accepted.

In fact, testing is not enough, we need automated testing. Every package on Ubuntu has a set of autopkgtests which, as the name suggests, are automatically-run package tests. These tests run on a range of architectures and releases. For example, here are systemd’s and snapd’s autopkgtest logs. We also have a list of currently running tests. 

However, there is one platform missing on that list. It looks very much like regular Ubuntu but has its own set of particulars – Ubuntu on Windows Subsystem for Linux. When we started support for the platform we relied on two factors: the fact that we have robust automated test suites for existing Ubuntu images, and a battery of manual tests before every release to ensure that everything works as expected. However, we are only so many people, and a whole distro has a very large surface area to cover. Furthermore, even if we catch an error, it may take a while for us to understand and resolve the issue.

Introducing new GitHub Actions

Ideally, package developers themselves would identify these issues ahead of time and attempt to fix them before updating their packages. However, the most popular CI runner at the moment – GitHub Actions – does not support WSL2 as a platform.

Here is where we come in! The Ubuntu WSL team has developed a suite of GitHub Actions to set up your WSL machine for you. They may not look that fancy, but that is because our aim was to keep them as simple as possible and help others avoid the headache of creating them themselves.

These actions enable us to run an automated suite of tests every time we submit a PR to the Ubuntu/WSL repository. This makes us more certain that we aren’t introducing any regressions. We also use it in Ubuntu/GoWSL, an interface between your Go code and the WSL API that offers wrappers around common actions to manage WSL distros. These are particularly complicated workflows, as they live on both Windows and Linux. Most packages only need to run on Linux so they should be simpler to set up. You can view Ubuntu/wsl-actions-example for a simpler example workflow.

How you can get involved

We cannot test every package every time we want to make a small change to WSL, so we’d like to ask the community for help. In order to make both Ubuntu and your packages more stable, we would encourage developers to ensure that they test on WSL as well.

To help you get started, we’ve created a tutorial on how to set up a simple testing pipeline. If you’re stuck on any step, or you have feedback, please get in touch via a GitHub issue on our issues page.

Testing WSL with an example

Let’s say you have a very simple CI where you only need to run a script. Traditionally, the structure would look like the following:

name: Test on regular Ubuntu

on:
  pull_request:
  workflow_dispatch:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v3
      - name: Run tests
        shell: bash
        run: python3 test.py

But now you want to run this test on WSL. Unfortunately it is not as easy as changing line 11 to wsl. Changing it to windows-latest and attempting to use WSL doesn’t work either. If you run wsl --install --d Ubuntu in a GitHub workflow you’ll get a message saying an error occurred during installation.

To address this you’ll need a self-hosted runner (such as an Azure Windows machine) that supports WSL. Again we cannot simply install the GitHub actions runner there. If run as a service, the actions runner will run as session 0, so it cannot run Windows store applications such as WSL or Ubuntu.

The trick is to do the following:

1. Set the Github actions runner as a startup application.
2. Set up your machine to log on automatically.

This will make the Github actions runner run on session 1, which is enough for it to use WSL. Notice in the snippet below that we start the virtual machine on demand. Most of the time the VM will be shutdown, only starting when there is a WSL specific job to run. This helps with minimising costs, but also prevents security concerns around automatic logon, as most of the time that machine cannot be reached or breached. Next, we can set up the workflow:

name: Test on WSL

# Prevent multiple runs from interfering with each other
concurrency: azure-vm

on:
  pull_request:
  workflow_dispatch:
  push:
    branches: [main]

env:
  az_name: the_name_of_your_machine
  az_resource_group: the_resource_group_on_azure

jobs:
  vm-setup:
    name: Set up Azure VM
    runs-on: ubuntu-latest
    steps:
      - name: Azure login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_VM_CREDS }}
      - name: Start the Runner
        shell: bash
        run: |
          az vm start --name ${{ env.az_name }} --resource-group ${{ env.az_resource_group }}

  test:
    name: Run QA checks on the Azure VM
    runs-on: self-hosted
    needs: vm-setup
    steps:
      - name: Set up WSL
        uses: Ubuntu/WSL/.github/actions/wsl-install@main
        with:
            distro: Ubuntu
      - name: Checkout your repo into WSL
        uses: Ubuntu/WSL/.github/actions/wsl-checkout@main
        with:
            distro: Ubuntu
            working-dir: "~/myrepo"
      - name: Run your tests
        uses: Ubuntu/WSL/.github/actions/wsl-bash@main
        with:
            distro: Ubuntu
            working-dir: "~/myrepo"
            exec: python3 test.py WSL

  vm-stop:
    name: Deallocate the Azure VM
    runs-on: ubuntu-latest
    needs: [vm-setup, tests]
    if: always()
    steps:
      - name: Azure login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_VM_CREDS }}
      - name: Deallocate the Runner
        shell: bash
        run: |
          az vm deallocate --name ${{ env.az_name }} --resource-group ${{ env.az_resource_group }}

That is a lot of text, so here is a simplified diagram of what is going on:

The workflow starts by powering up your Windows machine. The auto logon starts a user session. Windows then launches the startup programs, including the actions-runner. Importantly, the runner runs on session 1, which means Windows Store applications can be used.

The runner picks up the test job and starts running its tests. The actions called are from the Ubuntu/WSL repo, but you can run any action you need, bearing in mind that they’ll run on Windows.

The actions used in the example above are:

• wsl-install updates WSL and the distro you want to use, to ensure you’re in the latest version. You can use Ubuntu-18.04, Ubuntu-20.04, or Ubuntu-22.04 or simply Ubuntu to run on the latest LTS. You can even push it to Ubuntu-Preview to use the daily build.
• wsl-checkout checks out your repository into WSL. Simple enough but convenient nonetheless.
• wsl-bash runs an arbitrary bash script in WSL. This may seem redundant but it deals with the challenge of passing a script from the original Yaml to Windows then to WSL without introducing the feared carriage return character that Bash is easily confused by. It also sets the WSL_UTF8 environment variable so that logs are readable on GitHub.

For simplicity’s sake the diagram above shows only the wsl-bash script.

Finally, once the test job is done, the vm-stop job powers off the machine, which also halts the now-idle actions runner as well.

You can find these actions at https://github.com/ubuntu/WSL/tree/main/.github/actions. 

And don’t forget to check out the examples in this repository: https://github.com/ubuntu/wsl-actions-example.

Thanks for reading!

We hope that as many developers as possible to feel confident when testing WSL, either via these actions or their own implementation. This way other Linux developers on Windows will benefit from a more robust experience, and you can sleep soundly knowing that fewer bugs will need your attention.

To find out more about Ubuntu on WSL check out the following links:

• ubuntu.com/wsl
• ubuntu.com/blog/wsl-ubuntu-2022-year-in-review
• ubuntu.com/tutorials