Caching pipx packages in GitHub Actions

An example on how to set up pipx to cache packages in GitHub Actions.

We use pipx to install common Python tools in our CI/CD system. Installing those tools can be sped up by caching. Since GitHub Actions comes with pipx pre-installed on the runner images¹, setting up caching for pipx is trickier than usual².

How pipx works under the hood is covered extensively on its website. Note that the explanation assumes default path configurations, which is not the case for the GitHub runners. The runner images override the default pipx paths³.

The gist is that virtual environments are placed at $PIPX_HOME/venvs and binary apps are placed in $PIPX_BIN_DIR.

In theory, caching these paths should be enough:

# Don't do this!
steps:
  - uses: actions/checkout@v4
  - uses: "actions/setup-python@v5"
    with:
      python-version: "3.13"
  - id: setup-pipx
    shell: bash
    run: |
      echo "cache_home_dir=$PIPX_HOME" >> $GITHUB_OUTPUT
      echo "cache_bin_dir=$PIPX_BIN_DIR" >> $GITHUB_OUTPUT
      echo "version=$(pipx --version)" >> $GITHUB_OUTPUT
  - id: cache-pipx
    uses: actions/cache@v3
    with:
      # Restoring cache will fail, since we try to to write to /opt/pipx_bin.
      # Pipx will also use the system Python, not the one we installed above.
      path: |
        ${{ steps.setup-pipx.outputs.cache_home_dir }}
        ${{ steps.setup-pipx.outputs.cache_bin_dir }}
      key: pipx-${{ steps.setup-pipx.outputs.version }}
  - name: Install
    shell: bash
    run: |
      pipx install poetry==2.1.1
      pipx install awscli==1.38.4

However, this approach is riddled with issues:

Cache restoration fails, since Ubuntu runners set PIPX_BIN_DIR=/opt/pipx_bin³, which is not writable
Cache restoration leaves PATH in a bad state
- Pipx installed packages are first on PATH, which interferes with system site-packages
- Pipx installed packages may be missing from PATH, which requires running pipx ensurepath
We install packages irregardless of whether the cache was hit
Pre-installed packages are installed with the system Python version
Pipx installed packages are also installed with the system Python version
The cache key does not contain the Python version, which breaks when Python is upgraded
Caching pre-installed packages intereferes with the hosted cache of the runners⁴.

To put it concisely, the pre-installed packages and the pipx paths set by runners shouldn’t be cached.

Solution

Instead, we should instruct pipx to use a different path for our packages, and cache it. We’ll add the cached binaries directory to our PATH using $GITHUB_PATH to make the installed apps available.

steps:
  - uses: actions/checkout@v4
  - uses: "actions/setup-python@v5"
    id: setup-python
    with:
      python-version: "3.13"
  - id: setup-pipx
    shell: bash
    run: |
      # custom cached dir
      cache_home_dir="$HOME/.local/pipx"
      cache_bin_dir="$HOME/.local/pipx_bin"

      # add cached dir to PATH
      echo "$cache_bin_dir" >> $GITHUB_PATH

      echo "cache_home_dir=$cache_home_dir" >> $GITHUB_OUTPUT
      echo "cache_bin_dir=$cache_bin_dir" >> $GITHUB_OUTPUT
      echo "version=$(pipx --version)" >> $GITHUB_OUTPUT
  - id: cache-pipx
    uses: actions/cache@v3
    with:
      path: |
        ${{ steps.setup-pipx.outputs.cache_home_dir }}
        ${{ steps.setup-pipx.outputs.cache_bin_dir }}
      # cache based on versions and current file contents - please adjust filename
      key: pipx-${{ steps.setup-pipx.outputs.version}}-${{ steps.setup-python.outputs.python-version}}-${{ hashFiles('./.github/workflows/change-to-current-workflow-file.yaml') }}
  - name: Install
    if: ${{ steps.cache-pipx.outputs.cache-hit != 'true' }}
    shell: bash
    env:
      # install in cached dir
      PIPX_HOME: ${{ steps.setup-pipx.outputs.cache_home_dir }}
      PIPX_BIN_DIR: ${{ steps.setup-pipx.outputs.cache_bin_dir }}
      # use correct python
      PIPX_DEFAULT_PYTHON: ${{ steps.setup-python.outputs.python-path }}
    run: |
      pipx install poetry==2.1.1
      pipx install awscli==1.38.4

Albeit verbose, we don’t do much more than adjust paths and ensure that they are cached. You’ll have to update the file path used in the cache key: to suit your needs.

We can check the installed packages using:

steps:
  # ...
  - name: Verify install paths
    shell: bash
    run: |
      echo "pipx packages in normal location:"
      pipx list

      echo "pipx packages in cached location:"
      PIPX_HOME=${{ steps.setup-pipx.outputs.cache_home_dir }} \
      PIPX_BIN_DIR=${{ steps.setup-pipx.outputs.cache_bin_dir }} \
      pipx list

You should see something along:

pipx packages in normal location:

venvs are in /opt/pipx/venvs
apps are exposed on your $PATH at /opt/pipx_bin
manual pages are exposed at /home/runner/.local/share/man
   package ansible-core 2.18.2, installed using Python 3.12.3
    - ansible
    - ansible-config
    - ansible-console
    - ansible-doc
    - ansible-galaxy
    - ansible-inventory
    - ansible-playbook
    - ansible-pull
    - ansible-test
    - ansible-vault
   package yamllint 1.35.1, installed using Python 3.12.3
    - yamllint

pipx packages in cached location:

venvs are in /home/runner/.local/pipx/venvs
apps are exposed on your $PATH at /home/runner/.local/pipx_bin
manual pages are exposed at /home/runner/.local/share/man
   package awscli 1.38.4, installed using Python 3.13.2
    - aws
    - aws.cmd
    - aws_bash_completer
    - aws_completer
    - aws_zsh_completer.sh
   package poetry 2.1.1, installed using Python 3.13.2
    - poetry

The output reveals which packages in the runner come pre-installed, using system Python (3.12), and which are installed by us in the cached location, using a custom Python version (3.13).

Our pipx installs are now a lot faster and they don’t interfere with the pre-installed pipx packages.