Missing Guide: Alembic with Github Actions

Reading Time: 6 min / Published: 27/3/2020 min /

Recently I've started integrating Github's Actions CI into my existing and new python projects. This also included tests on database driven applications such as a blog with flask. These databases where managed by alembic. You will need an working alembic structure for this guide. This tutorial is only aimed for pure alembic and not for wrappers such as flask-alembic. For the sake of this tutorial we assume you have exposed you database url as variable or function in env.py.

This guide assumes you know about:

We will start with an clean python oriented github workflow for python 3.8 (adjust this to your requirements) for tests:

# This workflow will install Python dependencies, run tests with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Python package

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.8]  # Adjust this to your requirements

    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install alembic pytest
          pip install -r requirements.txt
      - name: Tests with pytest
        run: |
          pytest

To start with alembic a database environment for creating our schemas is required. For this guide postgres is used but you may use any github actions compatible database. To integrate postgres into our workflow add it as a service.

workflow.yaml:

# This workflow will install Python dependencies, run tests with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Python package

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.8]

    # Service containers to run with `container-job`
    services:
      # Label used to access the service container
      postgres:
        image: postgres
        # Provide the password for postgres
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: github_actions
        ports:
          - 5432:5432
        # Set health checks to wait until postgres has started
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps: { steps }

Now you will just need another step to add alembic migrations:

workflow.yaml

# This workflow will install Python dependencies, run tests with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Python package

on:
  push:
    branches: [master]
  pull_request:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.8]

    # Service containers to run with `container-job`
    services:
      # Label used to access the service container
      postgres: { service }

    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install pytest alembic
          pip install -r requirements.txt
      - name: Run migrations
        env:
          DB_USER: postgres
          DB_HOST: localhost
          DB_PASSWORD: postgres
        run: |
          alembic upgrade head
      - name: Tests with pytest
        env:
          DB_USER: postgres
          DB_HOST: localhost
          DB_PASSWORD: postgres
        run: |
          pytest

With this the github workflow is setup an is ready to rock. Now let#s work on the env.py and the project structure. You will need to alter your strucutre to allow migrations to access app when changing the working directory. alembic.ini needs to stay in the root next to project-folder.

project-folder
  • alembic.ini
  • your-app-folder
    • models.py
    • __init__.py
    • tests
    • migrations
      • versions
      • script.py.mako
      • env.py
      • README

First the alembic.ini requires an update:

alembic.ini:

# A generic, single database configuration.

[alembic]
# path to migration scripts
script_location = <your-app-folder>/migrations

# template used to generate migration files
# file_template = %%(rev)s_%%(slug)s

To take advantage of thi new structure change the env.py to provide the database url from a function:

def get_url() -> str:
    return <your-url>

Replace the url now with the function call in the two url references in env.py:

connectable = engine_from_config(
    { kwargs }
    url=get_url(),
)
context.configure(
    url=get_url(),
    { kwargs }
)

Now we need to adjust the env.py to change it's working path to your project folder (where alembic.ini is located). This is required because alembic will not see the your project folder in it's PYTHONPATH.

Change the top section of env.py:

import sys
import os
from os.path import abspath, dirname

sys.path.insert(0, dirname(dirname(dirname(abspath(__file__)))))

{ imports }

Test now by applying alembic upgrade head on your local machine before proceeding.

The next step is to check if alembic is run inside of github CI. That's where we can take advantage of our new get_url function and implement an check against the os.environ vars. Github nicely supplies us with a bunch of it including CI which is True inside of the Action Runner Environment.

env.py:

def get_url() -> str:
    if os.environ.get("CI"):
        return "postgres://postgres:postgres@127.0.0.1"
    else:
    return <your-url>

The supplied credentials (postgres://postgres:postgres@127.0.0.1) are to be directly taken from the workflow.yaml env part of the postgres service definition. Now you're ready to commit and push. These credentials are also supplied to pytest and alembic with the env configuration.

The postgres container may add up to 30s to your action execution time. Keep the limited execution time in mind for workflows

You may want to remove/ keep the pytest part of the workflow.yaml. You system and project may vary a lot from my project and so troubleshooting may differ. There will be an example repository with an appropriate workflow (coming soon!). For in depth help about some mentioning features and strategies take a look at my references below.

My references: