(Co-written with Alan Seiden)
CI has been invaluable to us. As maintainers of PHP’s PDO_IBM and ibm_db2 extensions, we use CI to ensure high quality across platforms.
Platforms supported by PHP’s extensions for Db2 include IBM i, Unix/Linux, and Windows. While our focus is IBM i, we must ensure compatibility with the others. Reliance on manual testing would introduce the risk of broken builds or subtle bugs. Automation is a must.
In this post, we explain how we use CI principles when building and testing PHP’s Db2 extensions.
This article refers to CI steps from these workflow files:
Building extensions on Linux using GitHub Actions
We chose GitHub Actions for our CI implementation. Not only is GitHub the host for the ibm_db2 and PDO_IBM projects, but we knew we could get ideas from existing workflows hosted on GitHub. We selected Linux for its existing Github CI runners and Db2 Docker containers.
Note: As you read about how we used CI and GitHub Actions, you can learn more from these resources:
Defining our workflows
Each workflow’s definition is defined in a .yml file. The .yml file contains such keywords as:
- on: specifies which Git event (e.g. push to a branch) will trigger the workflow, which consists of one or more jobs
- jobs: these are series of steps that run when a workflow is triggered
- strategy.matrix: Lets you run a job under multiple conditions, such as several operating systems or software versions
- steps: these can be a mix of ordinary commands and complex actions
A complex action can be packaged in its own repository for reuse. GitHub checkout, for example, is packaged in github.com/actions/checkout. To use version 3 of the checkout action, for example, specify
uses: actions/checkout@v3. The step will execute the checkout action. This packaging capability makes GitHub Actions easy to extend while keeping workflows concise.
Another action sets up an entire PHP build environment across a matrix of PHP versions. This action enables us to test our extensions under multiple versions of PHP. Each PHP version is passed by the job into the
php-version variable when calling the action. The action will automatically run multiple jobs to handle all PHP versions, as well as including any specified extension dependencies.
Obtain and use the Db2 Client library
When our action builds extensions on a Unix variant, such as Linux, the action needs to obtain the Db2 client library. We can make this process easier by configuring an action to download the library, extract it, and save it for later use. (This action is not needed with IBM i, because IBM i comes complete with its own variant of the Db2 client library.)
Next, the workflow specifies the steps to build the extension. As with any other PHP extension, we generate the build system using
phpize, configure the build system with
./configure (with a flag to point to the Db2 library), and finally, make (compile) it. These steps are sufficient to tell us whether we’ve broken the Unix build.
Building the extensions on Windows
Since the Db2 CLI driver already supports Windows (even with Microsoft’s different compiler), and GitHub Actions provides Windows builders, Windows support isn’t much extra work.
First, we’ll need a Windows-based PHP build environment. We can use the PHP builder action for Windows by PHP developer Christoph Becker.
The process of the Windows build is basically the same as on Linux, with minor differences. Instead of POSIX shell or bash, we use a PowerShell script that downloads and extracts the driver. The difference in commands is small: instead of
wget, we use
Invoke-WebRequest; instead of
unzip, we use
We found one bug with the Windows build system for the PHP Db2 extensions. The build system only checked for a single generic CLI driver name. On 64-bit Windows, the CLI driver library is named differently than on other platforms, so it wasn’t found. The fix was simple; we checked for the additional name.
Testing: Docker helps us test with Db2
Although we’ve been building the extensions successfully, we haven’t been testing them yet. To test, we’ll need an active Db2 instance. Setting up Db2 on Linux would normally be a lot of effort, but there’s an IBM Db2 Docker container that does much of the work for us. We are fortunate that GitHub Actions doesn’t seem to place any restriction on using Docker containers, because they are very helpful for testing services. All that’s needed is to pull the container from Docker, then start the container, accepting the default arguments.
More challenging was setting many subtle environment variables that influence test behaviour. Two examples:
- The Db2 CLI requires an INI file to specify the DSN pointing to our database, then an environment variable to point to the path of the INI.
- Complication: the connection string must be specified in full (“
ibm:DSN=SAMPLE“), because the Db2 CLI tests expect a connection string rather than just a DSN name. If the tests encounter only a DSN name, the tests try to append values to it, causing the tests to fail.
- Complication: the connection string must be specified in full (“
- PHP 8.1’s test runner script introduced the “skip cache.” Tests can include a “skip” section, which determine if the test can be run. PHP 8.1’s test runner script can cache these skip tests to avoid running these same tests every time. Unfortunately, the tests for PDO_IBM weren’t written with the skip cache in mind, so we had to disable the skip cache.
Once we established these settings, many of the tests were fine out of the box. Most problems revealed by tests were usually small or insignificant – issues related to the order of array entries, or PHP 8.1 tightening up the type system. In newer PHP releases such as 8.1, PHP blocks many of our “nonsense” test values before they ever reach the extension we were trying to test.
Caching downloaded files
While we could download supporting files on every job run, repeated downloads tax the Content Distribution Network (CDN). We reduced these download delays by configuring oft-used files to be cached. When testing Db2, for example, we have chosen to cache the IBM Db2 driver and the Db2 Docker container.
Cache the Db2 driver
To cache the driver, we added a caching rule before the normal download/extract commands. The caching rule declares the path to be cached and a key name for the cache. The key name should be unique per operating system (i.e. Windows, Ubuntu, etc.); we use a variable specific to each OS to make it obvious. Caching is achieved using three rules:
- The first rule checks if the data is already cached. If so, the rule restores the data from cache. Regardless if any data was restored, the rule sets a variable regarding the state of the cache.
- The second rule checks if the the data wasn’t cached. If so, the rule downloads the desired ZIP file from the CDN and extracts the data.
- A cache storage rule is automatically added to the end of a successful run. This rule caches the data, if not already cached.
Cache the Docker container
Caching the Docker container is similar, but restoring it only restores an image; we need to do some more work to load its contents. (If we didn’t need to worry about caching, we could just do a docker pull and create the container immediately.) The process is similar to downloading the driver, above, but with additional steps.
- Check if the data is already cached and, if so, restore it.
- Depending whether the data was restored from cache, one of these two rules will run:
- If it was cached, we tell Docker to load the container image from the archive that was cached.
- If not cached, we tells Docker to pull (download) the container image, then save it to an archive for later use.
- Either way, an additional rule then runs, to create the instance from the image.
- Upon success, we cache the data.
Continuous Integration is a great tool for detecting problems proactively. Collaboration on ibm_db2 and PDO_IBM has been made easier by using CI to test for high quality on all supported platforms. CI can also go beyond smoke tests and unit tests. We have been considering CI to build binary packages for Windows. We would reuse most of the same techniques from tests, but adding additional rules and triggers.