GitLab Pipelines

As seen in the CI overview, developers can easily interact and iterate while their changes are in their local machines or in the GitLab pipelines. In this section we will cover the versatility and possible uses of these pipelines to get the most out of them.

Pipelines consist of stages and jobs. Stages are logical groupings of jobs. Our pipeline does not rely on stages to sequence the jobs. Instead, it uses the Directed Acyclic Graph feature from GitLab. This allows us to start executing jobs as soon as their conditions are met. This improves parallelism and reduces the runtime of our pipelines.

Pipeline stages and jobs

A job is the building block of the CI and jobs are logically grouped into stages. Our CI consists of the following stages and jobs, which may or may not be executed depending on the type of pipeline:

• Validation:
  • pipeline-variable-check: performs a series of checks on the CI input variables for the pipeline type that is going to be executed.
  • check-versionlock-diff: checks if the version-locked EOS version of a specific tag matches the EOS version in the main branch. Used for EOS regression pipelines against the last CTA tag with the aim of warning that some tests might fail due to interface changes.
• Setup:
  • modify-versionlock: this job modifies the version lock file so that the ctageneric container image for the current commit is generated with some specific version of CTA, EOS and/or XRootD.

• Analysis: series of static analysis.

  • cppcheck: lightweight static analysis tool. For cppcheck, a number of errors are suppressed based on the .cppcheck-supression file.
  • clang-format-report: check the clang format for the lines that differ from the main branch and generates a git patch.
  • clang-format-apply: apply the git patch generated by the clang-format-report job. You can oonly trigger this job manually.
  • catalogue-schema-version-check: checks that the Catalogue version specified in the CTA repository matches the version of the Catalogue repo.
  • SonarCloud (external): To complement this stage, we also analyze the project with SonarCloud, the analysis results can be found here. It is not executed synchronously with the pipeline because the analysis is heavy and takes too long to be integrated into the developer workflow. To run it we use a GitHub mirror of the CTA repository that does the analysis for every commit on the main branch.

  Since it is not directly integrated into the pipeline, and as part of the development workflow, it is recommended to check the files you are modifying, to see if there are any issues that can be easily fixed in the same merge request. You should also check the results of the analysis run after your commits reach the main branch to check if the committed code generated any new issues.

• Build:

  • build-cta-srpm: builds the srpms for the current commit. The output is used by the rpm build stage.
  • build-cta-rpm: build the rpms for the current commit. The output can used to build a container image for the development setup or uploaded to a repository as a tagged CTA version.
  • export-docs: converts the man pages of the project as markdown and uploads then as an artifact to be conusmed by the eoscta-docs project.
• Build Image: A "base" container image (ctageneric) is built with common software used by all the containers in the kubernetes cluster spawned for testing, including the RPMSs generated in the previous stage. These RPMs are included in the image as a local repository, they are not installed. After the image is built it is uploaded to a private registry.
  • build-ctageneric-from-artifacts: build the ctageneric container image from the RPMs generated in the build stage.
  • build-ctageneric-from-repos: build the ctageneric container image from the specified repositories.
• Tests: set of tests that do not require a full development setup but do not fit into the unit tests of the project, i.e requiring some external resources.
  • test-cta-valgrind: runs valgrind tests to check for memory leaks.
  • integration-test-cta: tests executable invocation and CTA's threading code.
  • unit-test-postgresql: series of CTA Catalogue unit tests run against a live Postgres DB.
  • unit-test-oracle: series of CTA Catalogue unit tests run against a live Oracle DB.
  • test-cta-release: checks that the cta-release RPM works as expected.

• System Tests: The tests run in this stage require the existence of a working CTA deployment, a virtualized environment is the best option as it can be recreated quickly. They are mainly used to test workflows, compliance of APIs and integration of external software. These tests run inside our dedicated GitLab runner machines with a custom minikube an mhvtl setup.

  • test-client: tests rest API compliance; file immutability; archival, retrieval, eviction, retrieval abort and deletion of 10.000 files; multiple retrieve test; idempotent prepare; deletion on closew errors; eviction before archival; EOS evict command; ObjectStore queue cleanup.
  • test-client-gfal2: archival, retrieval, eviction and deletion of 10.000 files. Using the gfal2 library, core library for FTS, 5.000 files are tested against the XRootD protocol and the other 5.000 against the HTTP protocol. It also checks for activity passing through the gfal2 stack.
  • test-repack: tests of repacking workflows.
  • test-cta-admin: exercises the execution and tests the different cta-amdin commands.
  • test-liquibase-update: tests the upgrade and downgrade of the different schema versions of the Catalogue.
  • test-external-tape-formats: tests the support of tapes configured by other tape software.

  • test-regression-dCache: dCache regression tests.

  • System tests organization and desing constraints

    For system tests we have at our disposal 3 runners, each runner can only run one test at a time. The current run time of the system tests is around 20 minutes using the 3 available runners. Whenever a test is run the virtual environment is created and it is destroyed after the test. The creation of the environment has an overhead of ~2 minutes, the destruction is much faster.

    Ideally, the tests should be logically grouped together, this means that related workflows should be tested in the creation of the same environment, this helps to better understand the source of the failure, nevertheless the logs produced by the tests should be clear enough about what was being tested and why it failed.

    This ideal is not always achivable, as it is of utter importance to find the right balance of number of tests and execution length to minimize execution time. Having a single test containing everything leads to resource under-utilization and longer pipelines, specially when there are not many developers pushing to the repository at the same time; and splitting them too much will create an excessive amount of overhead which leads to wasted time, specially when many pipelines are being executed at the same time.

• Release Changelog:

  • changelog-preview: produces a preview of the changelog based on all the commits between two commits (the latest commit and the latest tag by default)
  • changelog-update: generates a merge request with an update to the CHANGELOG.md and cta.spec.in file.
• Release Internal:
  • release-cta: publishes the RPMs to a CTA internal repo, making them available to be deployed in the stress tests and later stages.
• Release External:
  • release-cta-testing: publishes the RPMs to the public testing repository.
  • release-cta-stable: publishes the RPMs to the public stable repository.
  • release-cta-gitlab-prepare: prepares the relases notes of the current tag from all the commit messages since the previous tag. And uploads them as an artifact.
  • release-cta-gitlab: creates a GitLab release for the current tag.

Pipeline Types

We introduced the logical concept of pipeline type to our CI to address the growing requirements for additional functionality and regression checks. Currently we have the following types of pipelines:

Commit Pipeline

The pipeline that runs when pushing changes to the repository, and can be considered the default pipeline. This type of pipeline is also executed when a new tag is added to the repository.

Commit pipeline's job dependency graph

The pipeline includes the following stages:

1. For the Analysis stage all tests, but the clang-format-apply, are run. It is recommended to apply the CLang format changes after the code review has been done to simplify it.
2. Build stage, both, the SRPM and RPM build jobs are executed.
3. The Build Image stage builds the container image containing the binaries generated in the previous step.
4. Test. unit-test-oracle; unit-test-postgres; integration-test-cta; test-cta-release are run automatically. The test-cta-valgrind job can be executed manually, be aware that it takes over 30 minutes to complete and it is always executed on the nightly pipelines, alongside the other tests.
5. System Tests. Currently the set of tests enabled by default on the commit pipeline are test-client; test-client-gfal2; test-repack; test-cta-admin. The following system tests can be triggered manually test-liquibase-update; test-external-tape-format; test-regression-dCache. All these tests are run for nightly scheduled pipelines.

This pipeline also runs when a tag is pushed to the GitLab repository. It will run all the catalogue-schema-version-check analysis and build stages. On top, it enables the Release Changelog, Release Internal and Release External stage's jobs to be run manually. In this case the dependency graph looks like:

Tag pipeline's job dependency graph

More details on Tagging Releases.

EOS Tag against CTA main

Whenever EOS publishes a new version they will trigger a pipeline on our side to test that the new version does not introduce any braking changes.

EOS Regression with CTA main pipeline's job dependency graph

For this type the stages/jobs are straight forward. Validate the variables, modify the version lock so that the ctageneric image generated contains the new EOS version, build the SRPM/RPM; build the container image and run a specific the test-client, test-client-gfal2, test-cta-repack.

EOS Tag against latest CTA tag

Similar to the previous type but tests with the latest available CTA tagged version, but in this case the container image pulls the CTA RPMs from the public repository instead of building them from source. The reason for this implementation is to facilitate the testing of old CTA tags using the latests version of the CI code.

EOS Regression with CTA tag pipeline's job dependency graph

CTA Generic Image Pipeline

Allows to create a CTA Generic container image containing the CTA, EOS and XRootD versions specified through the CUSTOM_CTA_TAG, CUSTOM_EOS_TAG and CUSTOM_XRD_TAG variables respectively.

CTA Generic Image pipeline's job dependency graph

Pipeline Variables

By default the commit pipelines (and any pipeline), runs with a set of preset variables, the defaults are defined in .gitlab-ci.yml. These can be modified either when doing a push to the repository by means of git push options, i.e., git push -o ci.variable="ORACLE_SUPPORT=OFF" or by triggering a manual pipeline from the GitLab Pipelines web UI and specifying the desired flags. Next you can find a list with the most relevant variables from a development point of view:

Category	Variable	Options / Format	Default	Description
CTA Configuration	SCHED_TYPE	objectstore | pgsched	objectstore	The scheduler backend
CTA Configuration	ORACLE_SUPPORT	ON | OFF	ON	Catalogue backend; ON: uses an oracle database; OFF: uses a postgresql database
System Tests	SYSTEMTESTS_ONLY	TRUE | FALSE	FALSE	Run only the system tests
System Tests	SYSTEMTESTS_IMAGETAG	{PIPELINE_ID}git{COMMIT_ID}	null	CTA Generic Image tag, used in combination with SYSTEMTESTS_ONLY
Pipeline Type	PIPELINE_TYPE	COMMIT | EOS_REGR_AGAINST_CTA_MAIN | EOS_REGR_AGAINST_CTA_TAG | CTAGENERIC_IMAGE	COMMIT	A
Pipeline Type Custom Var	CUSTOM_XRD_TAG	^\d+:\d+\.\d+\.\d+(-\d+)*$	null	XRootD version available in the projects stable EL9 repo. Example: 1:5.7.2-1
Pipeline Type Custom Var	CUSTOM_EOS_TAG	^\d+\.\d+\.\d+$	null	EOS version available in the project's CI testing EL9 repo. Example: 5.3.1
Pipeline Type Custom Var	CUSTOM_CTA_TAG	^v\d+\.\d+\.\d+\.\d+-\d+$	null	CTA version available in the project's public testing EL9 repo. Example: v5.12.1.1-1

The full set of available variables can be found in https://gitlab.cern.ch/cta/CTA/-/blob/main/.gitlab-ci.yml

Additionally, when pushing, it is possible to entirely skip the execution of the CI. This is useful when a developer wants to synchronize their local changes, but does not want to trigger the pipeline because they are not ready for testing yet. To do so, you can run: git push -o ci.skip. Merge requests which last commit skipped the pipeline cannot be merged into the main branch.

The SYSTEMTESTS_ONLY flag can be useful when working directly on the system tests themselves as it avoids running the entire pipeline, when this flag is set, the latest generic image generated from main will be used to create the virtual environment for the tests. It can also be helpful to debug some situations like race conditions that are hard to reproduce manually and require several executions, see https://gitlab.cern.ch/cta/CTA/-/issues/662.

Local testing and modifications

Waiting for the GitLab pipeline to reach the systemtests in an iterative way is too time consuming, specially when modifying the system tests themselves. For this use case, or when there is heavy resource contention in the pipelines, developers can use the development setup to speedup the process by running the CI tests locally.

To modify the tests, and make the modifications effective, one must take into account the following considerations:

• Modifying the container configuration: all the container configuration files are injected during the image build process, so, the image must be rebuilt for changes done to the initialization scripts or anything else that goes into the container.There is no need to rebuild the CTA binaries, only the image.
• Modifying the test behaviour: this is more trivial to do as they are run from the VM and not integrated in the container image. For this, you just need to modify the desired scripts. Sometimes it might be even useful to trim down the test set and only run the parts of the test that are of interest. Or even move into the container and run the specific commands used by the test.

While testing this way one must check if the cluster is in a clean state after a test failure or even after a clean execution, inconsistent states can misled you when debugging problems. Currently the tests are not designed to leave cluster in the same state as it was before being launched, this is why in CI we reinitialize the cluster for every set of tests.

Nightly Scheduled Commit Pipelines

The main developer workflow is based on pushing and merging into main. This workflow only triggers the default configuration of the pipeline. This default configuration matches the production state, or, at most, the version upgrades for the upcoming release. At night, pipelines for all the possible supported configurations are executed, i.e. different scheduler backends, compilation and testing for different OS versions when OS migrations are ongoing, etc.

These scheduled pipelines help keep the developer workflow as streamlined as possible while checking new changes don't break compatibility with different configurations. There is drawback for this approach, which is that the developer must check if its changes caused some of the nightly pipelines to fail.

Scheduled pipelines can be checked at: https://gitlab.cern.ch/cta/CTA/-/pipeline_schedules