Skip to content

CI Guidelines

To ensure that the continuous integration setup is maintainable and flexible, there are a number of guidelines to follow. These are outlined below. When working on anything related to the CI, be sure to thoroughly read through this first.

Scripting

  • Every script must have a copyright statement on top
  • Scripts should take their input from flags and not from environment variables
    • This ensures it is clear for any user of the script what is expected from them
    • If ENV variables are expected, their presence should be verified
  • For each input flag, a script must provide a long-form version. E.g. --image-tag
    • If applicable, an additional short-form option can be provided. E.g. -t
  • Each script that takes any sort of input must provide a usage() function
  • Keep scripts short and task-specific
    • Scripts should follow the single-responsibility principle.
    • If a script is doing too many things, split it into smaller scripts and call these scripts from a wrapper/parent script
  • Scripts must not make any assumption on what the calling script is to ensure reusability
    • This includes assumptions on pwd when the script starts
  • Always validate (the existence of) input arguments to ensure proper behaviour
  • Make sure scripts clean up after themselves. If temporary files are created that are no longer necessary after the script finishes, then using trap to remove these on script exit
  • Scripts should be able to handle errors gracefully
  • Use tools like ShellCheck to catch potential bugs and improve script quality.

Container Images

  • Be extremely careful never to install Oracle RPMs in any image that might ever be publicly available!!
    • We cannot distribute Oracle RPMs in any way, shape or form
  • Never include any secrets in a Dockerfile
  • Do not hardcode configuration values in a Dockerfile
  • Do not do at runtime what can be done at buildtime
  • No if-else branching in a Dockerfile. If this is needed, you probably need a separate Dockerfile instead
  • Naming for a Dockerfile should be [<purpose>.]Dockerfile
    • If there is only one Docker image, then naming this file Dockerfile is fine
    • Otherwise, prefix the purpose/name accordingly
  • If there are files only needed during the building of the image that are not needed at runtime, look into multi-stage builds to prevent these files from ending up in the final image
  • Group similar commands into a single layer, but prevent overly complex RUN commands
  • Put frequently running/changing instructions towards the end of the Dockerfile to ensure earlier layers can still be cached effectively
  • Remove temporary files, caches, and other unnecessary items during the build process to keep images lightweight
  • Add comments to explain any non-trivial steps or design decisions in the Dockerfile

GitLab Pipeline

  • Any job definitions go into .gitlab/ci/*.gitlab-ci.yaml
  • Similar jobs should be grouped into the same file.
  • A single .gitlab/ci/*.gitlab-ci.yaml should not contain jobs belonging to multiple stages
    • The exception being different stages with the same prefix (e.g. test: or release:)
  • Use the following order for defining the job properties:
    • allow_failure
    • retry
    • stage
    • extends
    • tags
    • needs
    • dependencies
    • rules
    • image
    • variables
    • before_script
    • script
    • after_script
    • artifacts
    • release
  • Use needs to specify which jobs a specific job is relying on. If there are none, specify an empty list.
    • This ensures that the job does not needlessly download all artifacts of jobs that came before
    • It also ensures its correct position in the DAG
  • Job naming should be all lowercase and contain only alphanumeric characters. Words can be split using hyphens: example-job
    • It is allowed to use a slash (/), colon (:) or space in the name if it is required to group jobs together
  • In any script part, each line should be a separate entry in yaml. I.e. do not do - | and dump the entire script
    • This ensures the steps remain readable when the pipeline is executed
    • For if-statement, using - | is fine to ensure readability. Try to keep any if-statements like this as short as possible
  • Do not put secrets in any job definitions. Secrets should be defined as GitLab CI/CD variables and referenced as such
  • Global pipeline variables are defined in .gitlab-ci.yaml
  • Jobs should refer to the image through a global variable to ensure that the images can be easily updated in a centralized placed
  • A job should only run on_success (see the rules section) unless there is a very good reason not to
  • Try to prevent too much scripting logic ending up in a job definition. If applicable, create a script out of it
  • Any calls to a script should use the long flag names (so --namespace instead of -n) to improve readability
  • Jobs in a file should be grouped logically together. In general: default or general jobs go towards the top of the file, whereas edge-case/specific jobs go towards the bottom
  • Don't do installs in jobs if this can be prevented. Ensure these dependencies are in the image used by the job instead (see the ci/cta-ci-images repo)

Orchestration

  • Don't do at runtime what can be done at build time.
  • Least priviledge: don't give pods more priviledges than they need.
  • For defining different configuration options, create an alternative values.yaml file, not a variation of an existing configmap (unless absolutely necessary).
  • Always check if things can be done natively in Helm/kubernetes before adding any scripting logic.
  • Don't use plain pod definitions, use deployments/statefulsets/replicasets instead. This ensures pods can be easily redeployed.
  • Restrict communication between pods with network policies to control ingress and egress when applicable.
  • Store sensitive data in Kubernetes Secrets and reference them securely in pods.
  • Ensure probes are correctly set up to monitor pod health and availability.
  • Redirect application logs to stdout and stderr so that logs can be easily collected by a logging agent.
  • Add meaningful labels and annotations for organization, tracking, and automation (see the common chart).
  • Upgrade the chart version when changes are passed through.
  • Keep the main container startup script as simple as possible. In ideal world, the only thing this script is doing is starting the relevant service.
    • When possible, use initContainers to do any initialisation
  • Do not hardcode in the manifest files what someone might reasonably want to change. Use templating and the values.yaml for this instead.

Helm Structure

Adhere to the following directory structure per chart:

my-chart/
├── charts/                # Directory for dependent subcharts
├── templates/             # YAML templates for Kubernetes resources
│   ├── _helpers.tpl       # Helper templates (e.g., for names, labels)
│   ├── deployment.yaml    # Deployment manifest (optional)
│   ├── service.yaml       # Service manifest (optional)
│   └── ingress.yaml       # Ingress manifest (optional)
├── Chart.yaml             # Chart metadata (name, version, etc.)
├── values.yaml            # Default configuration values
├── values.schema.json     # JSON schema for validating values (optional)
├── README.md              # Documentation

Naming Guidelines

  • Use lowercase letters
  • Use hyphens (-) for separators
  • Keep names descriptive but concise: Names should indicate the purpose of the resource without being excessively long.
  • Avoid special characters: Only use alphanumeric characters and hyphens.

Naming patterns:

  • Chart Directory: <name>/
  • Example: my-app/, cta
  • Values File: values.yaml (default values file)
  • Custom Values Files: <env>[-<resource>]-values.yaml
  • Example: dev-values.yaml, prod-values.yaml, ci-catalogue-oracle-values.yaml
  • Templates Directory: templates/

Resource templates

Each resource should reside in its own separate file. Do not define multiple resources in a single file. If the resources are really tightly coupled, create a directory instead.

Stick to simple names for the main different resource types:

  • Pod: pod.yaml
  • Deployment: deployment.yaml
  • Service: service.yaml
  • Ingress: ingress.yaml

If required, prefix the above with a descriptive name.

For the following resource types, use a descriptive name, followed by the shortname of the resource type:

  • ConfigMap: <name>-configmap.yaml
  • Secret: <name>-secret.yaml
  • Job: <name>-job.yaml
  • CronJob: <name>-cronjob.yaml

If there already is a directory for said type (e.g. configmaps/, secrets/), then the suffix is no longer necessary and <name>.yaml is sufficient.