Upgrading the EOS version used by the CI¶
Introduction¶
This document presents how to upgrade the EOS version used by the CI, used for automatic testing of CTA changes and for the packaging of CTA rpms.
The following steps must be performed: - Create a CTA dev ticket for the EOS version upgraded - Update the yum repository folder with the new EOS RPMs - Update the yum repository folder with the compatible EOS xrootd RPMs - Update the yum repository - Update the CTA CI versionlock - Merge the changes
Create a CTA dev ticket for the EOS version upgraded¶
Create a ticket in the CTA development repository. Add the appropriate EOS label to the ticket. The upgrade should be done in a separate feature branch, created from the gitlab issue page.
Update the yum repo folder with the new EOS RPMs¶
The yum repository used by the CTA CI can be accessed at /eos/user/c/ctareg/www/cta-ci-repo from AIADM. The EOS RPMs are naturally at /eos/user/c/ctareg/www/cta-ci-repo/eos/x86_64. If the version you desire to upgrade to isn't here you must update the repository with the new RPMS. There is an auxiliar script to help you in this task at /eos/user/c/ctareg/www/cta-ci-repo/eos/x86_64/tmp/get_eos.sh. It takes as an argument the EOS version you desire to download and outputs the necessary commands to download the respective RPMs. For example, for EOS version 4.8.67 the result is the following:
[mvelosob@c2adm01 tmp]$ ./get_eos.sh 4.8.67
--2021-11-18 13:37:06-- https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64/
Resolving storage-ci.web.cern.ch (storage-ci.web.cern.ch)... 188.185.90.87, 188.185.87.216, 137.138.31.235
Connecting to storage-ci.web.cern.ch (storage-ci.web.cern.ch)|188.185.90.87|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
[ <=> ] 536,348 --.-K/s in 0.03s
2021-11-18 13:37:07 (19.3 MB/s) - ‘index.html’ saved [536348]
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-archive-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-cleanup-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-client-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-fuse-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-fuse-core-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-fuse-sysv-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-fusex-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-fusex-core-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-fusex-selinux-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-ns-inspect-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-server-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-srm-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-test-4.8.67-1.el7.cern.x86_64.rpm
wget https://storage-ci.web.cern.ch/storage-ci/eos/citrine/tag/testing/el-7/x86_64//eos-testkeytab-4.8.67-1.el7.cern.x86_64.rpm
Validate the commands outputed are what you desire, and no extra EOS versions will be downloaded (i.e. if you run get-eos.sh 4 the command will list all EOS RPMs with 4 in the name).
After this, run the wget commands in the /eos/user/c/ctareg/www/cta-ci-repo/eos/x86_64 directory.
Note: Always check first if the EOS version is already in the repository folder. In that case no update is needed
Update the yum repo folder with the new EOS Xrootd RPMs¶
The EOS xrootd version has it's own RPM, allowing EOS and CTA to use different Xrootd versions. When upgrading EOS, it may also be necessary to update the EOS xrootd RPMs to the compatible version. To know the compatible version, go to the EOS git repository and look for the tag of the desired EOS release. Click on the build pipeline of the respective tag and look at the raw output of the build_cc7 stage. Grep for eos-xrootd and you should get the EOS Xrootd version used.
The update method is then the same, but using the /eos/user/c/ctareg/www/cta-ci-repo/eos/x86_64/tmp/get_eos-xrootd.sh script. For example, for EOS version 4.8-67, we get eos-xrootd-4.12.8-1.el7.cern.x86_64.
Running the command gives the following ouput:
mvelosob@c2adm01 tmp]$ ./get_eos-xrootd.sh 4.12.8-1
--2021-11-18 13:48:07-- http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64/
Resolving storage-ci.web.cern.ch (storage-ci.web.cern.ch)... 188.185.90.87, 188.185.87.216, 137.138.31.235
Connecting to storage-ci.web.cern.ch (storage-ci.web.cern.ch)|188.185.90.87|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64/ [following]
--2021-11-18 13:48:07-- https://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64/
Connecting to storage-ci.web.cern.ch (storage-ci.web.cern.ch)|188.185.90.87|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’
[ <=> ] 271,676 --.-K/s in 0.02s
2021-11-18 13:48:07 (14.0 MB/s) - ‘index.html’ saved [271676]
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//eos-xrootd-4.12.8-1.el7.cern.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//eos-xrootd-debuginfo-4.12.8-1.el7.cern.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//python2-xrootd-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//python3-xrootd-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrdcl-http-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-client-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-client-devel-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-client-libs-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-debuginfo-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-devel-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-doc-4.12.8-1.el7.noarch.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-fuse-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-libs-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-private-devel-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-selinux-4.12.8-1.el7.noarch.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-server-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-server-devel-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-server-libs-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-tests-4.12.8-1.el7.x86_64.rpm
wget http://storage-ci.web.cern.ch/storage-ci/eos/citrine-depend/el-7/x86_64//xrootd-voms-4.12.8-1.el7.x86_64.rpm
Note: Always check first if the Xrootd version is already in the repository folder. In that case no update is needed
Update the yum repository¶
You must run a script to update the yum repository metadata:
cd /eos/user/c/ctareg/www/cta-ci-repo
./update_repos.sh
Update the CTA CI versionlock¶
The versionlock file used by the CI is in the continuousintegration/docker/ctafrontend/cc7/etc/yum/pluginconf.d directory of the CTA development repository. Update the respective RPM versions. Note that eos-xrootd is the xrootd version used by EOS and xrootdis the EOS version used by CTA, which you probably do not want to upgrade.
In the case of upgrading to EOS version 4.8-67, before we have the following EOS versionlocks:
0:eos-archive-4.8.45-1.el7.cern.x86_64
0:eos-cleanup-4.8.45-1.el7.cern.x86_64
0:eos-client-4.8.45-1.el7.cern.x86_64
0:eos-debuginfo-4.8.45-1.el7.cern.x86_64
0:eos-fuse-4.8.45-1.el7.cern.x86_64
0:eos-fuse-core-4.8.45-1.el7.cern.x86_64
0:eos-fuse-sysv-4.8.45-1.el7.cern.x86_64
0:eos-fusex-4.8.45-1.el7.cern.x86_64
0:eos-fusex-core-4.8.45-1.el7.cern.x86_64
0:eos-fusex-selinux-4.8.45-1.el7.cern.x86_64
0:eos-ns-inspect-4.8.45-1.el7.cern.x86_64
0:eos-server-4.8.45-1.el7.cern.x86_64
0:eos-srm-4.8.45-1.el7.cern.x86_64
0:eos-test-4.8.45-1.el7.cern.x86_64
0:eos-testkeytab-4.8.45-1.el7.cern.x86_64
0:eos-xrootd-4.12.8-1.el7.cern.*
0:eos-xrootd-debuginfo-4.12.8-1.el7.cern.*
0:eos-folly-2019.11.11.00-1.el7.cern.*
0:eos-folly-deps-2019.11.11.00-1.el7.cern.*
1:python2-xrootd-4.12.6-1.el7.*
1:python3-xrootd-4.12.6-1.el7.*
1:xrootd-4.12.6-1.el7.*
1:xrootd-client-4.12.6-1.el7.*
1:xrootd-client-devel-4.12.6-1.el7.*
1:xrootd-client-libs-4.12.6-1.el7.*
1:xrootd-debuginfo-4.12.6-1.el7.*
1:xrootd-devel-4.12.6-1.el7.*
1:xrootd-doc-4.12.6-1.el7.*
1:xrootd-fuse-4.12.6-1.el7.*
1:xrootd-libs-4.12.6-1.el7.*
1:xrootd-private-devel-4.12.6-1.el7.*
1:xrootd-selinux-4.12.6-1.el7.*
1:xrootd-server-4.12.6-1.el7.*
1:xrootd-server-devel-4.12.6-1.el7.*
1:xrootd-server-libs-4.12.6-1.el7.*
1:xrootd-tests-4.12.6-1.el7.*
2:ceph-14.2.20-0.el7.x86_64
2:ceph-base-14.2.20-0.el7.x86_64
2:ceph-common-14.2.20-0.el7.x86_64
2:ceph-debuginfo-14.2.20-0.el7.x86_64
2:cephfs-java-14.2.20-0.el7.x86_64
2:ceph-fuse-14.2.20-0.el7.x86_64
2:ceph-mds-14.2.20-0.el7.x86_64
2:ceph-mgr-14.2.20-0.el7.x86_64
2:ceph-mon-14.2.20-0.el7.x86_64
2:ceph-osd-14.2.20-0.el7.x86_64
2:ceph-radosgw-14.2.20-0.el7.x86_64
2:ceph-resource-agents-14.2.20-0.el7.x86_64
2:ceph-selinux-14.2.20-0.el7.x86_64
2:ceph-test-14.2.20-0.el7.x86_64
2:libcephfs2-14.2.20-0.el7.x86_64
2:libcephfs-devel-14.2.20-0.el7.x86_64
2:libcephfs_jni1-14.2.20-0.el7.x86_64
2:libcephfs_jni-devel-14.2.20-0.el7.x86_64
2:librados2-14.2.20-0.el7.x86_64
2:librados-devel-14.2.20-0.el7.x86_64
2:libradospp-devel-14.2.20-0.el7.x86_64
2:libradosstriper1-14.2.20-0.el7.x86_64
2:libradosstriper-devel-14.2.20-0.el7.x86_64
2:librbd1-14.2.20-0.el7.x86_64
2:librbd-devel-14.2.20-0.el7.x86_64
2:librgw2-14.2.20-0.el7.x86_64
2:librgw-devel-14.2.20-0.el7.x86_64
2:python3-ceph-argparse-14.2.20-0.el7.x86_64
2:python3-cephfs-14.2.20-0.el7.x86_64
2:python3-rados-14.2.20-0.el7.x86_64
2:python3-rbd-14.2.20-0.el7.x86_64
2:python3-rgw-14.2.20-0.el7.x86_64
2:python-ceph-argparse-14.2.20-0.el7.x86_64
2:python-ceph-compat-14.2.20-0.el7.x86_64
2:python-cephfs-14.2.20-0.el7.x86_64
2:python-rados-14.2.20-0.el7.x86_64
2:python-rbd-14.2.20-0.el7.x86_64
2:python-rgw-14.2.20-0.el7.x86_64
2:rados-objclass-devel-14.2.20-0.el7.x86_64
2:rbd-fuse-14.2.20-0.el7.x86_64
2:rbd-mirror-14.2.20-0.el7.x86_64
2:rbd-nbd-14.2.20-0.el7.x86_64
And after the upgrade we have:
0:eos-archive-4.8.67-1.el7.cern.x86_64.rpm
0:eos-cleanup-4.8.67-1.el7.cern.x86_64.rpm
0:eos-client-4.8.67-1.el7.cern.x86_64.rpm
0:eos-fuse-4.8.67-1.el7.cern.x86_64.rpm
0:eos-fuse-core-4.8.67-1.el7.cern.x86_64.rpm
0:eos-fuse-sysv-4.8.67-1.el7.cern.x86_64.rpm
0:eos-fusex-4.8.67-1.el7.cern.x86_64.rpm
0:eos-fusex-core-4.8.67-1.el7.cern.x86_64.rpm
0:eos-fusex-selinux-4.8.67-1.el7.cern.x86_64.rpm
0:eos-ns-inspect-4.8.67-1.el7.cern.x86_64.rpm
0:eos-server-4.8.67-1.el7.cern.x86_64.rpm
0:eos-srm-4.8.67-1.el7.cern.x86_64.rpm
0:eos-test-4.8.67-1.el7.cern.x86_64.rpm
0:eos-testkeytab-4.8.67-1.el7.cern.x86_64.rpm
0:eos-xrootd-4.12.8-1.el7.cern.*
0:eos-xrootd-debuginfo-4.12.8-1.el7.cern.*
0:eos-folly-2019.11.11.00-1.el7.cern.*
0:eos-folly-deps-2019.11.11.00-1.el7.cern.*
1:python2-xrootd-4.12.6-1.el7.*
1:python3-xrootd-4.12.6-1.el7.*
1:xrootd-4.12.6-1.el7.*
1:xrootd-client-4.12.6-1.el7.*
1:xrootd-client-devel-4.12.6-1.el7.*
1:xrootd-client-libs-4.12.6-1.el7.*
1:xrootd-debuginfo-4.12.6-1.el7.*
1:xrootd-devel-4.12.6-1.el7.*
1:xrootd-doc-4.12.6-1.el7.*
1:xrootd-fuse-4.12.6-1.el7.*
1:xrootd-libs-4.12.6-1.el7.*
1:xrootd-private-devel-4.12.6-1.el7.*
1:xrootd-selinux-4.12.6-1.el7.*
1:xrootd-server-4.12.6-1.el7.*
1:xrootd-server-devel-4.12.6-1.el7.*
1:xrootd-server-libs-4.12.6-1.el7.*
1:xrootd-tests-4.12.6-1.el7.*
2:ceph-14.2.20-0.el7.x86_64
2:ceph-base-14.2.20-0.el7.x86_64
2:ceph-common-14.2.20-0.el7.x86_64
2:ceph-debuginfo-14.2.20-0.el7.x86_64
2:cephfs-java-14.2.20-0.el7.x86_64
2:ceph-fuse-14.2.20-0.el7.x86_64
2:ceph-mds-14.2.20-0.el7.x86_64
2:ceph-mgr-14.2.20-0.el7.x86_64
2:ceph-mon-14.2.20-0.el7.x86_64
2:ceph-osd-14.2.20-0.el7.x86_64
2:ceph-radosgw-14.2.20-0.el7.x86_64
2:ceph-resource-agents-14.2.20-0.el7.x86_64
2:ceph-selinux-14.2.20-0.el7.x86_64
2:ceph-test-14.2.20-0.el7.x86_64
2:libcephfs2-14.2.20-0.el7.x86_64
2:libcephfs-devel-14.2.20-0.el7.x86_64
2:libcephfs_jni1-14.2.20-0.el7.x86_64
2:libcephfs_jni-devel-14.2.20-0.el7.x86_64
2:librados2-14.2.20-0.el7.x86_64
2:librados-devel-14.2.20-0.el7.x86_64
2:libradospp-devel-14.2.20-0.el7.x86_64
2:libradosstriper1-14.2.20-0.el7.x86_64
2:libradosstriper-devel-14.2.20-0.el7.x86_64
2:librbd1-14.2.20-0.el7.x86_64
2:librbd-devel-14.2.20-0.el7.x86_64
2:librgw2-14.2.20-0.el7.x86_64
2:librgw-devel-14.2.20-0.el7.x86_64
2:python3-ceph-argparse-14.2.20-0.el7.x86_64
2:python3-cephfs-14.2.20-0.el7.x86_64
2:python3-rados-14.2.20-0.el7.x86_64
2:python3-rbd-14.2.20-0.el7.x86_64
2:python3-rgw-14.2.20-0.el7.x86_64
2:python-ceph-argparse-14.2.20-0.el7.x86_64
2:python-ceph-compat-14.2.20-0.el7.x86_64
2:python-cephfs-14.2.20-0.el7.x86_64
2:python-rados-14.2.20-0.el7.x86_64
2:python-rbd-14.2.20-0.el7.x86_64
2:python-rgw-14.2.20-0.el7.x86_64
2:rados-objclass-devel-14.2.20-0.el7.x86_64
2:rbd-fuse-14.2.20-0.el7.x86_64
2:rbd-mirror-14.2.20-0.el7.x86_64
2:rbd-nbd-14.2.20-0.el7.x86_64
Merge the changes¶
Push the changes to the feature branch. If the CI passes and everything seems to work well, merge the changes to master.