Create a native EOSCTA development environment¶
Introduction¶
This document describes how to set up a complete EOSCTA development environment from scratch. This is a complex procedure, but it is a useful exercise for new CTA developers, to gain an overview of all the EOSCTA components and how they interact.
The procedure is based on the native server installation and also by looking at how the CI works. If you encounter any errors or omissions, please update the documentation accordingly.
The following steps will be performed:
- Compile and install EOS from source
- Compile and install CTA from source
- Install and configure mhVTL (Virtual Tape Library), which simulates the tape hardware (library, drives and tapes).
- Wire the EOS instance to the CTA software
- Archive and Retrieve some files to check everything works
Compile and install EOS from source¶
Follow these instructions.
You should be able to have a running EOS instance after this tutorial.
Note: You may get a protobuf error during cmake generation, similar to this:
-- Found protoc: /bin/protoc3
-- Found Protobuf3: /opt/eos/include/protobuf3
CMake Warning at /usr/share/cmake3/Modules/FindProtobuf.cmake:500 (message):
Protobuf compiler version 3.3.1 doesn't match library version 3.5.1
Call Stack (most recent call first):
cmake/FindProtobuf3.cmake:64 (find_package)
cmake/EosFindLibs.cmake:71 (find_package)
CMakeLists.txt:95 (include)
If this is the case, you need to pass cmake the correct protobuf path. Add the following flag to the cmake command:
-DPROTOBUF_ROOT=/opt/eos
Compile and install CTA from source¶
Follow the instructions here: Building CTA but instead of doing the last part (Install), just type:
make install
Make sure that you're using the OS native compiler and not one from a software collection devtoolset defined during the EOS build.
In order to check the installation, you should be able to run
cta-unitTests
[==========] Running 885 tests from 76 test cases.
[----------] Global test environment set-up.
[----------] 13 tests from RAOTest
[ RUN ] RAOTest.InterpolationFilePositionEstimatorWrap0Test
[ OK ] RAOTest.InterpolationFilePositionEstimatorWrap0Test (1 ms)
[ RUN ] RAOTest.InterpolationFilePositionEstimatorWrap1Test
[ OK ] RAOTest.InterpolationFilePositionEstimatorWrap1Test (0 ms)
[ RUN ] RAOTest.InterpolationFilePositionEstimatorThrowsExceptionIfBlockIdGreaterThanLastEOWP
[ OK ] RAOTest.InterpolationFilePositionEstimatorThrowsExceptionIfBlockIdGreaterThanLastEOWP (52 ms)
[ RUN ] RAOTest.RAOHelpersImproveLastEOWPIfPossible
[ OK ] RAOTest.RAOHelpersImproveLastEOWPIfPossible (0 ms)
[ RUN ] RAOTest.RAOHelpersDetermineBand
[ OK ] RAOTest.RAOHelpersDetermineBand (0 ms)
[ RUN ] RAOTest.RAOHelpersDetermineLandingZone
[ OK ] RAOTest.RAOHelpersDetermineLandingZone (0 ms)
[ RUN ] RAOTest.RAOHelpersDoesWrapChange
[ OK ] RAOTest.RAOHelpersDoesWrapChange (0 ms)
[ RUN ] RAOTest.RAOHelpersDoesBandChange
[...]
Install and configure mhVTL¶
Follow the instructions here: Install the Virtual Tape Library (mhVTL).
Configure CTA¶
Configure the CEPH objectstore¶
Follow the instructions here: Configuration for a production system
Configure the CTA Catalogue¶
Follow the instructions here: Configuration the Catalogue
Configure the CTA Frontend¶
Create the Kerberos keytab file used by the CTA Frontend to authenticate clients¶
First, create the CTA user (the frontend and the tapeserver will be launched as if they were the CTA user)
useradd cta -g tape
If you do not have any krb5.keytab file in the /etc/ directory, the following instructions have to be followed in order to create it:
cd /etc/cta
cern-get-keytab --service cta --force
cp krb5.keytab cta-frontend.krb5.keytab
chown cta:tape cta-frontend.krb5.keytab
You can check the content of this file by using the following command:
echo -e "read_kt cta-frontend.krb5.keytab\nlist\nquit" | ktutil
Create SSS keys so that the Frontend can talk to the local EOS instance¶
The EOS instance name will be used as the user name in the sss keys that will be created. As you previously installed EOS, you should be able to see the EOS instance name that you have setup by typing:
grep EOS_INSTANCE_NAME /etc/sysconfig/eos_env
EOS_INSTANCE_NAME=eosdev
Create the sss key that will be used by the CTA Frontend to allow the communication between it and EOS:
$ cd /etc/cta
#k = keyname, u=eosdev(EOS_INSTANCE_NAME), g=cta(group)
$ xrdsssadmin -k cta_eosdev -u eosdev -g cta add ctafrontend_server_sss.keytab
xrdsssadmin: Keyfile 'ctafrontend_server_sss.keytab' does not exist. Create it? (y | n): y
xrdsssadmin: 1 key out of 1 kept (0 expired).
# As the frontend will be launched under the cta user, change the owner of the sss keytab file
$ chown cta ctafrontend_server_sss.keytab
$ chmod 600 ctafrontend_server_sss.keytab
Your sss keytab file should look like this:
cat ctafrontend_server_sss.keytab
0 u:eosdev g:cta n:cta_eosdev N:XXXXX c:XXXXX e:0 f:0 k:XXXXX
Create the cta-frontend-xrootd.conf configuration file¶
Create the file /etc/cta/cta-frontend-xrootd.conf and put the following information in it:
#
# CTA options
#
# CTA ObjectStore options
cta.objectstore.backendpath rados://OBJECTSTORE_ID@OBJECTSTORE_POOL:OBJECTSTORE_NAMESPACE
# CTA Scheduler DB options
cta.schedulerdb.numberofthreads 500
# CTA Catalogue options
cta.catalogue.numberofconnections 10
# Maximum file size (in GB) that the CTA Frontend will accept for archiving
cta.archivefile.max_size_gb 1000
# CTA Frontend log URL
cta.log.url file:/var/log/cta/cta-frontend.log
# CTA Logger log level
# Valid log levels are EMERG, ALERT, CRIT, ERR, WARNING, NOTICE (==USERERR), INFO, DEBUG
# cta.log.level DEBUG
# CTA XRootD SSI Protobuf log level
# cta.log.ssi debug protobuf
cta.log.ssi warning
# CTA Repack buffer URL
# cta.repack.repack_buffer_url root://ctaeos//eos/ctaeos/repack
# Keytab containing gRPC endpoints and tokens for each disk instance
#cta.ns.config /etc/cta/eos.grpc.keytab
#
# XRootD/SSI options
#
# Enable SSI debug logging
#ssi.trace debug
# Load the CTA SSI plugin
xrootd.fslib -2 libXrdSsi.so
# Load the SSI module
ssi.svclib libXrdSsiCta.so
# Use the security module
xrootd.seclib libXrdSec.so
# Security protocol specification
#
# Note: the XRootD process needs to be able to read the keytab file
sec.protocol krb5 /etc/cta/cta-frontend.krb5.keytab host/<your_dev_machine>.cern.ch@CERN.CH
sec.protocol sss -s /etc/ctafrontend_server_sss.keytab
# Only Kerberos 5 and sss are allowed
sec.protbind * only sss krb5
# sec.protbind * only sss
# Turn off asynchronous I/O
xrootd.async off
# Use a port other than 1094, already used by EOS xroot server
xrd.port 10955
xrd.sched avlt 512 mint 768 maxt 4096 idle 300
# Export the SSI resource
all.export /ctafrontend nolock r/w
Replace the cta.objectstore.backendpath and the sec.protocol krb5 values with the ones you have.
Also, create log file where the frontend will log what it is doing:
touch /var/log/cta/cta-frontend-xrootd.log
chmod a+w /var/log/cta/cta-frontend-xrootd.log
chown cta /var/log/cta
Configure the cta-admin command¶
Edit the file located in /etc/cta/cta-cli.conf:
# The CTA frontend address in the form <FQDN>:<TCPPort>
cta.endpoint <your_dev_machine>.cern.ch:10955
Be careful to have the same FQDN:TCPPort as the ones you configured above in the /etc/cta/cta-frontend-xrootd.conf file.
Add an admin user to CTA to allow you to execute cta-admin commands¶
A command-line tool has been developed to allow this. Just run it by giving your user name with the -u parameter. These instructions assume kerberos authentication here, so use your kerberos username.
cta-catalogue-admin-user-create /etc/cta/cta-catalogue.conf -u your_username -m "Administrative User"
Configure the cta-frontend systemd service¶
Create the file /etc/systemd/system/cta-frontend.service and put the following content:
[Unit]
Description=CTA frontend plugin for the XRoot deamon
After=network-online.target
[Service]
ExecStart=/usr/bin/xrootd -l /var/log/cta-frontend-xrootd.log -c /etc/cta/cta-frontend-xrootd.conf -k fifo -n cta
User=cta
Type=simple
Restart=always
RestartSec=10
StartLimitInterval=0
StartLimitBurst=0
OOMScoreAdjust=-1000
LimitCORE=infinity
LimitNOFILE=65536
[Install]
WantedBy=default.target
Boot the cta-frontend and test it¶
In order to start the cta-frontend, run
systemctl daemon-reload
systemctl enable cta-frontend
systemctl restart cta-frontend
Check that the cta-frontend runs by typing:
systemctl status cta-frontend
● cta-frontend.service - CTA frontend plugin for the XRoot deamon
Loaded: loaded (/etc/systemd/system/cta-frontend.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2021-03-30 18:13:43 CEST; 1min 39s ago
Main PID: 7367 (xrootd)
Tasks: 801
Memory: 45.6M
CGroup: /system.slice/cta-frontend.service
└─7367 /usr/bin/xrootd -l /var/log/cta-frontend-xrootd.log -c /etc/cta/cta-frontend-xrootd.conf -k fifo -n cta
Mar 30 18:13:43 eos-ccaffy-dev01.cern.ch systemd[1]: Started CTA frontend plugin for the XRoot deamon.
You should now be able to run the command cta-admin version.
cta-admin version
CTA Admin Client xrd-ssi-protobuf CTA Frontend Server xrd-ssi-protobuf Catalogue schema DB connection string
0-1 v0.0 0-1 v0.0 4.0 oracle:cta_dev04/******@castorint
If this is not the case, make sure you have kinit. Also, you can check the logs located in /var/log/cta-frontend-xrootd.log and in /etc/cta/cta-frontend-xrootd.conf to spot any eventual problem.
Configure the tapeserver (cta-taped daemon)¶
Configure cta-rmcd¶
Edit the /etc/sysconfig/cta-rmcd file:
# Arguments for the rmcd server daemon
# -smc_ldr is the picker device as defined in /dev.
# -f keep process in the foreground, do not fork.
#CTA_RMCD_OPTIONS="smc_ldr"
DAEMON_COREFILE_LIMIT=unlimited
RUN_RMCD=yes
CTA_RMCD_OPTIONS=/dev/smc
Edit the /etc/systemd/system/cta-rmcd.service file:
[Unit]
Description=CERN Tape Archive (CTA) rmcd daemon
After=syslog.target network-online.target
[Service]
User=cta
EnvironmentFile=-/etc/sysconfig/cta-rmcd
ExecStart=/usr/bin/cta-rmcd ${CTA_RMCD_OPTIONS}
LimitCORE=infinity
Type=forking
Restart=no
[Install]
WantedBy=default.target
Start the cta-rmcd daemon:
systemctl daemon-reload
systemctl enable cta-rmcd
systemctl restart cta-rmcd
Create the /etc/cta/TPCONFIG file¶
Define the tape drive(s) that will be used by the cta-taped daemon in the /etc/cta/TPCONFIG file:
vi /etc/cta/TPCONFIG
VDSTK1 VLSTK /dev/nst0 smc0
Here we configure only one drive.
chmod a+r /etc/cta/TPCONFIG
Create the cta-taped configuration file¶
vi /etc/cta/cta-taped.conf
taped BufferSizeBytes 262144
taped BufferCount 200
taped MountCriteria 2000000, 100
taped UseEncryption no
ObjectStore BackendPath rados://OBJECTSTORE_ID@OBJECTSTORE_POOL:OBJECTSTORE_NAMESPACE
Create the log file¶
Make sure to create the file that will be used by the tapeserver daemon to log what it is doing:
touch /var/log/cta/cta-taped.log
chmod a+w /var/log/cta/cta-taped.log
Create the sss key that will be used by the tapeserver to communicate with EOS¶
cd /etc/cta
xrdsssadmin -k cta-taped -u cta -g cta add ctafrontend_client_sss.keytab
chmod 600 ctafrontend_client_sss.keytab
chown cta:tape ctafrontend_client_sss.keytab
Configure systemd cta-taped service¶
Edit the file /etc/sysconfig/cta-taped:
# Arguments for the tape server daemon
# -f keep process in the foreground, do not fork
# -h display help and exit
# -c config file location (cta.conf)
#
# CTA_TAPED_OPTIONS="--foreground --log-to-file=/var/log/cta/cta-taped.log"
# Environment variables to configue the libXrdCl.so linked with cta-taped
#
# These two variables set the authentication mechanism to Simple Shared Secret
# (SSS) and set the SSS key to be that of the tape server
XrdSecPROTOCOL=sss
XrdSecSSSKT=/etc/cta/ctafrontend_client_sss.keytab
Edit the file /etc/systemd/system/cta-taped.service:
[Unit]
Description=CERN Tape Archive (CTA) tape server daemon
After=syslog.target network-online.target
[Service]
EnvironmentFile=/etc/sysconfig/cta-taped
ExecStart=/usr/bin/cta-taped --log-to-file=/var/log/cta/cta-taped.log
LimitCORE=infinity
OOMScoreAdjust=-1000
Type=forking
Restart=no
[Install]
WantedBy=default.target
Start the cta-taped daemon and test that it is running¶
systemctl daemon-reload
systemctl enable cta-taped
systemctl restart cta-taped
Verify that the cta-taped daemon is running by typing:
systemctl status cta-taped
● cta-taped.service - CERN Tape Archive (CTA) tape server daemon
Loaded: loaded (/etc/systemd/system/cta-taped.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2021-03-31 08:52:44 CEST; 18min ago
Process: 3702 ExecStart=/usr/bin/cta-taped --log-to-file=/var/log/cta/cta-taped.log (code=exited, status=0/SUCCESS)
Main PID: 3703 (cta-tpd-master)
Tasks: 75
Memory: 34.6M
CGroup: /system.slice/cta-taped.service
├─3703 /usr/bin/cta-taped --log-to-file=/var/log/cta/cta-taped.log...
├─3704 /usr/bin/cta-taped --log-to-file=/var/log/cta/cta-taped.log...
└─3705 /usr/bin/cta-taped --log-to-file=/var/log/cta/cta-taped.log...
Mar 31 08:52:44 eos-ccaffy-dev01.cern.ch systemd[1]: Starting CERN Tape Archi...
Mar 31 08:52:44 eos-ccaffy-dev01.cern.ch systemd[1]: Started CERN Tape Archiv...
Hint: Some lines were ellipsized, use -l to show in full.
Verify the drive registered itself by typing:
cta-admin dr ls
library drive host desired request status since vid tapepool vo files data MB/s session priority activity age reason
VLSTK VDSTK1 eos-ccaffy-dev01 Down - Down 1206 - - - - - - - 0 - 5 [cta-taped] INFO Setting the drive down at da[...]
If the cta-taped daemon did not start, you can look at the logs in the /var/log/cta/cta-taped.log.
Prepare the CTA test environment (cta-admin commands)¶
You can put the following code to a script file and run it:
LOGICAL_LIBRARY_NAME=VLSTK
cta-admin logicallibrary add \
--name $LOGICAL_LIBRARY_NAME \
--comment "ctasystest library mapped to drive VDSTK1"
VO_NAME="vo"
cta-admin virtualorganization add \
--vo $VO_NAME \
--readmaxdrives 1 \
--writemaxdrives 1 \
--comment "vo"
TAPE_POOL_NAME="ctasystest"
cta-admin tapepool add \
--name $TAPE_POOL_NAME \
--vo $VO_NAME \
--partialtapesnumber 5 \
--encrypted false \
--comment "ctasystest"
# add the media types of the tapes in production
cta-admin mediatype add \
--name T10K500G \
--capacity 500000000000 \
--primarydensitycode 74 \
--cartridge "T10000" \
--comment "Oracle T10000 cartridge formated at 500 GB (for developers only)"
cta-admin mediatype add \
--name 3592JC7T \
--capacity 7000000000000 \
--primarydensitycode 84 \
--cartridge "3592JC" \
--comment "IBM 3592JC cartridge formated at 7 TB"
cta-admin mediatype add \
--name 3592JD15T \
--capacity 15000000000000 \
--primarydensitycode 85 \
--cartridge "3592JD" \
--comment "IBM 3592JD cartridge formated at 15 TB"
cta-admin mediatype add \
--name 3592JE20T \
--capacity 20000000000000 \
--primarydensitycode 87 \
--cartridge "3592JE" \
--comment "IBM 3592JE cartridge formated at 20 TB"
cta-admin mediatype add \
--name LTO7M \
--capacity 9000000000000 \
--primarydensitycode 93 \
--cartridge "LTO-7" \
--comment "LTO-7 M8 cartridge formated at 9 TB"
cta-admin mediatype add \
--name LTO8 \
--capacity 12000000000000 \
--primarydensitycode 94 \
--cartridge "LTO-8" \
--comment "LTO-8 cartridge formated at 12 TB"
# add all tapes
TAPES=(V01001 V01002 V01003 V01004 V01005 V01006 V01007)
for ((i=0; i<${#TAPES[@]}; i++)); do
VID=${TAPES[${i}]}
cta-admin tape add \
--mediatype "T10K500G" \
--vendor vendor \
--logicallibrary $LOGICAL_LIBRARY_NAME \
--tapepool ctasystest \
--comment "ctasystest" \
--vid ${VID} \
--full false \
--comment "ctasystest"
done
STORAGE_CLASS_NAME="ctaStorageClass"
cta-admin storageclass add \
--name $STORAGE_CLASS_NAME \
--numberofcopies 1 \
--vo $VO_NAME \
--comment "ctasystest"
cta-admin archiveroute add \
--storageclass $STORAGE_CLASS_NAME \
--copynb 1 \
--tapepool $TAPE_POOL_NAME \
--comment "ctasystest"
MOUNT_POLICY_NAME="ctasystest"
cta-admin mountpolicy add \
--name $MOUNT_POLICY_NAME \
--archivepriority 1 \
--minarchiverequestage 1 \
--retrievepriority 1 \
--minretrieverequestage 1 \
--comment "ctasystest"
RMR_NAME="username"
EOS_INSTANCE="eosdev"
cta-admin requestermountrule add --instance $EOS_INSTANCE --name $RMR_NAME --mountpolicy $MOUNT_POLICY_NAME --comment "ctasystest"
###
echo "Labeling tapes:"
# add all tapes
for ((i=0; i<${#TAPES[@]}; i++)); do
VID=${TAPES[${i}]}
echo " cta-tape-label --vid ${VID}"
sudo cta-tape-label --vid ${VID}
if [ $? -ne 0 ]; then
echo "ERROR: failed to label the tape ${VID}"
exit 1
fi
done
Configure EOS to run with CTA¶
Now that EOS, the CTA Frontend and the tapeserver are running, we have to configure our EOS instance to communicate with CTA.
Set the cta user as sudo¶
The username specified in the SSS key shared between the EOS MGM and CTA Tape Server must be configured as an EOS superuser:
EOS Console [root://localhost] |/eos/username/> vid set membership cta +sudo
success: set vid [ eos.rgid=0 eos.ruid=0 mgm.cmd=vid mgm.subcmd=set mgm.vid.cmd=membership mgm.vid.key=cta:root mgm.vid.source.uid=cta mgm.vid.target.sudo=true ]
Create a tape directory on your EOS instance¶
This directory will be the one used to archive files onto tape and retrieve files from them.
EOS Console [root://localhost] |/eos/username/> mkdir tape
EOS Console [root://localhost] |/eos/username/>
Set the ACL on the newly created tape directory¶
We have to authorize ourselves and the cta-taped daemon (here uid=99) to write to this directory. We also need to authorize ourserlves to prepare files (retrieve them from tape).
EOS Console [root://localhost] |/eos/username/> whoami
Virtual Identity: uid=12345 (0,99,12345) gid=2763 (99,2763) [authz:krb5] sudo* host=localhost domain=localdomain
EOS Console [root://localhost] |/eos/username/> attr set sys.acl=u:12345:rwx+dp,u:99:rwx+dp,z:!u,u:0:+u .
Set the correct extended attributes on the tape directory¶
sudo eos attr set sys.workflow.sync::create.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::closew.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::archived.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::archive_failed.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::prepare.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::abort_prepare.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::evict_prepare.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::closew.retrieve_written="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::retrieve_failed.default="proto" /eos/username/tape
sudo eos attr set sys.workflow.sync::delete.default="proto" /eos/username/tape
sudo eos attr set sys.archive.storage_class=ctaStorageClass /eos/username/tape
sudo eos attr set sys.forced.layout=replica /eos/username/tape
sudo eos attr set sys.forced.nstripes=1 /eos/username/tape
You should see the following attributes on your tape directory:
eos attr ls /eos/username/tape
sys.acl="u:12345:rwx+dp,u:99:rwx+dp,z:!u,u:0:+u"
sys.archive.storage_class="ctaStorageClass"
sys.attr.link="/eos/dev/proc/workflow/"
sys.eos.btime="1615202872.649464979"
sys.forced.checksum="adler"
sys.forced.layout=""replica"
sys.forced.nstripes="1"
sys.workflow.sync::abort_prepare.default="proto"
sys.workflow.sync::archive_failed.default="proto"
sys.workflow.sync::archived.default="proto"
sys.workflow.sync::closew.default="proto"
sys.workflow.sync::closew.retrieve_written="proto"
sys.workflow.sync::create.default="proto"
sys.workflow.sync::delete.default="proto"
sys.workflow.sync::evict_prepare.default="proto"
sys.workflow.sync::prepare.default="proto"
sys.workflow.sync::retrieve_failed.default="proto"
Link the tape directory to the workflow directory¶
eos mkdir -p /eos/dev/proc/workflow
eos attr link /eos/dev/proc/workflow /eos/username/tape
Edit the eos.keytab file¶
Edit the /etc/eos.keytab file so that you add the two sss keys created (cta-taped and the eos_ctaeosdev) ones.
sudo vi /etc/eos.keytab
0 u:daemon g:daemon n:eosdev N:XXXX c:XXXX e:0 f:0 k:XXXXX
0 u:cta g:cta n:cta-taped N:XXXX c:XXXX e:0 f:0 k:XXXX
0 u:eosdev g:cta n:cta_eosdev N:XXXX c:XXXX e:0 f:0 k:XXXX
Enable the tape functionality in the MGM configuration file¶
vi /etc/xrd.cf.mgm
#####################################################################################
# IMPORTANT MAKE SURE YOU HAVE the -2 otherwise you will not be able to query prepare
xrootd.fslib -2 libXrdEosMgm.so
#-------------------------------------------------------------------------------
# Confguration parameters for tape
#-------------------------------------------------------------------------------
mgmofs.tapeenabled true
mgmofs.prepare.dest.space default
mgmofs.protowfendpoint localhost:10955
mgmofs.protowfresource /ctafrontend
NOTE: Make sure to have -2 in xrootd.fslib -2 libXrdEosMgm.so, to be able to run query prepare and evict.
Also, verify the following parameters are configured in the correct way. Kerberos can be enabled here if desired (see commented lines)
###########################################################
sec.protocol krb5 host/<hostname>.cern.ch@CERN.CH
sec.protbind * only krb5 sss unix
sec.protbind localhost krb5 unix sss
sec.protbind localhost.localdomain krb5 unix sss
###########################################################
Restart the MGM
sudo systemctl restart eos@mgm
Configure each of your FSTs to communicate with the CTA frontend, adding the following to each of /etc/xrd.cf.fst*
fstofs.protowfendpoint localhost:10955
fstofs.protowfresource /ctafrontend
Restart the all the FSTs, e.g.
sudo systemctl restart eos@fst1
Enable the Workflow Engine (WFE) for the default EOS space¶
# eos space config default space.wfe=on
# eos space set default on
# eos node set yourdevbox.cern.ch on
# eos space config default space.wfe.ntx=200
Enable file archived garbage collector¶
# eos space config default space.filearchivedgc=on
Create the tape space¶
eos space define tape
eos fs add -m 65535 tape localhost:1234 /does_not_exist tape
You should see the following:
eos space ls
┌──────────┬────────────────┬────────────┬────────────┬──────┬─────────┬───────────────┬──────────────┬─────────────┬─────────────┬──────┬──────────┬───────────┬───────────┬──────┬────────┬───────────┬──────┬────────┬───────────┐
│type │ name│ groupsize│ groupmod│ N(fs)│ N(fs-rw)│ sum(usedbytes)│ sum(capacity)│ capacity(rw)│ nom.capacity│ quota│ balancing│ threshold│ converter│ ntx│ active│ wfe│ ntx│ active│ intergroup│
└──────────┴────────────────┴────────────┴────────────┴──────┴─────────┴───────────────┴──────────────┴─────────────┴─────────────┴──────┴──────────┴───────────┴───────────┴──────┴────────┴───────────┴──────┴────────┴───────────┘
spaceview default 1 8 8 8 415.76 GB 1.37 TB 1.37 TB 0 B off off 20 off 2 0 on 200 0 off
spaceview tape 0 24 1 0 0 B 0 B 0 B 0 B off off 20 off 2 0 off 1 0 off
fs ls
┌────────────────────────┬────┬──────┬────────────────────────────────┬────────────────┬────────────────┬────────────┬──────────────┬────────────┬────────┬────────────────┐
│host │port│ id│ path│ schedgroup│ geotag│ boot│ configstatus│ drain│ active│ health│
└────────────────────────┴────┴──────┴────────────────────────────────┴────────────────┴────────────────┴────────────┴──────────────┴────────────┴────────┴────────────────┘
yourdevbox.cern.ch 2001 1 /fst/data1 default.0 schedgroup booted rw nodrain online N/A
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
localhost 1234 65535 /does_not_exist tape.0 off nodrain offline
eos vid set membership $(id -u daemon) +sudo
Archive files onto tape¶
Once everything is configured you can try to archive files onto tape. On your devbox, create a test file with some content in it:
vi test.txt
This is a test
Now, we will use the xrdcp command to archive it onto tape: we basically will upload this test.txt file on the /eos/username/tape directory to trigger the OPEN and CLOSEW workflows.
First, put the drive down so that we can see if the file is queued:
cta-admin dr down VDSTK1 --reason "Drive put down for test"
cta-admin dr ls
library drive host desired request status since vid tapepool vo files data MB/s session priority activity age reason
VLSTK VDSTK1 yourdevbox Down - Free 97 - - - - - - - 0 - 5 Drive put down for test
Upload the file to the tape directory:
kinit
xrdcp test.txt root://localhost//eos/username/tape
[9B/9B][100%][==================================================][9B/s]
Now verify the file is queued for archival:
eosdev@eos-ccaffy-dev01 ~ $ cta-admin sq
type tapepool vo logical library vid files queued data queued oldest age priority min age read max drives write max drives cur. mounts cur. files cur. data MB/s tapes capacity files on tapes data on tapes full tapes writable tapes
ArchiveForUser ctasystest vo - - 1 9 4 1 1 1 1 0 0 0 0 3.5T 5 45 0 7
Put the drive up again:
cta-admin dr up VDSTK1
The file should be archived:
EOS Console [root://localhost] |/eos/ccaffy/tape/> ls -y
d0::t1 -rw-r--r-- 1 username group 9 Apr 1 14:53 test.txt
Retrieve a file from tape¶
Put the drive down to see if the Retrieve request will be queued:
cta-admin dr down VDSTK1 --reason "Test retrieval"
Drive VDSTK1: set Down.
cta-admin dr ls
library drive host desired request status since vid tapepool vo files data MB/s session priority activity age reason
VLSTK VDSTK1 yourdevbox Down - Down 3 - - - - - - - 0 - 3 Test retrieval
Then retrieve the file we previously archived:
kinit
xrdfs root://localhost prepare -s /eos/ccaffy/tape/test.txt
0446bcb97f32:17fb95cd.606476aa:8
The file should be queued for retrieve.
cta-admin sq
type tapepool vo logical library vid files queued data queued oldest age priority min age read max drives write max drives cur. mounts cur. files cur. data MB/s tapes capacity files on tapes data on tapes full tapes writable tapes
Retrieve ctasystest vo VLSTK V01001 1 9 10 1 1 1 1 0 0 0 0 500.0G 6 54 0 1
Now let's put the drive back up:
cta-admin dr up VDSTK1
The file should be retrieved (d1::t1):
eos ls -y /eos/ccaffy/tape
d1::t1 -rw-r--r-- 2 ccaffy it 9 Apr 1 14:53 test.txt
If everything is working, save all the configuration files and the keytab files in a specific folder in your home directory¶
mkdir -p ~/cta-config-files
sudo cp -p /etc/cta/* ~/cta-config-files
mkdir -p ~/eos-config-files
sudo cp -p /etc/xrd.cf.* ~/eos-config-files
mkdir -p ~/keytabs
sudo cp -p /etc/*.keytab ~/keytabs
mkdir -p ~/sysconfig-files
sudo cp -p /etc/sysconfig/cta-* ~/sysconfig-files
sudo cp -p /etc/sysconfig/eos* ~/sysconfig-files
mkdir -p systemd-system-config-files
sudo cp -p /etc/systemd/system/cta-* systemd-system-config-files
This will allow to give back the configuration files and keytabs that would be overriden by the make install command.
Log files to look at to troubleshoot errors¶
Look at the different log files:
# CTA logs
/var/log/cta/cta-frontend.log
/var/log/cta/cta-taped.log
/var/log/cta/cta-frontend-xrootd.log
# EOS logs
/var/log/eos/mgm/xrdlog.mgm
/var/log/eos/fst*/xrdlog.fst* # depends on the FST you are interested in