Verify CTA installation¶
Here are the various pods in our cta namespace:
[root@ctadevjulien orchestration]# kubectl get namespace
NAME STATUS AGE
cta Active 4m
default Active 88d
kube-system Active 88d
[root@ctadevjulien orchestration]# kubectl get pods -a --namespace cta
NAME READY STATUS RESTARTS AGE
ctacli 1/1 Running 0 3m
ctaeos 1/1 Running 0 3m
ctafrontend 1/1 Running 0 3m
init 0/1 Completed 0 4m
kdc 1/1 Running 0 3m
tpsrv 2/2 Running 0 3m
Everything looks fine, you can even check the logs of eos mgm:
[root@ctadevjulien CTA]# kubectl --namespace $NAMESPACE logs ctaeos
...
### ctaeos mgm ready ###
Or the Completed init pod:
[root@ctadevjulien CTA]# kubectl --namespace $NAMESPACE logs init
...
Using this configuration for library:
Configuring mhvtl library
DRIVESLOT is not defined, using driveslot 0
export LIBRARYTYPE=mhvtl
export LIBRARYNAME=VLSTK60
export LIBRARYDEVICE=sg35
export DRIVENAMES=(VDSTK61 VDSTK62 VDSTK63)
export DRIVEDEVICES=(nst15 nst16 nst17)
export TAPES=(V06001 V06002 V06003 V06004 V06005 V06006 V06007)
export driveslot=0
Configuring objectstore:
Configuring ceph objectstore
Wiping objectstore
Rados objectstore rados://cta-julien@tapetest:cta-julien is not empty: deleting content
New object store path: rados://cta-julien@tapetest:cta-julien
Rados objectstore rados://cta-julien@tapetest:cta-julien content:
driveRegister-cta-objectstore-initialize-init-295-20170307-23:20:49-1
cta-objectstore-initialize-init-295-20170307-23:20:49
agentRegister-cta-objectstore-initialize-init-295-20170307-23:20:49-0
schedulerGlobalLock-cta-objectstore-initialize-init-295-20170307-23:20:49-2
root
Configuring database:
Configuring oracle database
Wiping database
Aborting: unlockSchema failed: executeNonQuery failed for SQL statement... ORA-00942: table or view does not exist
Aborting: schemaIsLocked failed: executeQuery failed for SQL statement ... ORA-00942: table or view does not exist
Aborting: schemaIsLocked failed: executeQuery failed for SQL statement ... ORA-00942: table or view does not exist
Database contains no tables. Assuming the schema has already been dropped.
Labelling tapes using the first drive in VLSTK60: VDSTK61 on /dev/nst15:
V06001 in slot 1 Loading media from Storage Element 1 into drive 0...done
1+0 records in
1+0 records out
80 bytes (80 B) copied, 0.00448803 s, 17.8 kB/s
Unloading drive 0 into Storage Element 1...done
OK
...
Running a simple test¶
The test will archive and retrieve 10000 files.
cd continuousintegration/orchestration/tests
./archive_retrieve.sh -n cta
It configures CTA instance for the tests and xrdcp a file to EOS instance, checks that it is on tape and recalls it.
If something goes wrong, please check the logs from the various containers running on the pods that were defined during dev meetings:
init- Initializes the system, e.g. labels tapes using cat, creates/wipes the catalog database, creates/wipes the CTA object store and makes sure all drives are empty and tapes are back in their home slots.
kdc- Runs the KDC server for authenticating EOS end-users and CTA tape operators.
eoscli- Hosts the command-line tools to be used by an end-user of EOS, namely xrdcp, xrdfs and eos.
ctaeos- One EOS mgm.
- One EOS fst.
- One EOS mq.
- The EOS Simple Shared Secret (SSS) to be used by the EOS mgm and EOS fst to authenticate each other.
- The CTA SSS to be used by the EOS MGM to authenticate itself with the CTA frontend.
- The tape server SSS to be used by the EOS mgm to authenticate file transfer requests from the tape servers.
ctacli- The cta-admin command-line tool to be used by tape operators.
- This pod has the keytab of
ctaadmin1who is allowed to typectaadmin commands.
ctafrontend- One CTA front-end.
- The CTA SSS of the EOS instance that will be used by the CTA frontend to authenticate the EOS MGM.
tpsrvXXNo two pods in the same namespace can have the same name, hence each tpsrv pod will be numbered differently- One
cta-tapeddaemon running intapedcontainer oftpsrvxxxpod. - One
cta-rmcddaemon running inrmcdcontainer oftpsrvxxxpod. - The tape server SSS to be used by cta-taped to authenticate its file transfer requests with the EOS mgm (all tape servers will use the same SSS).
- One
client- The
ctacommand-line tool to be used byeosusers. - The
eoscommand-line tool. - This pod has the keytab of
user1who is allowed to read-write file inroot://ctaeos//eos/ctaeos/cta.
- The
Get an interactive access to the pod¶
$ kubectl --namespace $NAMESPACE exec <pod> -ti bash
Post-mortem analysis¶
logs¶
An interesting feature is to collect the logs from the various processes running in containers on the various pods.
Kubernetes allows to collect stdout and stderr from any container and add time stamps to ease post-mortem analysis.
Those are the logs of the init pod first container:
[root@ctadev01 CTA]# kubectl logs init --timestamps --namespace $NAMESPACE
2016-11-08T20:00:28.154608000Z New object store path: file:///shared/test/objectstore
2016-11-08T20:00:31.490246000Z Loading media from Storage Element 1 into drive 0...done
2016-11-08T20:00:32.712066000Z 1+0 records in
2016-11-08T20:00:32.712247000Z 1+0 records out
2016-11-08T20:00:32.712373000Z 80 bytes (80 B) copied, 0.221267 s, 0.4 kB/s
2016-11-08T20:00:32.733046000Z Unloading drive 0 into Storage Element 1...done
Dump the objectstore content¶
Connect to a pod where the objectstore is configured, like ctafrontend:
kubectl --namespace $NAMESPACE exec ctafrontend -ti bash
From there, you need to configure the objecstore environment variables, sourcing /tmp/objectstore-rc.sh install the missing protocolbuffer tools like protoc binary, and then dump all the objects you want:
[root@ctafrontend /]# yum install -y protobuf-compiler
...
Installed:
protobuf-compiler.x86_64 0:2.5.0-7.el7
Complete!
[root@ctafrontend /]# . /tmp/objectstore-rc.sh
[root@ctafrontend /]# cta-objectstore-list
OStoreDBFactory-tpsrv-340-20161216-14:17:51
OStoreDBFactory-ctafrontend-188-20161216-14:15:35
schedulerGlobalLock-makeMinimalVFS-init-624-20161216-14:14:17-2
driveRegister-makeMinimalVFS-init-624-20161216-14:14:17-1
makeMinimalVFS-init-624-20161216-14:14:17
agentRegister-makeMinimalVFS-init-624-20161216-14:14:17-0
root
RetrieveQueue-OStoreDBFactory-ctafrontend-188-20161216-14:15:35-3
[root@ctafrontend /]# cta-objectstore-dump-object RetrieveQueue-OStoreDBFactory-ctafrontend-188-20161216-14:15:35-3
1: 10
2: 0
3: "root"
4: "root"
5 {
10100: "V02001"
10131 {
9301: 1
9302: 1
}
10132 {
9301: 1
9302: 1
}
10133 {
9301: 1
9302: 1
}
10140: 406
10150: 0
}
Delete a kubernetes test instance¶
Deletion manages the removal of the infrastructure that had been created and populated during the creation procedure: it deletes the database content and drop the schema, remove all the objects in the objectstore independently of the type of resources (ceph objectstore, file based objectstore, oracle database, sqlite database...).
In our example:
[root@ctadevjulien CTA]# ./delete_instance.sh -n $NAMESPACE
Deleting cta instance
Unlock cta catalog DB
Delete cta catalog DB for instance
Deleted 1 admin users
Deleted 1 admin hosts
Deleted 1 archive routes
Deleted 1 requester mount-rules
Deleted 0 requester-group mount-rules
Deleted 1 archive files and their associated tape copies
Deleted 7 tapes
Deleted 1 storage classes
Deleted 1 tape pools
Deleted 1 logical libraries
Deleted 1 mount policies
Status cta catalog DB for instance
UNLOCKED
Drop cta catalog DB schema for instance
namespace "cta" deleted
................................OK
Deleting PV sg34 with status Released
persistentvolume "sg34" created
OK
Status of library pool after test:
NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE
sg30 1Mi RWO Available 19h
sg31 1Mi RWO Available 19h
sg32 1Mi RWO Available 19h
sg33 1Mi RWO Available 18h
sg34 1Mi RWO Available 0s
sg35 1Mi RWO Available 19h
sg36 1Mi RWO Available 19h
sg37 1Mi RWO Available 19h
sg38 1Mi RWO Available 19h
sg39 1Mi RWO Available 18h
When this deletion script is finished, the cta instance is gone: feel free to start another one...
[root@ctadevjulien CTA]# kubectl get namespace
NAME STATUS AGE
default Active 6d
kube-system Active 6d
Long running tests¶
If some tests run for long, the kerberos token in the cli pod should be renewed with:
kubectl --namespace=${instance} exec ctacli -- kinit -kt /root/admin1.keytab admin1@TEST.CTA