CTA Tape Verify¶

The CTA Tape Verify framework monitors tapes in the CTA system and alerts in case of potential data loss/corruption. It does so by periodically identifying tapes to retrieve data from, and then reading a selection of files back for individual verification using the CTA tool.

Installation¶

Requirements¶

This tool is intended to be run periodically using an external scheduler such as Cron or Rundeck.

Configuration¶

The tool can be configured in the standard config file like so:

tools:
  # -------------------------------
  # CTA Tape Verification
  # -------------------------------
  cta-ops-tape-verify:
    debug: false
    logger:
      log_dir: "/var/log/cta-ops/verification/"
    cta-ops-verify-tape:
      default_read_data_size: '0B'
      default_read_time: 0
      default_first: 10
      default_random: 10
      default_last: 10
    cta-ops-verification-feeder:
      verification_mount_policy: 'verification'
      default_min_age: 0
      default_max_verify: 10
      default_min_data_on_tape: 0
      default_min_relative_capacity: 0
      default_verify_options: '--first 10 --last 10 --read_time 30'
      default_verify_policy: 'last_verified'
      default_tape_verify_path: 'cta-verify-tape'
      ts_format: '%Y-%m-%d %H:%M:%S'
      sleep_time: 120  # 2* 60

Usage¶

cta-ops-verify-tape¶

The cta-ops-verify-tape command submits a tape for verification by selecting some files from it and issuing a cta-verify-file command for each of them.

When a tape is submitted for verification, it's verification_status is updated to {date: <current_date>, status: ongoing, files_submitted: <number of files>, files_verified: 0, files_failed: 0}.

The options for this command are:

vid: The vid of the tape to be verified.
read_time: The minimum amount of time it should take to read a tape (assuming a read speed of 300 MB/s).
data_size: The minimum amount of data that should be read from the tape.
first: The number of files that should be read from the beginning of the tape (fseq order)
last: The number of files that should be read from the end of the tape (fseq order)
random: The number of files that should be read from the middle of the tape
all: If present, all tape files will be verified
options: Options to pass to the cta-verify-file commands.

The cta-ops-verify-tape command should be run from the CTA frontend.

cta-ops-verification-feeder¶

This command is responsible for selecting which tapes are to be verified according to user supplied parameters. It is intended to be run periodically by a scheduler such as Cron or Rundeck, and it handles the following:

Check the tapes that have an ongoing verification status. If the verification has finished (cta-admin --json sq does now show the verification mount policy for the queue) change the verification status to finished).
Compare the number of ongoing verifications with a user supplied maximum. If it is less than the maximum, submit new tapes for verification until the limit is reached.

The feeder can use three parameters to choose which tape to submit for verification:

last_read: Choose the tape that has not been read in the longest time.
last_write: Choose the tape that has not been written in the longest time.
last_verified: Choose the tape that has not been verified in the longest time.

The options for the cta-verification-feeder are as follows:

filter: Filter out tapes matching the states passed.
[ ] tapepool: Only verify tapes in the specified tapepool
min_data_on_tape: Minimum number of bytes on tape for it to be verified
min_relative_capacity: Minimum relative filled capacity of tape for it to be verified
verify_path: Location of cta-verify-tape executable.
verify_policy: One of last_read, last_written, last_verified.
full_tape: Verify only tapes that are full.
noaction: Dry run, do not submit the tapes for verification.
minage: Verify only tapes which have not been verified for so many days.
maxverify: maximum number verify processes to run concurrently.
verify_options: Options to pass to cta-verify-tape.
logfile: Log file path.

The feeder calls the cta-ops-verify-tape script for each tape it decides to verify.