Skip to content

Repack lifecycle notes:

Repack structure

  • An index referencing all the repack requests (one per VID).
  • Only one request per VID is allowed an any point in time.
  • The repack request has a status and contains global counters for tape files on the tape, plus references to the individual requests to allow canceling.
  • When requests require processing, they get queued. They remain referenced in the index at all times.

    • Feeder queue: requests are queue there initially (status=Pending)
    • To expand queue: requests are fed in this queue in a controlled way, to be picked up for expansion in file-level requests (status=ToExpand)
    • The next status does not require queueing: (status=Starting). All file level requests are created, but none are reported as executed yet.
    • (status=Running). At least one file request is reported as done (for retrieve or archive, failure or success). We can go straight from ToExpand to Running if the file jobs get processed before the request creation completes.
    • (status=Success): all jobs done.
    • (status=Failed): at least a job not done, but all are complete.
  • Orthogonally, a repack request can be of type Expand, Repack or both.

  • The semantic of repack is to move the data out of the tape and to migrate it data to new tapes. It will hence create a replacement for any copy on this tape.
  • The semantic of expand is to generate any missing copy (if any) according to the file's storage class parameters. This can be needed is the storage class was changed after files were created (addition of a second copy of existing data).

  • File level sub requests are retrieve jobs converting into archive job(s) on completion. Job conversion is handled by the maintenance process. Success and failure reporting is also handled by it like for the EOS reporting. We can get several archive jobs after a single retrieve in the case of an expand type repack (expand meaning re-creation of missing tape files). This is done with a single archive request object as for initial archives.

  • Aborting: the file requests are referenced by the repack request, and can be found and removed through it. Any reporting on them will fail instantly. Disk copies should be cleared if present.

  • Disk destination: the disk destination is decided per repack request and is provided as a base URL which will be complemented by the file location (VID/fseq(08%d)).
  • During expansion, disk buffer is scanned for already existing files before creating the retrieve requests (or directly the archive ones). Files with correct size and checksum are kept and archive requests are created directly. Non-matching files get deleted.
  • Data safety and crashes. When expanding a repack request, we need to be able to determine if a sub request reference was created, yet the sub request was not created or a request was created, processed and deleted. In order to discriminate between the two cases, a status is kept together with the reference, and the request is marked as deleted in the reference before the actual deletion, preventing a re-creation (see cta.proto).