The cta-statistics-save tool¶

Danger

This tool only works for MySQL Statistics database. The compatibility with PostgreSQL or Oracle can be added later.

Danger

This tool is no longer actively used at CERN. We instead recommend that you generate your own statistics from the output of cta-admin --json tape ls --all.

Introduction¶

The cta-statistics-save tool is a C++ command line tool that allows the user to save the per-Tape statistics computed by the cta-statistics-update tool in a Statistics database (MySQL only).

Using the cta-statistics-save tool¶

A configuration file that describes the database connection is needed. The content of the configuration file has to be done according to the catalogue/cta-catalogue.conf.example file.

$ cta-statistics-save --catalogueconf catalogueDbConnectionFile

Computation and storage of the CTA statistics¶

How the cta-statistics-save tool computes and save the CTA statistics¶

Computation of the per-VO and total CTA file statistics¶

As the per-Tape statistics have already been computed by the cta-statistics-update tool, this tool will just query the catalogue database and compute the total statistics per Virtual Organization

Info

A tape belongs to a tape pool that belongs to a Virtual Organization. The SELECT of the statistics will be grouped by Virtual Organization.

The cta-statistics-save tool will query the Catalogue database and compute, per Virtual Organization, the following TAPE fields:

The sum of the NB_MASTER_FILES
The sum of the MASTER_DATA_IN_BYTES
The sum of the NB_COPY_NB_1
The sum of the COPY_NB_1SIZE_IN_BYTES
The sum of the NB_COPY_NB_GT_1
The sum of the COPY_NB_GT_1_IN_BYTES

A description of these fields can be found in the Executes a SQL UPDATE query on the TAPE table

These per Virtual Organization statistics will be stored in a Statistics object that will compute the total CTA statistics by summing each per Virtual Organization statistics.

This object will be used to save the per Virtual Organization and the total CTA files statistics.

Dump the per Virtual Organization and Total CTA statistics in a json output¶

The CTA statistics are displayed in json format on stdout.

$ cta-statistics-save --catalogueconf cta-catalogue.conf | jq
{
  "statisticsPerVo": [
    {
      "vo": "VO",
      "nbMasterFiles": 8,
      "masterDataInBytes": 60,
      "nbCopyNb1": 6,
      "copyNb1InBytes": 50,
      "nbCopyNbGt1": 2,
      "copyNbGt1InBytes": 10
    },
    {
      "vo": "VO2",
      "nbMasterFiles": 1,
      "masterDataInBytes": 100,
      "nbCopyNb1": 1,
      "copyNb1InBytes": 100,
      "nbCopyNbGt1": 0,
      "copyNbGt1InBytes": 0
    }
  ],
  "totalFiles": 9,
  "totalBytes": 160,
  "totalFilesCopyNb1": 7,
  "totalBytesCopyNb1": 150,
  "totalFilesCopyNbGt1": 2,
  "totalBytesCopyNbGt1": 10,
  "updateTime": 1585740174 
}

In case the cta-statistics-save tool fails¶

In case of failure, the exception that generated the failure will be printed to stderr. If the statistics have been retrieved and computed but not saved, a JSON output of these not-saved statistics will be printed on stderr.

Here is an example of failure message:

$ cta-statistics-save --catalogueconf cta-catalogue.conf
Failed on purpose.
StatisticsJson:{"statisticsPerVo": [{"vo": "VO","nbMasterFiles": 8,"masterDataInBytes": 60,"nbCopyNb1": 6,"copyNb1InBytes": 50,"nbCopyNbGt1": 2,"copyNbGt1InBytes": 10},{"vo": "VO2","nbMasterFiles": 1,"masterDataInBytes": 100,"nbCopyNb1": 1,"copyNb1InBytes": 100,"nbCopyNbGt1": 0,"copyNbGt1InBytes": 0}],"totalFiles": 9,"totalBytes": 160,"totalFilesCopyNb1": 7,"totalBytesCopyNb1": 150,"totalFilesCopyNbGt1": 2,"totalBytesCopyNbGt1": 10}

# jq print of the failed statistics json
$ cat statistics.json | jq
{
  "statisticsPerVo": [
    {
      "vo": "VO",
      "nbMasterFiles": 8,
      "masterDataInBytes": 60,
      "nbCopyNb1": 6,
      "copyNb1InBytes": 50,
      "nbCopyNbGt1": 2,
      "copyNbGt1InBytes": 10
    },
    {
      "vo": "VO2",
      "nbMasterFiles": 1,
      "masterDataInBytes": 100,
      "nbCopyNb1": 1,
      "copyNb1InBytes": 100,
      "nbCopyNbGt1": 0,
      "copyNbGt1InBytes": 0
    }
  ],
  "totalFiles": 9,
  "totalBytes": 160,
  "totalFilesCopyNb1": 7,
  "totalBytesCopyNb1": 150,
  "totalFilesCopyNbGt1": 2,
  "totalBytesCopyNbGt1": 10,
  "updateTime": 1585740174 
}