EOS MGM query rates

It has been observered that hammering the EOS MGM with client requests may cause starvation between different types of request. We have observered that a high ratecombination of archive and retrieve related requests have caused the EOS MGM to unfairly starve either tape archivals or retrievals. It is not known which request types encourage this starvation behaviour more than others. We do not know if "read only" query requests are completely innocent.

Given the above story of uncertainty it should be noted that the more FST garbage collectors there are, the higher the rate of "read only" query requests they are going to make against the EOS MGM. Assuming an FST garbage collector is single threaded, there will be a big difference between deploying a single FST garbage collector for each host and deploying one per file system. One possible solution to this dilemma would be to introduced explicit rate limiting within the FST garbage collectors themselves.