Director Test

The Director periodically runs file transfer tests against every Origin and Cache in the federation to verify they are healthy and responding (every Director.OriginCacheHealthTestInterval, 15s by default). A server that repeatedly fails these tests is marked unhealthy and taken out of client redirection.

The test mechanism differs between Origins and Caches because of what each server can do with data. An Origin exports a writable monitoring namespace, so the Director can exercise the full data path with a real file: upload it, read it back, and delete it. A Cache is a read-only proxy: it accepts no uploads or deletes and only holds objects it has fetched from an upstream server, so the Director can only ask it to fetch an object. To keep that fetch from depending on the health of any particular Origin, the Director itself acts as the upstream for the /pelican/monitoring namespace and serves the test object from its own API, as described below.

How the Director Tests an Origin

For an Origin, the Director performs a full write–read–delete cycle with a real file:

PUT a small test file to the Origin under the /pelican/monitoring/directorTest namespace, which maps to a directory on the Origin’s local disk.
GET the file back and verify its content.
DELETE the file.

The Director runs these three steps in order and stops at the first failure, so the DELETE only runs after a successful GET. A cycle that fails after the upload therefore leaves that one test file behind on the Origin’s disk.

Unlike on Caches (see below), the Origin layout is flat and is not organized per Director. In a federation with several Directors, all of them write into the same /pelican/monitoring/directorTest/ directory, distinguished only by the timestamp in the file name, and each Director deletes its own file at the end of a successful cycle.

How the Director Tests a Cache

For a Cache, the Director performs a single GET of a synthetic path such as /pelican/monitoring/directorTest/<director-hostname>/<YYYY-MM-DD>/director-test-<timestamp>.txt and checks the response body against a known test string. No Origin is involved: the Director itself serves the file content from its /api/v1.0/director/healthTest API, acting as the upstream for the request.

Although the Director never uploads anything to the Cache, the Cache’s XRootD proxy file cache (PFC) stores each fetched test object on disk as a side effect of caching it: a small (~100 byte) data file plus a companion .cinfo metadata file. After each successful test cycle, the Director evicts the previous cycle’s test files by POSTing to the Cache.

Test File Cleanup on Caches

Before Pelican v7.27, a Cache’s test files accumulated in a single flat /pelican/monitoring/directorTest/ directory, growing to tens of thousands of entries and leading to disk/inode exhaustion. The files were only ever removed when the cache’s total disk usage crossed the XRootD PFC high watermark and the LRU purge swept them out as the oldest-accessed objects. Since v7.27, test files are organized per Director and per day, and are cleaned up by two complementary mechanisms: a Director-initiated eviction after every successful test cycle, and a Cache-side daily sweep that catches anything the Director missed. Both are described below.

Test File Layout

Director test files on a Cache live under:


/pelican/monitoring/directorTest/<director-hostname>/<YYYY-MM-DD>/director-test-<timestamp>.txt

In federations with multiple Directors, each Director writes only into its own <director-hostname> subtree and only cleans up files it created, so Directors never interfere with each other’s in-flight tests. The daily subdirectories keep any single directory small and allow whole days to be removed at once.

How Cleanup Works

Director-initiated eviction (primary). After each successful test cycle, the Director asks the Cache to evict the previous cycle’s test file by calling POST /api/v1.0/cache/evictTestFile with a federation-issued token. The Cache validates the token and the requested path (it must match the director test layout above), then calls the evict endpoint of its own XRootD server, /pelican/api/v1.0/evict, which is provided by the xrdhttp-pelican plugin loaded into the Cache’s XRootD. The plugin asks XRootD’s PFC to mark the file as a purge candidate.

Cache-side daily sweep (backup). Each Cache also runs its own cleanup at startup and every 24 hours thereafter. It deletes day-directories older than today in each Director’s subtree and trims today’s directory down to the most recent test object (the data file and its .cinfo are always removed together), catching files the Director missed due to evict API errors, network failures or restarts.

What to Expect on Disk

A successful eviction does not immediately remove the file from disk. It marks the file as a purge candidate in XRootD’s cache (PFC), and the bytes are unlinked later by PFC’s periodic purge cycle, which only acts once disk usage crosses the configured high-water mark. Old test files remaining visible on the filesystem is therefore normal and harmless; they are reclaimed automatically under disk pressure.

When Server.DropPrivileges is enabled, the Pelican process cannot delete XRootD-owned files directly, so the daily sweep also routes removals through the evict API and empty day-directories may persist until XRootD’s own housekeeping removes them.

Director.OriginCacheHealthTestInterval: how often the Director tests each Origin/Cache (default 15s).
Server.Hostname (on the Director): the hostname used for the per-director subtree; the Director errors at startup if it cannot be determined.
Director tests are distinct from the Origin/Cache’s own self-tests, which are controlled by Origin.SelfTest and Cache.SelfTest.