Query Pelican Server Metrics via Prometheus

For Pelican >= 7.16.0. Older version of Pelican may not include all of the metrics listed.

Pelican servers have Prometheus  embedded by default and provide a handful of Prometheus metrics to monitor server status. You can access the metrics endpoint at https://<pelican-server-host>:<server-web-port>/metrics to see all the available metrics and their current values. By default, /metrics is a protected endpoint and you are required to login and get authenticated to view the page. You can change Monitoring.MetricAuthorization to false in config to turn off the authentication.

Pelican also exposes Prometheus PromQL query engine  at https://<pelican-server-host>:<server-web-port>/api/v1.0/prometheus where you can query the metrics against Prometheus powerful query language.

Example: https://<pelican-server-host>:<server-web-port>/api/v1.0/prometheus/query?query=pelican_component_health_status[10m] queries the pelican_component_health_status metric and shows data collected in past 10 min.

However, Pelican does not support Prometheus native /graph endpoint nor other Prometheus native web services other than the two above. For custom data visualizations, Grafana  is one of the popular software to use.

Pelican included metrics from built-in gin  web server, as well as Go runtime. For all metrics available, visit https://<pelican-server-host>:<server-web-port>/api/v1.0/prometheus/label/__name__/values.

Pelican also has a set of built-in metrics to monitor Pelican server’s status, listed below.

Scrape Pelican Metrics with an External Prometheus

If you operate a separate Prometheus instance and want it to scrape metrics from one or more Pelican services, you can point that Prometheus at the Pelican /metrics endpoint. Because /metrics requires authentication by default, you must issue a Pelican API key for the scraping job and configure Prometheus to send it as a bearer credential.

The built-in Prometheus instance (controlled by Monitoring.EnablePrometheus, on by default) is sufficient when you only need to inspect a single server: it keeps metrics for the period set by Monitoring.DataRetention (15 days by default) and needs no extra infrastructure. Use an external Prometheus when you want to collect metrics from several Pelican servers in one place, retain them for longer, or feed a shared Grafana or alerting stack. The two are not mutually exclusive: /metrics is served regardless of Monitoring.EnablePrometheus, so you can keep the built-in instance and scrape the same server externally.

Generate an API key

Use pelican apikey generate on the Pelican server you want to scrape. The command reads the server’s configuration to sign its request with the server’s issuer key, so if the server’s configuration file is not at the default location, pass the same --config <path> the server was started with. The key must include the monitoring.scrape scope, which grants access to the /metrics endpoint. (The separate monitoring.query scope is only needed when issuing PromQL queries against /api/v1.0/prometheus; it does not grant access to /metrics.)

pelican apikey generate \
  --scopes "monitoring.scrape" \
  --expiration "2038-01-19T03:14:07Z" \
  --server https://<server-web-url>

• --scopes: must include monitoring.scrape for scraping /metrics. To allow the same key to also run PromQL queries, use --scopes "monitoring.scrape,monitoring.query".
• --expiration: RFC3339 timestamp at which the key stops being valid.
• --server: must match the Pelican server’s Server.ExternalWebUrl. When that value is not set explicitly, it falls back to https://<Server.Hostname>[:<WebPort>], with the default :443 port stripped. If the server is reachable by multiple hostnames, use the one the server is configured with. For example, if aaa.com is a CNAME pointing at bbb.com and the server’s configuration sets Server.Hostname: bbb.com with Server.ExternalWebUrl unset, use --server https://bbb.com; if the configuration instead sets Server.ExternalWebUrl: https://aaa.com, use --server https://aaa.com. A mismatch will be rejected.

The command prints the API key to standard output. Save it to a file that the scraping Prometheus instance can read (for example, /etc/prometheus/secrets/pelican-apikey) and restrict the file’s permissions; treat it as a credential.

Note: An API key is validated against the database of the server that issued it. If the configured hostname is a CNAME that may fail over to a different deployment, the key will not be accepted by the failover target. Re-issue the key against the failover server, or generate it on each backing deployment ahead of time.

Configure the scrape job

Add a job similar to the following to your Prometheus configuration:

scrape_configs:
  - job_name: pelican
    metrics_path: /metrics
    scheme: https
    static_configs:
      - targets: ['<server-hostname>:<server-web-port>']
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/pelican-apikey

• metrics_path is /metrics, which is also the Prometheus default.
• scheme must be https; Pelican does not serve metrics over plain HTTP.
• targets should be the same host[:port] you used for --server (without the https:// scheme). For a server reachable on the default HTTPS port, the :443 suffix can be omitted.
• authorization.credentials_file should point to the file containing the API key value, with no Bearer prefix or trailing newline. To inline the secret instead of reading it from a file, use credentials: <api-key> in place of credentials_file. (Older Prometheus syntax exposes the equivalent bearer_token / bearer_token_file fields; the authorization block is the current preferred form.)

Note: For a test deployment that uses a self-signed certificate, you can turn off server certificate verification for this job by adding the following under the job entry. Do not use this in production.

    tls_config:
      insecure_skip_verify: true

Start Prometheus

Start Prometheus with the configuration file above (or reload a running instance so it picks up the new job):

prometheus --config.file=/etc/prometheus/prometheus.yml

Once Prometheus is running, the new target should appear as UP on the Prometheus /targets page (http://localhost:9090/targets by default). If the target instead reports a 403, double-check that the API key was issued with the monitoring.scrape scope and that the --server value used when generating the key matched the Pelican server’s Server.ExternalWebUrl.

Counter Metrics

Many metrics in this documentation are counters (typically identified by names ending in _total or _count). Counter metrics are monotonically increasing values that accumulate over time. Important notes about counters:

• Counters reset on restart: When a Pelican server restarts, all counter values reset to 0. The counter then begins accumulating from 0 again.
• Time ranges are required: To get meaningful data from counters, you must use PromQL functions that calculate changes over time, such as:
  • rate() - calculates the per-second average rate of increase
  • irate() - calculates the per-second instant rate of increase
  • increase() - calculates the increase over a time range
• Example usage: Instead of querying xrootd_server_bytes_total directly, use rate(xrootd_server_bytes_total[5m]) to get bytes per second, or increase(xrootd_server_bytes_total[1h]) to get total bytes transferred in the last hour.

Gauge Metrics

Many metrics in this documentation are gauges. Gauge metrics represent a single numerical value that can go up or down over time. Important notes about gauges:

• Gauges represent current state: Unlike counters, gauges show the current value of something at a point in time (e.g., number of active connections, current CPU usage, available disk space).
• Gauges can be queried directly: You can query gauge metrics directly without time-range functions like rate() or increase(). The value represents the current state.
• Gauges persist across restarts: When a server restarts, gauge values reset to their initial state (often 0), but they don’t accumulate like counters. The gauge will reflect the new current state after restart.
• Example usage: Query xrootd_server_io_active directly to see the current number of active IO operations, or use avg_over_time(xrootd_server_io_active[5m]) to see the average over the last 5 minutes.

All Servers

All of the Pelican servers have the following metrics:

process_start_time_seconds

The UNIX epoch time in seconds when the Pelican process started.

To get the duration of the Pelican server running time, use the following PromQL:

time() - process_start_time_seconds

This yields the duration in seconds.

pelican_component_health_status

The health status of Pelican server components. The metric value can be converted into following status:

Note: This is a gauge metric representing the current health status. Query directly to see the current state of each component.

1: Critical
2: Warning
3: OK
4: Unknown

Label: component

pelican_component_health_status_last_update

The timestamp of last update of health status of Pelican server components. The value is UNIX time in seconds. It shares the same label as pelican_component_health_status

Note: This is a gauge metric representing the last update timestamp. Query directly to see when each component’s health status was last updated.

pelican_server_xrootd_last_crash

The timestamp (seconds) of the last crash of the XRootD server.

Note: This is a gauge metric representing the timestamp of the last crash. Query directly to see when XRootD last crashed. A value of 0 indicates no crashes have occurred since the server started.

Registry

pelican_registry_federation_namespaces

The number of namespace registrations in the registry.

Note: This is a gauge metric representing the current number of namespace registrations. Query directly to see the current count.

Label: status

pelican_osdf_institution_count

Total number of contributing institutions. This is only available when running in OSDF mode.

Note: This is a gauge metric representing the current number of institutions. Query directly to see the current count.

Storage Servers (Origin and Cache)

xrootd_monitoring_packets_received_total

The total number of XRootD monitoring  UDP packets received.

Note: This is a counter metric. Use rate(xrootd_monitoring_packets_received_total[5m]) to get packets per second, or increase(xrootd_monitoring_packets_received_total[1h]) to get total packets in the last hour.

xrootd_sched_thread_count

The number of XRootD scheduler threads. Ref: https://xrootd.web.cern.ch/doc/dev6/xrd_monitoring.htm#_Toc204013493 

Note: This is a gauge metric representing the current number of threads. Query directly or use avg_over_time(xrootd_sched_thread_count[5m]) to see the average over a time range.

Label: state

xrootd_sched_thread_creations

Number of scheduler thread creations.

Note: This is a counter metric. Use rate(xrootd_sched_thread_creations[5m]) to get thread creations per second, or increase(xrootd_sched_thread_creations[1h]) to get total thread creations in the last hour.

xrootd_sched_thread_destructions

Number of scheduler thread destructions.

Note: This is a counter metric. Use rate(xrootd_sched_thread_destructions[5m]) to get thread destructions per second, or increase(xrootd_sched_thread_destructions[1h]) to get total thread destructions in the last hour.

xrootd_sched_thread_limit_reached

Number of times the scheduler thread limit has been reached.

Note: This is a counter metric. Use rate(xrootd_sched_thread_limit_reached[5m]) to get limit hits per second, or increase(xrootd_sched_thread_limit_reached[1h]) to get total limit hits in the last hour.

xrootd_sched_jobs

Number of scheduler jobs requiring a thread.

Note: This is a gauge metric representing the current number of jobs. Query directly or use avg_over_time(xrootd_sched_jobs[5m]) to see the average over a time range.

xrootd_sched_queue_longest_length

Length of the longest run-queue.

Note: This is a gauge metric representing the current longest queue length. Query directly or use avg_over_time(xrootd_sched_queue_longest_length[5m]) to see the average over a time range.

xrootd_sched_queued

Number of jobs queued.

Note: This is a gauge metric representing the current number of queued jobs. Query directly or use avg_over_time(xrootd_sched_queued[5m]) to see the average over a time range.

xrootd_server_bytes_total

The total number of bytes XRootD sent/received. Ref: https://xrootd.web.cern.ch/doc/dev6/xrd_monitoring.htm#_Toc204013487  (See link.in and link.out)

Note: This is a counter metric. Use rate(xrootd_server_bytes_total[5m]) to get bytes per second, or increase(xrootd_server_bytes_total[1h]) to get total bytes in the last hour.

Label: direction

xrootd_server_connections_total

The total number of server connections to XRootD.

Note: This is a counter metric. Use rate(xrootd_server_connections_total[5m]) to get connections per second, or increase(xrootd_server_connections_total[1h]) to get total connections in the last hour.

xrootd_storage_volume_bytes

The storage volume usage on the storage server.

Note: This is a gauge metric representing the current storage volume. Query directly or use avg_over_time(xrootd_storage_volume_bytes[5m]) to see the average over a time range.

Label: type

Label: server_type

Label: ns

The top-level namespace the XRootD is serving for. Example: /foo

xrootd_transfer_bytes

The bytes of transfers for individual object. Ref: https://xrootd.web.cern.ch/doc/dev6/xrd_monitoring.htm#_Toc204013508  (See XrdXrootdMonStatXFR)

Note: This is a counter metric. Use rate(xrootd_transfer_bytes[5m]) to get bytes per second, or increase(xrootd_transfer_bytes[1h]) to get total bytes in the last hour.

Label: path

The path to the object (filename).

Label: ap

Authentication protocol name used to authenticate the client. Default is https

Label: dn

Client’s distinguished name as reported by ap. If no name is present, the variable data is null.

Label: role

Client’s role name as reported by prot. If no role name is present, the variable data is null.

Label: org

Client’s group names in a space-separated list. If no groups are present, the tag variable data is null.

Label: proj

Client’s User-Agent header when requesting the file. This is used to label the project name that accesses the file.

Label: type

xrootd_transfer_operations_total

The number of transfer operations performed for individual object. The labels for this metric is the same as the ones in xrootd_transfer_bytes

Note: This is a counter metric. Use rate(xrootd_transfer_operations_total[5m]) to get operations per second, or increase(xrootd_transfer_operations_total[1h]) to get total operations in the last hour.

xrootd_transfer_readv_segments_total

The number of segments in readv operations for individual object. The labels for this metric is the same as the ones in xrootd_transfer_bytes except that type label isn’t available in this metric.

Note: This is a counter metric. Use rate(xrootd_transfer_readv_segments_total[5m]) to get segments per second, or increase(xrootd_transfer_readv_segments_total[1h]) to get total segments in the last hour.

xrootd_cache_access_bytes

Number of bytes the data requested is in the cache or not.

Note: This is a gauge metric representing the current cache access state. Query directly or use avg_over_time(xrootd_cache_access_bytes[5m]) to see the average over a time range.

Label: path

The path to the object (filename).

Label: type

xrootd_server_io_total

Total storage operations in origin/cache server.

Note: This is a counter metric. Use rate(xrootd_server_io_total[5m]) to get operations per second, or increase(xrootd_server_io_total[1h]) to get total operations in the last hour.

xrootd_server_io_active

Number of ongoing storage operations in origin/cache server.

Note: This is a gauge metric representing the current number of active IO operations. Query directly or use avg_over_time(xrootd_server_io_active[5m]) to see the average over a time range.

xrootd_server_io_wait_seconds_total

The aggregate time spent in storage operations in origin/cache server.

Note: This is a counter metric. Use rate(xrootd_server_io_wait_seconds_total[5m]) to get average wait time per second, or increase(xrootd_server_io_wait_seconds_total[1h]) to get total wait time in the last hour.

xrootd_cpu_utilization

CPU utilization of the XRootD server, represented as the average number of CPU cores utilized (e.g., 1.0 = one full core, 2.5 = two and a half cores).

Note: This is a gauge metric representing the current CPU utilization. Query directly to see the current utilization, or use avg_over_time(xrootd_cpu_utilization[5m]) to see the average over a time range.

OSS Layer Metrics

The following metrics are available from the XRootD OSS layer.

Note: All metrics ending in _total are counter metrics. Use rate() or increase() with a time range to query them. All metrics ending in _time_seconds are histogram metrics that track operation duration distributions.

S3 Cache Plugin Metrics

The following metrics are available from the XRootD S3 cache plugin.

Note: All metrics ending in _total are counter metrics. Use rate() or increase() with a time range to query them.

XrdCl Client Metrics

The following metrics are available from the XrdCl client.

Note: All metrics ending in _total are counter metrics. Use rate() or increase() with a time range to query them. Metrics ending in _timestamp_seconds are gauge metrics representing timestamps and can be queried directly. xrootd_xrdcl_queue_pending is a gauge metric representing the current queue size.

Cache Eviction Metrics

The following metrics are available from the XRootD cache eviction process.

Note: All cache eviction metrics are gauge metrics representing current state. Query them directly or use avg_over_time() to see averages over a time range.

Director

up

The Pelican director scrapes Prometheus metrics from all origins and cache servers that successfully advertise to the director. This metric reflects the Pelican origin or cache servers that are scraped by the director.

Label: server_name

The name of the storage server. By default it’s the hostname.

Label: server_type

Label: server_url

The storage server XRootD url.

Label: server_web_url

The storage server web url.

Label: server_auth_url

The storage server authentication url.

Label: server_lat

The storage server latitude.

Label: server_long

The storage server longitude.

# of Active Origins and Caches

With the up metric, it is possible to count number of active origin and cache servers in the federation by a simple Prometheus query: count(up{server_type=<"Origin">}) for counting origin servers, or count(up{server_type=<"Cache">}) for counting cache servers.

pelican_director_advertisements_received_total

The accumulated number of origin/cache advertisements to the director. This metric shows if an origin/cache server successfully joins the federation or not. For origin servers, it also shows if each federation namespace prefix it exports passed director verification.

Note: This is a counter metric. Use rate(pelican_director_advertisements_received_total[5m]) to get advertisements per second, or increase(pelican_director_advertisements_received_total[1h]) to get total advertisements in the last hour.

Label: server_name

The name of the storage server. By default it’s the hostname.

Label: server_type

Label: server_web_url

The storage server web url.

Label: namespace_prefix

The federation namespace prefix the storage server exported.

Label: status_code

The status code of the director’s response. The most useful value is 403, which means the server advertisement didn’t pass director’s verification.

pelican_director_stat_total

The accumulated number of stat query the director made to origin/cache servers to check for object availability. Only available when Director.EnableStat is set to true. This metric is a good indicator of object availability and origin/cache service quality.

Note: This is a gauge metric representing the current accumulated count. Query directly to see the current total. Note that this metric accumulates but is a gauge (not a counter), so it may reset on restart.

Label: server_name

The name of the storage server. By default it’s the hostname.

Label: server_type

Label: server_url

The storage server XRootD url.

Label: result

The stat query result.

Label: cached_result

Whether the result was cached.

pelican_director_stat_active

The ongoing stat queries at the server. Note that Prometheus samples the metric value per 15s, and each stat request only takes ~10-100ms to finish. The value of this metric can’t capture per-second transient requests.

Note: This is a gauge metric representing the current number of active stat queries. Query directly to see the current count.

Label: server_name

The name of the storage server. By default it’s the hostname.

Label: server_type

Label: server_url

The storage server XRootD url.

pelican_director_total_ftx_test_suite

The number of file transfer test suite the director issued. In Pelican, director creates a test file and sent to origin servers to as a health test. It issues such test suite when it receives the registration from the origin server. In a test suite, a timer was set to run a cycle of uploading, getting, and deleting the test file every 15 seconds. Such cycle is called a “test run”. In theory, director should issue only one test for each origin servers; however, since the registration information was stored in a TTL cache in director, and it expires after certain period of time, and the test suite issued will be cancelled. A new test suite is issued with the new registration. Thus, director can issue multiple test suites to an origin server.

Note: This is a counter metric. Use rate(pelican_director_total_ftx_test_suite[5m]) to get test suites per second, or increase(pelican_director_total_ftx_test_suite[1h]) to get total test suites in the last hour.

Label: server_name

The name of the storage server. By default it’s the hostname.

Label: server_type

Label: server_web_url

The storage server web url.

pelican_director_active_ftx_test_suite

The number of active director file transfer test suite. As mentioned in previous metric, the test suites are individual tasks running independently from the main program logic. This can cause race condition in some condition where an expired test suite was not cleared but a new test suite is issued for the same origin. This metric records such condition for debugging and monitoring. The value of the metric should be 1 for all the time.

This metric shares the same label as pelican_director_total_ftx_test_suite

Note: This is a gauge metric representing the current number of active test suites. Query directly to see the current count.

pelican_director_total_ftx_test_runs

The number of file transfer test runs the director issued. A “test run” is a set of upload/get/delete of test files to a origin. It executes in a cycle of 15s (by default).

Note: This is a counter metric. Use rate(pelican_director_total_ftx_test_runs[5m]) to get test runs per second, or increase(pelican_director_total_ftx_test_runs[1h]) to get total test runs in the last hour.

This metric shares the same label as pelican_director_total_ftx_test_suite, with two additions:

Label: status

Label: report_status

pelican_director_map_items_total

The total number of map items in the director, by the name of the map.

Note: This is a gauge metric representing the current number of map items. Query directly to see the current count.

Label: name

The name of the map. One of healthTestUtils, filteredServers, serverStatUtils, serverStatEntries.

pelican_director_ttl_cache

The statistics of various TTL caches.

Note: This is a gauge metric representing the current TTL cache statistics. Query directly to see the current values.

Label: name

The name of the cache. One of serverAds, jwks.

Label: type

The type of the statistic. One of evictions, insertions, hits, misses, total.

pelican_director_server_count

The number of servers currently recognized by the Director, delineated by pelican/non-pelican and origin/cache.

Note: This is a gauge metric representing the current number of servers. Query directly to see the current count.

Label: server_name

The name of the server.

Label: server_type

Label: from_topology

Whether the server was discovered from the topology.

pelican_director_client_requests_total

The total number of requests from clients.

Note: This is a counter metric. Use rate(pelican_director_client_requests_total[5m]) to get requests per second, or increase(pelican_director_client_requests_total[1h]) to get total requests in the last hour.

Label: version

The client version.

Label: service

The service that received the request.

pelican_director_redirects_total

The total number of redirects the director issued.

Note: This is a counter metric. Use rate(pelican_director_redirects_total[5m]) to get redirects per second, or increase(pelican_director_redirects_total[1h]) to get total redirects in the last hour.

Label: destination

The destination of the redirect.

Label: status_code

The status code of the redirect.

Label: version

The client version.

Label: network

The network of the client.

pelican_director_maxmind_server_errors_total

The total number of errors encountered trying to resolve server coordinates using the GeoIP MaxMind database.

Note: This is a counter metric. Use rate(pelican_director_maxmind_server_errors_total[5m]) to get errors per second, or increase(pelican_director_maxmind_server_errors_total[1h]) to get total errors in the last hour.

Label: network

The network address that was being resolved.

Label: server_name

The name of the server that was being resolved.

pelican_director_maxmind_client_errors_total

The total number of errors encountered trying to resolve client coordinates using the GeoIP MaxMind database.

Note: This is a counter metric. Use rate(pelican_director_maxmind_client_errors_total[5m]) to get errors per second, or increase(pelican_director_maxmind_client_errors_total[1h]) to get total errors in the last hour.

Label: network

The network address that was being resolved.

Label: project

The project of the client that was being resolved.

pelican_director_rejected_advertisements

The total number of advertisements rejected by the director.

Note: This is a counter metric. Use rate(pelican_director_rejected_advertisements[5m]) to get rejections per second, or increase(pelican_director_rejected_advertisements[1h]) to get total rejections in the last hour.

Label: hostname

The hostname of the server that sent the advertisement.

pelican_director_server_statusweight

The EWMA-smoothed status weight generated by the Director for each server.

Note: This is a gauge metric representing the current status weight. Query directly to see the current weight value.

Label: server_name

The name of the server.

Label: server_url

The URL of the server.

Label: server_type

Removed Metrics

The previously deprecated metrics below have been removed. Use the replacement listed for each.

XRootD metrics:

• xrootd_monitoring_packets_received: Removed; use xrootd_monitoring_packets_received_total.
• xrootd_transfer_readv_segments_count and xrootd_transfer_readv_segments_toal: Removed; use xrootd_transfer_readv_segments_total.
• xrootd_transfer_operations_count: Removed; use xrootd_transfer_operations_total.
• xrootd_server_connection_count: Removed; use xrootd_server_connections_total.
• xrootd_server_bytes: Removed; use xrootd_server_bytes_total.
• xrootd_server_io_wait_time: Removed; use xrootd_server_io_wait_seconds_total.

Director metrics:

• pelican_director_client_version_total: Removed; use pelican_director_client_requests_total.
• pelican_director_redirections_total: Removed; use pelican_director_redirects_total.
• pelican_director_geoip_errors and pelican_director_geoip_errors_total: Removed; these were split into pelican_director_maxmind_server_errors_total and pelican_director_maxmind_client_errors_total.