Parameters

ConfigLocations provides administrators a way to define a list of directories containing Pelican configuration files. Within a given directory, files are read in lexicographical order, and any keys that are defined in multiple files will take the value from the last file read. Directories are read in the order provided by the list. For example, specifying:

ConfigLocations: ["/configs1", "/configs2"]

will read files first from /configs1 and then from /configs2. If a key is defined in both /configs1 and /configs2, the value from /configs2 will be used. If /configs1 contains files a.yaml and b.yaml where both define the same key, the value from b.yaml will be used.

Subdirectories of the provided directories are not read. Only the root config file's ConfigLocations key is used, and any redefinitions are ignored.

A bool indicating whether Pelican should emit debug messages in its log. NOTE: this will override whatever is set within your configuration file under Logging.Level!

When set to true, Pelican will skip TLS verification. This allows a "man in the middle" attack on the connection but can simplify testing. Intended for developers.

A filepath to the file containing a PEM-encoded ecdsa private key which later will be parsed into a JWK and serves as the private key to sign various JWTs issued by this server.

A public JWK will be derived from this private key and used as the key for token verification.

A list of IP addresses whose GeoIP resolution should be overridden with the supplied Lat/Long coordinates (in decimal form). This affects both server ads (for determining the location of origins and caches) and incoming client requests (for determing where a client request is coming from).

Configuration takes an IP address (both regular and CIDR) and a Coordinate made up of a lat/long pair in decimal format. For example:

GeoIPOverrides:
  - IP: "123.234.123.234"
    Coordinate:
      Lat: 43.073904
      Long: -89.384859
  - IP: "ABCD::1234/112"
    Coordinate:
      Lat: 39.8281
      Long: -98.5795

Will result in the IP address "123.234.123.234" being mapped to Madison, WI, and IP addresses in the range ABCD::0000-FFFF will be mapped to a field in Kansas.

[Deprecated] A legacy configuration for disabling the client's HTTP proxy. See Client.DisableHttpProxy for new config.

[Deprecated] A legacy configuration for disabling the client's proxy fallback. See Client.DisableProxyFallback for new config.

[Deprecated] A legacy configuration for setting the client's minimum download speed. See Client.MinimumDownloadSpeed for new config.

Maximum time allowed for establishing a connection to target host.

Maximum time a TCP connection should be kept alive without any activity.

Maximum number of idle connections that the HTTP client should maintain in its connection pool.

Maximum duration an idle connection should remain open in the connection pool.

Maximum time allowed for the TLS handshake to complete when making an HTTPS connection.

Timeout to control how long the client should wait for the "Expect: 100-continue" response from the server before sending the request body.

Maximum time the client should wait for the response headers to be received from the server.

A string defining the log level of the client. Options include (going from most info to least): Trace, Debug, Info, Warn, Error, Fatal, Panic.

A filename defining a file to write log outputs to, if the user desires.

A bool defining if progress bars should be enabled or not.

Trace level of XRootD cluster management service, one of the main XRootD executables. Cms basically is a file (or asset) discovery service. Each server has a cmsd daemon which connect to a master one informing it if a server is available. XRootD asks cms where a file could be found and cms works to report back the server for where the file is located. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace level of scitokens debug output within XRootD configuration. This entails token management and security credentials within XRootD. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace level of the eXtended Request Daemon within XRootD, another main XRootD executable. This reports information the XRootD protocol and works with cms. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace options for XRootD debug output within XRootD configuration. This prefix is reserved for the xroot protocol, which is the component that sits on sockets and talks to clients as they query file-system info, open files, and read data. This is the protocol for XRootD (like http) and handles connections and requests. Accepted values: trace, debug, info, warn, error, fatal, panic

Logging level for the HTTP component of the origin. Increasing to debug will cause the Xrootd daemon to log all headers and requests. Accepted values: trace, debug, info, warn, error, fatal, panic

Logging level of Xrootd's "Open File System" (ofs) subsystem. The OFS manages the file descriptor table and redirection/ error handling. Accepted values: trace, debug, info, warn, error, fatal, panic

Logging level of Xrootd's "Open Storage System" (oss) subsystem. The OSS manages the interaction with the underlying POSIX storage (open, read, write, close, etc). Accepted values: trace, debug, info, warn, error, fatal, panic

Logging level for the HTTP component of the cache. Increasing to debug will cause the Xrootd daemon to log all headers and requests. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace level of XRootD's Open File System. This component cares about files and directories from the administrative perspective. This component is build on top of the Open Storage System component, which deals with things like file creation and reads and writes for files and directories. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace level of XRootD Proxy File Cache (XCache), the caching mechanism used by XRootD. This component entails information for caches/caching within XRootD. This component instantiates its own Open Storage System (OSS) to write local files to. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace level of XRootD Proxy System Service. Variables this component reports include: number of remotes file opens, number of opens that failed, number of remote file closes, and number of closes that failed. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace level of scitokens debug output within XRootD configuration. This entails token management and security credentials within XRootD. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace level of the eXtended Request Daemon within XRootD, another main XRootD executable. This reports information the XRootD protocol and works with cms. Accepted values: trace, debug, info, warn, error, fatal, panic

Trace options for XRootD debug output within XRootD configuration. This prefix is reserved for the xroot protocol, which is the component that sits on sockets and talks to clients as they query file-system info, open files, and read data. This is the protocol for XRootD (like http) and handles connections and requests. Accepted values: trace, debug, info, warn, error, fatal, panic

A URL pointing to the federation's metadata discovery host. NOTE: this does not work if the url contains a path!

A URL for the top level OSG Topology location (a legacy integration). This URL is needed to retrieve authorization file information.

A URL containing namespace information for origins and caches configured via the OSG Topology application (a legacy integration). The URL should point to the hosted namespace.json.

A URL for determining OSG topology server downtime information. The result of querying this URL is an XML file containing downtime information.

The frequency, in minutes, that topology should be reloaded.

A timeout indicating when a "stopped transfer" event should be triggered.

A duration indicating the ramp up period for a slow transfer.

A duration indicating the sliding window over which to consider transfer speeds for slow transfers.

A bool indicating whether the client's HTTP proxy should be disabled.

An integer indicating the number of file transfer tasks that should be executed in parallel.

A bool indicating whether the a proxy fallback should be used by the client.

The minimum speed (in bytes per second) allowed for a client download before an error is thrown.

The maximum speed allowed for a client to download a given file (enforced via rate limits). This is not intended for use by production clients but rather for unit tests; 0 disables the rate limit.

A filepath to the intended location of the origin's database.

The origin's configured URL, as reported to XRootD. This is the file transfer endpoint for the origin.

The TCP port to be used by the origin service for serving files. If set to 0, then a random open port will be used.

A list describing the origin's exports. Each item in the list describes a single namespace the origin exports:

• StoragePrefix: The relevant path from the object store, e.g. for posix /my/dir

• FederationPrefix: The namespace prefix that data from StoragePrefix is made available under within the federation

• Capabilities: A list of the capabilities the origin is willing to support for the given export. Capabilities include: ["Reads", "PublicReads", "Writes", "Listings", "DirectReads"] where each of these has the same effect as the corresponding "Origin.Enable*" configuration, except scoped to the given export. If "PublicReads" is included, "Reads" is inferred.

• SentinelLocation: A filename under StoragePrefix path for Pelican to check the storage directory exists and is correctly mounted. The value must be a file and contain no directory. Leave it empty to skip the check.

  You should always choose a distinct name for SentinelLocation. It should not be reused for other servers. If running in a containerized environment it should not be the name of the underlying physical host as that may change and lead to confusion. You need to manually create a file under path to StoragePrefix with the same name as SentinelLocation.

  Note that this parameter is only available for the POSIX backend.

  Example:

  Origin.Exports
    - StoragePrefix: /home/foo/bar
      FederationPrefix: /demo/project
      Capabilities: ["Reads", "PublicReads", "Writes", "Listings", "DirectReads"]
      SentinelLocation: demoproject_origin_A

If Origin.StorageType == "s3", the following additional fields are available:

• S3Bucket: [OPTIONAL] See Origin.S3Bucket for details
• S3AccessKeyfile: [OPTIONAL] See Origin.S3AccessKeyfile for details
• S3SecretKeyfile: [OPTIONAL] See Origin.S3SecretKeyfile for details

If Origin.StorageType == "globus", the following additional fields are available:

• GlobusCollectionID: [REQUIRED] See Origin.GlobusCollectionID for details
• GlobusCollectionName: [OPTIONAL] See Origin.GlobusCollectionName for details

The type of storage underpinning the origin. Currently supported types are "posix", "https", "s3", "globus", and "xroot".

The namespace prefix of the origin's contents within the federation.

NOTE: This config option is incompatible with multiple exports defined via Origin.Exports and is ignored when the origin exports multiple prefixes.

A string indicating the path to the volume exported by an origin's underlying storage. For example, if the origin has a StorageType of "posix", this constitutes the path on disk exported by the origin for the federation. If the origin has a StorageType of "s3", this value is not currently used.

NOTE: This config option is incompatible with multiple exports defined via Origin.Exports and is ignored when the origin exports multiple prefixes.

A list of docker-style export volumes for the origin. Each item in the list describes a single volume the origin exports. This configuration is meant mostly to be used by passing the -v flag from the command line. Paths exported with this configuration will inherit the origin's abilities, so individual export configurations are not possible.

A boolean indicating whether the origin permits reads without valid authorization. When false, reads from the origin will require a properly-scoped authorization token signed by the origin's issuer.

NOTE: This config option is meant to configure an origin's capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration.

A boolean indicating whether the origin permits any reads. When false, the origin may still allow writes.

NOTE: This config option is meant to configure an origin's capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration.

A boolean indicating whether the origin permits writes. All writes require authorization.

NOTE: This config option is meant to configure an origin's capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration.

A boolean indicating whether the origin permits object listings. When true, clients can list the contents of the origin.

NOTE: This config option is meant to configure an origin's capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration.

A boolean indicating whether the origin permits direct reads. When true, the origin indicates that it is willing to interact directly with clients. When false, the origin is indicating it is only willing to interact with clients via a cache service.

NOTE: This config option is meant to configure an origin's capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration.

[Deprecated] Origin.ExportVolume is being deprecated and will be removed in a future release. It is replaced by Origin.ExportVolumes. A path to the volume exported by an origin.

A directory where temporary configurations will be stored for the XRootD daemon started by the origin.

For non-root servers, if $XDG_RUNTIME_DIR is not set, a temporary directory will be created (and removed on shutdown).

[Deprecated] Origin.NamespacePrefix is being deprecated and will be removed in a future release. It's configuration is being replaced by either Origin.Exports.FederationPrefix or by Origin.FederationPrefix. Note that Origin.FederationPrefix is incompatible with multiple exports and requires that the origin exports only a single path.

The filepath prefix at which an origin's contents are made globally available, eg /pelican/PUBLIC.

[Deprecated] Origin.EnableWrite is being deprecated and will be removed in a future release. It is replaced by Origin.EnableWrites.

A boolean indicating if an origin allows write access.

[Deprecated] Origin.EnableFallbackRead is being deprecated and will be removed in a future release. It is replaced by Origin.EnableDirectReads.

Set to true if the origin permits clients to directly read from it when no cache service is available.

A bool indicating whether an origin is "multiuser", ie whether the underlying XRootD instance must be configured in multi user mode.

A bool indicating whether the origin should enable the cmsd daemon.

A bool indicating whether the origin allows clients to authenticate using macaroons.

A bool indicating whether the director should send file transfer tests to the origin.

If Origin.StorageType is set to values other than POSIX, this parameter is set to false.

A bool indicating whether the origin should perform self health checks.

If Origin.StorageType is set to values other than POSIX, this parameter is set to false.

The interval of which the origin starts a new file transfer test to itself.

Indicate whether the origin should enable its web UI.

Indicate whether the origin should allow users to login to the admin website via OAuth2/OIDC with third-party authentication providers such as CILogon.

If set to true, it is recommended that you also set Server.UIAdminUsers to a list of users to give admin privilege. This is because origin admin website doesn't have a public, non-admin view, and an empty AdminUsers list will lead to "permission denied" error for all users logged into origin admin website via OAuth.

Indicate whether the origin should utilize the broker service to avoid the need for incoming connections.

Enable the built-in issuer daemon for the origin.

This parameter is used to configure XRootD's SciTokens authorization plugin.

Any restrictions on the paths that the issuer can authorize inside their namespace. This is meant to be a mechanism to help with transitions, where the underlying storage is setup such that an issuer's namespace contains directories that should not be managed by the issuer.

This parameter is used to configure XRootD's SciTokens authorization plugin.

If set to true, the contents of the token's sub claim will be copied into the XRootD username. When Origin.Multiuser is also set to true, this will allow XRootD to read and write files using the Unix username specified in the token.

This parameter is used to configure XRootD's SciTokens authorization plugin.

If set, then all authorized operations will be performed under the provided username when interacting with the file system. This is useful when all files owned by an issuer should be mapped to a particular Unix user account.

This parameter is used to configure XRootD's SciTokens authorization plugin.

If set, then the provided claim will be used to determine the XRootD username, and it will override the Origin.ScitokensMapSubject and Origin.ScitokensDefaultUser parameters.

This parameter is used to configure XRootD's SciTokens authorization plugin.

If set, then the referenced file is parsed as a JSON object and the specified mappings are applied to the username inside the XRootD framework. See the XrdSciTokens documentation for more information on the mapfile's format.

The directory prefix for the XRootD origin configuration files.

Enable X.509 / VOMS-based authentication. This allows HTTP clients to present X.509 client credentials in order to authenticate. The configuration of the authorization for these clients must be done by the admin; Pelican does not support automatic VOMS authorization configuration.

[Deprecated] Origin.EnableDirListing is being deprecated and will be removed in a future release. It is replaced by Origin.EnableListings.

Allows the origin to enable collection listings. Needs to be enabled for recursive downloads to work properly and for directories to be visible.

[Deprecated] Origin.Mode is being deprecated and will be removed in a future release. It is replaced by Origin.StorageType.

The backend mode to be used by an origin. Current values that can be selected from are either "posix" or "s3".

[Deprecated] Origin.S3ServiceName was previously used in part to determine an export's FederationPrefix, but upstream changes no longer rely on this value. As of Pelican 7.7.0, setting this value no longer has any effect. AWSv4 signatures used by S3 servers to handle authentication now hardcode "s3" as their service name.

When constructing signed URLs for S3, this value is used as a part of the signature. It is almost always "s3". For more information about S3 service names, see https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

Objects in S3 are associated with a "region", which is specifically a part of AWS's infrastructure. Often, S3 endpoints that are not provided by Amazon use "us-east-1" as their region. This value is used when constructing signed URLs for getting authenticated objects from a bucket.

For more information about how Amazon uses regions, see https://docs.aws.amazon.com/general/latest/gr/s3.html

This value is REQUIRED for S3 origins.

Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use Origin.Exports

Objects in S3 are stored in "buckets", which have unique names at each S3 service URL (ie the URL that provides access to your objects). Setting a bucket restricts the origin to only serving objects from that bucket.

However, if the origin is meant to export all of the buckets associated with a given service URL, this value can be left unset IF all of those buckets are public and the origin is using path-style URLS. When this is the case, objects can be fetched from the origin at the path /federation/prefix/bucket-name/object-name.

The URL that provides API access your objects. When the S3 instance is hosted by Amazon, this is often "https://s3.us-east-1.amazonaws.com".

This value is REQUIRED for S3 origins.

Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use Origin.Exports

A path to a file containing an S3 access keyfile (also sometimes called an API key) for authenticated buckets when an origin is run in S3 mode.

This value is OPTIONAL for S3 origins, and only applies when an exported bucket requires authentication. It should not be used if the bucket is public or if the origin is meant to export all public buckets from the S3 service URL.

Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use Origin.Exports

A path to a file containing an S3 secret keyfile for authenticated buckets when an origin is run in S3 mode.

This value is OPTIONAL for S3 origins, and only applies when an exported bucket requires authentication. It should not be used if the bucket is public or if the origin is meant to export all public buckets from the S3 service URL.

The style of S3 urls used by the service URL host. This can be either "path" if objects are fetched at <service-url>/<bucket>/<object> or "virtual" if objects are fetched at <bucket>.<service-url>/<object>.

This value is REQUIRED for S3 origins, but defaults to "path" if not set.

If Origin.StorageType is set to https, the service URL is used as the base for requests to the backend. To generate the request, the Origin.FederationPrefix is removed from the object name, then the result is joined with the service URL and storage prefix. For example, if one sets Origin.HTTPServiceUrl=https://example.com, Origin.StoragePrefix=/testfiles and Origin.FederationPrefix=/foo, then a request for an object named /foo/bar will generate a request to https://example.com/testfiles/bar.

When set, all requests from the origin to the http backend will include the contents of the file as a bearer token in the Authorization header.

If the origin backend is configured with the globus storage type, any value set here will be overridden with the filepath to the first file ending in .tok found in the $(Origin.GlobusConfigLocation)/tokens directory

When the origin is configured to export another XRootD storage backend by setting Origin.StorageType = xroot, the XRootServiceUrl is used as the base for root protocol requests and should point at the upstream XRootD server.

Note: This value is only for setting up an origin that exports one Globus collection. For multiple exports, use Origin.Exports

Required if Origin.StorageType == "globus" AND Origin.Exports is not set.

Globus stores objects in Collections. The unique identifier of a Collection is the Collection UUID. You can find the Collection UUID at the bottom of your Collection's overview page.

Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use Origin.Exports

An optional human-readable name to describe the Collection. This should set to the "Display Name" of your Collection in Globus. It is recommended to set this name; otherwise the UUID will be used as the Collection name.

Required if Origin.StorageType == "globus" and OIDC.Issuer is not Globus

A filepath to the file containing the Globus ClientID. You need to create a new project and register a new confidential OAuth client: https://app.globus.org/settings/developers/registration/confidential_client/select-project Once registered, in the client page, find "Client UUID" and copy it to a file.

If the server uses Globus as the OIDC authentication provider, Pelican will use OIDC configuration for Globus storage access.

Required if Origin.StorageType == "globus" and OIDC.Issuer is not Globus

A filepath to the file containing the Globus ClientID. You need to create a new project and register a new confidential OAuth client following the instruction in Origin.GlobusClientIDFile. In the client page, you need to "Add Client Secret" and copy the secret to a file.

If the server uses Globus as the OIDC authentication provider, Pelican will use OIDC configuration for Globus storage access.

A filepath to the folder containing Globus config and access tokens

The directory for the runtime files of the local cache.

The directory for the location of the cache data files - this is where the actual data in the cache is stored for the local cache.

The location of the socket used for client communication for the local cache.

The maximum size of the local cache. If not set, it is assumed the entire device can be used. This parameter can be provided with units (e.g., 20GB, 150MB); if no unit is provided, then it is assumed to be in bytes.

A percentage value where the cache cleanup routines will triggered. Once the cache usage of completed files hits the high water mark, files will be deleted until the usage hits the low water mark.

A percentage value where the cache cleanup routines will complete. Once the cache usage of completed files hits the high water mark, files will be deleted until the usage hits the low water mark.

An absolute path to the directory where xrootd will create its default namespace, meta, and data directories. For example, setting Cache.StorageLocation=/run/pelican/cache without specifying further Cache.DataLocations or Cache.MetaLocations values will result in the cache creating a directory structure like:

.
└── /run/pelican/cache/
    ├── data/
    │   ├── 00 # hexadecimal name values
    │   ├── 01
    │   ├── ...
    │   └── FF
    ├── meta/
    │   ├── 00 # hexadecimal name values
    │   ├── 01
    │   ├── ...
    │   └── FF
    └── namespace/
        ├── namespace1/
        │   ├── foo1.txt --> /run/pelican/cache/data/00
        │   └── foo2.txt --> /run/pelican/cache/data/01
        └── namespace2/
            └── bar.txt --> /run/pelican/cache/data/FF

In this setup, actual data files live at /run/pelican/cache/data and are given hexadecimal names, while references (symbolic links) to those files are stored in /run/pelican/cache/namespace. The meta directory is used for object metadata. Object requests to XRootD will be served from the namespace directories, and resolve the underlying object through these symbolic links.

We recommend tying the Cache.StorageLocation to a fast storage device, such as an SSD, to ensure optimal cache performance. If this directory does not already exist, it will be created by Pelican.

WARNING: The default value of /var/run/pelican should never be used for production caches, as this directory is typically cleared on system restarts, and may interfere with system services if it becomes full. Running a cache with the default value set will generate a warning at cache startup.

A cache's namespace directory is used to duplicate/recreate the federation's namespace structure, and stores symbolic links from object names to the actual data files (see Cache.StorageLocation for extra information). For example, requesting /foo/bar.txt from a cache will check for the existence of a symbolic link at ${Cache.NamespaceLocation}/foo/bar.txt, and if it exists, the cache will serve the data file at the location the symbolic link points to.

If this directory does not already exist, it will be created by Pelican.

WARNING: It's important that any values for Cache.DataLocations and Cache.MetaLocations are NOT subdirectories of Cache.NamespaceLocation, as this will make the raw data/meta files accessible through the cache's namespace structure, which is undefined behavior.

A list of absolute filesystem paths/directories where the cache's object data will be stored. This list of directories can be used to string together multiple storage devices to increase the cache's storage capacity, as long as each of the directories is accessible by the cache service. For example, setting Cache.DataLocations=["/mnt/cache1", "/mnt/cache2"] will result in splitting cache data between two mounted drives, /mnt/cache1 and /mnt/cache2. As such, these drives should be fast storage devices, such as SSDs.

For more information, see the xrootd oss documentation for the oss.space directive as well as the xrootd pfc documentation for the pfc.spaces directive.

If this directory does not already exist, it will be created by Pelican.

WARNING: It's important that any values for Cache.DataLocations are NOT subdirectories of Cache.NamespaceLocation, as this will make the raw data files accessible through the cache's namespace structure, which is undefined behavior.

A list of absolute filesystem paths/directories where the cache's object metadata will be stored. Values in this list may point to separate drives as long as they're accessible by the cache service. For example, setting Cache.MetaLocations=["/mnt/meta1", "/mnt/meta2"] will result in splitting cache metadata between two the mounted drives. As such, these drives should be fast storage devices, such as SSDs.

For more information, see the xrootd oss documentation for the oss.space directive as well as the xrootd pfc documentation for the pfc.spaces directive.

If this directory does not already exist, it will be created by Pelican.

WARNING: It's important that any values for Cache.MetaLocations are NOT subdirectories of Cache.NamespaceLocation, as this will make the raw metadata files accessible through the cache's namespace structure, which is undefined behavior.

[Deprecated] Cache.LocalRoot is deprecated and replaced by Cache.StorageLocation.

[Deprecated] Cache.DataLocation is being deprecated and will be removed in a future release. It is replaced by Cache.StorageLocation

A path that's relative to the Cache.NamespaceLocation where the cache will expose its contents. This path can be used to control which namespaces are available through the cache. For example, setting Cache.ExportLocation: /foo will only expose the /foo namespace to clients.

A directory where temporary configurations will be stored for the XRootD daemon started by the cache.

For non-root servers, if $XDG_RUNTIME_DIR is not set, a temporary directory will be created (and removed on shutdown).

A filename under Cache.DataLocation path for Pelican to check the storage directory exists and is correctly mounted. The value must be a file and contain no directory. Leave it empty to skip the check.

You should always choose a distinct name for Cache.SentinelLocation. It should not be reused for other servers. If running in a containerized environment it should not be the name of the underlying physical host as that may change and lead to confusion. You need to manually create a file under path to Cache.DataLocation with the same name as Cache.SentinelLocation.

The directory prefix for the XRootD cache configuration files.

The cache's configured URL, as reported to XRootD. This is the file transfer endpoint for the cache.

The TCP port the cache service should use. If set to 0, then a random open port will be used.

A value of cache disk usage that stops the purging of cached files.

The value should be either a percentage integer of total available disk space (default is 90), or a number suffixed by k, m, g, or t. In which case, they must be absolute sizes in k (kilo-), m (mega-), g (giga-), or t (tera-) bytes, respectively.

A value of cache disk usage that triggers the purging of cached files.

The value should be either a percentage integer of total available disk space (default is 95), or a number suffixed by k, m, g, or t. In which case, they must be absolute sizes in k (kilo-), m (mega-), g (giga-), or t (tera-) bytes, respectively.

Enable X.509 / VOMS-based authentication for the cache. This allows HTTP clients to present X.509 client credentials in order to authenticate. The configuration of the authorization for these clients must be done by the admin; Pelican does not support automatic VOMS authorization configuration.

This value represents the maximum number of connections to a cache for XRootD throttling. When this value is set, it enables the XRootD throttling plugin and will set the maximum number of connections to this value.

LotMan is a library that provides management of storage space in the cache.

A list of namespaces the cache is allowed to pull from. If the list is empty or this option is unset, it's assumed that the cache is allowed to access any namespace that's advertised to the director. Otherwise, it will only be allowed to access the listed namespaces.

A bool indicating whether the cache should perform self health checks.

The interval of which the cache starts a new file transfer test to itself.

Indicate whether the cache should allow users to login to the admin website via OAuth2/OIDC with third-party authentication providers such as CILogon.

If set to true, it is recommended that you also set Server.UIAdminUsers to a list of users to give admin privilege. This is because cache admin website doesn't have a public, non-admin view, and an empty AdminUsers list will lead to "permission denied" error for all users logged into cache admin website via OAuth.

The number of 128 kilobyte blocks the cache will read ahead when recieving requests. This will put the data in the cache potentially before it is needed and reduce the latency to the client when a request is made. However, it can also cause many extra requests to an origin and potentially overload it when unnessesary. As such, this is turned off by default.

The default value of the cache operation timeout if one is not specified by the client.

Newer clients should always specify a timeout; changing this default is rarely necessary.

A filepath to the intended location of the director's database, where server downtime info is stored.

The default response type of a redirect for a director instance. Can be either "cache" or "origin". If a director is hosted at https://director.com, then a GET request to https://director.com/foo/bar.txt will either redirect to the nearest cache for namespace /foo if Director.DefaultResponse is set to "cache" or to the origin for /foo if it is set to "origin".

In the "origin" response, the director returns a list of origins that can serve the object. If Director.CachesPullFromCaches is set to true (default is false), the director then appends a list of caches that can serve the object to the original response.

A list of virtual hostnames for the director. If a request is sent by the client to one of these hostnames, the director assumes it should respond with a redirect to a cache.

If present, the hostname is taken from the X-Forwarded-Host header in the request. Otherwise, Host is used.

When the director recieves a client request that needs to be redirected to a cache, it will use this method to determine the ordering of the caches. The default method is "distance", which sorts caches by their spherical distance from the client.

Available methods include:

• "distance": Sorts caches by their spherical distance from the client.
• "distanceAndLoad": Sorts caches according to both their distance and a calculated load. This is currently a placeholder, and returns the same ordering as "distance".
• "random": Sorts caches randomly.
• "adaptive": Sorts caches according to stochastically-generated weights that consider a combination of factors, including a cache's distance from the client, its IO load, and whether the cache already has the requested object. See details at https://github.com/PelicanPlatform/pelican/discussions/1198. Note that if Director.CheckCachePresence is set to false, then the adaptive algorithm cannot use the cache locality information.

A list of virtual hostnames for the director. If a request is sent by the client to one of these hostnames, the director assumes it should respond with a redirect to an origin.

If present, the hostname is taken from the X-Forwarded-Host header in the request. Otherwise, Host is used.

A filepath to a MaxMind API key. The director service uses the MaxMind GeoLite City database (available here) to determine which cache is nearest to a client's IP address. The database, if not already found, will be downloaded automatically when a director is served and a valid key is present.

A filepath to the intended location of the MaxMind GeoLite City database. This option can be used either to load an existing database, or to configure the preferred download location if Pelican has a MaxMind API key.

A positive integer indicating minimum number of origin's responses required for a stat call.

A positive integer indicating maximum number of origin's responses required for a stat call. stat call will cancel the rest of the ongoing query if max response is hit.

Before redirecting a cache (or, for direct reads or writes, a client) to an origin, query the origin to see if the object is present.

Enabling this option generates slightly more load on the origin; however, it provides improved error messages and allows a namespace to effectively be split across multiple origins.

Before redirecting a cache (or, for direct reads or writes, a client) to an origin, query the origin to see if the object is present.

Enabling this option generates slightly more load on the origin; however, it provides improved error messages and allows a namespace to effectively be split across multiple origins.

If Director.CheckOriginPresence is enabled, the director will check for object presence at the origin before redirecting a client.

If this option is enabled and there's only one possible origin for the object, then the check will be skipped.

Enabling this option will reduce load on single-origin namespaces but clients will get less informative error messages.

Before redirecting a client to a cache, query the cache to see if the object is present at the cache.

Enabling this option improves the cache selection algorithm, allowing the director to prefer caches nearby the client with the object over caches without the object.

The timeout for a single stat request.

The maximum number of concurrent stat request to a single origin server. Additional requests are blocked until total requests for the origin is below limit. See golang.org/x/sync/errgroup for detail

The time to live (TTL) of director's internal cache to store origins and caches advertisement.

The interval of which director issues a new file transfer test to all the registered origins and caches.

Whether the director should also run the connection brokering service.

A list of server resource names that the Director should consider in downtime, preventing the Director from issuing redirects to them. Additional downtimes are aggregated from Topology (when the Director is served in OSDF mode), and the Web UI.

An Email address to receive issues and help requests for the federation the director is hosting. The values will be displayed on the director web interface if provided. We highly recommend director admin to fill out this field.

A URL where user can find support information. Can be your website, GitHub discussion, or third-party support portal for the federation the director is hosting. The values will be displayed on the director web interface if provided. We highly recommend director admin to fill out this field.

Indicate whether the director should allow users to login to the admin website via OAuth2/OIDC with third-party authentication providers such as CILogon.

If set to true, it is recommended that you also set Server.UIAdminUsers to a list of users to give admin privilege. This is because origin admin website doesn't have a public, non-admin view, and an empty AdminUsers list will lead to "permission denied" error for all users logged into origin admin website via OAuth.

A list of object prefixes where the origin uses X.509 client authentication.

If a cache requests an object starting with one of these prefixes, then it will be instructed by the director to use X.509 client authentication if available.

This setting allows for compatibility with specific legacy OSDF origins and is not needed for new origins.

If Director.CheckCachePresence is enabled, the director will check with remote cache to see if the object is present before redirecting a client.

This parameter controls how long the director will cache the result of the lookup. Longer values will reduce the load generated on the caches but may reduce the accuracy of the result (as the contents of the cache will change over time).

If Director.CheckCachePresence is enabled, the director will check with remote cache to see if the object is present before redirecting a client.

This parameter controls how many responses, per-service, will be cached.

A filepath to the intended location of the namespace registry's database.

Specifies whether namespaces requesting registration must possess a key matching any already-registered super/sub namespaces. For example, if true and a namespace /foo/bar is already registered, then registration of /foo or /foo/bar/baz can only be done using keys registered to /foo/bar.

[Deprecated] Registry.AdminUsers is deprecated and will be removed in the future releases. Please migrate to use Server.UIAdminUsers instead.

A string slice of "subject" claim of users to give admin permission for registry UI.

The "subject" claim should be the "CILogon User Identifier" from CILogon user page: https://cilogon.org/

A array of institution objects available to register. Users can only select from this list when they register a new namespace. Each object has name and id field where name is a human-readable name for the institution and id is a unique identifier for the institution. For Pelican running in OSDF alias, the id will be OSG ID.

For example:

  - name: University of Wisconsin - Madison
    id: https://osg-htc.org/iid/01y2jtd41

Note that this value will take precedence over Registry.InstitutionsUrl if both are set.

An array of objects specifying additional fields when registering namespaces.

The schema of the object is as follows:

  - name: department_name
    type: enum
    required: true
    options:
      - name: Math
        id: math
      - name: Computer Science
        id: cs
    optionsUrl: https://example.com/options
    description: The department of the organization that holds this namespace

Note the following requirements:

• name must be snake case with underline connecting words, i.e. department_name. The name displayed in the registration table will be converted from this field into a human-readble, space-separated name with the first letter(s) capitalized. i.e. department_name -> Department Name

• type must be one of string, int, bool, datetime (Unix time in seconds), or enum.

• options must be a non-empty yaml array for field with type enum. optionsUrl will be ignored if options is set.

  Example:

  options:
    - name: "Option A"
      id: "optionA"

• description will show up in the web UI as helper text to help user understand the field

• optionsUrl is a URL to provide a list of options for enum type field. The URL should respond to an anonymous GET request and return JSON response in the same format as the options field above

A url to get a list of available institutions for users to register their namespaces to. The url must accept a GET request with 200 response in JSON/YAML content with the following format:

JSON:

  [
    {
      "name": "University of Wisconsin - Madison",
      "id": " https://osg-htc.org/iid/01y2jtd41"
    }
  ]

YAML:

  - name: University of Wisconsin - Madison
    id: " https://osg-htc.org/iid/01y2jtd41"

Where the id field will be stored in registry database and must be unique, and name field will be displayed in UI as the option.

Note that Pelican will cache the response of the url in a TTL cache with default refresh time of 15 minutes. Also note that `Registry.Institutions`` will take precedence over this value if both are set.

Number of minutes that the Registry.InstitutionsUrl will be reloaded into the TTL cache.

Only allow approved caches to join the federation and serve files. If set to true, caches can successfully self-register or registered via registry, but director won't direct traffic to the cache.

Only allow approved origins to join the federation and serve files. If set to true, origins can successfully self-register or registered via registry, but director won't direct traffic to the origin, nor would files on the origin show up in the federation.

A filepath to a file containing an X.509 host certificate to use for TLS authentication when running server components of Pelican.

If you override this filepath, you need to provide the matched-pair private key via Server.TLSKey and a Certificate Authority (CA) certificate via Server.TLSCACertificateFile.

A filepath to a file containing the full X.509 certificate chain, including the host certificate followed by any intermediate certificates, to use for TLS authentication when running server components of Pelican.

If you override this filepath, you need to provide the matched-pair private key via Server.TLSKey and a Certificate Authority (CA) certificate via Server.TLSCACertificateFile.

A filepath to the TLS Certificate Authority (CA) certificate file, to be used by XRootD and internal HTTP client requests.

Do not override this filepath unless you want to provide your TLS host certificate

A filepath to the directory used for storing TLS Certificate Authority (CA) certificate to be used by XRootD only.

This is exclusive with Server.TLSCACertificateFile for XRootD and this value takes priority over Server.TLSCACertificateFile.

The name of a file containing a private key corresponding to the TLSCACertificate. Used when running server components of Pelican.

The name of a file containing the private key corresponding to the host certificate in the TLSCertificateChain. Used when running server components of Pelican.

Indicate whether a server should enable its web UI.

The port number the Pelican web interface and internal web APIs will be bound to.

A string-encoded IP address that the Pelican web engine is configured to listen on.

A URL indicating the Pelican web interface and internal web APIs address as it appears externally.

Port number will be stripped if it's 443, from Server.WebPort or directly set through Server.ExternalWebUrl.

The server's hostname, by default it's os.Hostname().

The URL and port at which the server's issuer can be accessed.

The hostname at which the server's issuer can be accessed.

The port at which the server's issuer can be accessed.

A filepath indicating where the server's public JSON web keyset can be found.

A list of modules to enable when running pelican in pelican serve mode.

If the server's web UI has not yet been configured, this file will contain the activation code necessary to turn it on.

A filepath specifying where the server's web UI password file should be stored.

The filepath to the secret for encrypt/decrypt session data for Pelican web UI to initiate a session cookie.

This is used for sending redirect request for OAuth2 authentication follow. This is also used for CSRF auth key.

The duration of delay in origin/cache registration retry attempts if the initial registration call to registry was failed.

The maximum number of requests a user can be made under the same IP address per second against the login endpoint

A filepath to the file where web-based configuration changes are stored

A string slice of "subject" claim of users to give admin permission for the server admin website, who are authenticated through OAuth/OIDC.

The "subject" claim should be the "CILogon User Identifier" from CILogon user page: https://cilogon.org/

The amount of time the pelican server will wait for its components and services to startup. If the timeout is hit while waiting on a component, the server will shutdown.

A boolean to enable or disable the pprof endpoints for debugging.

Location of the system tomcat installation

Location of the scitoken server installation

Location of the QDL language and scripts install on the system

Contents of the issuer (iss) claim in the generated tokens

How users should authenticate with the issuer. Currently-supported values are:

• none (default): No authentication is performed. All requests are successful and assumed to be a user named nobody.
• OIDC: Use the server's OIDC configuration to authenticate with an external identity provider.

A list of claim-value pairs that indicate required values from the OIDC ID token to authenticate.

Example:

- claim: idp_name
  value: University of Wisconsin-Madison

Would only allow tokens with "idp_name": "University of Wisconsin-Madison" set to authenticate.

The claim in the OIDC ID token to be used as the "username" for the issuer.

Some OIDC issuers generate a username of the form user@domain (such as john.doe@gmail.com); when UserStripDomain is enabled, Pelican will strip the domain when determining the username.

For example, the OIDC identity john.doe@gmail.com would map to john.doe.

How the issuer should determine group information based on the authenticated identity. Valid values are:

• none (default): No group information should be used.
• file: Read groups from an external, JSON-formatted file. The file should contain a single JSON object with keys corresponding to the "user" name and the value a list of strings that are interpreted as the user's groups.
• oidc: Take group information from the identity token provided by the OIDC identity provider. Parses the value of the claim specified by Issuer.OIDCGroupClaim (defaults to "groups") as a list of groups. The value may either be a comma-separated string or an array of strings.

The claim in the OIDC ID token to be used as the group for the issuer. If the value is a string, it is assumed that a comma is used as a group delimiter; otherwise, an array of strings is assumed. Check the documentation of your OIDC provider to determine the appropriate claim name.

The location of a file containing group information. The file should contain a single JSON object with keys corresponding to the "user" name and the value a list of strings that are interpreted as the user's groups.

Group membership requirements. A request must be mapped to one of the groups in this list to successfully authenticate.

The authorizations that should be generated for an authenticated request. Value should be a list of authorized actions.

Each action is a key-value pair with the following keys defined:

• actions: A list of authorized actions. Valid string values are read, modify, and create.
• prefix: A prefix where the actions are authorized. If the prefix contains the substring $USER, the string is replaced with the authenticated username. If the prefix contains the substring $GROUP, then an authorization is emitted for each group authenticated.

For example, if the request is authenticated as user bbockelm with groups dept_a and dept_b, then the following configuration:

- actions: ["read", "create"]
  prefix: /projects/$GROUP
- actions: ["read", "modify"]
  prefix: /home/$USER

will result in the following authorizations:

• read /projects/dept_a
• create /projects/dept_a
• read /projects/dept_b
• create /projects/dept_b
• read /home/bbockelm
• modify /home/bbockelm

A filepath to a file containing an OIDC Client ID.

This is used by the namespace registry to allow OAuth2/OIDC login and authenticated namespace registration. By default, Pelican uses CILogon as the authentication provider. You need to first register an OIDC client at CILogon: https://cilogon.org/oauth2/register. If you'd like to use other authentication providers, you need to change other endpoint parameters under OIDC configuration to the endpoints of your provider, such as OIDC.AuthorizationEndpoint, OIDC.UserInfoEndpoint, etc.

OIDC.ClientIDFile is mutually exclusive with OIDC.ClientID. The value of OIDC.ClientID will override the value of OIDC.ClientIDFile if both are set.

This is a required parameter for the registry server. This is a required parameter for the origin/cache/director server if Origin.EnableOIDC/Cache.EnableOIDC/Director.EnableOIDC is set to true, respectively.

The OIDC ClientID to use for the server. This is mutually exclusive with OIDC.ClientIDFile. The value of OIDC.ClientID will override the value of OIDC.ClientIDFile if both are set.

This is a required parameter for the registry server. This is a required parameter for the origin/cache/director server if Origin.EnableOIDC/Cache.EnableOIDC/Director.EnableOIDC is set to true, respectively.

A filepath to a file containing an OIDC Client Secret. This is used by the namespace registry to establish OIDC information for authenticated registration.

This is a required parameter for the registry server. This is a required parameter for the origin/cache/director server if Origin.EnableOIDC/Cache.EnableOIDC/Director.EnableOIDC is set to true, respectively.

A URL describing an OIDC Device Auth Endpoint. This is used by the namespace registry to establish OIDC information for authenticated registration. The default value is set to the URL from CILogon.

A URL describing an OIDC Token Endpoint. This is used by the namespace registry to establish OIDC information for authenticated registration. The default value is set to the URL from CILogon.

A URL describing an OIDC User Info Endpoint. This is used by the namespace registry to establish OIDC information for authenticated registration. The default value is set to the URL from CILogon.

A URL containing the OIDC authorization endpoint. The default value is set to the URL from CILogon.

The URL of the OIDC issuer. If set, OIDC auto-discovery may be used to find other endpoints (token, user info, device auth). The URL should not contain a path unless your authentication server enables multi-tenant support.

If the OIDC auto-discovery failed, Pelican will fall back to use individual endpoints set in the configuration. For any unset endpoints, Pelican will use default values, which are from CILogon.

For CILogon, it's https://cilogon.org For Globus, it's https://auth.globus.org

The hostname for the OIDC client redirect URL that the OIDC provider will redirect to after the user is authenticated.

For development use only. Useful when developing in a container and you want to expose localhost instead of container hostname to your OAuth provider.

[Deprecated] Xrootd.Port is deprecated and will be removed in the future release. Please migrate to use Origin.Port or Cache.Port instead.

The port over which XRootD should be made available. This setting is deprecated; please use the Cache.Port or Origin.Port, as appropriate, for the server.

[Deprecated] Xrootd.RunLocation is deprecated and will be removed in a future release. Please migrate to use Cache.RunLocation or Origin.RunLocation instead.

A directory where temporary configurations will be stored for the XRootD daemon started by the origin or cache. For non-root servers, if $XDG_RUNTIME_DIR is not set, a temporary directory will be created (and removed on shutdown). This setting is deprecated; please use the Cache.RunLocation or Origin.RunLocation, as appropriate, for the server.

The absolute path to an XRootD configuration file for customized XRootD configuration. This should only be used by admins with experience in configuring XRootD directly.

Xrootd.ConfigFile will be used as the continuation of the Pelican generated XRootD configuration, via the continue directive. Existing configuration values may be overwritten or appended.

Refer to Configuration File Continuation for details

Origins may be indexed by web search engines; to control the behavior of search engines, one may provide local policy via a robots.txt file.

If this file is not present, it will be auto-created with a default policy of blocking all indexing.

The location of a file configuring XRootD's token-based authorization subsystem. This file allows arbitrary changes to the authorization configuration and will be merged with any auto-generated configuration; it's recommended for use by experts only.

The mount path for an instance of XRootD.

The filepath to a Macaroons key for setting up authorization in XRootD.

The filepath to an auth file for setting up authorization in XRootD.

The interval used by XRootD (cache/origin) for refreshing Authfiles. This affects how often the server polls for upstream changes that might affect the authorization policy. For example, when applied to a cache, this affects how often origin permissions are polled for changes.

A URL pointing toward the XRootD instance's Manager Host.

The port at which the XRootD instance's Manager Host is available.

A URL pointing toward the XRootD instance's Summary Monitoring Host.

The port at which the XRootD instance's Summary Monitoring Host is available.

A URL pointing toward the XRootD instance's Detailed Monitoring Host.

The port at which the XRootD instance's Detailed Monitoring Host is available.

A URL pointing toward the XRootD instance's Local Monitoring Host.

The sitename, as configured for XRootD.

The maximum amount of time pelican will wait for the xrootd daemons to successfully start

A filepath where Prometheus should host its monitoring data.

The lower end of a range of monitoring ports for Prometheus configuration.

The lower end of a range of monitoring ports for Prometheus configuration.

A list of path-like prefixes, potentially containing a glob (wildcard character), indicating how the Prometheus-based monitoring should aggregate records when reporting. For example, if /foo/* is on the aggregate path list, then the monitoring data for a download of objects /foo/bar and /foo/baz will be aggregated into a single series, /foo.

The duration of which the tokens for various Prometheus endpoints expire.

This includes tokens for director's Prometheus origin discovery endpoint, director's origin scraper, and server's self-scraper.

The interval of which the token issuer for various Prometheus endpoints refreshes the token for monitoring.

The tokens that are affected by this config are the same as the one in Monitoring.TokenExpiresIn. This value must be less than Monitoring.TokenExpiresIn.

If authorization (Bearer token) is required for accessing /metrics endpoint.

If authorization (Bearer token or cookie) is required for accessing /prometheus/query endpoint.

The duration of which Prometheus should retain the monitoring data.

Enable XRootD monitoring shoveler: https://github.com/opensciencegrid/xrootd-monitoring-shoveler.

Select which protocol to use in order to connect to the MQ. Options are amqp, stomp.

For amqp, the following configurations are required:

• URL: amqps://username:password@example.com/vhost
• Topic: mytopic
• AMQPExchange: shoveled-xrd
• AMQPTokenLocation: /etc/pelican/xrootd-monitoring-shoveler-token

For stomp, the following configurations are required:

• URL: messagebroker.org:port
• Topic: mytopic
• StompUsername: username
• StompPassword: password
• StompCert: path/to/cert/file
• StompCertKey: path/to/certkey/file

For amqp and stomp.

The URL to connect to the shoveler.

For amqp and stomp.

The topic of the messages. For stomp, it defaults to xrootd.shoveler.

For amqp only.

The exchange to shovel messages.

For amqp only.

A filepath to the location of the JWT used for authenticating amqp connection.

For stomp only.

Username for authentication.

For stomp only.

Password for authentication.

For stomp only.

A filepath to the location of the TLS certifcate.

For stomp only.

A filepath to the location of the private key associated with the certificate.

The lower end of a range of Shoveler ports for Shoveler to set up UDP server.

The lower end of a range of Shoveler ports for Shoveler to set up UDP server.

A list of IP:Port destinations to forward XRootD monitoring packet to.

Whether to verify the header of the packet matches XRootD's monitoring packet format.

Directory to store overflow of queue onto disk. The queue keeps 100 messages in memory. If the shoveler is disconnected from the message bus, it will store messages over the 100 in memory onto disk into this directory. Once the connection has been re-established the queue will be emptied. The queue on disk is persistent between restarts, so a persistent directory should be used.

IP Mapping for remote IP addresses in forwarding to the destinations. You may either pass one IP address to map all messages to the configured origin, or a list of key-value pairs for one-to-one mapping.

One-to-all mapping:

IPMapping:
  - All: "172.0.0.4"

If a packet comes in with the private ip address of 192.168.0.4, the packet origin will be changed to 172.0.0.4 The port is always preserved.

One-to-one mapping:

IPMapping:
  - Source: "192.168.0.5"
    Dest: "172.0.0.5"
  - Source: "192.168.0.6"
    Dest: "129.93.10.7"

The specified token for pelican plugin staging.

Flag to specify HTCondor hook behavior.

Prefix corresponding to the local mount point of the origin.

Prefix corresponding to the local origin.

Prefix corresponding to the shadow origin.

The prefix indicating where LotMan should store its lot database. For the provided path, the database will be stored at <path>/.lot/lotman_cpp.sqlite.

The location of the system's installed LotMan library (libLotMan.so). When unset, the system will attempt to find Lotman at these fallback paths:

• /usr/lib64/libLotMan.so
• /usr/local/lib64/libLotMan.so
• /opt/local/lib64/libLotMan.so

Whether Lotman should enable its CRUD web endpoints. If true, administrators with an appropriately-signed token can interface with Lotman via HTTP. Otherwise, lots are only configurable via the Pelican configuration file at the cache.

A list of named Lotman purge policy definitions that may be enabled by the cache administrator through setting the Lotman.EnabledPolicy configuration. Each policy definition is an object with the following fields:

• PolicyName: The name of the policy. This is used to identify the policy in the Lotman.EnabledPolicy configuration.
• PurgeOrder: An ordered list of strings indicating the order in which lots should be purged. The strings should be one of the following:
  • del: Purge lots that have passed their deletion time.
  • exp: Purge lots that have passed their expiration time.
  • opp: Purge lots that have passed their opportunistic storage quota.
  • ded: Purge lots that have passed their dedicated storage quota.
• DiscoverPrefixes: A boolean indicating whether Lotman should automatically discover prefixes from the Director. If true, Lotman will attempt to create lots for all discovered federation prefixes. Locally-defined lots will take precedence over discovered lots if the two have the same name.
• MergeLocalWithDiscovered: A boolean indicating whether Lotman should merge locally-defined lot configurations with discovered namespaces. Most Lot configuration fields will take precedence from local configuration, but the Paths and Parents fields are additive.
• DivideUnallocated: A boolean indicating whether Lotman should attempt to make intelligent decisions regarding management policy attributes for lots that have not provided explicit values. These decisions are based on the cache's total storage capacity and the number of lots that have been explicitly configured, and are intended to maximize potential cache utilization. This should be set to "true" in most cases.
• Lots: A list of lot objects, each of which describes a "lot". Every lot can be defined with the following:
  • LotName: REQUIRED. The name of the lot. This is used to identify the lot in the LotMan database.
  • Owner: REQUIRED. A string identifying the owner of the lot's data (as opposed to someone who can modify the lot itself). The Owner field should generally be set to the issue for the lot's namespace path. For example, if the lot tracks namespace /foo/bar, the owner might be set to https://registry.com/api/v1.0/registry/foo/bar.
  • Paths: OPTIONAL. A list of path objects, each of which describes a path that should be managed by the lot.
    • Path: REQUIRED. The path to be managed by the lot.
    • Recursive: REQUIRED. A boolean indicating whether the path should be managed recursively. If true, the lot will manage all files and directories under the specified path.
  • ManagementPolicyAttrs: REQUIRED. The lot's management policy attributes object. This contains information about resources the lot should be allocated, and how it should be managed.
    • DedicatedGB: REQUIRED. The amount of storage, in GB, that should be dedicated to the lot. This means the lot can assume it always has access to this quantity.
    • OpportunisticGB: REQUIRED. The amount of opportunistic storage, in GB, the lot should have access to, when storage is available.
    • MaxNumObjects: REQUIRED. The maximum number of objects a lot is allowed to store.
    • CreationTime: REQUIRED. A unix timestamp indicating when the lot should begin being considered valid. Times in the future indicate the lot should not be considered valid until that time.
    • ExpirationTime: REQUIRED. A unix timestamp indicating when the lot expires. Lots may continue to function after expiration, but lot data owners should recognize the storage is at-will and may be pre-empted at any time.
    • DeletionTime: REQUIRED. A unix timestamp indicating when the lot and its associated data should be deleted.

For example, Lotman could be configured with the "my-policy" policy with the following:

Lotman:
  EnabledPolicy: "my-policy"
  PolicyDefinitions:
  - PolicyName: "my-policy"
    DivideUnallocated: true
    PurgeOrder: ["del", "exp", "opp", "ded"]
    DiscoverPrefixes: true
    MergeLocalWithDiscovered: true
    Lots:
    - LotName: "/foo/bar"
      Owner: "https://registry.com/api/v1.0/registry/foo/bar"
      Paths:
        Path: "/foo/bar"
        Recursive: true
      ManagementPolicyAttrs:
        DedicatedGB: 100
        OpportunisticGB: 100
        MaxNumObjects: 1000
        CreationTime: 1614556800
        ExpirationTime: 1614556800
        DeletionTime: 1614556800
    - LotName ... <additional lots>

Additional example configurations can be found in lotman/resources/lots-config.yaml For more information about LotMan configuration, see: https://github.com/pelicanplatform/lotman

The name of the policy to use with Lotman's purge logic. Policy names are defined in the Lotman.PolicyDefinitions list object. If unset, the "fairshare" policy is used, which evenly divides the cache's space amongst all top-level namespaces discoverable through the Director and purges data according in order of lots past deletion, lots past expiration, lots past opportunistic storage, and lots past dedicated storage. The "fairshare" policy is defined as follows:

Lotman:
  EnabledPolicy: "fairshare"
  DefaultLotExpirationLifetime: "2016h"
  DefaultLotDeletionLifetime: "4032h"
  PolicyDefinitions:
  - PolicyName: "fairshare"
    DivideUnallocated: true
    PurgeOrder: ["del", "exp", "opp", "ded"]
    DiscoverPrefixes: true
    MergeLocalWithDiscovered: false

The default expiration lifetime for lots that have not provided an explicit expiration time. Valid time units are:

• ns for nanoseconds
• us (or µs) for microseconds
• ms for milliseconds
• s for seconds
• m for minutes
• h for hours

This value is fed to Lotman as a unix timestamp in microseconds, adjusted from the current time.

The default deletion lifetime for lots that have not provided an explicit deletion time. Valid time units are:

• ns for nanoseconds
• us (or µs) for microseconds
• ms for milliseconds
• s for seconds
• m for minutes
• h for hours

This value is fed to Lotman as a unix timestamp in microseconds, adjusted from the current time.