Managing Administrator Credentials

Pelican services maintain multiple types of credentials:

• The admin password provides access to the server’s web interface.
• The namespace private key can create arbitrary new credentials (user tokens) for a portion of the federation’s namespace.

This page covers various topics in managing these credentials.

Pre-generate Credentials

Namespace Private Key (Issuer Key)

Note: The namespace private key is a powerful secret and should never be shared with others. With the private key, one can create user credentials that can arbitrarily read or write into your namespace.

Pelican services rely on a private/public key pair for creating (signing) and verifying the user credentials (tokens) used to read or write objects. When a new prefix in the data federation is created, an associated public key must be provided; the corresponding private key can generate any arbitrary permissions (including delete!) within the namespace. For more information about namespace registration, see details here.

When a Pelican origin or other service is started, the server automatically generates a new private key if one is not present. However, if someone is running the origin on your behalf, you will need to generate the keypair on your own and send the public portion to the origin operator. If you’re doing the deployment, it can also be convenient to generate the key-pair in advance and mount the private key file to the Pelican server via configuration variable, especially in a container environment.

To generate a private/public key pair, run the following command:

pelican key create

Resulting in public and private keys generated under the current working directory:

Successfully generated keys at:
Private key: /path/to/private-key.pem
Public Key: /path/to/issuer-pub.jwks

You can change the location of the generated keys by passing in arguments private-key and/or public-key:

pelican key create --private-key /tmp/foo --public-key /tmp/bar

You can also use this command to derive the public key of an existing private key:

pelican key create --private-key /path/to/existing/private/key

To configure a Pelican server to use the newly-generated private key, move it into the directory specified by the IssuerKeysDirectory parameter in the configuration yaml file:

IssuerKeysDirectory: /path/to/issuer-keys

This directory supports multiple private keys. A token or payload signature will be considered valid if it can be verified by any key found either within IssuerKeysDirectory or at IssuerKey.

If there is a remote operator for your origin, email them the public key portion. Never send anyone your private key. A sample public key looks like the following:

$ cat issuer-pub.jwks
{
	"keys": [
		{
			"alg": "ES256",
			"crv": "P-256",
			"kid": "N3Tks9H791dG7bdzjjlQzV-fwwCrnXd_mcd_2D5ZvYE",
			"kty": "EC",
			"x": "9IIkGEzyWywecBsIisrJsH-_XqMal-U25dthHewUC5w",
			"y": "h_1DUotM2qnLPPRia6hGXNKZdpi92VJ5u_NSrZ-pd0c"
		}
	]
}

A sample private key has the following PEM format:

$ cat private-key.pem
-----BEGIN PRIVATE KEY-----
M0G+AgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgzeomq6yJvek0fg2Z
g+NY6qklGz4ZjVlrRgTDT9hk5EShRANCAAT0giQYTPJbLB5wGwiKysmwf79eoxqX
5Tbl22Ed7BQLnIf9Q1KLTNqpyzz0YmuoRlzSmXaYvdlSebvzUq2fqXdH
-----END PRIVATE KEY-----

Web Interface Administrator Password

By default, Pelican’s administrator web interface requires a username/password to be setup to access. This password can be enabled by a one-time passcode the server writes into the log file. However, by pre-generating the credential, Pelican server administrators can skip the initial setup and automate installation.

The pelican generate command generates the htpasswd -formatted password file for storing the web interface administrator login password. To generate a password file, run:

pelican generate password

and Pelican will prompt you to enter and confirm the password:

Enter your password:
Confirm the password:
 
Successfully generated the admin password file at: /path/to/server-web-passwd

You can change the location of the generated htpasswd file by the -o or --output argument. You can also provide the file that contains your password through the -p or --password argument.

pelican generate password -o /path/to/generated-htpasswd-file -p /path/to/password

The generated htpasswd file can then be passed to the Pelican server via Server.UIPasswordFile configuration parameter.

Server:
  UIPasswordFile: /path/to/generated-htpasswd-file

Key Auto-Generation, Multiple Keys, and Key Replacement

Key Auto-Generation

Starting with Pelican v7.13.0, if no pre-generated private key is found at either IssuerKey or within IssuerKeysDirectory, Pelican will automatically generate a new private key file in PEM format inside IssuerKeysDirectory.

IssuerKeysDirectory/
├── pelican_generated_<TIMESTAMP_IN_MILLISECONDS>_<RANDOM_CHARS>.pem
└── ...

The Privacy-Enhanced Mail (PEM) format  is a standardized format for sending and storing cryptographic data.

Multiple Keys

Since v7.13.0, Pelican has supported multiple issuer keys. Server administrator can place additional .pem file into IssuerKeysDirectory, and any valid private key in this directory or at IssuerKey can be used to verify token or payload signatures.

When selecting a key for signing, Pelican automatically chooses the private key with the lexicographically smallest filename.

ConfigBase
├──IssuerKey
└──IssuerKeysDirectory/
	├── pelican_generated_<TIMESTAMP_IN_MILLISECONDS>_<RANDOM_CHARS>.pem
	├── <ANY_NAME>.pem
	└── ...

Pelican scans the above directory every 60 seconds to detect newly added or removed private keys. When retrieving public keys, Pelican dynamically generate them from the current set of private keys on-the-fly. If Server.IssuerJwks is specified, Pelican also includes public keys from that .jwks file.

Key Replacement

With support for Multiple Keys, Pelican allows the administrator to replace private keys at anytime. Follow these steps to replace an existing key:

1. Generate a new private/public keypair by using the pelican key create command, see details here.
2. Ensure that the Pelican server is running. Copy the new private key file to IssuerKeysDirectory.
3. Visit your federation’s Registry in your browser (you can find it in your Pelican server’s web interface) and download the latest registered public key set for your server under “Origins” section. Ensure the new public key is included. If it isn’t, wait for 60 seconds and retry.
4. Remove the old private key from IssuerKeysDirectory
5. Wait for 60s, then go back to the Registry, download the latest registered public key set again. Verify the old public key has been removed.

If you are using a Credmon  in HTCondor, increase the wait time in the above steps to the credmon’s configured token lifetime (15 minutes by default).

PKCS#11 Enhanced TLS Security

By default, the XRootD subprocess requires read access to the TLS private key file. This introduces a potential security risk: if an attacker could exploit XRootD to read arbitrary files (for example, through a symlink attack), they could potentially extract the private key. Starting with Pelican v7.23.0, Pelican can use PKCS#11  to protect the TLS private key used by the XRootD subprocess. When enabled, the TLS private key never leaves the Pelican process memory. XRootD performs TLS signing operations via a PKCS#11 interface without directly reading the key file.

How It Works

When PKCS#11 is enabled, Pelican starts an internal p11proxy server that implements the PKCS#11 protocol over a Unix socket. The architecture works as follows:

1. Pelican starts a PKCS#11 RPC server (via go-p11-kit) that holds the TLS private key in memory
2. XRootD is configured to use a PKCS#11 URI instead of a direct key file path
3. When XRootD needs to perform TLS signing, OpenSSL calls out to the PKCS#11 provider/engine
4. The p11-kit-client library forwards the signing request over a Unix socket to Pelican
5. Pelican signs the request and returns only the signature—never the key itself

Prerequisites

Before enabling PKCS#11, you must install the required system packages. The packages differ based on your OpenSSL version:

If you are using the official Pelican Origin container image, these OpenSSL 3.x packages are already included.

For OpenSSL 3.x (e.g., AlmaLinux 9, Fedora 37+, Ubuntu 22.04+):

dnf install -y pkcs11-provider p11-kit-server

For OpenSSL 1.x (e.g., CentOS 7, RHEL 8):

dnf install -y openssl-pkcs11 p11-kit-server

On Debian/Ubuntu systems, use apt instead:

# OpenSSL 3.x
apt install -y pkcs11-provider p11-kit
 
# OpenSSL 1.x
apt install -y libengine-pkcs11-openssl p11-kit

Note: Pelican automatically detects your OpenSSL version and uses the appropriate mode (Provider API for OpenSSL 3.x, ENGINE API for OpenSSL 1.x).

Enabling PKCS#11

To enable PKCS#11 TLS key protection, add the following to your pelican.yaml:

Server:
  EnablePKCS11: true

Once enabled, Pelican will:

1. Start the internal p11proxy server
2. Generate a PKCS#11-compatible OpenSSL configuration
3. Configure XRootD to use the PKCS#11 URI for TLS operations

Verifying PKCS#11 is Active

When PKCS#11 is enabled and working correctly, you will see log messages similar to:

p11proxy: accepted connection ...

You can also run Pelican with verbose p11-kit debugging to see detailed RPC activity:

P11_KIT_DEBUG=rpc pelican origin serve

This will produce detailed output showing the PKCS#11 communication:

(p11-kit:PID) rpc_unix_init: initialized rpc socket: ...
(p11-kit:PID) rpc_C_GetSlotList: C_GetSlotList: enter ...

Restricting Key File Permissions

With PKCS#11 enabled, you can further harden your deployment by making the TLS key unreadable to the XRootD process:

chmod 600 /etc/pelican/certificates/tls.key
chown pelican:pelican /etc/pelican/certificates/tls.key

Troubleshooting

If PKCS#11 fails to initialize, Pelican will log a warning and fall back to the traditional method where XRootD reads the key file directly. Common issues include:

Note: If you explicitly set Server.EnablePKCS11: true but the required dependencies are missing, Pelican will log a warning. If the parameter is not set (defaulting to false), Pelican will silently skip PKCS#11 setup.