# Pelican Documentation — Full Text > Concatenated, Markdown-converted content of the Pelican documentation site, generated from the built HTML for LLM consumption. > Source: https://docs.pelicanplatform.org Source: https://docs.pelicanplatform.org/about-pelican # What Is the Pelican Platform? Pelican is an open-source software platform for building data federations that works by connecting a broad range of data repositories under a unified architecture. Whether data lives on a POSIX filesystem, in S3, or behind an HTTP server, Pelican aims to bring this data together and simplify its access by abstracting away the need to know where it comes from. **Pelican’s goals are to**: - Enable access to data wherever it is needed, regardless of where it comes from — without having to learn multiple backend technologies. This access could take place in a Jupyter notebook, a campus cluster, or from national-scale computing infrastructure like the [OSPool](https://osg-htc.org/services/open_science_pool.html) . - Support Open Science initiatives by supporting inter-disciplinary data sharing - Encourage and support [FAIR](https://www.go-fair.org/fair-principles/)  data practices - Allow computing providers to stage data on-site as it’s needed **Pelican enables researchers to**: - Integrate their data with national-scale computing infrastructure, with a focus on easy setup, distributed data caching and object delivery efficiency - Make their data accessible to a broad range of users while maintaining control over how their data is accessed and by whom - Coalesce disparate data repositories like S3, Globus and Posix under a common namespace The flagship federation underpinned by Pelican is called the [Open Science Data Federation](https://osdf.osg-htc.org/)  (OSDF), which serves a variety of large scientific collaborations across more than fifty data providers and approximately two dozen caches located throughout the world, often at points of presence within the global Research and Education networks such as ESNet and Internet2. ## Core Concepts and Terminology Before proceeding, we recommend reading through the [Core Concepts and Terminology](./about-pelican/core-concepts) page to get familiar with terms used in the documentation. ## Making Bytes Accessible and Moving Them — A First Look Under The Hood This section provides a simplified example of how data is made accessible and moved within the OSDF. In particular, it elides the OSDF’s Caching infrastructure and any discussion of authorization tokens. Pelican serves two sides of the same coin — Data owners who want to federate their data from wherever it lives natively, and data consumers who want to access and compute on data wherever they need it. ![Pelican and OSDF](/pelican/optimized_images/arch-repo-and-consumer-opt-1920.WEBP) The federation’s core goal is connecting data owners and data consumers. As such, the primary prerequisite for data to be moved via a Pelican federation is for a data owner to make their data accessible to the federation. This happens when an Origin is placed in front of the repository and registered with the federation. While federations like the OSDF _may_ wish to control or filter any Origin registrations to vet the data they make available, this example assumes the Origin’s registration is automatically approved. The red arrow in the following graphic represents the vetting/approval step, should the federation require it. ![Pelican and OSDF](/pelican/optimized_images/arch-origin-registration-opt-1920.WEBP) The Origin’s owner configures a federation root before starting the service. After startup, the Origin then discovers the hostnames for its Registry and Directory by using the federation root to construct the URL “[https://osg-htc.org/.well-known/pelican-configuration](https://osg-htc.org/.well-known/pelican-configuration) ”, the federation’s _discovery endpoint_ containing a JSON that details the federation’s central services. Next, the Origin registers its namespace and public key with the Registry, proving that it owns the corresponding private key. Finally, the Origin begins advertising its namespace information and hostname to the Director. While somewhat simplified, this example illustrates the process origins must take to make themselves known within the federation. After completing these steps, the objects from the Data Repository are available via Pelican. The next step is for the data consumer to actually _move_ the data. Pelican assumes the data consumer already knows the federation that provides the data they want, along with the name of the object within the federation. These two pieces of information are combined and provided to the Client as a `pelican://`\-schemed URL ![Pelican and OSDF](/pelican/optimized_images/arch-origin-discovery-opt-1920.WEBP) The data consumer provides their Pelican client of choice the pelican:// URL that defines the object they want to download, where `osg-htc.org` is the federation and (`/weather/cloud.jpg`) is the object. Just as the origin discovered the Director’s hostname by visiting the discovery endpoint, so too does the client. After the client has performed federation metadata discovery, it issues an HTTP GET request to the Director, using the object name as a URL path. The Director responds with an HTTP 307 Redirect, forwarding the client on to the a server that can provide the object, in this example an Origin. Finally, the Client follows the redirect and downloads the object by issuing an HTTP GET request to “[https://my-origin.com/weather/cloud.jpg](https://my-origin.com/weather/cloud.jpg) ” Notice that the Origin continues advertising with the Director throughout. Once again, this example is simplified, mainly because the Director typically sends the client to a Cache capable of fetching the object, not directly to the Origin. In any case, the object is delivered to the Client without passing through the federation’s Central Services. When the object is fetched through a Cache, the Cache performs the same discovery step as the Client by asking the Director for an Origin that exports the object. --- Source: https://docs.pelicanplatform.org/install # Install Pelican This document lists Pelican’s operating system requirements and explains how you can download and install the correct Pelican executable. Know what you want? Visit our [Downloads Repository](https://dl.pelicanplatform.org) . ## Before Starting Pelican is distributed as two separate executables: | Binary | Purpose | | --- | --- | | `pelican` | **Client** — download and upload data, manage credentials, submit jobs | | `pelican-server` | **Server** — run an Origin, Cache, Director, or Registry | Most users only need the `pelican` client. If you intend to run a federation service (Origin, Cache, Director, or Registry), you also need `pelican-server`. On RPM-based Linux systems both binaries are installed together via the `pelican` and `pelican-server` RPM packages. For containerized server deployments we recommend our [Pelican Docker images](./install/docker), which ship both binaries. ## Supported Operating Systems Pelican supports the following operating systems, when running as a **client**: - [Red Hat Enterprise Linux (CentOS, Fedora, Alma, Rocky, or openSUSE)](./install/rhel) - [Debian or Ubuntu](./install/debian) - [Alpine Linux](./install/alpine) - [macOS](./install/macos) - [Windows](./install/windows) - [Conda-forge (Linux, macOS, Windows)](./install/conda) > **Note**: Installation of Pelican on other operating systems is possible, but not recommended or supported. ## Determine Which Executable To Download Each section of the following chart contains a link to a specific Pelican executable based on your operating system and CPU architecture. If you don’t know your computer’s architecture, refer to any specific instructions for your operating system under [Install Pelican Executable](#install-pelican-executable). The vast majority of non-MacOS operating systems will use an x86\_64 architecture. Once you’ve determined which executable to use, copy the provided link and continue to [Install Pelican Executable](#install-pelican-executable). ### What Version Should I Download? In most cases, you should use the highest version of Pelican available because it will contain our latest features and bug fixes. This is the default link provided unless you specify otherwise. If you need a specific version that isn’t the latest, you can select it using the chart’s “Version” dropdown. Pelican follows the [semantic versioning scheme](https://semver.org/) , so, for example, 7.10.5 means: - 7 represents the major release (changes in this number represent large changes that may break backwards compatibility) - 10 represents feature releases (each increment within the same major release represents new, backward-compatible functionality) - 5 represents a bug fix/patch release The download table above includes Pelican versions as old as `v7.6.x` To download older versions, please refer to our [GitHub release archive](https://github.com/PelicanPlatform/pelican/releases) . ## Install Pelican Executable ### Linux - [Install Pelican on Red Hat Enterprise Linux (CentOS, Fedora, Alma, Rocky, or openSUSE)](./install/rhel) - [Install Pelican on Debian or Ubuntu](./install/debian) - [Install Pelican on Alpine Linux](./install/alpine) - [Install Pelican as a standalone executable](./install/linux-binary) ### MacOS - [Install Pelican on macOS](./install/macos) ### Windows - [Install Pelican on Windows](./install/windows) ### Cross-Platform Package Manager - [Install Pelican via Conda-forge (Linux, macOS, Windows)](./install/conda) ### Install OSDF or Server Package Pelican has two special RPM packages: `pelican-osdf-compat` (for running Pelican in the Open Science Data Federation) and `pelican-server` (which installs the `pelican-server` binary needed to run federation services such as an Origin or Cache). The `pelican-server` RPM **requires the `pelican` RPM to be installed first**. You may get the following error message if you only install the special package: ``` Problem: conflicting requests - nothing provides pelican needed by pelican-osdf-compat-7.8.5-1.aarch64 from @commandline ``` To fix the issue, install the Pelican package first by following [the preceding instructions](#install-pelican-executable). Are you planning to contribute to the OSDF? Check out the OSDF documentation for system administrators here: [osg-htc.org/docs/data/osdf/overview/](https://osg-htc.org/docs/data/osdf/overview/) . ## Verify Pelican Is Installed > If you installed Pelican as a standalone executable, or if you installed Pelican on MacOS or Windows operating systems, you need to add Pelican to your `PATH` environment variable before proceeding. See instructions for your operating system under the [os-specific installation instructions section](#install-pelican-executable) 1. Run the following command after you installed Pelican - **Linux and macOS**: ``` which pelican ``` - **Windows**: ``` where pelican ``` If this command outputs a filepath, Pelican is installed correctly. If no output is returned, revisit instructions for your operating system. 2. If you installed the `pelican-server` RPM or a server Docker image, verify the server binary is also available: - **Linux and macOS**: ``` which pelican-server ``` - **Windows**: ``` where pelican-server ``` 3. Test the Client’s functionality by running an **object get** command that downloads a test file from [Open Science Data Federation (OSDF)](https://osg-htc.org/services/osdf)  to your current directory ``` $ pelican object get pelican://osg-htc.org/pelicanplatform/test/hello-world.txt . hello-world.txt 27.00b / 27.00b [=============================================================================================] Done! ``` ## Server Prerequisites If you plan to run a Pelican server (Origin, Cache, Director, or Registry), the following prerequisites apply to all server types. ### Enable Time Synchronization Pelican servers use tokens for authorization, and these tokens contain time-sensitive claims (such as `iat`, the “issued at” time). If your server’s clock is significantly out of sync with the federation’s other services, tokens may be rejected and your server may fail to join the federation. Ensure that Network Time Protocol (NTP) is enabled and actively synchronizing your server’s clock. If your site has a firewall, make sure outgoing UDP traffic on port 123 (NTP) is permitted. To check whether your system clock is synchronized, run: ``` timedatectl status ``` Look for `System clock synchronized: yes` and `NTP service: active` in the output. If NTP is not active, you can enable it with: ``` timedatectl set-ntp true ``` If your system uses `chrony`, you can also check synchronization status with: ``` chronyc tracking ``` If your system uses `ntpd`, you can check synchronization status with: ``` ntpq -p ``` ## Next Steps - [Access data with Pelican client](/getting-data-with-pelican/client) - [Serve your data with Pelican Origin server](/federating-your-data) - [Serve a new Pelican Pelican federation](/operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/rhel # Install Pelican on CentOS, Fedora, openSUSE, Alma, or Rocky This document explains how to install Pelican on a Red Hat-based Linux distribution system such as: Red Hat Enterprise Linux, CentOS, Fedora, or openSUSE. ## Install Pelican RPM You can install Pelican from the standalone RPM, or with the binary `.tar.gz` file. You only need to follow one section below to install. ### Install the Pelican RPM package manually 1. Navigate to [Pelican download page](../install#download-pelican-binary) and select the Pelican RPM you want to install. 2. In the **Operating System** section, select **Linux**. In **Architectures** section, select **X86\_64** or **ARM64** depending on the architecture of your machine. 3. In the list of download candidates, copy the link to `pelican-x.x.x-1.x86_64.rpm` if you select **X86\_64**, or `pelican-x.x.x-aarch64.rpm` if you select **ARM64** (ARM), where `x.x.x` is the version number. 4. Change the following command with the link to the binary you copied in the previous step and run the command ``` sudo yum install -y ``` Example to install Pelican `v7.5.8` RPM package on an `X86_64` machine: ``` sudo yum install -y https://github.com/PelicanPlatform/pelican/releases/download/v7.5.8/pelican-7.5.8-1.x86_64.rpm ``` ### Install Pelican as a standalone binary Refer to installing [Linux Standalone Binary](./linux-binary) page for how to install Pelican as a standalone binary. ## Uninstall on CentOS, Fedora, or openSUSE 1. If you configured Pelican server to run with systemd, stop the systemd service for Pelican server: ``` sudo systemctl stop grafana ``` 2. To uninstall Pelican ``` sudo dnf remove pelican ``` ## Next steps - [Verify Pelican is installed](../install#verify-pelican-is-installed) - [Read our quick start guide on accessing data using Pelican client](../getting-started/accessing-data) - [Federate your data via Pelican Origin server](../federating-your-data) - [Operate a Pelican data federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/debian # Install Pelican on Debian or Ubuntu This document explains how to install Pelican on a Debian or Ubuntu Linux operating system. ## Install Pelican DEB You can install Pelican by downloading a `.deb` package or by downloading a binary `.tar.gz` file. You only need to follow one section below to install. > **Note**: Pelican Debian package does not contain dependencies such as `XRootD` for Pelican servers. Use [Pelican Docker image](./docker) instead if you want to serve a Pelican server. ### Install the Pelican using a DEB package 1. Navigate to [Pelican download page](../install#download-pelican-binary) and select the Pelican DEB you want to install. 2. In the **Operating System** section, select **Linux**. In **Architectures** section, select **X86\_64** or **ARM64** depending on the architecture of your machine. 3. In the list of download candidates, copy the link to `pelican-x.x.x-1.amd64.deb` if you select **X86\_64**, or `pelican-x.x.x-1_arm64.deb` if you select **ARM64**, where `x.x.x` is the version number. 4. Change the following command with the link to the binary you copied in the previous step and run the command ``` wget sudo dpkg -i pelican_7.5.8-1_amd64.deb ``` > **Note**: You need to replace `pelican_7.5.8-1_amd64.deb` with `pelican_7.5.8-1_arm64.deb` if you are running an `ARM64` machine. Example to install Pelican `v7.5.8` DEB package on an `X86_64` machine: ``` wget https://github.com/PelicanPlatform/pelican/releases/download/v7.5.8/pelican_7.5.8-1_amd64.deb sudo dpkg -i pelican_7.5.8-1_amd64.deb ``` ### Install Pelican as a standalone binary Refer to installing [Linux Standalone Binary](./linux-binary) page for how to install Pelican as a standalone binary. ## Uninstall on Debian or Ubuntu To uninstall Pelican, run the following command ``` sudo apt-get remove pelican ``` ## Next steps - [Verify Pelican is installed](../install#verify-pelican-is-installed) - [Read our quick start guide on accessing data using Pelican client](../getting-started/accessing-data) - [Federate your data via Pelican Origin server](../federating-your-data) - [Operate a Pelican data federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/alpine # Install Pelican on Alpine This document explains how to install Pelican on a Alpine Linux operating system. ## Install Pelican APK You can install Pelican by downloading an `.apk` package or by downloading a binary `.tar.gz` file. You only need to follow one section below to install. > **Note**: Pelican Alpine Linux package does not contain dependencies such as `XRootD` for Pelican servers. Use [Pelican Docker image](./docker) instead if you want to serve a Pelican server. ### Install the Pelican using an APK package 1. Navigate to [Pelican download page](../install#download-pelican-binary) and select the Pelican APK you want to install. 2. In the **Operating System** section, select **Linux**. In **Architectures** section, select **X86\_64** or **ARM64** depending on the architecture of your machine. 3. In the list of download candidates, copy the link to `pelican_x.x.x_p1_x86_64.apk` if you select **X86\_64**, or `pelican_x.x.x_p1_aarch64.apk` if you select **ARM64**, where `x.x.x` is the version number. 4. Change the following command with the link to the binary you copied in the previous step and run the command ``` wget sudo apk add --allow-untrusted pelican_7.5.8_p1_x86_64.apk ``` > **Note**: You need to replace `pelican_7.5.8_p1_x86_64.apk` with `pelican_7.5.8-1_aarch64.apk` if you are running an `ARM64` machine. Example to install Pelican `v7.5.8` APK package on an `X86_64` machine: ``` wget https://github.com/PelicanPlatform/pelican/releases/download/v7.5.8/pelican_7.5.8_p1_x86_64.apk sudo apk add --allow-untrusted pelican_7.5.8_p1_x86_64.apk ``` ### Install Pelican as a standalone binary Refer to installing [Linux Standalone Binary](./linux-binary) page for how to install Pelican as a standalone binary. ## Uninstall on Alpine To uninstall Pelican, run the following command ``` sudo apk del pelican ``` ## Next steps - [Verify Pelican is installed](../install#verify-pelican-is-installed) - [Read our quick start guide on accessing data using Pelican client](../getting-started/accessing-data) - [Federate your data via Pelican Origin server](../federating-your-data) - [Operate a Pelican data federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/linux-binary # Install Pelican as unprivileged Linux user This document explains how to install Pelican on a Linux operating system as a standalone binary without any special administrator privileges. ## Quickstart To download the latest version of Pelican and unpack it into the default location for user binaries on most Linux hosts, copy/paste the following into a terminal: ``` wget -O - "https://dl.pelicanplatform.org/latest/pelican_$(uname -s)_$(uname -m).tar.gz" | tar zx -C ~/.local/bin/ --strip-components=1 ``` To test the binary, execute: ``` pelican --version ``` If a version number did not print, then you may have a special platform or configuration on your host; follow the subsequent sections. Otherwise, you’re done and may follow the [Next Steps](#next-steps). ## Download Pelican Binary 1. Navigate to the [Pelican download page](../install#download-pelican-binary) to select the Pelican standalone binary you want to install. 2. In **Operating System** section, select **Linux**. In **Architectures** section, select **X86\_64** or **ARM64** depending on the architecture of your machine. 3. In the list of candidates, copy the link to `pelican_Linux_x86_64.tar.gz` or `pelican_Linux_arm64.tar.gz`, as appropriate. 4. Change the following command with the link to the binary you copied in the previous step and run the command ``` wget mkdir -p ~/.local/bin tar -zxvf -C ~/.local/bin/ --strip-components=1 pelican_Linux_$(uname -m).tar.gz ``` > **Note**: The shell should expand `$(uname -m)` with the machine’s hardware platform. If it fails, you may need to replace the filename with `pelican_Linux_x86_64.tar.gz` with `pelican_Linux_arm64.tar.gz`, as appropriate. Example to install Pelican standalone binary on an `X86_64` machine: ``` $ wget https://dl.pelicanplatform.org/latest/pelican_Linux_x86_64.tar.gz $ mkdir -p ~/.local/bin $ tar -zxvf -C ~/.local/bin/ --strip-components=1 pelican_Linux_x86_64.tar.gz ``` ## Make Pelican Binary Available The above command extracted the binary from the `tar` file and placed it inside `.local/bin/` in your home directory. On most Linux distributions, this makes the binary automatically available. You may test this by running: ``` command -v pelican ``` If it outputs a path like this example: ``` $ command -v pelican /home/username/.local/bin/pelican ``` then you may skip this section. If not, you need to alter the `PATH` environment variable that controls which directories are searched for binaries. ### Add Pelican binary to your `PATH` for the current terminal To change the `PATH` variable for only the currently-running terminal session, execute the following line: ``` export PATH="$HOME/.local/bin/:$PATH" ``` Example outputs: ``` $ export PATH="$HOME/.local/bin/:$PATH" # Add ~/.local/bin/ to the PATH $ pelican --version # Run Pelican binary Version: 7.12.0 Build Date: 2025-01-14T21:33:23Z Build Commit: 57748c37af7574ec182e5a21db741c4c5a1e61a8 Built By: goreleaser ``` ### Add Pelican binary to your `PATH` permanently To add the `~/.local/bin/` directory to your `PATH` variable permanently, execute the following line (assuming you are using the ‘bash’ shell): ``` echo "export PATH="\$HOME/.local/bin/:\$PATH" >> ~/.bashrc ``` Example outputs: ``` $ echo "export PATH=\$HOME/.local/bin:\$PATH" >> ~/.bashrc # Add the .local/bin folder to your .bashrc file $ source ~/.bashrc # Apply the change $ pelican --version # Run Pelican binary Version: 7.12.0 Build Date: 2025-01-14T21:33:23Z Build Commit: 57748c37af7574ec182e5a21db741c4c5a1e61a8 Built By: goreleaser ``` ## Next steps - [Verify Pelican is installed](../install#verify-pelican-is-installed) - [Read our quick start guide on accessing data using Pelican client](../getting-started/accessing-data) - [Federate your data via Pelican Origin server](../federating-your-data) - [Operate a Pelican data federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/macos # Install Pelican on MacOS This document explains how to install Pelican on MacOS. ## Install Pelican as a Standalone Executable Pelican provides a binary executable file instead of a `DMG` installer for MacOS, so installation must happen through a terminal. This process entails downloading/extracting the executable and adding it to your computer’s `PATH` environment variable. 1. If you’ve not yet determined which executable you need, refer to the [download page](../install#determine-which-executable-to-download) and copy the relevant link: In the **Operating System** section, select **macOS**. In **Architectures** section, select **X86\_64** or **ARM64** depending on the architecture of your machine. > **Note:** If you don’t know your computer’s architecture, click the **Apple menu**  in the top-left corner of your screen, then click **About This Mac** and check the **Chip** or **Processor** field. If it mentions **Intel** your computer needs Pelican’s **X86\_64** executable, and if it mentions Apple M1, M2, or later, you’ll need the **ARM64** executable. Intel Macs (x86\_64) should see the file `pelican_Darwin_x86_64.tar.gz` and Apple Silicon Macs (ARM64) should see `pelican_Darwin_arm64.tar.gz`. 2. Open a terminal and navigate to a directory where you can download, extract and store the Pelican executable. Then use your command line tools of choice to download and extract the link you copied in the previous step. The rest of these instructions assume `curl` for downloading and `tar` for extraction: ``` curl -LO tar -zxvf ``` Example to install Pelican executable for an Apple Silicon Mac: ``` curl -LO https://github.com/PelicanPlatform/pelican/releases/download/v7.10.5/pelican_Darwin_arm64.tar.gz tar -zxvf pelican_Darwin_arm64.tar.gz ``` 3. There should now be another directory called `pelican-` containing a Pelican executable that you can run directly using a relative `./` path. For example: ``` $ cd pelican-7.10.5 $ ./pelican --version # Print the executable version Version: 7.10.8 Build Date: 2024-10-09T14:56:56Z Build Commit: 1ef2c25a1585803e0c74a9e41fceb214b80bf3da Built By: goreleaser ``` However, to make the `pelican` command available everywhere on your Mac without needing a relative path, you still need to add it to your system’s `PATH` environment variable. ## Add Pelican Executable to Your `PATH` - Add the Pelican executable to your `PATH` for the current terminal session (disappears after closing the terminal) ``` $ cd pelican-7.10.5 # Go to the executable folder $ export PATH=$PWD:$PATH # Add current folder to the PATH $ pelican --version # Run Pelican executable Version: 7.10.8 Build Date: 2024-10-09T14:56:56Z Build Commit: 1ef2c25a1585803e0c74a9e41fceb214b80bf3da Built By: goreleaser ``` - Add the Pelican executable to your `PATH` permanently (persists through terminal restarts) ``` $ cd pelican-7.10.5 # Go to the executable folder $ echo "export PATH=$PWD:\$PATH" >> ~/.zshrc # Add the current folder to your .zshrc file $ source ~/.zshrc # Apply the change $ pelican --version # Run Pelican executable Version: 7.10.8 Build Date: 2024-10-09T14:56:56Z Build Commit: 1ef2c25a1585803e0c74a9e41fceb214b80bf3da Built By: goreleaser ``` ## Next steps - [Verify Pelican is installed](../install#verify-pelican-is-installed) - [Read our quick start guide on accessing data using Pelican client](../getting-started/accessing-data) - [Federate your data via Pelican Origin server](../federating-your-data) - [Operate a Pelican data federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/windows # Install Pelican on Windows This document explains how to install Pelican on Windows. ## Install Pelican as a standalone binary Pelican provides a binary file instead of a `MSI` installer for Windows. You need to download and extract the binary and add the binary to your `PATH` environment variable. 1. Navigate to [Pelican download page](../install#download-pelican-binary) and select the Pelican Windows binary. 2. In the **Operating System** section, select **Windows**. Download the file `pelican_Windows_x86_64.zip` to your computer. 3. Using Windows Explorer, navigate to folder containing the downloaded `.zip` file and extract it. 4. You may run the binary in the extracted folder, but it is recommended that you add Pelican binary to your `PATH` environment variable to allow `pelican` to be called directly from your PowerShell. To run Pelican binary from the extracted folder, do the following in [PowerShell](https://learn.microsoft.com/en-us/powershell/scripting/learn/ps101/01-getting-started?view=powershell-7.4) : ``` rem Go to the downloaded Pelican folder $ cd D:\Downloads\pelican_Windows_x86_64 rem Run Pelican binary $ pelican.exe --version Version: 7.5.8 Build Date: 2024-03-01T18:13:00Z Build Commit: d260a07d3b057d19b7fdd36125f91a8768531258 Built By: goreleaser ``` ## Add Pelican binary to your `PATH` - Add Pelican binary to your `PATH` for the current [PowerShell](https://learn.microsoft.com/en-us/powershell/scripting/learn/ps101/01-getting-started?view=powershell-7.4)  ``` rem Go to the binary folder $ cd pelican_Linux_arm64 rem Add current folder to the PATH environment variable $ $env:PATH += ";$(Get-Location)" rem Run Pelican binary $ pelican --version Version: 7.5.8 Build Date: 2024-03-01T18:13:00Z Build Commit: d260a07d3b057d19b7fdd36125f91a8768531258 Built By: goreleaser ``` - Add Pelican binary to your `PATH` permanently 1. **Open System Properties**: - Press `Win + R`, type `sysdm.cpl`, and press `Enter`. 2. **Access Environment Variables**: - In the System Properties window, go to the `Advanced` tab. - Click on the `Environment Variables` button. 3. **Edit the PATH Variable**: - In the Environment Variables window, under the `System variables` section, scroll down and find the `Path` variable, then select it and click `Edit`. - In the Edit Environment Variable window, click `New` and add the path to the directory containing `pelican.exe` (e.g., `C:\path\to\directory`). - Click `OK` to close all windows. 4. Verify Pelican is added to `PATH` - In a **new** PowerShell window, run the following command: ``` rem Run Pelican binary $ pelican --version Version: 7.5.8 Build Date: 2024-03-01T18:13:00Z Build Commit: d260a07d3b057d19b7fdd36125f91a8768531258 Built By: goreleaser ``` ## Next steps - [Verify Pelican is installed](../install#verify-pelican-is-installed) - [Read our quick start guide on accessing data using Pelican client](../getting-started/accessing-data) - [Federate your data via Pelican Origin server](../federating-your-data) - [Operate a Pelican data federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/conda # Install Pelican via Conda-forge This document explains how to install Pelican using conda-forge, a community-led conda channel that provides the `pelicanplatform` package. ## Prerequisites You’ll need either [conda](https://docs.conda.io/en/latest/) , [mamba](https://mamba.readthedocs.io/) , or [pixi](https://pixi.sh/)  installed on your system. These package managers work across Linux, macOS, and Windows. > **Note**: The package on conda-forge is named `pelicanplatform` (not `pelican`, which is a different package). ## Install Pelican using Conda 1. First, add the `conda-forge` channel and set channel priority: ``` conda config --add channels conda-forge conda config --remove channels defaults conda config --set channel_priority strict ``` 2. Install `pelicanplatform`: ``` conda install pelicanplatform ``` ## Install Pelican using Mamba [Mamba](https://mamba.readthedocs.io/)  is a faster alternative to conda: 1. Add the `conda-forge` channel and set channel priority: ``` conda config --add channels conda-forge conda config --remove channels defaults conda config --set channel_priority strict ``` 2. Install `pelicanplatform`: ``` mamba install pelicanplatform ``` ## Install Pelican using Pixi [Pixi](https://pixi.sh/)  is a modern, fast package management tool that provides fully reproducible multi-platform environments: 1. Initialize a pixi project (if not already initialized): ``` pixi init ``` 2. Add `pelicanplatform` to your project: ``` pixi add pelicanplatform ``` 3. Run Pelican commands in the project environment: ``` pixi run pelican --version ``` ## Available Platforms The `pelicanplatform` package is available on conda-forge for the following platforms: - Linux (x86\_64, aarch64, ppc64le) - macOS (x86\_64, arm64) - Windows (x86\_64) ## Version Management To search for available versions: ``` conda search pelicanplatform --channel conda-forge ``` Or with mamba: ``` mamba search pelicanplatform --channel conda-forge ``` Or with Pixi: ``` pixi search pelicanplatform ``` ## Additional Information For more details about the conda-forge package, including build status and maintainer information, visit the [pelicanplatform-feedstock repository](https://github.com/conda-forge/pelicanplatform-feedstock) . ## Next steps - [Verify Pelican is installed](../install#verify-pelican-is-installed) - [Read our quick start guide on accessing data using Pelican client](../getting-started/accessing-data) - [Federate your data via Pelican Origin server](../federating-your-data) - [Operate a Pelican data federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/install/docker # Run Pelican Server with Docker Image This document explains how to run a Pelican server using Pelican docker image. If you want to use Pelican client functionalities, such as to download or upload an object, refer to the [install page](../install) to download and install a Pelican binary instead. ## Before starting Pelican builds separate images for each Pelican server components, e.g. origin, cache, director, and registry. Depending on which Pelican server you want to run, you need to select a different Docker image from the list below. - Pelican origin server: `hub.opensciencegrid.org/pelican_platform/origin:latest` - Pelican cache server: `hub.opensciencegrid.org/pelican_platform/cache:latest` - Pelican director server: `hub.opensciencegrid.org/pelican_platform/director:latest` - Pelican registry server: `hub.opensciencegrid.org/pelican_platform/registry:latest` The `latest` tag will pull the Pelican server image with the latest released version. You may pull an image with explicit Pelican version by passing the version as the tag (e.g. `v7.6.4`) instead. For a list of available tags, refer to [Pelican Harbor repository](https://hub.opensciencegrid.org/harbor/projects/883/repositories/origin/artifacts-tab?publicAndNotLogged=yes) . ## Run Pelican server via Docker CLI This section describes how to run various Pelican server images using the Docker CLI. If you haven’t installed Docker engine, follow the [documentation from Docker](https://docs.docker.com/get-docker/)  to install and start the Docker engine first. ### Run Pelican origin server To run the latest pelican origin server, run the following command: ``` docker run -it -p 8444:8444 -p 8443:8443 -v /path/to/your/data/:/tmp/pelican --name=pelican-origin hub.opensciencegrid.org/pelican_platform/origin:latest serve -v /tmp/pelican:/foo/bar -f ``` Where: - `docker run` is a Docker CLI command that runs a new container from an image - `-it` (`--interactive --tty`) runs the container in interactive mode and uses a tty terminal - `-p :` (`--publish`) publishes a container’s port(s) to the host, allowing you to reach the container’s port via a host port. In this case, we can reach the container’s port `8444` via the host’s port `8444`. Note that the admin website of Pelican servers run on port `8444` by default, and the objects transfer endpoint of the Pelican origin server runs on port `8443` by default. - `-v :` (`--volume`) binds mount a volume from the host location(s) to the container’s location(s). This allow you to share files in your host machine to the container. In this case, we bind `/path/to/your/data/` on your host machine to `/tmp/pelican` in the container. You need to replace `/path/to/your/data/` to the directory where your data to publish is located. - `--name` assigns a logical name to the container (e.g. pelican-origin). This allows you to refer to the container by name instead of by ID. - `hub.opensciencegrid.org/pelican_platform/origin:latest` is the image to run - `serve` is the command to run the Pelican binary - `-v /tmp/pelican:/foo/bar` is the Pelican argument to bind `/tmp/pelican` directory in the container as namespace `/foo/bar` in Pelican. You need to change `/foo/bar` to a meaningful path that can represent your data, e.g. `/chtc/public-data`. You may pass additional arguments to Pelican server by appending them after this argument. - `-f ` sets the federation discovery URL, where `` is the URL to the federation the origin will be joining in. For instructions on how to find a federation to join, refer to [Serve a Pelican Origin](../serving_an_origin#find-a-federation-to-join) ### Run Pelican cache server To run the latest pelican cache server, run the following command: ``` docker run -it -p 8444:8444 -p 8442:8442 --name=pelican-cache hub.opensciencegrid.org/pelican_platform/cache:latest serve -f ``` Where most of the command overlaps with the one to run the Pelican origin server, with the following differences: - `-p 8444:8444 -p 8442:8442` publishes port `8444` and `8442` from the container to the same port on the host machine. Note that the admin website of the Pelican server runs on port `8444` by default, and the objects transfer endpoint of the Pelican cache server runs on port `8442` by default. ### Run Pelican director server To successfully run a Pelican director server, additional configuration is required. Follow [Serve a Pelican Director](../serving_a_federation#serve-a-director) for instructions. For how to pass configurations to Docker container, refer to [the next section](#configure-pelican-server-in-a-container). To run the latest pelican director server, run the following command: ``` docker run -it -p 8444:8444 --name=pelican-director hub.opensciencegrid.org/pelican_platform/director:latest serve -f ``` ### Run Pelican registry server To successfully run a Pelican registry server, additional configuration is required. Follow [Serve a Pelican Registry](../serving_a_federation#serve-a-registry) for instructions. For how to pass configurations to Docker container, refer to [the next section](#configure-pelican-server-in-a-container). To run the latest pelican registry server, run the following command: ``` docker run -it -p 8444:8444 --name=pelican-registry hub.opensciencegrid.org/pelican_platform/registry:latest serve -f ``` ## Stop Pelican container To stop the Pelican container, run the following command: ``` # The `docker ps` command shows the processes running in Docker docker ps # This will display a list of containers that looks like the following: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0be1a304b5d7 hub.opensciencegrid.org/pelican_platform/director:latest "/bin/sh" 1 hour ago Up 1 hour 0.0.0.0:8444->8444/tcp pelican-director # To stop the pelican container run the command # docker stop or use # docker stop , which is `pelican-director` as previously defined docker stop pelican-director ``` ## Configure Pelican server in a container There are two ways to configure a Pelican server running in a container. One is through the environment variables, the other is by passing a `pelican.yaml` configuration file to the container. This section includes instructions for both. ### Configure via environment variables Most Pelican configuration parameters described in [parameters page](../parameters) can be passed to a Pelican server as environment variables. For a configuration parameter, the corresponding environment variable is `PELICAN_`, where `PELICAN` is the prefix, and `` is the name of the parameter, with dots `.` replaced by underscores `_`. For example, a Pelican configuration named `Logging.Level` has a corresponding environment variable named `PELICAN_LOGGING_LEVEL`. To pass environment variables to the Docker container, append `-e` flag to your `docker run` command. For example, if you want to pass `PELICAN_LOGGING_LEVEL` to the container, run: ``` docker run -it -e PELICAN_LOGGING_LEVEL=debug -p 8444:8444 -p 8443:8443 -v /path/to/your/data/:/tmp/pelican --name=pelican-origin hub.opensciencegrid.org/pelican_platform/origin:latest serve -v /tmp/pelican:/foo/bar -f ``` There are other ways to pass environment variables to a Docker container, see details in [Docker documentation](https://docs.docker.com/reference/cli/docker/container/run/#env) . Some of Pelican configuration parameters with the `object` type can not be passed as environment variables, such as [GeoIPOverrides](../parameters#GeoIPOverrides) and [Origin.Exports](../parameters##Origin-Exports). For these parameters, use a configuration file instead. ### Configure via the configuration file Pelican servers can also be configured via a configuration file in `yaml`. The default location that Pelican looks for a configuration file is `/etc/pelican/pelican.yaml` if Pelican runs as the root user (which is the case when the Pelican server is running in a container). If the Pelican server runs as a non-root user, the default location is `~/.config/pelican/pelican.yaml`. It is recommended that admins bind a Pelican configuration file on the host machine to the container running the Pelican server. Follow the steps below for instructions. 1. Prepare a `yaml` file on the host machine: ``` touch pelican.yaml ``` 2. Modify and save the `pelican.yaml` file on host machine with the following content: pelican.yaml ``` Logging: Level: debug ``` 3. Bind the configuration file on the host machine to the container when running the Pelican server image: ``` docker run -it -p 8444:8444 -p 8443:8443 -v /path/to/your/data/:/tmp/pelican -v $PWD/pelican.yaml:/etc/pelican/pelican.yaml --name=pelican-origin hub.opensciencegrid.org/pelican_platform/origin:latest serve -v /tmp/pelican:/foo/bar -f ``` where - `-v $PWD/pelican.yaml:/etc/pelican/pelican.yaml` is the flag to bind the `pelican.yaml` file under current working directory `$PWD` to `/etc/pelican/pelican.yaml` directory inside the container 4. You should note that your running Pelican origin server is logging debug level messages. ## Additional Configurations for Running Pelican in a Container ### Server hostname and port Pelican server has a built-in web server to handle various API requests. By default, it uses server hostname and a port number as the server address (e.g. `https://127.0.0.1:8444`). In a container environment, the hostname is an alphanumeric value, e.g. `6ee28c5df997`, and the corresponding Pelican server address is `https://6ee28c5df997:8444`. This value is for container internal communications and is not accessible from outside of the container. In local deployment, to access the container ports from the host machine, one can publish a port from inside the container to the host machine. ``` docker run -p 8444:8444 origin ``` However, Pelican server still recognize `6ee28c5df997` as its hostname, causing a mismatch. In practice, there is usually a reverse-proxy service (such as Nginx, Traefik, etc.), to direct traffic from a human-readable domain name (e.g. [https://example-origin.org](https://example-origin.org) ) to your container (e.g. `6ee28c5df997:8444`). If Pelican server doesn’t catch this information and uses it as its server address, there will be a mismatch when the server advertises itself to other Pelican services. Therefore, to keep things consistent, Pelican internally should use the same human-readable domain name as its web server’s _external address_. To do so, configure `Server.ExternalWebURL` to the domain name the outside world uses to access the Pelican web service, e.g. [https://example-origin.org:8444](https://example-origin.org:8444)  for a production service or [https://localhost:8444](https://localhost:8444)  for a local deployment. Notice that there is a port number, `8444`, following the hostname in the URLs above. This is the default port for Pelican web server. When publishing ports from the container, you may choose a different port on the host machine to publish to. Two conventional options are `443` or `8080`, which are default ports for a web service. If you publish `8444` port from the container to `443`, or `8080` on the host machine, i.e. `docker run -p 8444:443`, you may remove the port number from `Server.ExternalWebURL`, i.e. `https://example-origin.org`. For other ports, you need to leave the port number in the URL. For more information about how Docker networking works, refer to [Docker Networking Documentation](https://docs.docker.com/network/) . ## Persisting Data Across Container Restarts When running Pelican servers in containers, certain files and directories need to persist across container restarts to maintain server identity, configurations, and data. Without proper volume mounts, restarting a container will cause data loss, including cryptographic keys, databases, and configuration files. This section describes which data should be persisted for each Pelican server type and how to configure volume mounts appropriately. ### Quick Reference: Essential Volume Mounts The table below shows the minimum recommended volume mounts for each server type at the directory level. For additional information about using config params to set the location of specific files in your container, see the section for your service. | Server Type | Essential Volume Mounts | | --- | --- | | **All** | `-v /host/pelican/config:/etc/pelican` `-v /host/pelican/lib:/var/lib/pelican` `-v /host/pelican/certificates:/etc/pelican/certificates` | | **Origin** | ’All’ + `-v /host/data:/data` (your data directory) | | **Cache** | ’All’ + `-v /host/cache/storage:/var/cache/pelican` (cache storage) | | **Director** | ’All’ + `-v /host/pelican/maxmind:/var/cache/pelican/maxmind` (optional, for GeoIP) | | **Registry** | ’All’ (no additional mounts required) | The `/etc/pelican` directory contains cryptographic keys that establish your server’s identity in the federation. Loss of these keys will cause the “namespace is registered with a different key” error for Caches/Origins, and loss of the Director’s issuer key will cause federation-wide issues with Cache/Origin Director tests. See [“Issuer Keys Directory”](./docker#issuer-keys-directory) for more information. ### Understanding Default Paths Pelican uses different default paths depending on whether the server runs as root (which is the case in containers): - **Root user** (including root-ly containers such as in Docker, and in unprivileged containers where pelican still appears to run as PID 0): Configuration and data are stored in `/etc/pelican` and `/var/lib/pelican` - **Non-root user**: Configuration and data are stored in `~/.config/pelican` All examples below assume the server runs as root inside a container. ### Mounting Directories vs Individual Files The volume mount examples in this document show mounting entire directories (e.g., `-v /host/pelican/config:/etc/pelican`). This is the recommended approach for Docker deployments as it’s simpler and ensures all related files are persisted together. However, in some deployment scenarios (such as Kubernetes with sealed secrets), you may need to mount individual files instead of entire directories. In such cases, consult the configuration parameter documentation linked for each file to determine or customize the file’s path. Each file’s default location is also specified in the sections below. ### Data That Should Persist for All Server Types The following data should be persisted for all Pelican server types (Origin, Cache, Director, Registry): #### Issuer Keys Directory **Configuration**: To customize the path, see the [`IssuerKeysDirectory`](../parameters#IssuerKeysDirectory) parameter. **Purpose**: Contains PEM-encoded ECDSA private keys used to sign credentials issued by the server. These keys establish the server’s cryptographic identity within the federation. **Default Path**: `/etc/pelican/issuer-keys` **Best Practice**: Pre-generate these keys outside the container and mount them into the container’s issuer keys directory. This prevents accidental re-generation of keys in the event of a typo in volume mounts, as the keys remain persisted on the host. **Volume Mount**: ``` -v /host/pelican/issuer-keys:/etc/pelican/issuer-keys ``` For mounting individual key files: ``` -v /host/pelican/issuer.pem:/etc/pelican/issuer-keys/issuer.pem ``` **Why it must persist**: If these keys are lost, the server will generate new ones, causing authentication failures with other federation services that trust the old keys. For Caches/Origins, this is the source of the “namespace/service is registered with a different key” error. When this error is encountered, you must contact your federation’s administrators to have them delete the old registration. For security, this cannot be fixed without help from the federation administrators. #### Server Database **Configuration**: To customize the path, see the [`Server.DbLocation`](../parameters#Server-DbLocation) parameter. **Purpose**: Stores server state and operational data. **Default Path**: `/var/lib/pelican/pelican.sqlite` **Volume Mount**: ``` -v /host/pelican/lib:/var/lib/pelican ``` For mounting individual database files: ``` -v /host/pelican/pelican.sqlite:/var/lib/pelican/pelican.sqlite ``` #### Web UI Password File **Configuration**: To customize the path, see the [`Server.UIPasswordFile`](../parameters#Server-UIPasswordFile) parameter. **Purpose**: Stores the hashed+salted password for accessing the server’s web UI. **Default Path**: `/etc/pelican/server-web-passwd` **Volume Mount**: ``` -v /host/pelican/config:/etc/pelican ``` For mounting individual password files: ``` -v /host/pelican/server-web-passwd:/etc/pelican/server-web-passwd ``` #### TLS Certificates and Keys **Configuration**: To customize paths, see [`Server.TLSCertificateChain`](../parameters#Server-TLSCertificateChain), [`Server.TLSKey`](../parameters#Server-TLSKey), and [`Server.TLSCACertificateFile`](../parameters#Server-TLSCACertificateFile) parameters. **Purpose**: TLS certificate chain and private key for HTTPS communication. Generally you _must_ mount these into your container because while Pelican will generate them if they’re not provided, the Pelican-generated versions are self-signed and will not be trusted by most other web services. **Default Paths**: - Certificate chain: `/etc/pelican/certificates/tls.crt` - Private key: `/etc/pelican/certificates/tls.key` - CA certificate: `/etc/pelican/certificates/tlsca.pem` **Volume Mount**: ``` -v /host/pelican/certificates:/etc/pelican/certificates ``` For mounting individual certificate files: ``` -v /host/tls.crt:/etc/pelican/certificates/tls.crt -v /host/tls.key:/etc/pelican/certificates/tls.key -v /host/tlsca.pem:/etc/pelican/certificates/tlsca.pem ``` #### Session Secret File **Configuration**: To customize the path, see the [`Server.SessionSecretFile`](../parameters#Server-SessionSecretFile) parameter. **Purpose**: Secret used for encrypting/decrypting session data for the web UI and CSRF protection. **Default Path**: `/etc/pelican/session-secret` **Volume Mount**: Included in the `/etc/pelican` mount above. #### Log Files (Optional) **Configuration**: To send service logs to a file rather than stdout/stderr, see the [`Logging.LogLocation`](../parameters#Logging-LogLocation) parameter. **Purpose**: Server log files for troubleshooting and monitoring. **Default Path**: Logs are written to stderr/stdout by default, but can be redirected to a file. **Volume Mount**: ``` -v /host/pelican/logs:/var/log/pelican ``` #### Prometheus Monitoring Data (Optional) **Configuration**: To customize the path, see the [`Monitoring.DataLocation`](../parameters#Monitoring-DataLocation) parameter. **Purpose**: Prometheus monitoring data and metrics. **Default Path**: `/var/lib/pelican/monitoring/data` **Volume Mount**: ``` -v /host/pelican/monitoring:/var/lib/pelican/monitoring ``` #### OIDC Client Credentials (If Using OIDC) **Configuration**: To customize paths, see [`OIDC.ClientIDFile`](../parameters#OIDC-ClientIDFile) and [`OIDC.ClientSecretFile`](../parameters#OIDC-ClientSecretFile) parameters. **Purpose**: OIDC client ID and secret for OAuth2/OIDC authentication. **Default Paths**: - Client ID: `/etc/pelican/oidc-client-id` - Client secret: `/etc/pelican/oidc-client-secret` **Volume Mount**: ``` -v /host/pelican/config:/etc/pelican ``` For mounting individual credential files: ``` -v /host/oidc-client-id:/etc/pelican/oidc-client-id -v /host/oidc-client-secret:/etc/pelican/oidc-client-secret ``` ### Origin-Specific Persistent Data In addition to the common data above, Origins need to persist: #### Origin Database **Configuration**: To customize the path, see the [`Origin.DbLocation`](../parameters#Origin-DbLocation) parameter. **Purpose**: Origin-specific operational data and state. **Default Path**: `/var/lib/pelican/origin.sqlite` **Volume Mount**: Included in the `/var/lib/pelican` mount above. #### Origin Data Directory **Configuration**: Use the [`Origin.Exports`](../parameters#Origin-Exports) parameter to define exports. For example, in `/etc/pelican/pelican.yaml`: ``` Origin: Exports: - StoragePrefix: /data FederationPrefix: /my-namespace ``` **Purpose**: The actual data being served by the Origin (if using POSIX storage). **Default Path**: Configured via the [`Origin.Exports`](../parameters#Origin-Exports) parameter. **Volume Mount Example**: ``` -v /host/data:/data ``` #### Example: Running an Origin with Persistent Data ``` docker run -d \ --name pelican-origin \ -p 8444:8444 \ -p 8443:8443 \ -v /host/pelican/config:/etc/pelican \ -v /host/pelican/lib:/var/lib/pelican \ -v /host/pelican/certificates:/etc/pelican/certificates \ -v /host/data:/data \ hub.opensciencegrid.org/pelican_platform/origin:latest \ serve ``` All Origin configuration, including exports and federation URL, should be provided via `/etc/pelican/pelican.yaml` which is mounted into the container. ### Cache-Specific Persistent Data In addition to the common data, Caches need to persist: #### Cache Storage Location **Configuration**: Set [`Cache.StorageLocation`](../parameters#Cache-StorageLocation) to point to the mounted cache storage. In `/etc/pelican/pelican.yaml`: ``` Cache: StorageLocation: /var/cache/pelican ``` **Purpose**: Directory where cached objects are stored. **Default Path**: `/run/pelican/xrootd/cache` **Important**: The default path is in `/run`, which is typically a temporary filesystem. All Cache administrators should change this value to point at their mounted cache disks to ensure cached data persists across container restarts. If this configuration is not changed, cached data will fill up `/run`, potentially causing system instability and the Cache will lose all cached objects on restart. **Volume Mount**: ``` -v /host/cache/storage:/var/cache/pelican ``` **Why it must persist**: Without persistence, the Cache loses all cached objects on restart, increasing load on Origins and slowing down data access due cache misses. #### Example: Running a Cache with Persistent Data ``` docker run -d \ --name pelican-cache \ -p 8444:8444 \ -p 8442:8442 \ -v /host/pelican/config:/etc/pelican \ -v /host/pelican/lib:/var/lib/pelican \ -v /host/pelican/certificates:/etc/pelican/certificates \ -v /host/cache/storage:/var/cache/pelican \ hub.opensciencegrid.org/pelican_platform/cache:latest \ serve ``` Note: All Cache configuration, including federation URL, should be provided via `/etc/pelican/pelican.yaml` which is mounted into the container. ### Director-Specific Persistent Data In addition to the common data, Directors need to persist: #### GeoIP Database (Optional) **Configuration**: To customize paths, see [`Director.GeoIPLocation`](../parameters#Director-GeoIPLocation) and [`Director.MaxMindKeyFile`](../parameters#Director-MaxMindKeyFile) parameters. **Purpose**: MaxMind GeoIP database for geographic routing optimization. **Default Path**: `/var/cache/pelican/maxmind/GeoLite2-City.mmdb` **Volume Mount**: ``` -v /host/pelican/maxmind:/var/cache/pelican/maxmind ``` #### Example: Running a Director with Persistent Data ``` docker run -d \ --name pelican-director \ -p 8444:8444 \ -v /host/pelican/config:/etc/pelican \ -v /host/pelican/lib:/var/lib/pelican \ -v /host/pelican/certificates:/etc/pelican/certificates \ -v /host/pelican/maxmind:/var/cache/pelican/maxmind \ hub.opensciencegrid.org/pelican_platform/director:latest \ serve ``` Note: All Director configuration, including federation URL, should be provided via `/etc/pelican/pelican.yaml` which is mounted into the container. ### Registry-Specific Persistent Data Registries only require persisting the directories/files discussed in the [“Data That Should Persist for All Server Types”](./docker#data-that-should-persist-for-all-server-types) section. #### Example: Running a Registry with Persistent Data ``` docker run -d \ --name pelican-registry \ -p 8444:8444 \ -v /host/pelican/config:/etc/pelican \ -v /host/pelican/lib:/var/lib/pelican \ -v /host/pelican/certificates:/etc/pelican/certificates \ hub.opensciencegrid.org/pelican_platform/registry:latest \ serve ``` All Registry configuration, including federation URL, should be provided via `/etc/pelican/pelican.yaml` which is mounted into the container. ### Using Docker Compose for Persistent Data For production deployments, Docker Compose provides a cleaner way to manage volumes. Here’s an example for an Origin: ``` version: '3.8' services: pelican-origin: image: hub.opensciencegrid.org/pelican_platform/origin:latest container_name: pelican-origin ports: - "8444:8444" - "8443:8443" volumes: - pelican-config:/etc/pelican - pelican-lib:/var/lib/pelican - pelican-certs:/etc/pelican/certificates - /path/to/data:/data command: serve restart: unless-stopped volumes: pelican-config: pelican-lib: pelican-certs: ``` Note: Configuration including exports and federation URL should be provided in a `pelican.yaml` file placed in the `pelican-config` volume. ### Best Practices 1. **Always persist `/etc/pelican` and `/var/lib/pelican`**: These directories contain most of the critical persistent data. 2. **Use named volumes or bind mounts consistently**: Avoid mixing approaches to prevent confusion. 3. **Back up databases regularly**: The SQLite databases in `/var/lib/pelican` should be backed up regularly. 4. **Protect private keys**: The issuer keys and TLS private keys in `/etc/pelican` are sensitive and should be protected with appropriate file system permissions on the host. 5. **Monitor disk usage**: Cache storage and monitoring data can grow over time. Monitor disk usage and configure appropriate limits. 6. **Test container restarts**: After setting up persistent volumes, test that the container can be stopped and restarted without losing data or generating new keys. 7. **Document your volume mounts**: Keep a record of which host directories are mounted to which container paths for each server. --- Source: https://docs.pelicanplatform.org/parameters # Configuration ## Yaml Configuration Pelican’s preferred configuration mechanism is via a [YAML](https://yaml.org/)  file found at `/etc/pelican/pelican.yaml`. Each config value is represented by a key-value pair. Below is an example configuration file with a key-value pair, a nested key-value pair, and a list of key-value pairs. /etc/pelican/pelican.yaml ``` IssuerKeysDirectory: /some/directory OIDC: ClientIDFile: /etc/pelican/oidc-client-id Registry: Institutions: - id: 0 name: institution0 - id: 1 name: institution1 ``` ## Environment Variable Configuration Pelican configuration is typically set via the `pelican.yaml` file, in part because config parameters that have the `object` type cannot be fully represented with environment variables. If, however, there is a corresponding environment variable set in the shell where the command is executed, that will override the configuration provided in the `pelican.yaml` file. Using environment variables to change the configuration means the state of the Pelican component is not easily reproducible, especially if inheriting the environment from multiple parents. Furthermore, the changes will be lost upon restart of the host machine/container. While this approach is useful for testing changes to the configuration, any “permanent” changes should be set via the appropriate `pelican.yaml` file. ### Syntax The environment variable name is constructed from the one-line address of the configuration parameter, e.g., `Level1.Level2`, as follows: 1. Capitalize all letters. 2. Replace the period `.` with an underscore `_`. 3. Prefix with `PELICAN_` This leads to an environment variable name of the form `PELICAN_LEVEL1_LEVEL2`. ### Examples To demonstrate the syntax for setting the environment variable, consider the following examples. To set the logging level for a Pelican component to “Debug” in the `pelican.yaml` file, you would use the following: ``` # /etc/pelican/pelican.yaml Logging: Level: Debug ``` To set the logging level for a Pelican component to “Debug” via an environment variable, you would use ``` export PELICAN_LOGGING_LEVEL=Debug ``` Note that the environment variable is prefixed by `PELICAN_` and that the nested keys are capitalized and separated by underscores (`_`). ## Configurable Parameters Filter by component ​ Filter by component Filter by configuration ​ Filter by configuration ##### ConfigLocations Type:stringSlice Default: `ConfigLocations` provides administrators a way to define a list of directories containing Pelican configuration files. Within a given directory, files are read in lexicographical order, and any keys that are defined in multiple files will take the value from the last file read. Directories are read in the order provided by the list. For example, specifying: `ConfigLocations: ["/configs1", "/configs2"]` will read files first from `/configs1` and then from `/configs2`. If a key is defined in both `/configs1` and `/configs2`, the value from `/configs2` will be used. If `/configs1` contains files `a.yaml` and `b.yaml` where both define the same key, the value from `b.yaml` will be used. Subdirectories of the provided directories are not read. Only the root config file's `ConfigLocations` key is used, and any redefinitions are ignored. \* * * * ##### RuntimeDir Type:filename Default:"" Directory where Pelican writes runtime artifacts such as address files; if unset it defaults to /run/pelican for root, $XDG\_RUNTIME\_DIR/pelican when XDG\_RUNTIME\_DIR is set, or a temporary directory that Pelican cleans up on shutdown. cachedirectororiginregistry * * * ##### Debug Type:bool Default:false \[Deprecated\] To enable debug logging, set ${Logging.Level} to "debug". A bool indicating whether Pelican should emit debug messages in its log. NOTE: this will override whatever is set within your configuration file under Logging.Level! \* * * * ##### TLSSkipVerify Type:bool Default:false When set to true, Pelican will skip TLS verification. This allows a "man in the middle" attack on the connection but can simplify testing. Intended for developers. cachedirectororiginregistry * * * ##### IssuerKey Type:filename Default:$ConfigBase/issuer.jwk Root Default:/etc/pelican/issuer.jwk \[Deprecated\] Use IssuerKeysDirectory instead. A filepath to the file containing a PEM-encoded ECDSA private key which will be used to sign credentials issued by this server. A public key will be derived from this private key and used as the key for token verification by external services. The use of `IssuerKeysDirectory` is preferred as it allows administrators to have more than one signing key. origincacheregistrydirector * * * ##### IssuerKeysDirectory Type:filename Default:$ConfigBase/issuer-keys Root Default:/etc/pelican/issuer-keys A filepath to the directory used for storing one or multiple PEM-encoded ecdsa private keys. The most recent modified private key will be parsed into a JWK and serves as the active private key to sign various JWTs issued by this server. A public JWK will be derived from this private key and used as the key for token verification. \* * * * ##### GeoIPOverrides Type:object Default:none A list of IP addresses whose GeoIP resolution should be overridden with the supplied Lat/Long coordinates (in decimal form). This affects both server ads (for determining the location of origins and caches) and incoming client requests (for determining where a client request is coming from). Configuration takes an IP address (both regular and CIDR) and a Coordinate made up of a lat/long pair in decimal format. For example: ``` GeoIPOverrides: - IP: "123.234.123.234" Coordinate: Lat: 43.073904 Long: -89.384859 - IP: "ABCD::1234/112" Coordinate: Lat: 39.8281 Long: -98.5795 ``` Will result in the IP address "123.234.123.234" being mapped to Madison, WI, and IP addresses in the range ABCD::0000-FFFF will be mapped to a field in Kansas. director * * * ##### DisableHttpProxy Type:bool Default:false \[Deprecated\] A legacy configuration for disabling the client's HTTP proxy. See Client.DisableHttpProxy for new config. client * * * ##### DisableProxyFallback Type:bool Default:false \[Deprecated\] A legacy configuration for disabling the client's proxy fallback. See Client.DisableProxyFallback for new config. client * * * ##### MinimumDownloadSpeed Type:int Default:102400 \[Deprecated\] A legacy configuration for setting the client's minimum download speed. See Client.MinimumDownloadSpeed for new config. client * * * #### Transport * * * ##### Transport.DialerTimeout Type:duration Default:10s Maximum time allowed for establishing a connection to target host. clientregistryorigin * * * ##### Transport.DialerKeepAlive Type:duration Default:30s Maximum time a TCP connection should be kept alive without any activity. clientregistryorigin * * * ##### Transport.MaxIdleConns Type:int Default:30 Maximum number of idle connections that the HTTP client should maintain in its connection pool. clientregistryorigin * * * ##### Transport.IdleConnTimeout Type:duration Default:90s Maximum duration an idle connection should remain open in the connection pool. clientregistryorigin * * * ##### Transport.TLSHandshakeTimeout Type:duration Default:15s Maximum time allowed for the TLS handshake to complete when making an HTTPS connection. clientregistryorigin * * * ##### Transport.ExpectContinueTimeout Type:duration Default:1s Timeout to control how long the client should wait for the "Expect: 100-continue" response from the server before sending the request body. clientregistryorigin * * * ##### Transport.ResponseHeaderTimeout Type:duration Default:10s Maximum time the client should wait for the response headers to be received from the server. clientregistryorigin * * * #### Logging * * * ##### Logging.Level Type:string Default:none Client Default:warn Server Default:info A string defining the log level of the client. Options include (going from most info to least): Trace, Debug, Info, Warn, Error, Fatal, Panic. Log levels are inherited by all components unless explicitly overridden. Levels are case-insensitive. The default logging level for clients is "warn", whereas the default for servers is "info". \* * * * ##### Logging.LogLocation Type:filename Default:none A filename defining a file to write log outputs to, if the user desires. \* * * * ##### Logging.DisableProgressBars Type:bool Default:false A bool defining if progress bars should be enabled or not. client * * * #### Logging.Client * * * ##### Logging.Client.ProgressInterval Type:duration Default:1m Interval at which the client's download progress is logged. client * * * #### Logging.Origin * * * ##### Logging.Origin.Cms Type:string Default:error Trace level of XRootD cluster management service, one of the main XRootD executables. Cms basically is a file (or asset) discovery service. Each server has a cmsd daemon which connect to a master one informing it if a server is available. XRootD asks cms where a file could be found and cms works to report back the server for where the file is located. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. origin * * * ##### Logging.Origin.Scitokens Type:string Default:fatal Trace level of scitokens debug output within XRootD configuration. This entails token management and security credentials within XRootD. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. origin * * * ##### Logging.Origin.Xrd Type:string Default:error Trace level of the eXtended Request Daemon within XRootD, another main XRootD executable. This reports information the XRootD protocol and works with cms. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. origin * * * ##### Logging.Origin.Xrootd Type:string Default:info Trace options for XRootD debug output within XRootD configuration. This prefix is reserved for the xroot protocol, which is the component that sits on sockets and talks to clients as they query file-system info, open files, and read data. This is the protocol for XRootD (like http) and handles connections and requests. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. origin * * * ##### Logging.Origin.Http Type:string Default:error Logging level for the HTTP component of the origin. Increasing to debug will cause the Xrootd daemon to log all headers and requests. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. origin * * * ##### Logging.Origin.Ofs Type:string Default:error Logging level of Xrootd's "Open File System" (ofs) subsystem. The OFS manages the file descriptor table and redirection/ error handling. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. origin * * * ##### Logging.Origin.Oss Type:string Default:error Logging level of Xrootd's "Open Storage System" (oss) subsystem. The OSS manages the interaction with the underlying POSIX storage (open, read, write, close, etc). Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. origin * * * #### Logging.Cache * * * ##### Logging.Cache.Http Type:string Default:error Logging level for the HTTP component of the cache. Increasing to debug will cause the Xrootd daemon to log all headers and requests. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.Lotman Type:string Default:error Trace level of Lotman, the Lot Manager plugin for XRootD cache eviction. This component manages cache eviction policies and storage allocation. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic`. Note that lotman trace levels are additive (e.g., info also includes warning and error). If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.Ofs Type:string Default:error Trace level of XRootD's Open File System. This component cares about files and directories from the administrative perspective. This component is build on top of the Open Storage System component, which deals with things like file creation and reads and writes for files and directories. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.Pfc Type:string Default:info Trace level of XRootD Proxy File Cache (XCache), the caching mechanism used by XRootD. This component entails information for caches/caching within XRootD. This component instantiates its own Open Storage System (OSS) to write local files to. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.Pss Type:string Default:error Trace level of XRootD Proxy System Service. Variables this component reports include: number of remotes file opens, number of opens that failed, number of remote file closes, and number of closes that failed. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.PssSetOpt Type:string Default:error Trace level of XRootD Proxy System Service Set Options. This component reports detailed information about the configuration and operational settings of the Proxy System Service. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.Scitokens Type:string Default:fatal Trace level of scitokens debug output within XRootD configuration. This entails token management and security credentials within XRootD. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.Xrd Type:string Default:error Trace level of the eXtended Request Daemon within XRootD, another main XRootD executable. This reports information the XRootD protocol and works with cms. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * ##### Logging.Cache.Xrootd Type:string Default:error Trace options for XRootD debug output within XRootD configuration. This prefix is reserved for the xroot protocol, which is the component that sits on sockets and talks to clients as they query file-system info, open files, and read data. This is the protocol for XRootD (like http) and handles connections and requests. Accepted values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic` If a non-default value is configured for `Logging.Level`, that level will be inherited unless explicitly overridden here. Levels are case-insensitive. cache * * * #### Federation * * * ##### Federation.DiscoveryUrl Type:url Default:none A URL pointing to the federation's metadata discovery host. NOTE: this does not work if the url contains a path! \* * * * ##### Federation.DirectorUrl Type:url Default:none A URL indicating where a director service is hosted. clientorigincacheregistry * * * ##### Federation.RegistryUrl Type:url Default:none A URL indicating where the namespace registry service is hosted. clientdirectororigincache * * * ##### Federation.JwkUrl Type:url Default:none A URL indicating where the JWKS for the Federation is hosted. \* * * * ##### Federation.TopologyUrl Type:url Default:none A URL for the top level OSG Topology location (a legacy integration). This URL is needed to retrieve authorization file information. origincache * * * ##### Federation.TopologyNamespaceUrl Type:url Default:none A URL containing namespace information for origins and caches configured via the OSG Topology application (a legacy integration). The URL should point to the hosted namespace.json. directorregistry * * * ##### Federation.TopologyDowntimeUrl Type:url Default:none A URL for determining OSG topology server downtime information. The result of querying this URL is an XML file containing downtime information. director * * * ##### Federation.TopologyReloadInterval Type:duration Default:none The frequency, in minutes, that topology should be reloaded. directorregistry * * * ##### Federation.BrokerUrl Type:url Default:none The URL of the broker endpoint used by the origin. If left unset, it will be populated by the federation metadata discovery. origin * * * #### Client * * * ##### Client.StoppedTransferTimeout Type:duration Default:100s A timeout indicating when a "stopped transfer" event should be triggered. client * * * ##### Client.SlowTransferRampupTime Type:duration Default:100s A duration indicating the ramp up period for a slow transfer. client * * * ##### Client.SlowTransferWindow Type:duration Default:30s A duration indicating the sliding window over which to consider transfer speeds for slow transfers. client * * * ##### Client.DisableHttpProxy Type:bool Default:false A bool indicating whether the client's HTTP proxy should be disabled. When false (the default), the client uses an HTTP proxy if any of the standard proxy environment variables are set to a non-empty value: `http_proxy`, `HTTP_PROXY`, `https_proxy`, or `HTTPS_PROXY`. Setting this parameter to true disables proxy usage regardless of those environment variables. client * * * ##### Client.WorkerCount Type:int Default:5 An integer indicating the number of file transfer tasks that should be executed in parallel. client * * * ##### Client.DisableProxyFallback Type:bool Default:false A bool indicating whether the a proxy fallback should be used by the client. client * * * ##### Client.DirectorRetries Type:int Default:5 A positive integer indicating the number of retries a client should attempt when contacting a non-responsive Director. Each retry will happen after a delay of 3\*(1 + retries attempted) to give the Director time to recover. Some randomness is also added to this interval to avoid the potential for thundering herd problems. Plugin clients will retry twice the configured value because plugin failure is considered more "costly". client * * * ##### Client.MinimumDownloadSpeed Type:int Default:102400 The minimum speed (in bytes per second) allowed for a client download before an error is thrown. client * * * ##### Client.CredentialFile Type:filename Default:none Override the default path to the client credential file used for token generation. When set, the client uses this file instead of the default credential location. client * * * ##### Client.PreferredCaches Type:stringSlice Default:none A list of preferred cache hostname/ports the Pelican client/plugin should use when interacting with a remote object. There are two configuration options: - A list containing `+` as its last element - A list with no `+` When `+` is last in the list, the client should _first_ try the provided caches, then fall back to any discovered via the Director. If no `+` is included, the client should try _only_ the preferred caches and fail if none is able to provide the object. A `+` anywhere else in the list will generate an error. > Use of this configuration, especially when omitting the `+`, _bypasses the Director and any of its potential logic for cache selection_. As such, this configuration should only be used for testing or for preferring an on- or near-premises cache. Caches are generally tried in the order they're presented. For example, the configuration: ``` Client: PreferredCaches: ["https://cache1.com:8443", "https://cache2.com:8443", "+"] ``` should result in trying `cache1`, then `cache2`, and finally any caches discovered by the Director (if needed). When set via the `PELICAN_CLIENT_PREFERREDCACHES` environment variable, caches can be space-separated or comma-separated, e.g.: `PELICAN_CLIENT_PREFERREDCACHES="https://cache1.com:8443 https://cache2.com:8443 +"` or `PELICAN_CLIENT_PREFERREDCACHES="https://cache1.com:8443,https://cache2.com:8443,+"` Surrounding quotes (both single and double) are automatically trimmed from the entire value and from individual elements. client * * * #### ClientAgent * * * ##### ClientAgent.DbLocation Type:filename Default:"" The filepath to the SQLite database used by the client API server for persisting transfer job state, history, and recovery information. If not specified, defaults to `~/.pelican/client-agent.db`. Set this to an empty string to disable database persistence entirely and run the client API server in memory-only mode. client * * * ##### ClientAgent.HistoryRetentionDays Type:int Default:30 The number of days to retain completed job history in the database before automatic pruning. Historical records older than this threshold will be deleted during the daily maintenance cycle. Set to 0 to disable automatic pruning (history will be retained indefinitely). client * * * ##### ClientAgent.MaxConcurrentJobs Type:int Default:5 The maximum number of concurrent transfer jobs that the client API server can process simultaneously. This limits resource usage and prevents overwhelming the system with too many parallel transfers. Set to 0 or negative to use the default value (5). client * * * ##### ClientAgent.PidFile Type:filename Default:"" The filepath to the PID file used by the client agent daemon process. If not specified, defaults to `~/.pelican/client-agent.pid`. The PID file is used to track the running daemon process and prevent multiple instances from starting simultaneously. client * * * ##### ClientAgent.Socket Type:filename Default:"" The filepath to the Unix domain socket used by the client API server for inter-process communication. Clients connect to this socket to submit transfer jobs and query their status. If not specified, the default location is `~/.pelican/client-api.sock`. client * * * ##### ClientAgent.IdleTimeout Type:duration Default:10m The duration of inactivity (no active jobs or requests) after which the client API server will automatically shut down to conserve resources. Set to 0 to disable automatic shutdown. When running as a daemon spawned by object transfer commands, the server will shut down after this period of inactivity. The server will NOT shut down while there are active transfer jobs in progress. This is particularly useful for reducing resource usage when the client API server is automatically started by commands using the `--async` flag. client * * * #### Origin * * * ##### Origin.DbLocation Type:filename Default:$ConfigBase/origin.sqlite Root Default:/var/lib/pelican/origin.sqlite A filepath to the intended location of the origin's database. origin * * * ##### Origin.Url Type:url Default:https://${Server.Hostname}:${Origin.Port} The origin's configured URL, as reported to XRootD. This is the file transfer endpoint for the origin. origin * * * ##### Origin.Port Type:int Default:8443 The TCP port to be used by the origin service for serving files. If set to 0, then a random open port will be used. origin * * * ##### Origin.Exports Type:object Default:none A list describing the origin's exports. Each item in the list describes a single namespace the origin exports: - StoragePrefix: The relevant path from the object store, e.g. for posix /my/dir - FederationPrefix: The namespace prefix that data from StoragePrefix is made available under within the federation - IssuerUrls: A list of URLs that token requests to the federation prefix can use as issuers. These issuer URLs are used to craft the Origin's Scitokens configuration file. If unset, the Origin will fall back to its own external web URL and assume its server keys are also used for minting data access tokens. When clients need to bootstrap access tokens using OAuth2 flows, they'll use the first URL in this list for bootstrapping. If no URLs are provided but a defined namespace capability for the export requires tokens (`Reads`, `Writes`), the derived value from `${Server.IssuerUrl}` will be used. - Capabilities: A list of the capabilities the origin is willing to support for the given export. Capabilities include: \["Reads", "PublicReads", "Writes", "Listings", "DirectReads"\] where each of these has the same effect as the corresponding "Origin.Enable\*" configuration, except scoped to the given export. If "PublicReads" is included, "Reads" is inferred. - SentinelLocation: A filename under `StoragePrefix` path for Pelican to check the storage directory exists and is correctly mounted. The value must be a file and contain no directory. Leave it empty to skip the check. You should always choose a distinct name for `SentinelLocation`. It should not be reused for other servers. If running in a containerized environment it should not be the name of the underlying physical host as that may change and lead to confusion. You need to manually create a file under path to `StoragePrefix` with the same name as `SentinelLocation`. Note that this parameter is only available for POSIX and S3 backends. - AuthorizationTemplates: \[OPTIONAL\] Per-export authorization templates that override the global `Issuer.AuthorizationTemplates` for this export's namespace. When set, only these templates are used for scope calculation in this namespace; the global templates are ignored entirely (no merging). The template format is the same as `Issuer.AuthorizationTemplates`. Example: ``` Origin: Exports: - StoragePrefix: /home/foo/bar FederationPrefix: /demo/project Capabilities: ["Reads", "PublicReads", "Writes", "Listings", "DirectReads"] SentinelLocation: demoproject_origin_A IssuerUrls: ["https://issuer1.example.com", "https://issuer2.example.com"] AuthorizationTemplates: - actions: ["read", "modify"] prefix: /home/$USER - actions: ["read"] prefix: /data/$GROUP groups: ["/physics"] ``` If Origin.StorageType == "s3", the following additional fields are available: - S3Bucket: \[OPTIONAL\] See `Origin.S3Bucket` for details - S3AccessKeyfile: \[OPTIONAL\] See `Origin.S3AccessKeyfile` for details - S3SecretKeyfile: \[OPTIONAL\] See `Origin.S3SecretKeyfile` for details If Origin.StorageType == "globus", the following additional fields are available: - GlobusCollectionID: \[REQUIRED\] See `Origin.GlobusCollectionID` for details - GlobusCollectionName: \[OPTIONAL\] See `Origin.GlobusCollectionName` for details If Origin.StorageType == "xroot", the following additional field is available: - XrootServiceUrl: \[REQUIRED\] See `Origin.XrootServiceUrl` for details origin * * * ##### Origin.StorageType Type:string Default:posix The type of storage underpinning the origin. Currently supported types are "posix", "https", "s3", "globus", and "xroot". origin * * * ##### Origin.FederationPrefix Type:string Default:none The namespace prefix of the origin's contents within the federation. NOTE: This config option is incompatible with multiple exports defined via `Origin.Exports` and is ignored when the origin exports multiple prefixes. origin * * * ##### Origin.StoragePrefix Type:string Default:none A string indicating the path to the volume exported by an origin's underlying storage. For example, if the origin has a StorageType of "posix", this constitutes the path on disk exported by the origin for the federation. If the origin has a StorageType of "s3", this value is not currently used. NOTE: This config option is incompatible with multiple exports defined via `Origin.Exports` and is ignored when the origin exports multiple prefixes. origin * * * ##### Origin.ExportVolumes Type:stringSlice Default: A list of docker-style export volumes for the origin. Each item in the list describes a single volume the origin exports. This configuration is meant mostly to be used by passing the -v flag from the command line. Paths exported with this configuration will inherit the origin's abilities, so individual export configurations are not possible. origin * * * ##### Origin.EnablePublicReads Type:bool Default:false A boolean indicating whether the origin permits reads without valid authorization. When false, reads from the origin will require a properly-scoped authorization token signed by the origin's issuer. NOTE: This config option is meant to configure an _origin's_ capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration. origin * * * ##### Origin.EnableReads Type:bool Default:true A boolean indicating whether the origin permits any reads. When false, the origin may still allow writes. NOTE: This config option is meant to configure an _origin's_ capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration. origin * * * ##### Origin.EnableWrites Type:bool Default:true A boolean indicating whether the origin permits writes. All writes require authorization. NOTE: This config option is meant to configure an _origin's_ capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration. origin * * * ##### Origin.EnableListings Type:bool Default:true A boolean indicating whether the origin permits object listings. When true, clients can list the contents of the origin. NOTE: This config option is meant to configure an _origin's_ capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration. origin * * * ##### Origin.EnableDirectReads Type:bool Default:true A boolean indicating whether the origin permits direct reads. When true, the origin indicates that it is willing to interact directly with clients. When false, the origin is indicating it is only willing to interact with clients via a cache service. NOTE: This config option is meant to configure an _origin's_ capabilities, but can be used to configure a namespace when the origin exports only a single prefix or when every exported namespace should inherit the same configuration. origin * * * ##### Origin.ExportVolume Type:string Default:none \[Deprecated\] Origin.ExportVolume is being deprecated and will be removed in a future release. It is replaced by Origin.ExportVolumes. A path to the volume exported by an origin. origin * * * ##### Origin.DefaultChecksumTypes Type:stringSlice Default:crc32c A list of checksum algorithms that the origin will automatically compute and cache in extended attributes, even if not explicitly requested by the client. This allows the server to pre-compute commonly needed checksums for performance. Supported values are "md5", "sha1", "crc32", and "crc32c". origin * * * ##### Origin.RunLocation Type:filename Default:$XDG\_RUNTIME\_DIR/pelican/origin Root Default:/run/pelican/xrootd/origin A directory where temporary configurations will be stored for the XRootD daemon started by the origin. For non-root servers, if $XDG\_RUNTIME\_DIR is not set, a temporary directory will be created (and removed on shutdown). origin * * * ##### Origin.EnableAtomicUploads Type:bool Default:false A boolean that, when true, enables atomic uploads for the Origin. When true, the origin will enable atomic uploads. When false, the origin will not enable atomic uploads. Atomic uploads are only available for origins that have a StorageType of "posix". Enabling this on bare metal Pelican installations requires `xrootd-s3-http` `v0.6.4` or later. This can be verified by using `rpm -q xrootd-s3-http` to check the version. Containerized Pelican installations greater than `v7.24.0` will have the required dependencies already installed. Atomic uploads enables the use of a temporary location to store partially-written files until they are committed. If an upload fails or is cancelled partway through, the partially-written file is automatically cleaned up rather than leaving an incomplete object visible in the namespace. Without this, a PUT that fails mid-transfer would burn a name in the namespace, leaving behind a corrupt or truncated file that clients may attempt to read. For example, if a client uploads a 1 GiB file and the connection drops at 500 MiB, the origin will automatically remove the partial file from the temporary location. Without atomic writes, the 500 MiB fragment would remain at the intended path and be served to any client that requests it. The atomic upload mechanism uses `rename(2)` to move completed files from Origin.UploadTempLocation into the export's StoragePrefix. Because POSIX filesystems do not support renaming files across filesystem boundaries, Origin.UploadTempLocation must reside on the same filesystem as every configured export's StoragePrefix. Pelican will reject the configuration at startup if a cross-filesystem mismatch is detected. origin * * * ##### Origin.NamespacePrefix Type:string Default:none \[Deprecated\] Origin.NamespacePrefix is being deprecated and will be removed in a future release. It's configuration is being replaced by either Origin.Exports.FederationPrefix or by Origin.FederationPrefix. Note that Origin.FederationPrefix is incompatible with multiple exports and requires that the origin exports only a single path. The filepath prefix at which an origin's contents are made globally available, eg /pelican/PUBLIC. origin * * * ##### Origin.EnableWrite Type:bool Default:true \[Deprecated\] Origin.EnableWrite is being deprecated and will be removed in a future release. It is replaced by Origin.EnableWrites. A boolean indicating if an origin allows write access. origin * * * ##### Origin.EnableFallbackRead Type:bool Default:false \[Deprecated\] Origin.EnableFallbackRead is being deprecated and will be removed in a future release. It is replaced by Origin.EnableDirectReads. Set to `true` if the origin permits clients to directly read from it when no cache service is available. origin * * * ##### Origin.Multiuser Type:bool Default:false Root Default:true A bool indicating whether an origin is "multiuser", ie whether the underlying XRootD instance must be configured in multi user mode. origin * * * ##### Origin.MultiuserMinID Type:int Default:1000 The minimum UID/GID the multiuser origin will switch to when performing filesystem operations on behalf of a user. Any resolved user or group ID below this threshold is rejected, preventing accidental operations as root or other system accounts. Set to 0 to disable the guard entirely (not recommended). origin * * * ##### Origin.MultiuserUmask Type:int Default:\-1 The file-creation mask (umask) applied to the process at startup when multiuser mode is enabled. This umask controls which permission bits are masked off from the requested mode for all filesystem operations. The umask is set once when the multiuser filesystem is initialised and left in place for the lifetime of the process. A value of -1 (the default) means the process inherits the umask from its parent and does not change it. A value of 0 means the exact permissions requested by the caller are applied (no bits masked). Specify the value in YAML octal notation (e.g. 0o0022 removes group-write and other-write). Common values: -1 (inherit), 0 (no masking), 0o0022 (standard), 0o0077 (restrictive). origin * * * ##### Origin.EnableCmsd Type:bool Default:true A bool indicating whether the origin should enable the `cmsd` daemon. origin * * * ##### Origin.EnableMacaroons Type:bool Default:false A bool indicating whether the origin allows clients to authenticate using macaroons. origin * * * ##### Origin.Concurrency Type:int Default:none This value represents the maximum number of permitted IO operations in-progress per second. When this value is set, it enables the [XRootD throttling plugin's](https://github.com/xrootd/xrootd/tree/master/src/XrdThrottle) 'concurrency' throttle directive. For example, if there are two simultaneous read requests and each takes 1 second to complete, the concurrency is 2. Setting a concurrency limit of 1 would cause one of the requests to be delayed until the other completes. For POSIX Origins, this value should be approximately: - (HDDs) ~2x the number of underlying disks - (NVMe/SSDs) ~10x the number of cores available to the Origin origin * * * ##### Origin.ConcurrencyDegradedThreshold Type:int Default:90 The percentage of permissible concurrency that indicates when the Origin should enter the "degraded" state. For example, if `${Origin.Concurrency}` is set to 100 and `${Origin.ConcurrencyDegradedThreshold}` is set to 80, then the Origin will enter the degraded state when active IO exceeds 80. The Director will continue to deprioritize the Origin while in the degraded state until the active IO drops below the threshold. This setting only has effect when `${Origin.Concurrency}` is set. origin * * * ##### Origin.DirectorTest Type:bool Default:true A bool indicating whether the director should send file transfer tests to the origin. If `Origin.StorageType` is set to values other than `POSIX`, this parameter is set to false. origin * * * ##### Origin.SelfTest Type:bool Default:true A bool indicating whether the origin should perform self health checks. If `Origin.StorageType` is set to values other than `POSIX`, this parameter is set to false. origin * * * ##### Origin.SelfTestInterval Type:duration Default:15s The interval of which the origin starts a new file transfer test to itself. origin * * * ##### Origin.EnableOIDC Type:bool Default:false Indicate whether the origin should allow users to login to the admin website via OAuth2/OIDC with third-party authentication providers such as CILogon. If set to true, it is recommended that you also set `Server.UIAdminUsers` to a list of users to give admin privilege. This is because origin admin website doesn't have a public, non-admin view, and an empty AdminUsers list will lead to "permission denied" error for all users logged into origin admin website via OAuth. origin * * * ##### Origin.EnableBroker Type:bool Default:false Indicate whether the origin should utilize the broker service to avoid the need for incoming connections. origin * * * ##### Origin.EnableIssuer Type:bool Default:false Enable the built-in issuer daemon for the origin. origin * * * ##### Origin.IssuerMode Type:string Default:oa4mp Select the issuer implementation to use when Origin.EnableIssuer is true. "embedded" uses the fosite-based OIDC provider built directly into Pelican, which requires no external processes. This is under canary testing and is expected to become the default in a future release. "oa4mp" uses the external Java-based OA4MP issuer (legacy, will be deprecated in a future release). When switching between modes, state is not migrated between the two databases. Note: the Issuer.AccessTokenLifetime, Issuer.AuthorizationCodeLifetime, Issuer.IDTokenLifetime, and Issuer.RefreshTokenLifetime settings only take effect when this is set to "embedded". origin * * * ##### Origin.ScitokensRestrictedPaths Type:stringSlice Default: This parameter is used to configure [XRootD's SciTokens authorization plugin](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens). Any restrictions on the paths that the issuer can authorize inside their namespace. This is meant to be a mechanism to help with transitions, where the underlying storage is setup such that an issuer's namespace contains directories that should not be managed by the issuer. origin * * * ##### Origin.ScitokensMapSubject Type:bool Default:false This parameter is used to configure [XRootD's SciTokens authorization plugin](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens). If set to `true`, the contents of the token's `sub` claim will be copied into the XRootD username. When `Origin.Multiuser` is also set to `true`, this will allow XRootD to read and write files using the Unix username specified in the token. origin * * * ##### Origin.ScitokensDefaultUser Type:string Default:none This parameter is used to configure [XRootD's SciTokens authorization plugin](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens). If set, then all authorized operations will be performed under the provided username when interacting with the file system. This is useful when all files owned by an issuer should be mapped to a particular Unix user account. origin * * * ##### Origin.ScitokensUsernameClaim Type:string Default:none This parameter is used to configure [XRootD's SciTokens authorization plugin](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens). If set, then the provided claim will be used to determine the XRootD username, and it will override the `Origin.ScitokensMapSubject` and `Origin.ScitokensDefaultUser` parameters. origin * * * ##### Origin.ScitokensGroupsClaim Type:string Default:wlcg.groups The JWT claim to use for extracting group information from tokens during authorization. This is used by the Pelican authorization system to map token claims to local groups for access control decisions. Common values are "wlcg.groups" (WLCG tokens) or "groups" (generic OAuth2 tokens). origin * * * ##### Origin.ScitokensNameMapFile Type:string Default:none This parameter is used to configure [XRootD's SciTokens authorization plugin](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens). If set, then the referenced file is parsed as a JSON object and the specified mappings are applied to the username inside the XRootD framework. See the [XrdSciTokens documentation](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens#mapfile-format) for more information on the mapfile's format. origin * * * ##### Origin.UserMapfileRefreshInterval Type:duration Default:1m The interval at which the origin will check for updates to the user mapfile. When set, the origin will periodically reload the mapfile if it has been modified on disk. This allows administrators to update user mappings without restarting the origin server. Set to 0 to disable automatic refresh. origin * * * ##### Origin.XRootDPrefix Type:string Default:origin The directory prefix for the XRootD origin configuration files. origin * * * ##### Origin.EnableVoms Type:bool Default:true Enable X.509 / VOMS-based authentication. This allows HTTP clients to present X.509 client credentials in order to authenticate. The configuration of the authorization for these clients must be done by the admin; Pelican does not support automatic VOMS authorization configuration. origin * * * ##### Origin.EnableDirListing Type:bool Default:false \[Deprecated\] Origin.EnableDirListing is being deprecated and will be removed in a future release. It is replaced by Origin.EnableListings. Allows the origin to enable collection listings. Needs to be enabled for recursive downloads to work properly and for directories to be visible. origin * * * ##### Origin.Mode Type:string Default:posix \[Deprecated\] Origin.Mode is being deprecated and will be removed in a future release. It is replaced by Origin.StorageType. The backend mode to be used by an origin. Current values that can be selected from are either "posix" or "s3". origin * * * ##### Origin.S3ServiceName Type:string Default:none \[Deprecated\] Origin.S3ServiceName was previously used in part to determine an export's FederationPrefix, but upstream changes no longer rely on this value. As of Pelican `7.7.0`, setting this value no longer has any effect. AWSv4 signatures used by S3 servers to handle authentication now hardcode "s3" as their service name. When constructing signed URLs for S3, this value is used as a part of the signature. It is almost always "s3". For more information about S3 service names, see https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html origin * * * ##### Origin.S3Region Type:string Default:none Objects in S3 are associated with a "region", which is specifically a part of AWS's infrastructure. Often, S3 endpoints that are not provided by Amazon use "us-east-1" as their region. This value is used when constructing signed URLs for getting authenticated objects from a bucket. For more information about how Amazon uses regions, see https://docs.aws.amazon.com/general/latest/gr/s3.html This value is REQUIRED for S3 origins. origin * * * ##### Origin.S3Bucket Type:string Default:none Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use `Origin.Exports` Objects in S3 are stored in "buckets", which have unique names at each S3 service URL (ie the URL that provides access to your objects). Setting a bucket restricts the origin to only serving objects from that bucket. However, if the origin is meant to export all of the buckets associated with a given service URL, this value can be left unset _IF_ all of those buckets are public and the origin is using path-style URLS. When this is the case, objects can be fetched from the origin at the path `/federation/prefix/bucket-name/object-name`. origin * * * ##### Origin.S3ServiceUrl Type:string Default:none The URL that provides API access your objects. When the S3 instance is hosted by Amazon, this is often "https://s3.us-east-1.amazonaws.com". This value is REQUIRED for S3 origins. origin * * * ##### Origin.S3AccessKeyfile Type:filename Default:none Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use `Origin.Exports` A path to a file containing an S3 access keyfile (also sometimes called an API key) for authenticated buckets when an origin is run in S3 mode. This value is OPTIONAL for S3 origins, and only applies when an exported bucket requires authentication. It should not be used if the bucket is public or if the origin is meant to export all public buckets from the S3 service URL. origin * * * ##### Origin.S3SecretKeyfile Type:filename Default:none Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use `Origin.Exports` A path to a file containing an S3 secret keyfile for authenticated buckets when an origin is run in S3 mode. This value is OPTIONAL for S3 origins, and only applies when an exported bucket requires authentication. It should not be used if the bucket is public or if the origin is meant to export all public buckets from the S3 service URL. origin * * * ##### Origin.S3UrlStyle Type:string Default:path The style of S3 urls used by the service URL host. This can be either "path" if objects are fetched at `//` or "virtual" if objects are fetched at `./`. This value is REQUIRED for S3 origins, but defaults to "path" if not set. origin * * * ##### Origin.HttpServiceUrl Type:string Default:none If Origin.StorageType is set to `https`, the service URL is used as the base for requests to the backend. To generate the request, the Origin.FederationPrefix is removed from the object name, then the result is joined with the service URL and storage prefix. For example, if one sets `Origin.HTTPServiceUrl=https://example.com`, `Origin.StoragePrefix=/testfiles` and `Origin.FederationPrefix=/foo`, then a request for an object named `/foo/bar` will generate a request to https://example.com/testfiles/bar. origin * * * ##### Origin.HttpAuthTokenFile Type:filename Default:none When set, all requests from the origin to the http backend will include the contents of the file as a bearer token in the Authorization header. If the origin backend is configured with the `globus` storage type, any value set here will be overridden with the filepath to the file ending in `.tok` found in the $(Origin.GlobusConfigLocation)/tokens directory origin * * * ##### Origin.XRootServiceUrl Type:string Default:none When the origin is configured to export another XRootD storage backend by setting `Origin.StorageType = xroot`, the `XRootServiceUrl` is used as the base for `root` protocol requests and should point at the upstream XRootD server. origin * * * ##### Origin.GlobusCollectionID Type:string Default:none Note: This value is only for setting up an origin that exports one Globus collection. For multiple exports, use `Origin.Exports` Required if `Origin.StorageType == "globus"` AND `Origin.Exports` is not set. Globus stores objects in [Collections](https://docs.globus.org/guides/overviews/collections-and-endpoints/#collection). The unique identifier of a Collection is the Collection UUID. You can find the Collection UUID at the bottom of your Collection's overview page. origin * * * ##### Origin.GlobusCollectionName Type:string Default:none Note: This value is only for setting up an origin that exports one storage prefix. For multiple exports, use `Origin.Exports` An optional human-readable name to describe the Collection. This should set to the "Display Name" of your Collection in Globus. It is recommended to set this name; otherwise the UUID will be used as the Collection name. origin * * * ##### Origin.GlobusClientIDFile Type:filename Default:none Required if `Origin.StorageType == "globus"` and `OIDC.Issuer` is not Globus A filepath to the file containing the Globus ClientID. You need to create a new project and register a new confidential OAuth client: https://app.globus.org/settings/developers/registration/confidential\_client/select-project Once registered, in the client page, find "Client UUID" and copy it to a file. If the server uses Globus as the OIDC authentication provider, Pelican will use OIDC configuration for Globus storage access. origin * * * ##### Origin.GlobusClientSecretFile Type:filename Default:none Required if `Origin.StorageType == "globus"` and `OIDC.Issuer` is not Globus A filepath to the file containing the Globus ClientID. You need to create a new project and register a new confidential OAuth client following the instruction in `Origin.GlobusClientIDFile`. In the client page, you need to "Add Client Secret" and copy the secret to a file. If the server uses Globus as the OIDC authentication provider, Pelican will use OIDC configuration for Globus storage access. origin * * * ##### Origin.GlobusConfigLocation Type:filename Default:$XDG\_RUNTIME\_DIR/pelican/xrootd/origin/globus Root Default:/run/pelican/xrootd/origin/globus A filepath to the folder containing Globus config and access tokens origin * * * ##### Origin.GlobusTransferTokenFile Type:filename Default:none When set, all requests from the Globus backend to the Globus Transfer API will include the contents of the file as a bearer token in the authorization header. Any value set here will be overridden with the filepath to the file ending in `.transfer.tok` found in the $(Origin.GlobusConfigLocation)/tokens directory origin * * * ##### Origin.FedTokenLocation Type:filename Default:$ConfigBase/origin-fed-token A path to the file containing a token issued by the federation's issuer. This token may be consumed by other federation services to prove the origin's membership in the federation. For example, a third-party copy from one Origin to another that serves a namespace without DirectReads enabled may require a token to prove the origin's membership in the federation. origin * * * ##### Origin.SelfTestMaxAge Type:duration Default:1h Defines the maximum allowed time since the last successful self-test. If an Origin fails to complete a new successful test within this period, and Xrootd.AutoShutdownEnabled is true, the Origin will automatically shut down. This acts as a fail-safe mechanism. origin * * * ##### Origin.EnableDiskUsageCalculation Type:bool Default:false Enable periodic calculation of disk usage for origin exports. When enabled, the origin will periodically scan each export to count objects and sum sizes, and report these metrics via Prometheus. For POSIX backends, this performs a local directory walk. For Ceph-backed POSIX storage with extended attributes available, the xattr-based fast path is used. This feature is disabled by default to avoid unnecessary disk I/O. origin * * * ##### Origin.DiskUsageCalculationInterval Type:duration Default:24h How often to calculate disk usage for origin exports. This only has effect when Origin.EnableDiskUsageCalculation is true. origin * * * ##### Origin.DiskUsageCalculationRateLimit Type:int Default:1000 Maximum number of filesystem operations per second when calculating disk usage. This rate limit helps prevent the disk usage calculation from impacting normal origin operations. This only has effect when Origin.EnableDiskUsageCalculation is true. origin * * * #### Origin.SSH * * * ##### Origin.SSH.Host Type:string Default:none The hostname or IP address of the remote SSH server for the SSH backend. When Origin.StorageType is set to "ssh", this parameter is required. origin * * * ##### Origin.SSH.Port Type:int Default:22 The SSH port to connect to on the remote server. origin * * * ##### Origin.SSH.User Type:string Default:none The SSH username to use for authentication. origin * * * ##### Origin.SSH.AuthMethods Type:stringSlice Default:publickey,agent,keyboard-interactive,password A list of SSH authentication methods to try, in order. Supported methods are: - "publickey": Use SSH public key authentication (requires PrivateKeyFile) - "password": Use password authentication (requires PasswordFile) - "keyboard-interactive": Use keyboard-interactive authentication (allows admin to complete via WebSocket) - "agent": Use the SSH agent for authentication If not specified, defaults to trying: publickey, agent, keyboard-interactive, password origin * * * ##### Origin.SSH.PasswordFile Type:filename Default:none Path to a file containing the SSH password. The password should be the only content of the file. This file should have restricted permissions (e.g., 0600). Used when "password" is in the AuthMethods list. origin * * * ##### Origin.SSH.PrivateKeyFile Type:filename Default:none Path to the SSH private key file for public key authentication. Used when "publickey" is in the AuthMethods list. Supports RSA, ECDSA, and Ed25519 keys. origin * * * ##### Origin.SSH.PrivateKeyPassphraseFile Type:filename Default:none Path to a file containing the passphrase for an encrypted SSH private key. This file should have restricted permissions (e.g., 0600). Only needed if the private key is encrypted. origin * * * ##### Origin.SSH.KnownHostsFile Type:filename Default:none Path to the SSH known\_hosts file for host key verification. If not specified, defaults to ~/.ssh/known\_hosts. The remote host must be present in this file for the connection to succeed (unless Origin.SSH.AutoAddHostKey is true). origin * * * ##### Origin.SSH.AutoAddHostKey Type:bool Default:false Automatically add unknown host keys to the known\_hosts file. When false (default for server mode), the connection will fail if the remote host key is not already in the known\_hosts file. This provides better security by preventing man-in-the-middle attacks. Set to true only in test/development environments where the risk is acceptable. origin * * * ##### Origin.SSH.PelicanBinaryPath Type:filename Default:none Path to the Pelican binary to transfer to the remote host. If not specified, the currently running Pelican executable is used. This must be compatible with the remote host's OS and architecture. origin * * * ##### Origin.SSH.RemotePelicanBinaryDir Type:string Default:none Directory on the remote host where the Pelican binary should be placed. If not specified, a temporary directory is created on the remote host. origin * * * ##### Origin.SSH.RemotePelicanBinaryOverrides Type:stringSlice Default: A list of platform-specific binary overrides for the remote host. Format: "os/arch=/path/to/binary" Example: \["linux/amd64=/opt/pelican/pelican", "linux/arm64=/opt/pelican/pelican-arm64"\] Use this when the remote host already has Pelican installed, or when you need to use a different binary than the one that would be transferred automatically. The platform is detected by running "uname -s" and "uname -m" on the remote host. origin * * * ##### Origin.SSH.MaxRetries Type:int Default:5 Maximum number of times to retry the SSH connection if it fails. After exceeding this limit, the origin will fail to start. origin * * * ##### Origin.SSH.ConnectTimeout Type:duration Default:30s Timeout for establishing the SSH connection. origin * * * ##### Origin.SSH.KeepaliveInterval Type:duration Default:5s How often to send SSH keepalive packets to verify the connection is still alive. origin * * * ##### Origin.SSH.KeepaliveTimeout Type:duration Default:20s Maximum time to wait without receiving a keepalive response before considering the connection dead and shutting down. Both the SSH connection and the HTTP connection to the helper are monitored. origin * * * ##### Origin.SSH.ChallengeTimeout Type:duration Default:1m Timeout for individual SSH authentication challenges (password prompts, keyboard-interactive questions). This is the maximum time to wait for user input on a single authentication challenge. The overall authentication timeout is controlled by Origin.SSH.ConnectTimeout. origin * * * ##### Origin.SSH.ProxyJump Type:string Default:none Jump host(s) for SSH ProxyJump (similar to ssh -J flag). Format: \[user@\]host\[:port\] for a single jump host. For chained jumps, use comma-separated list: \[user@\]host1\[:port1\],\[user@\]host2\[:port2\] This allows connecting to a remote host through one or more intermediate hosts. origin * * * ##### Origin.SSH.SessionEstablishTimeout Type:duration Default:5m Maximum time allowed to establish a complete working SSH session. This includes connecting, authenticating, detecting the remote platform, transferring the helper binary (if needed), and starting the helper process. If this timeout is exceeded, the connection attempt is aborted and retried. This is an end-to-end timeout that bounds all session establishment operations. origin * * * ##### Origin.SSH.TunnelCallback Type:bool Default:false When true, use SSH remote port forwarding to tunnel the helper's callback connections back to the origin instead of requiring direct network connectivity from the remote SSH host to the origin. This is useful when the remote SSH host cannot reach the origin's web server directly (e.g., behind a firewall or NAT). When enabled, the origin opens an SSH remote port forward on the remote host (127.0.0.1 with a randomly-allocated port) and runs a local TCP proxy that forwards tunneled connections to the origin's web server. The helper is instructed to connect to the forwarded port on localhost instead of the origin's external URL. origin * * * #### LocalCache * * * ##### LocalCache.RunLocation Type:filename Default:$XDG\_RUNTIME\_DIR/pelican/localcache Root Default:/run/pelican/localcache The directory for the runtime files of the local cache. localcache * * * ##### LocalCache.DataLocation Type:filename Default:$PELICAN\_LOCALCACHE\_RUNLOCATION/cache The directory for the location of the cache data files - this is where the actual data in the cache is stored for the local cache. localcache * * * ##### LocalCache.Socket Type:filename Default:$PELICAN\_LOCALCACHE\_RUNLOCATION/cache.sock The location of the socket used for client communication for the local cache. localcache * * * ##### LocalCache.Size Type:string Default:0 The maximum size of the local cache. If not set, it is assumed the entire device can be used. This parameter can be provided with units (e.g., 20GB, 150MB); if no unit is provided, then it is assumed to be in bytes. localcache * * * ##### LocalCache.HighWaterMarkPercentage Type:int Default:89 A percentage value where the cache cleanup routines will triggered. Once the cache usage of completed files hits the high water mark, files will be deleted until the usage hits the low water mark. localcache * * * ##### LocalCache.LowWaterMarkPercentage Type:int Default:85 A percentage value where the cache cleanup routines will complete. Once the cache usage of completed files hits the high water mark, files will be deleted until the usage hits the low water mark. localcache * * * ##### LocalCache.MemoryCacheSize Type:string Default:0 Size of the in-memory block cache for the local cache module. When enabled, objects are cached in memory in addition to on disk, which can significantly improve performance for frequently accessed objects because disk objects are encrypted and require decryption on each access, while memory cache pages are stored decrypted, avoiding repeated decryption. Set to "0" to disable the memory cache (default). Accepts a plain number of bytes or a human-readable value with suffix (e.g. "8GB", "512MB", "1TB"). This parameter is used when running in local cache mode. For a full cache server, use Cache.MemoryCacheSize instead. localcache * * * ##### LocalCache.MaxConcurrentPrefetch Type:int Default:5 The maximum number of concurrent prefetch operations allowed. This limits how many blocks can be prefetched in parallel to avoid overwhelming the upstream server or local resources. localcache * * * ##### LocalCache.DefaultMaxAge Type:duration Default:24h The default freshness lifetime for cached objects when the origin server does not provide explicit Cache-Control headers. This value determines how long an object can be served from cache before revalidating with the origin. Setting this higher reduces origin load but may serve slightly stale data. Setting this lower ensures fresher data but increases origin requests. localcache * * * ##### LocalCache.StorageDirs Type:object Default:none A list of storage directory configurations for persistent cache object storage. Each entry describes a directory path and its per-directory size and eviction settings. When not set, the cache stores objects under the LocalCache.DataLocation directory using global size and watermark settings. A UUID file is placed in each directory so that storage IDs remain stable even if paths are reordered or changed. Each entry is an object with the following keys: - Path (string, required): absolute path to the storage directory. - MaxSize (string, optional): maximum size for this directory (e.g. "100GB"). When omitted or "0", auto-detected from the filesystem. - HighWaterMarkPercentage (int, optional): eviction trigger threshold as a percentage of MaxSize. When omitted or 0, the global LocalCache.HighWaterMarkPercentage is used. - LowWaterMarkPercentage (int, optional): eviction target threshold as a percentage of MaxSize. When omitted or 0, the global LocalCache.LowWaterMarkPercentage is used. Example YAML: ``` LocalCache: StorageDirs: - Path: /mnt/nvme/cache MaxSize: 500GB HighWaterMarkPercentage: 89 LowWaterMarkPercentage: 85 - Path: /mnt/hdd/cache MaxSize: 2TB ``` For backward compatibility, a plain list of strings (directory paths) is also accepted and treated as entries with only Path set. localcache * * * ##### LocalCache.ChunkSize Type:string Default:512MB The target chunk size for splitting large objects across multiple storage directories. When multiple storage directories are configured (via LocalCache.StorageDirs), objects larger than the chunk size will be split into multiple chunk files distributed across the directories to improve I/O parallelism and load balance storage utilization. Chunking is automatically disabled when only one storage directory is configured, as there would be no benefit to splitting files. The value can be specified with units (e.g., "64MB", "256MB", "1GB"). Set to "0" or "disabled" to disable chunking entirely. Reducing the maximum chunk size causes large objects to go across more directories and, ideally, increase overall throughput for a single object. Going below 64MB is discouraged. Objects smaller than the chunk size are stored in a single file. localcache * * * #### Cache * * * ##### Cache.StorageLocation Type:string Default:$XDG\_RUNTIME\_DIR/pelican/cache Root Default:/run/pelican/cache An absolute path to the directory where xrootd will create its default `namespace`, `meta`, and `data` directories. For example, setting `Cache.StorageLocation=/run/pelican/cache` without specifying further `Cache.DataLocations` or `Cache.MetaLocations` values will result in the cache creating a directory structure like: ``` . └── /run/pelican/cache/ ├── data/ │ ├── 00 # hexadecimal name values │ ├── 01 │ ├── ... │ └── FF ├── meta/ │ ├── 00 # hexadecimal name values │ ├── 01 │ ├── ... │ └── FF └── namespace/ ├── namespace1/ │ ├── foo1.txt --> /run/pelican/cache/data/00 │ └── foo2.txt --> /run/pelican/cache/data/01 └── namespace2/ └── bar.txt --> /run/pelican/cache/data/FF ``` In this setup, actual data files live at `/run/pelican/cache/data` and are given hexadecimal names, while references (symbolic links) to those files are stored in `/run/pelican/cache/namespace`. The `meta` directory is used for object metadata. Object requests to XRootD will be served from the namespace directories, and resolve the underlying object through these symbolic links. We recommend tying the `Cache.StorageLocation` to a fast storage device, such as an SSD, to ensure optimal cache performance. If this directory does not already exist, it will be created by Pelican. WARNING: The default value of /var/run/pelican should _never_ be used for production caches, as this directory is typically cleared on system restarts, and may interfere with system services if it becomes full. Running a cache with the default value set will generate a warning at cache startup. cache * * * ##### Cache.NamespaceLocation Type:string Default:${Cache.StorageLocation}/namespace A cache's namespace directory is used to duplicate/recreate the federation's namespace structure, and stores symbolic links from object names to the actual data files (see `Cache.StorageLocation` for extra information). For example, requesting `/foo/bar.txt` from a cache will check for the existence of a symbolic link at `${Cache.NamespaceLocation}/foo/bar.txt`, and if it exists, the cache will serve the data file at the location the symbolic link points to. If this directory does not already exist, it will be created by Pelican. WARNING: It's important that any values for `Cache.DataLocations` and `Cache.MetaLocations` are NOT subdirectories of `Cache.NamespaceLocation`, as this will make the raw data/meta files accessible through the cache's namespace structure, which is undefined behavior. cache * * * ##### Cache.DataLocations Type:stringSlice Default:${Cache.StorageLocation}/data A list of absolute filesystem paths/directories where the cache's object data will be stored. This list of directories can be used to string together multiple storage devices to increase the cache's storage capacity, as long as each of the directories is accessible by the cache service. For example, setting `Cache.DataLocations=["/mnt/cache1", "/mnt/cache2"]` will result in splitting cache data between two mounted drives, `/mnt/cache1` and `/mnt/cache2`. As such, these drives should be fast storage devices, such as SSDs. For more information, see the [XRootD oss documentation](https://xrootd.web.cern.ch/doc/dev56/ofs_config.htm) for the `oss.space` directive as well as the [XRootD pfc documentation](https://xrootd.web.cern.ch/doc/dev6/pss_config.htm) for the `pfc.spaces` directive. If this directory does not already exist, it will be created by Pelican. WARNING: It's important that any values for `Cache.DataLocations` are NOT subdirectories of `Cache.NamespaceLocation`, as this will make the raw data files accessible through the cache's namespace structure, which is undefined behavior. cache * * * ##### Cache.MetaLocations Type:stringSlice Default:${Cache.StorageLocation}/meta A list of absolute filesystem paths/directories where the cache's object metadata will be stored. Values in this list may point to separate drives as long as they're accessible by the cache service. For example, setting `Cache.MetaLocations=["/mnt/meta1", "/mnt/meta2"]` will result in splitting cache metadata between two the mounted drives. As such, these drives should be fast storage devices, such as SSDs. For more information, see the [XRootD oss documentation](https://xrootd.web.cern.ch/doc/dev56/ofs_config.htm) for the `oss.space` directive as well as the [XRootD pfc documentation](https://xrootd.web.cern.ch/doc/dev6/pss_config.htm) for the `pfc.spaces` directive. If this directory does not already exist, it will be created by Pelican. WARNING: It's important that any values for `Cache.MetaLocations` are NOT subdirectories of `Cache.NamespaceLocation`, as this will make the raw metadata files accessible through the cache's namespace structure, which is undefined behavior. cache * * * ##### Cache.LocalRoot Type:string Default:$XDG\_RUNTIME\_DIR/pelican/cache Root Default:/run/pelican/cache \[Deprecated\] Cache.LocalRoot is deprecated and replaced by Cache.StorageLocation. cache * * * ##### Cache.DataLocation Type:string Default:$XDG\_RUNTIME\_DIR/pelican/cache Root Default:/run/pelican/cache \[Deprecated\] Cache.DataLocation is being deprecated and will be removed in a future release. It is replaced by Cache.StorageLocation cache * * * ##### Cache.EnableBroker Type:bool Default:true Control whether the cache will use the connection broker to talk to the director. When enabled, the cache doesn't need an incoming network port open to communicate with the director. cache * * * ##### Cache.ExportLocation Type:string Default:/ A path that's relative to the `Cache.NamespaceLocation` where the cache will expose its contents. This path can be used to control which namespaces are available through the cache. For example, setting `Cache.ExportLocation: /foo` will only expose the `/foo` namespace to clients. cache * * * ##### Cache.RunLocation Type:filename Default:$XDG\_RUNTIME\_DIR/pelican/cache Root Default:/run/pelican/xrootd/cache A directory where temporary configurations will be stored for the XRootD daemon started by the cache. For non-root servers, if $XDG\_RUNTIME\_DIR is not set, a temporary directory will be created (and removed on shutdown). cache * * * ##### Cache.SentinelLocation Type:filename Default:none A filename under `Cache.DataLocation` path for Pelican to check the storage directory exists and is correctly mounted. The value must be a file and contain no directory. Leave it empty to skip the check. You should always choose a distinct name for `Cache.SentinelLocation`. It should not be reused for other servers. If running in a containerized environment it should not be the name of the underlying physical host as that may change and lead to confusion. You need to manually create a file under path to `Cache.DataLocation` with the same name as `Cache.SentinelLocation`. cache * * * ##### Cache.XRootDPrefix Type:string Default:cache The directory prefix for the XRootD cache configuration files. cache * * * ##### Cache.Url Type:url Default:https://${Server.Hostname}:${Cache.Port} The cache's configured URL, as reported to XRootD. This is the file transfer endpoint for the cache. cache * * * ##### Cache.Port Type:int Default:8442 The TCP port the cache service should use. If set to 0, then a random open port will be used. cache * * * ##### Cache.LowWatermark Type:string Default:85 Whenever the cache initiates file purging, it will attempt to clean files until its cumulative disk usages reaches this value. Note that "cache disk usage" is calculated based on the cache's entire set of configured disks, not just data directories from those disks. The value should be either a percentage integer of total available disk space (default is 90), or a number suffixed by k, m, g, or t. In which case, they must be absolute sizes in k (kilo-), m (mega-), g (giga-), or t (tera-) bytes, respectively. For more information, see the [xrootd pfc documentation](https://xrootd.web.cern.ch/doc/dev56/pss_config.pdf) for `pfc.diskusage`. cache * * * ##### Cache.HighWaterMark Type:string Default:89 When the cache's disk usage exceeds this value, file purging is triggered. Note that "cache disk usage" is calculated based on the cache's entire set of configured disks, not just data directories from those disks. The value should be either a percentage integer of total available disk space (default is 95), or a number suffixed by k, m, g, or t. In which case, they must be absolute sizes in k (kilo-), m (mega-), g (giga-), or t (tera-) bytes, respectively. For more information, see the [xrootd pfc documentation](https://xrootd.web.cern.ch/doc/dev56/pss_config.pdf) for `pfc.diskusage`. cache * * * ##### Cache.FilesMaxSize Type:string Default:none A value that sets the maximum cumulative size of files that can be stored in the cache's data directories (specified by `Cache.StorageLocation` and `Cache.DataLocations). When either this value or` Cache.HighWaterMark`is exceeded, the cache will begin purging files until it reaches the`Cache.FilesNominal`value. If the cache's disk usage is still in excess of the`Cache.LowWaterMark`, the cache will continue purging files until it reaches the` Cache.FilesBase\` value. Unlike watermark values, this value _must_ be suffixed by a unit of k, m, g, or t, which represent kilobytes, megabytes, gigabytes, and terabytes, respectively. All `Cache.Files*Size` parameters must be less than the cache's calculated low watermark, which may be configured as a percentage of total disk space from multiple disks. For more information, see the [xrootd pfc documentation](https://xrootd.web.cern.ch/doc/dev56/pss_config.pdf) for `pfc.diskusage`. cache * * * ##### Cache.FilesNominalSize Type:string Default:none A value that sets the "nominal" cumulative size of files that can be stored in the cache's data directory. When files in the cache exceed the `Cache.FilesMax` value, or if the cache's overall disk exceeds its `Cache.HighWaterMark` value, the cache will begin purging files until it reaches this value. If the cache's disk usage is still in excess of the `Cache.LowWaterMark`, the cache will continue purging files until it reaches the `Cache.FilesBase`. Unlike watermark values, this value _must_ be suffixed by a unit of k, m, g, or t, which represent kilobytes, megabytes, gigabytes, and terabytes, respectively. All `Cache.Files*Size` parameters must be less than the cache's calculated low watermark, which may be configured as a percentage of total disk space from multiple disks. For more information, see the [xrootd pfc documentation](https://xrootd.web.cern.ch/doc/dev56/pss_config.pdf) for `pfc.diskusage`. cache * * * ##### Cache.FilesBaseSize Type:string Default:none A value that sets the "base" cumulative size of files that can be stored in the cache's data directory. This is the stopping point for the cache's purging routines. Unlike watermark values, this value _must_ be suffixed by a unit of k, m, g, or t, which represent kilobytes, megabytes, gigabytes, and terabytes, respectively.All `Cache.Files*Size` parameters must be less than the cache's calculated low watermark, which may be configured as a percentage of total disk space from multiple disks. For more information, see the [xrootd pfc documentation](https://xrootd.web.cern.ch/doc/dev56/pss_config.pdf) for `pfc.diskusage`. cache * * * ##### Cache.EnableVoms Type:bool Default:false Enable X.509 / VOMS-based authentication for the cache. This allows HTTP clients to present X.509 client credentials in order to authenticate. The configuration of the authorization for these clients must be done by the admin; Pelican does not support automatic VOMS authorization configuration. cache * * * ##### Cache.Concurrency Type:int Default:none This value represents the maximum number of permitted IO operations in-progress per second. When this value is set, it enables the [XRootD throttling plugin's](https://github.com/xrootd/xrootd/tree/master/src/XrdThrottle) 'concurrency' throttle directive. For example, if there are two simultaneous read requests and each takes 1 second to complete, the concurrency is 2. Setting a concurrency limit of 1 would cause one of the requests to be delayed until the other completes. For Caches, this value should be approximately: - (HDDs) ~2x the number of underlying disks - (NVMe/SSDs) ~10x the number of cores available to the Cache cache * * * ##### Cache.ConcurrencyDegradedThreshold Type:int Default:90 The percentage of permissible concurrency that indicates when the Cache should enter the "degraded" state. For example, if `${Cache.Concurrency}` is set to 100 and `${Cache.ConcurrencyDegradedThreshold}` is set to 80, then the Cache will enter the degraded state when active IO exceeds 80. The Director will continue to deprioritize the Cache while in the degraded state until the active IO drops below the threshold. This setting only has effect when `${Cache.Concurrency}` is set. cache * * * ##### Cache.EnableLotman Type:bool Default:false LotMan is a library that provides management of storage space in the cache. cache * * * ##### Cache.PermittedNamespaces Type:stringSlice Default: A list of namespaces the cache is allowed to pull from. If the list is empty or this option is unset, it's assumed that the cache is allowed to access any namespace that's advertised to the director. Otherwise, it will only be allowed to access the listed namespaces. cache * * * ##### Cache.SelfTest Type:bool Default:true A bool indicating whether the cache should perform self health checks. cache * * * ##### Cache.SelfTestInterval Type:duration Default:15s The interval of which the cache starts a new file transfer test to itself. cache * * * ##### Cache.EnableOIDC Type:bool Default:false Indicate whether the cache should allow users to login to the admin website via OAuth2/OIDC with third-party authentication providers such as CILogon. If set to true, it is recommended that you also set `Server.UIAdminUsers` to a list of users to give admin privilege. This is because cache admin website doesn't have a public, non-admin view, and an empty AdminUsers list will lead to "permission denied" error for all users logged into cache admin website via OAuth. cache * * * ##### Cache.BlocksToPrefetch Type:int Default:0 The number of 128 kilobyte blocks the cache will read ahead when receiving requests. This will put the data in the cache potentially before it is needed and reduce the latency to the client when a request is made. However, it can also cause many extra requests to an origin and potentially overload it when unnecessary. As such, this is turned off by default. cache * * * ##### Cache.DbLocation Type:filename Default:$ConfigBase/cache.sqlite Root Default:/var/lib/pelican/cache.sqlite A filepath to the intended location of the cache's database. cache * * * ##### Cache.EnableTLSClientAuth Type:bool Default:false Turns client certificate authentication on or off in xrootd for the HTTPS protocol. When false (default) the cache will never request a TLS certificate. When true, the cache will always request a client certificate from the client. cache * * * ##### Cache.FedTokenLocation Type:filename Default:$ConfigBase/cache-fed-token A path to the file containing a token issued by the federation's issuer. This token may be consumed by other federation services to prove the cache's membership in the federation. For example, Origins serving a namespace without DirectReads enabled require that all clients prove they come from within the federation. cache * * * ##### Cache.DisableClientX509 Type:bool Default:true When true (default), prevents the Cache from sending its host TLS certificate to Origins when acting as a client (e.g., on Cache misses). This is necessary because certificates from popular CAs such as Let's Encrypt no longer include the TLS clientAuth Extended Key Usage (EKU), which causes SSL errors when the certificate is presented to an origin. Setting this to false restores the old behavior of sending the host certificate. Only do this if your host certificate explicitly includes the clientAuth EKU. Pelican will refuse to start if this is false and the configured certificate does not include clientAuth. cache * * * ##### Cache.EnableEvictionMonitoring Type:bool Default:true Enable cache eviction monitoring. The cache eviction monitoring data includes information like the total space available, the space used, and the space used by each namespace. cache * * * ##### Cache.EvictionMonitoringInterval Type:duration Default:60s The interval at which the eviction monitoring will be reported. Valid values are 60, 300, 600, 900, 1800, 3600. cache * * * ##### Cache.EvictionMonitoringMaxDepth Type:int Default:1 The maximum depth of the eviction monitoring within the namespace hierarchy. Depth is measured by the number of /-delimited levels in a namespace path. For example, a depth of 0 refers to the root of the namespace ('/'), while a depth of 1 refers to top-level namespaces like '/foo' and '/bar'. When eviction monitoring data is generated, it will aggregate usage statistics up to the specified depth. For example, if a cache serves the namespaces `/projectA/dataset1` and `/projectA/dataset2`, and `EvictionMonitoringMaxDepth` is set to 1, the usage for both namespaces will be aggregated and reported under `/projectA`. If the depth is set to 2, usage for `/projectA/dataset1` and `/projectA/dataset2` will be reported separately. A depth of 0 will report only the total usage for the entire cache, aggregated at the root level. cache * * * ##### Cache.ClientStatisticsLocation Type:filename Default:${Cache.RunLocation}/xrootd.stats If set, Pelican will pass this path to the XRootD cache process via the `XRD_CURLSTATISTICSLOCATION` environment variable to enable client-side curl statistics in `xrdcl-pelican` (v1.5.0+). Example: ``` Cache: ClientStatisticsLocation: "${Cache.RunLocation}/xrootd.stats" ``` The XRootD process will periodically write JSON-formatted statistics to the specified file, which can be read by external monitoring (e.g., `jq`). Leave unset to disable. cache * * * ##### Cache.SelfTestMaxAge Type:duration Default:1h Defines the maximum allowed time since the last successful self-test. If a Cache fails to complete a new successful test within this period, and Xrootd.AutoShutdownEnabled is true, the Cache will automatically shut down. This acts as a fail-safe mechanism. cache * * * ##### Cache.EnableSiteLocalMode Type:bool Default:false When true, the Cache will run without fully joining its configured federation. This means it will not register at its Registry and it will not advertise to its Director. This mode is intended for those who want the Cache to be used only by clients at their local site, without being part of a larger Pelican federation. Because this Cache will not be discoverable via the Director, Clients will need to be run with the ${Client.PreferredCaches} configuration set to the URL:port of this Cache's XRootD component in order to use it. If you need help finding this URL, see the Cache's ${Cache.Url} configuration. Note: When running in site-local mode, the operators of your federation will remain unaware of your Cache's existence, and thus will not be able to automatically detect its health or version. Furthermore, any statistics or usage data that would normally be collected by the Director will not include this Cache. Administrators of site-local Caches should watch closely for releases with cache-related security patches to ensure their systems remain secure. cache * * * #### Director * * * ##### Director.EnableFederationMetadataHosting Type:bool Default:true Controls whether or not the Director should host a copy of the Federation's metadata. This feature should be enabled whenever your Director is expected to serve as the federation's root discovery source through the `Federation.DiscoveryUrl` parameter, or whenever clients reference your Director's hostname with their Pelican URLs, e.g. `pelican object get pelican:// ...`. If your federation uses a "federation hostname" or "discovery URL" that is different from the Director hostname (for example, the OSDF uses https://osg-htc.org for discovery, whereas the Director is hosted at https://osdf-director.osg-htc.org), then this feature should be set to `false`. director * * * ##### Director.AdvertiseUrl Type:url Default:$(Server.ExternalWebUrl) The URL that director advertisements should be sent to. In a high-availability setup, the URL/hostname where the director receives advertisements from the cache & origin services may be distinct from the shared URL/hostname used by the clients for HA. This will set the advertisement URL separate from the external URL. If not set, Server.ExternalWebUrl will be used instead. director * * * ##### Director.DbLocation Type:filename Default:$ConfigBase/director.sqlite Root Default:/var/lib/pelican/director.sqlite A filepath to the intended location of the director's database, where server downtime info is stored. director * * * ##### Director.DefaultResponse Type:string Default:cache The default response type of a redirect for a director instance. Can be either "cache" or "origin". If a director is hosted at https://director.com, then a GET request to https://director.com/foo/bar.txt will either redirect to the nearest cache for namespace /foo if Director.DefaultResponse is set to "cache" or to the origin for /foo if it is set to "origin". director * * * ##### Director.CachesPullFromCaches Type:bool Default:false In the "origin" response, the director returns a list of origins that can serve the object. If `Director.CachesPullFromCaches` is set to true (default is false), the director then appends a list of caches that can serve the object to the original response. director * * * ##### Director.CacheResponseHostnames Type:stringSlice Default: A list of virtual hostnames for the director. If a request is sent by the client to one of these hostnames, the director assumes it should respond with a redirect to a cache. If present, the hostname is taken from the X-Forwarded-Host header in the request. Otherwise, Host is used. director * * * ##### Director.CacheSortMethod Type:string Default:distance When the director receives a client request that needs to be redirected to a cache, it will use this method to determine the ordering of the caches. The default method is "distance", which sorts caches by their spherical distance from the client. Available methods include: - "distance": Sorts caches by their spherical distance from the client. - "distanceAndLoad": Sorts caches according to both their distance and a calculated load. This is currently a placeholder, and returns the same ordering as "distance". - "random": Sorts caches randomly. - "adaptive": Sorts caches according to stochastically-generated weights that consider a combination of factors, including a cache's distance from the client, its IO load, server status and whether the cache already has the requested object. See details at https://github.com/PelicanPlatform/pelican/discussions/1198. Note that if `Director.CheckCachePresence` is set to false, then the adaptive algorithm cannot use the cache locality information. director * * * ##### Director.AdaptiveSortTruncateConstant Type:int Default:6 The first step in the Director's adaptive sorting algorithm is to sort all servers for the given request by their distance from the client and then truncate to the nearest N server before generating the other adaptive sort weights (load, status, locality). This constant sets the value of N. Higher values increase the pool of servers considered for adaptive sorting, while lower values restrict the pool to only the closest servers. The constant cannot be set to a value lower than 3, as Pelican clients expect to receive 3 servers at minimum, and the Director will reply with at most 6 servers regardless of this value. director * * * ##### Director.OriginResponseHostnames Type:stringSlice Default: A list of virtual hostnames for the director. If a request is sent by the client to one of these hostnames, the director assumes it should respond with a redirect to an origin. If present, the hostname is taken from the X-Forwarded-Host header in the request. Otherwise, Host is used. director * * * ##### Director.MaxMindKeyFile Type:filename Default:none A filepath to a MaxMind API key. The director service uses the MaxMind GeoLite City database (available [here](https://dev.maxmind.com/geoip/docs/databases/city-and-country)) to determine which cache is nearest to a client's IP address. The database, if not already found, will be downloaded automatically when a director is served and a valid key is present. director * * * ##### Director.GeoIPLocation Type:filename Default:$ConfigBase/maxmind/GeoLite2-city.mmdb Root Default:/var/cache/pelican/maxmind/GeoLite2-City.mmdb A filepath to the intended location of the MaxMind GeoLite City database. This option can be used either to load an existing database, or to configure the preferred download location if Pelican has a MaxMind API key. director * * * ##### Director.MinStatResponse Type:int Default:1 A positive integer indicating minimum number of origin's responses required for a `stat` call. director * * * ##### Director.MaxStatResponse Type:int Default:1 A positive integer indicating maximum number of origin's responses required for a `stat` call. `stat` call will cancel the rest of the ongoing query if max response is hit. director * * * ##### Director.CheckOriginPresence Type:bool Default:true Before redirecting a cache (or, for direct reads or writes, a client) to an origin, query the origin to see if the object is present. Enabling this option generates slightly more load on the origin; however, it provides improved error messages and allows a namespace to effectively be split across multiple origins. director * * * ##### Director.CheckCachePresence Type:bool Default:true Before redirecting a client to a cache, query the cache to see if the object is present at the cache. Enabling this option improves the cache selection algorithm, allowing the director to prefer caches nearby the client with the object over caches without the object. director * * * ##### Director.StatTimeout Type:duration Default:2000ms The timeout for a single `stat` request. director * * * ##### Director.StatConcurrencyLimit Type:int Default:100 The maximum number of concurrent `stat` request to a single origin server. Additional requests are blocked until total requests for the origin is below limit. See [golang.org/x/sync/errgroup](https://pkg.go.dev/golang.org/x/sync@v0.6.0/errgroup#Group.SetLimit) for detail director * * * ##### Director.AdvertisementTTL Type:duration Default:15m The time to live (TTL) of director's internal cache to store origins and caches advertisement. director * * * ##### Director.OriginCacheHealthTestInterval Type:duration Default:15s The interval of which director issues a new file transfer test to all the registered origins and caches. director * * * ##### Director.EnableBroker Type:bool Default:true Whether the director should also run the connection brokering service. director * * * ##### Director.FilteredServers Type:stringSlice Default: A list of server resource names that the Director should consider in downtime, preventing the Director from issuing redirects to them. Additional downtimes are aggregated from Topology (when the Director is served in OSDF mode), and the Web UI. director * * * ##### Director.SupportContactEmail Type:string Default:none An Email address to receive issues and help requests for the federation the director is hosting. The values will be displayed on the director web interface if provided. We highly recommend director admin to fill out this field. director * * * ##### Director.SupportContactUrl Type:string Default:none A URL where user can find support information. Can be your website, GitHub discussion, or third-party support portal for the federation the director is hosting. The values will be displayed on the director web interface if provided. We highly recommend director admin to fill out this field. director * * * ##### Director.EnableOIDC Type:bool Default:false Indicate whether the director should allow users to login to the admin website via OAuth2/OIDC with third-party authentication providers such as CILogon. If set to true, it is recommended that you also set `Server.UIAdminUsers` to a list of users to give admin privilege. This is because origin admin website doesn't have a public, non-admin view, and an empty AdminUsers list will lead to "permission denied" error for all users logged into origin admin website via OAuth. director * * * ##### Director.CachePresenceTTL Type:duration Default:1m If `Director.CheckCachePresence` is enabled, the director will check with remote cache to see if the object is present before redirecting a client. This parameter controls how long the director will cache the result of the lookup. Longer values will reduce the load generated on the caches but may reduce the accuracy of the result (as the contents of the cache will change over time). director * * * ##### Director.MetadataComparisonInterval Type:duration Default:10m Defines the interval at which the director compares its local federation metadata (director URL, registry URL, and JWKS) against the federation's discovery URL. This comparison helps detect configuration mismatches that could cause unexpected behavior for clients. When discrepancies are detected, alerts are displayed in the Director's web UI for administrators. This comparison is only performed when the Director is not itself the federation's discovery URL. If the Director serves as the discovery URL (i.e., `Server.ExternalWebUrl` matches `Federation.DiscoveryUrl`), the comparison is skipped. director * * * ##### Director.FedTokenLifetime Type:duration Default:15m The default lifetime assigned to tokens issued by the director on behalf of the federation. These tokens may be issued to caches to prove their authorization within the federation to origins that require it. director * * * #### Registry * * * ##### Registry.DbLocation Type:filename Default:$ConfigBase/ns-registry.sqlite Root Default:/var/lib/pelican/registry.sqlite A filepath to the intended location of the namespace registry's database. registry * * * ##### Registry.RequireKeyChaining Type:bool Default:true Specifies whether namespaces requesting registration must possess a key matching any already-registered super/sub namespaces. For example, if true and a namespace `/foo/bar` is already registered, then registration of `/foo` or `/foo/bar/baz` can only be done using keys registered to `/foo/bar`. registry * * * ##### Registry.AdminUsers Type:stringSlice Default: \[Deprecated\] `Registry.AdminUsers` is deprecated and will be removed in the future releases. Please migrate to use `Server.UIAdminUsers` instead. A string slice of "subject" claim of users to give admin permission for registry UI. The "subject" claim should be the "CILogon User Identifier" from CILogon user page: [https://cilogon.org/](https://cilogon.org/) registry * * * ##### Registry.Institutions Type:object Default:none A array of institution objects available to register. Users can only select from this list when they register a new namespace. Each object has `name` and `id` field where `name` is a human-readable name for the institution and `id` is a unique identifier for the institution. For Pelican running in OSDF alias, the `id` will be OSG ID. For example: ``` - name: University of Wisconsin - Madison id: https://osg-htc.org/iid/01y2jtd41 ``` Note that this value will take precedence over Registry.InstitutionsUrl if both are set. registry * * * ##### Registry.CustomRegistrationFields Type:object Default:none An array of objects specifying additional fields when registering namespaces. The schema of the object is as follows: ``` - name: department_name type: enum required: true options: - name: Math id: math - name: Computer Science id: cs optionsUrl: https://example.com/options description: The department of the organization that holds this namespace ``` Note the following requirements: - `name` must be snake case with underline connecting words, i.e. department\_name. The name displayed in the registration table will be converted from this field into a human-readable, space-separated name with the first letter(s) capitalized. i.e. department\_name -> Department Name - `type` must be one of `string`, `int`, `bool`, `datetime` (Unix time in seconds), or `enum`. - `options` must be a non-empty yaml array for field with type `enum`. `optionsUrl` will be ignored if `options` is set. Example: ``` options: - name: "Option A" id: "optionA" ``` - `description` will show up in the web UI as helper text to help user understand the field - `optionsUrl` is a URL to provide a list of options for `enum` type field. The URL should respond to an anonymous GET request and return JSON response in the same format as the options field above registry * * * ##### Registry.InstitutionsUrl Type:url Default:none A url to get a list of available institutions for users to register their namespaces to. The url must accept a GET request with 200 response in JSON/YAML content with the following format: `JSON`: ``` [ { "name": "University of Wisconsin - Madison", "id": " https://osg-htc.org/iid/01y2jtd41" } ] ``` `YAML`: ``` - name: University of Wisconsin - Madison id: " https://osg-htc.org/iid/01y2jtd41" ``` Where the id field will be stored in registry database and must be unique, and name field will be displayed in UI as the option. Note that Pelican will cache the response of the url in a TTL cache with default refresh time of 15 minutes. Also note that \`Registry.Institutions\`\` will take precedence over this value if both are set. registry * * * ##### Registry.InstitutionsUrlReloadMinutes Type:duration Default:15m Number of minutes that the Registry.InstitutionsUrl will be reloaded into the TTL cache. registry * * * ##### Registry.RequireCacheApproval Type:bool Default:false Only allow approved caches to join the federation and serve files. If set to true, caches can successfully self-register or registered via registry, but director won't direct traffic to the cache. registry * * * ##### Registry.RequireOriginApproval Type:bool Default:false Only allow approved origins to join the federation and serve files. If set to true, origins can successfully self-register or registered via registry, but director won't direct traffic to the origin, nor would files on the origin show up in the federation. registry * * * #### Server * * * ##### Server.TLSCertificate Type:filename Default:$ConfigBase/certificates/tls.crt Root Default:/etc/pelican/certificates/tls.crt \[Deprecated\] A filepath to a file containing an X.509 host certificate to use for TLS authentication when running server components of Pelican. If you override this filepath, you need to provide the matched-pair private key via Server.TLSKey and a Certificate Authority (CA) certificate via Server.TLSCACertificateFile. cachedirectororiginregistry * * * ##### Server.TLSCertificateChain Type:filename Default:$ConfigBase/certificates/tls.crt Root Default:/etc/pelican/certificates/tls.crt A filepath to a file containing the full X.509 certificate chain, including the host certificate followed by any intermediate certificates, to use for TLS authentication when running server components of Pelican. If you override this filepath, you need to provide the matched-pair private key via Server.TLSKey and a Certificate Authority (CA) certificate via Server.TLSCACertificateFile. cachedirectororiginregistry * * * ##### Server.TLSCACertificateFile Type:filename Default:$ConfigBase/certificates/tlsca.pem Root Default:/etc/pelican/certificates/tlsca.pem A filepath to the TLS Certificate Authority (CA) certificate file, to be used by XRootD and internal HTTP client requests. Do not override this filepath unless you want to provide your TLS host certificate cachedirectororiginregistry * * * ##### Server.TLSCACertificateDirectory Type:string Default:none A filepath to the directory used for storing TLS Certificate Authority (CA) certificate to be used by XRootD only. This is exclusive with Server.TLSCACertificateFile for XRootD and this value takes priority over Server.TLSCACertificateFile. cachedirectororiginregistry * * * ##### Server.TLSCAKey Type:filename Default:$ConfigBase/certificates/tlsca.key Root Default:/etc/pelican/certificates/tlsca.key The name of a file containing a private key corresponding to the TLSCACertificate. Used when running server components of Pelican. cachedirectororiginregistry * * * ##### Server.TLSKey Type:filename Default:$ConfigBase/certificates/tls.key Root Default:/etc/pelican/certificates/tls.key The name of a file containing the private key corresponding to the host certificate in the TLSCertificateChain. Used when running server components of Pelican. cachedirectororiginregistry * * * ##### Server.EnableUI Type:bool Default:true Indicate whether a server should enable its web UI. This only controls the serving of web UI resources and pages. Backend functionality such as OIDC authentication, OAuth endpoints, and API routes will remain enabled regardless of this setting. originregistrydirectorcache * * * ##### Server.WebPort Type:int Default:8444 The port number the Pelican web interface and internal web APIs will be bound to. cachedirectororiginregistry * * * ##### Server.TrustedProxies Type:stringSlice Default: A list of CIDR ranges or IP addresses of trusted reverse proxies. When set, the Gin web engine will use X-Forwarded-For headers only from these trusted sources to determine the client IP. When empty (the default), no proxies are trusted and the client IP is always taken from the network connection's remote address. Use "\*" to trust all sources (equivalent to 0.0.0.0/0 and ::/0). Both IPv4 and IPv6 addresses and CIDR ranges are supported. Example: \["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fd00::/8"\] cachedirectororiginregistry * * * ##### Server.WebHost Type:string Default:0.0.0.0 A string-encoded IP address that the Pelican web engine is configured to listen on. cachedirectororiginregistry * * * ##### Server.ExternalWebUrl Type:url Default:https://${Server.Hostname}:${Server.WebPort} (for ${Server.WebPort} != 443) A URL indicating the Pelican web interface and internal web APIs address as it appears externally. Port number will be stripped if it's 443, from `Server.WebPort` or directly set through `Server.ExternalWebUrl`. cachedirectororiginregistry * * * ##### Server.Hostname Type:string Default:none The server's hostname, by default it's os.Hostname(). cachedirectororiginregistry * * * ##### Server.HealthMonitoringPublic Type:bool Default:false Whether the server's health monitoring results should be made public without any authentication required cachedirectororiginregistry * * * ##### Server.IssuerUrl Type:string Default:none The URL and port at which the server's issuer can be accessed. cachedirectororiginregistry * * * ##### Server.IssuerHostname Type:string Default:none The hostname at which the server's issuer can be accessed. cachedirectororiginregistry * * * ##### Server.IssuerPort Type:int Default:none The port at which the server's issuer can be accessed. cachedirectororiginregistry * * * ##### Server.IssuerJwks Type:filename Default:none A filepath to a JWKS-formatted file containing pre-generated public keys. These keys will be appended to the public keys dynamically generated from the private keys in [Server.IssuerKeysDirectory](https://docs.pelicanplatform.org/parameters#IssuerKeysDirectory) and [Server.IssuerKey](https://docs.pelicanplatform.org/parameters#IssuerKey) (deprecated), to form the final exported public keys JWKS. cachedirectororiginregistry * * * ##### Server.UIActivationCodeFile Type:filename Default:$ConfigBase/server-web-activation-code If the server's web UI has not yet been configured, this file will contain the activation code necessary to turn it on. cachedirectororiginregistry * * * ##### Server.UIPasswordFile Type:filename Default:$ConfigBase/server-web-passwd A filepath specifying where the server's web UI password file should be stored. cachedirectororiginregistry * * * ##### Server.SessionSecretFile Type:filename Default:$ConfigBase/session-secret The default content of the file is the hash of the concatenation of "pelican" and the DER form of ${IssuerKey} The filepath to the secret for encrypt/decrypt session data for Pelican web UI to initiate a session cookie. This is used for sending redirect request for OAuth2 authentication follow. This is also used for CSRF auth key. cachedirectororiginregistry * * * ##### Server.WebReadOnly Type:bool Default:false When enabled, web UI routes that employ the read-only middleware will reject state changing HTTP methods such as, POST, PUT, PATCH, and DELETE. Read only methods such as GET, HEAD, and OPTIONS remain allowed as well as the following system critical routes: - /origin-api/directorTest - /origin/directorTest - /api/v1.0/auth/login - /api/v1.0/auth/logout If the issuer is enabled: - /api/v1.0/issuer/oidc-cm - /api/v1.0/issuer/token cachedirectororiginregistry * * * ##### Server.RegistrationRetryInterval Type:duration Default:10s The duration of delay in origin/cache registration retry attempts if the initial registration call to registry was failed. cacheorigin * * * ##### Server.UILoginRateLimit Type:int Default:1 The maximum number of requests a user can be made under the same IP address per second against the login endpoint cachedirectororiginregistry * * * ##### Server.WebConfigFile Type:filename Default:$ConfigBase/web-config.yaml Root Default:/etc/pelican/web-config.yaml A filepath to the file where web-based configuration changes are stored cachedirectororiginregistry * * * ##### Server.UIAdminUsers Type:stringSlice Default: A string slice of "subject" claim of users to give admin permission for the server admin website, who are authenticated through OAuth/OIDC. The "subject" claim should be the "CILogon User Identifier" from CILogon user page: https://cilogon.org/ cachedirectororiginregistry * * * ##### Server.AdminGroups Type:stringSlice Default: A string slice of group names that grant admin permission for the server admin website. Users who belong to any of these groups will be granted admin privileges, regardless of their username. This is useful when you want to grant admin access based on group membership rather than individual user identifiers. Group information is obtained from the issuer configuration (`Issuer.GroupSource`). Depending on the group source: - If `Issuer.GroupSource` is `file`: Groups are read from the file specified by `Issuer.GroupFile`. - If `Issuer.GroupSource` is `oidc`: Groups are extracted from the OIDC provider token using the claim specified by `Issuer.OIDCGroupClaim` (defaults to "groups"). - If `Issuer.GroupSource` is `internal`: Groups are read from the Pelican server's internal SQLite database. Note: This works in conjunction with Server.UIAdminUsers. A user can be granted admin access either by being listed in Server.UIAdminUsers or by belonging to a group listed in Server.AdminGroups. cachedirectororiginregistry * * * ##### Server.StartupTimeout Type:duration Default:10s The amount of time the pelican server will wait for its components and services to startup. If the timeout is hit while waiting on a component, the server will shutdown. cachedirectororiginregistry * * * ##### Server.EnablePprof Type:bool Default:false A boolean to enable or disable the [pprof](https://pkg.go.dev/runtime/pprof) endpoints for debugging. cachedirectororiginregistry * * * ##### Server.DropPrivileges Type:bool Default:false If the server has been started with root privileges, drop down to an unprivileged user. \* * * * ##### Server.UnprivilegedUser Type:string Default:pelican The user to run as after dropping root privileges. This is only relevant if `Server.DropPrivileges` is set to true \* * * * ##### Server.DirectorUrls Type:stringSlice Default: A list of director URLs known to the server. These are used by the service to forward the advertisements to multiple directors; a director service will advertise these URLs in its auto-generated configuration metadata. origincachedirector * * * ##### Server.DbLocation Type:filename Default:$ConfigBase/pelican.sqlite Root Default:/var/lib/pelican/pelican.sqlite A filepath to the intended location of the server's database. cachedirectororiginregistry * * * #### Server.DatabaseBackup * * * ##### Server.DatabaseBackup.Location Type:filename Default:$ConfigBase/backups Root Default:/var/lib/pelican/backups The directory where periodic SQLite database backups are stored. Each backup file is compressed and encrypted using the server's issuer keys. cachedirectororiginregistry * * * ##### Server.DatabaseBackup.Frequency Type:duration Default:24h How often the server creates a backup of its SQLite database. Set to 0 to disable periodic backups. cachedirectororiginregistry * * * ##### Server.DatabaseBackup.MaxCount Type:int Default:10 The maximum number of database backup files to retain. When the number of backups exceeds this count, the oldest backups are removed. Set to 0 to disable rotation and retain all backups indefinitely. cachedirectororiginregistry * * * #### Issuer * * * ##### Issuer.IssuerClaimValue Type:string Default:$(Server.ExternalWebUrl) Contents of the issuer (`iss`) claim in the generated tokens origin * * * ##### Issuer.AuthenticationSource Type:string Default:OIDC How users should authenticate with the issuer. Currently-supported values are: - `none` (default): No authentication is performed. All requests are successful and assumed to be a user named `nobody`. - `OIDC`: Use the server's OIDC configuration to authenticate with an external identity provider. origin * * * ##### Issuer.OIDCAuthenticationRequirements Type:object Default: A list of claim-value pairs that indicate required values from the OIDC ID token to authenticate. Example: ``` - claim: idp_name value: University of Wisconsin-Madison ``` Would only allow tokens with `"idp_name": "University of Wisconsin-Madison"` set to authenticate. origin * * * ##### Issuer.OIDCPreferClaimsFromIDToken Type:bool Default:false This applies to the claims specified by `Issuer.OIDCAuthenticationUserClaim` and `Issuer.GroupSource`. If set to `true`, then claims will be searched for in the OIDC ID token before falling back on claims from the UserInfo endpoint. If set to `false`, then only the UserInfo endpoint will be considered. origin * * * ##### Issuer.OIDCAuthenticationUserClaim Type:string Default:sub The claim to be used as the "username" for the issuer. origin * * * ##### Issuer.OIDCSubjectClaim Type:string Default:sub The claim to be used as the unique subject identifier for the user. For OIDC providers, this is typically "sub". For OAuth2 providers like GitHub, this might be "id". If the claim is not found in the user info response, the system will fall back to using the username. If the claim value is numeric, it will be converted to a string. originregistrycachedirector * * * ##### Issuer.OIDCIssuerClaim Type:string Default:iss The claim to be used to identify the authentication provider (issuer). For OIDC providers, this is typically "iss". For OAuth2 providers that don't provide an issuer claim, the system will fall back to using the value of OIDC.Issuer or the hostname from OIDC.AuthorizationEndpoint. originregistrycachedirector * * * ##### Issuer.UserStripDomain Type:bool Default:false Some OIDC issuers generate a username of the form user@domain (such as `john.doe@gmail.com`); when `UserStripDomain` is enabled, Pelican will strip the domain when determining the username. For example, the OIDC identity `john.doe@gmail.com` would map to `john.doe`. origin * * * ##### Issuer.GroupSource Type:string Default:none How the issuer should determine group information based on the authenticated identity. Valid values are: - `none` (default): No group information should be used. - `file`: Read groups from an external, JSON-formatted file. The file should contain a single JSON object with keys corresponding to the "user" name and the value a list of strings that are interpreted as the user's groups. - `oidc`: Take group information from the identity token provided by the OIDC identity provider. Parses the value of the claim specified by `Issuer.OIDCGroupClaim` (defaults to "groups") as a list of groups. The value may either be a comma-separated string or an array of strings. - `internal`: Take group information from the Pelican server's internal user database. - `github`: Fetch group information from GitHub organization. Each GitHub organization the user belongs to becomes a group. Requires the OAuth2 application to have the `read:org` scope. origin * * * ##### Issuer.OIDCGroupClaim Type:string Default:groups The claim to be used as the group for the issuer. If the value is a string, it is assumed that a comma is used as a group delimiter; otherwise, an array of strings is assumed. Check the documentation of your OIDC provider to determine the appropriate claim name. origin * * * ##### Issuer.GroupFile Type:string Default:none The location of a file containing group information. The file should contain a single JSON object with keys corresponding to the "user" name and the value a list of strings that are interpreted as the user's groups. origin * * * ##### Issuer.GroupRequirements Type:stringSlice Default: Group membership requirements. A request must be mapped to one of the groups in this list to successfully authenticate. origin * * * ##### Issuer.AuthorizationTemplates Type:object Default: The global authorizations that may be generated for an authenticated request, specified via a list of templates. These rules apply to every namespace served by the issuer unless a specific export overrides them with its own `AuthorizationTemplates` (see `Origin.Exports`). When an export defines per-namespace templates, only those templates are used for that namespace; the global templates are ignored entirely (no merging). Each template defines a set of authorizations that can depend on the authenticated username and groups. An authorization is an action and a prefix to which it applies. Concretely, a template is a collection of key-value pairs: - `actions`: A list of actions. Valid values are `read`, `create`, and `modify`. - `prefix`: The prefix to which those actions apply. If the prefix contains the substring `$USER`, the string is replaced with the authenticated username. If the prefix contains the substring `$GROUP`, then an authorization is generated for each authenticated group. - `users` (optional): A list of usernames. If non-empty, the authenticated username must be in this list in order for this template to generate any authorizations. - `groups` (optional): A list of groups. If non-empty, at least one authenticated group must be in this list in order for this template to generate any authorizations. If `prefix` contains the substring `$GROUP`, then authorizations will be generated only for the groups listed here. - `group_regexes` (optional): A list of regular expressions. If non-empty, at least one authenticated group must match one of the regular expressions in order for this template to generate any authorizations. If `prefix` contains the substring `$GROUP`, then authorizations will be generated only for the matching groups. An authenticated group will match if it is listed in either the `groups` or the `group_regexes` list. For example, if the request is authenticated as username `bbockelm` and groups `dept_a` and `dept_b`, then the list of templates ``` - actions: ["read", "modify"] prefix: /home/$USER - actions: ["read"] prefix: /staging/$USER users: ["alice", "bob"] - actions: ["read", "create"] prefix: /projects/$GROUP groups: ["dept_a", "dept_c"] - actions: ["read"] prefix: /data/$GROUP group_regexes: ["^dept_"] ``` will result in the following authorizations: - read /home/bbockelm - modify /home/bbockelm - read /projects/dept\_a - create /projects\_dept\_a - read /data/dept\_a - read /data/dept\_b origin * * * ##### Issuer.RedirectUris Type:stringSlice Default: Root Default: The list of redirect URIs for Issuer Clients to use in the Authorization Code Flow. Used to enable the use of a Pelican Web Client at the redirect target. The URIs must use the `https` scheme. More information on using the [Pelican Web Client](https://github.com/PelicanPlatform/web-client). origin * * * #### OIDC * * * ##### OIDC.ClientIDFile Type:filename Default:$ConfigBase/oidc-client-id Root Default:/etc/pelican/oidc-client-id A filepath to a file containing an OIDC Client ID. This is used by the namespace registry to allow OAuth2/OIDC login and authenticated namespace registration. By default, Pelican uses [CILogon](www.cilogon.org) as the authentication provider. You need to first register an OIDC _client_ at CILogon: https://cilogon.org/oauth2/register. If you'd like to use other authentication providers, you need to change other endpoint parameters under OIDC configuration to the endpoints of your provider, such as `OIDC.AuthorizationEndpoint`, `OIDC.UserInfoEndpoint`, etc. `OIDC.ClientIDFile` is mutually exclusive with `OIDC.ClientID`. The value of `OIDC.ClientID` will override the value of `OIDC.ClientIDFile` if both are set. This is a required parameter for the registry server. This is a required parameter for the origin/cache/director server if `Origin.EnableOIDC`/`Cache.EnableOIDC`/`Director.EnableOIDC` is set to `true`, respectively. registryorigincachedirector * * * ##### OIDC.ClientID Type:string Default:none The OIDC ClientID to use for the server. This is mutually exclusive with `OIDC.ClientIDFile`. The value of `OIDC.ClientID` will override the value of OIDC.ClientIDFile if both are set. This is a required parameter for the registry server. This is a required parameter for the origin/cache/director server if `Origin.EnableOIDC`/`Cache.EnableOIDC`/`Director.EnableOIDC` is set to `true`, respectively. registryorigincachedirector * * * ##### OIDC.ClientSecretFile Type:filename Default:$ConfigBase/oidc-client-secret Root Default:/etc/pelican/oidc-client-secret A filepath to a file containing an OIDC Client Secret. This is used by the namespace registry to establish OIDC information for authenticated registration. This is a required parameter for the registry server. This is a required parameter for the origin/cache/director server if `Origin.EnableOIDC`/`Cache.EnableOIDC`/`Director.EnableOIDC` is set to `true`, respectively. registryorigincachedirector * * * ##### OIDC.DeviceAuthEndpoint Type:url Default:https://cilogon.org/oauth2/device\_authorization A URL describing an OIDC Device Auth Endpoint. This is used by the namespace registry to establish OIDC information for authenticated registration. The default value is set to the URL from CILogon. registryorigincachedirector * * * ##### OIDC.TokenEndpoint Type:url Default:https://cilogon.org/oauth2/token A URL describing an OIDC Token Endpoint. This is used by the namespace registry to establish OIDC information for authenticated registration. The default value is set to the URL from CILogon. registryorigincachedirector * * * ##### OIDC.UserInfoEndpoint Type:url Default:https://cilogon.org/oauth2/userinfo A URL describing an OIDC User Info Endpoint. This is used by the namespace registry to establish OIDC information for authenticated registration. The default value is set to the URL from CILogon. registryorigincachedirector * * * ##### OIDC.AuthorizationEndpoint Type:url Default:https://cilogon.org/authorize A URL containing the OIDC authorization endpoint. The default value is set to the URL from CILogon. registryorigincachedirector * * * ##### OIDC.Issuer Type:url Default:https://cilogon.org The URL of the OIDC issuer. If set, OIDC auto-discovery may be used to find other endpoints (token, user info, device auth). The URL should not contain a path unless your authentication server enables multi-tenant support. If the OIDC auto-discovery failed, Pelican will fall back to use individual endpoints set in the configuration. For any unset endpoints, Pelican will use default values, which are from CILogon. Note: If you explicitly set the OIDC endpoints (AuthorizationEndpoint, TokenEndpoint, etc.), those values will take precedence over auto-discovery. This is useful for OAuth2 providers like GitHub that don't support OIDC discovery. For CILogon, it's https://cilogon.org For Globus, it's https://auth.globus.org For GitHub OAuth2, set this to https://github.com and explicitly configure the individual endpoints registryorigincachedirector * * * ##### OIDC.Scopes Type:stringSlice Default:openid,email,profile A list of scopes to request from the authentication provider. registryorigincachedirector * * * ##### OIDC.ClientRedirectHostname Type:string Default:none The hostname for the OIDC client redirect URL that the OIDC provider will redirect to after the user is authenticated. For development use only. Useful when developing in a container and you want to expose localhost instead of container hostname to your OAuth provider. registryorigincachedirector * * * #### Xrootd * * * ##### Xrootd.Port Type:int Default:8443 \[Deprecated\] `Xrootd.Port` is deprecated and will be removed in the future release. Please migrate to use `Origin.Port` or `Cache.Port` instead. The port over which XRootD should be made available. This setting is deprecated; please use the Cache.Port or Origin.Port, as appropriate, for the server. cacheorigin * * * ##### Xrootd.RunLocation Type:filename Default:$XDG\_RUNTIME\_DIR/pelican Root Default:/run/pelican/xrootd \[Deprecated\] `Xrootd.RunLocation` is deprecated and will be removed in a future release. Please migrate to use `Cache.RunLocation` or `Origin.RunLocation` instead. A directory where temporary configurations will be stored for the XRootD daemon started by the origin or cache. For non-root servers, if $XDG\_RUNTIME\_DIR is not set, a temporary directory will be created (and removed on shutdown). This setting is deprecated; please use the Cache.RunLocation or Origin.RunLocation, as appropriate, for the server. cacheorigin * * * ##### Xrootd.ConfigFile Type:filename Default:none The _absolute_ path to an XRootD configuration file for customized XRootD configuration. This should only be used by admins with experience in configuring XRootD directly. `Xrootd.ConfigFile` will be used as the continuation of the Pelican generated XRootD configuration, via the `continue` directive. Existing configuration values may be overwritten or appended. Refer to [Configuration File Continuation](https://xrootd.web.cern.ch/doc/dev55/Syntax_config.htm#_Toc520499864) for details cacheorigin * * * ##### Xrootd.RobotsTxtFile Type:filename Default:$ConfigBase/robots.txt Root Default:/etc/pelican/robots.txt Origins may be indexed by web search engines; to control the behavior of search engines, one may provide local policy via a [robots.txt file](https://en.wikipedia.org/wiki/Robots.txt). If this file is not present, it will be auto-created with a default policy of blocking all indexing. origin * * * ##### Xrootd.ScitokensConfig Type:filename Default:$ConfigBase/xrootd/scitokens.cfg Root Default:/etc/pelican/xrootd/scitokens.cfg The location of a file configuring XRootD's [token-based authorization subsystem](https://github.com/xrootd/xrootd/blob/master/src/XrdSciTokens/README.md). This file allows arbitrary changes to the authorization configuration and will be merged with any auto-generated configuration; it's recommended for use by experts only. cacheorigin * * * ##### Xrootd.Mount Type:string Default:none The mount path for an instance of XRootD. origin * * * ##### Xrootd.MacaroonsKeyFile Type:string Default:none The filepath to a Macaroons key for setting up authorization in XRootD. origin * * * ##### Xrootd.Authfile Type:string Default:none The filepath to an auth file for setting up authorization in XRootD. cacheorigin * * * ##### Xrootd.AuthRefreshInterval Type:duration Default:5m The interval used by XRootD (cache/origin) for refreshing Authfiles. This affects how often the server polls for upstream changes that might affect the authorization policy. For example, when applied to a cache, this affects how often origin permissions are polled for changes. cacheorigin * * * ##### Xrootd.ManagerHost Type:url Default:none A URL pointing toward the XRootD instance's Manager Host. cacheorigin * * * ##### Xrootd.ManagerPort Type:int Default:1213 The port at which the XRootD instance's Manager Host is available. cacheorigin * * * ##### Xrootd.SummaryMonitoringHost Type:url Default:none A URL pointing toward the XRootD instance's Summary Monitoring Host. cacheorigin * * * ##### Xrootd.SummaryMonitoringPort Type:int Default:9931 The port at which the XRootD instance's Summary Monitoring Host is available. cacheorigin * * * ##### Xrootd.DetailedMonitoringHost Type:url Default:none A URL pointing toward the XRootD instance's Detailed Monitoring Host. cacheorigin * * * ##### Xrootd.DetailedMonitoringPort Type:int Default:9930 The port at which the XRootD instance's Detailed Monitoring Host is available. cacheorigin * * * ##### Xrootd.LocalMonitoringHost Type:url Default:none A URL pointing toward the XRootD instance's Local Monitoring Host. cacheorigin * * * ##### Xrootd.Sitename Type:string Default:none The sitename, as configured for XRootD. It is generally used as a human readable way to convey something about the institution running the service. This value will also be used by caches when registering with the federation's Registry (overriding the service's hostname, which is the default). Origins currently always use the hostname at the Registry. For both cahces/origins the sitename will be displayed as the service's name in the federation's Director. cacheorigin * * * ##### Xrootd.ShutdownTimeout Type:duration Default:1m The maximum amount of time pelican will wait for the xrootd daemons to gracefully shutdown before killing ongoing transfers. During this period, the Director will stop redirecting clients to the Origin/Cache, while in-flight transfers are allowed to proceed until timeout. cacheorigin * * * ##### Xrootd.ConfigUpdateFailureTimeout Type:duration Default:1h If the Authfile and/or scitoken config file fails to update within this duration, and Xrootd.AutoShutdownEnabled is true, the server will be automatically shut down. origincache * * * ##### Xrootd.AutoShutdownEnabled Type:bool Default:true If enabled, the server will be automatically shut down if the Authfile and/or scitoken config file fails to update within the timeout. origincache * * * #### Monitoring * * * ##### Monitoring.EnablePrometheus Type:bool Default:true Enable the internal Prometheus server. This server is bound to the ${Server.WebPort}. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.DataLocation Type:string Default:$ConfigBase/monitoring/data Root Default:/var/lib/pelican/monitoring/data A filepath where Prometheus should host its monitoring data. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.PortLower Type:int Default:9930 The lower end of a range of monitoring ports for Prometheus configuration. origincache * * * ##### Monitoring.PortHigher Type:int Default:9999 The lower end of a range of monitoring ports for Prometheus configuration. origincache * * * ##### Monitoring.AggregatePrefixes Type:stringSlice Default:/\* A list of path-like prefixes, potentially containing a glob (wildcard character), indicating how the Prometheus-based monitoring should aggregate records when reporting. For example, if `/foo/*` is on the aggregate path list, then the monitoring data for a download of objects `/foo/bar` and `/foo/baz` will be aggregated into a single series, `/foo`. origincache * * * ##### Monitoring.TokenExpiresIn Type:duration Default:1h The duration of which the tokens for various Prometheus endpoints expire. This includes tokens for director's Prometheus origin discovery endpoint, director's origin scraper, and server's self-scraper. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.TokenRefreshInterval Type:duration Default:5m The interval of which the token issuer for various Prometheus endpoints refreshes the token for monitoring. The tokens that are affected by this config are the same as the one in Monitoring.TokenExpiresIn. This value must be less than Monitoring.TokenExpiresIn. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.MetricAuthorization Type:bool Default:true If authorization (Bearer token) is required for accessing /metrics endpoint. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.PromQLAuthorization Type:bool Default:true If authorization (Bearer token or cookie) is required for accessing /prometheus/query endpoint. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.DataRetention Type:duration Default:360h The duration of which Prometheus should retain the monitoring data. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.DataRetentionSize Type:string Default:0B The maximum number of bytes of storage blocks to retain. The oldest data will be removed first. This is used to limit the amount of data that can be stored in the database. This parameter is equivalent to the Prometheus `storage.tsdb.retention.size` flag. For more information, see [Prometheus Storage Documentation](https://prometheus.io/docs/prometheus/latest/storage/). Units supported: B, KB, MB, GB, TB, PB, EB Ex: "512MB" origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.LabelLimit Type:int Default:64 The maximum number of labels that can be attached to a single metric. 0 means no limit. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.LabelNameLengthLimit Type:int Default:128 The maximum length of a label name. 0 means no limit. The default value is 128 bytes, which allows for up to 32 characters. For an example where the LabelNameLengthLimit is set to 24, meaning the label name can only be 6 characters long (This is very unrealistic). ``` metric{name="Alice"} -> OK other_metric{very_long_label_name="Alice"} -> Error ``` It is worth noting that the picking sensible values is really important. The Prometheus server will reject any metrics that exceed the limit, meaning that the metrics will not be scraped and will not be available for querying. If it is too large we could exceed the memory limits of Prometheus. Be wary of modify this value. Similar considerations should be made for modifying Monitoring.LabelValueLengthLimit, Monitoring.LabelLimit, and Monitoring.SampleLimit. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.LabelValueLengthLimit Type:int Default:2048 The maximum length of a label value. 0 means no limit. The default value is 2048 bytes, which allows for up to 512 characters. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.SampleLimit Type:int Default:200 Per-scrape limit on the number of scraped samples that will be accepted. If more than this number of samples are present after metric relabeling the entire scrape will be treated as failed. 0 means no limit. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.StorageHealthCheckInterval Type:duration Default:5m The interval at which the server checks filesystem storage consumption for health monitoring. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.StorageWarningThreshold Type:int Default:92 The storage usage percentage threshold at which a warning health status is reported. The value should be between 0 and 100 representing the percentage of storage used. origincachedirectorregistrybrokerlocalcache * * * ##### Monitoring.StorageCriticalThreshold Type:int Default:97 The storage usage percentage threshold at which a critical health status is reported. The value should be between 0 and 100 representing the percentage of storage used. origincachedirectorregistrybrokerlocalcache * * * #### Shoveler * * * ##### Shoveler.Enable Type:bool Default:false Enable the XRootD monitoring shoveler. The shoveler gathers UDP monitoring messages from XRootD servers and sends them to a message bus, such as [RabbitMQ](https://www.rabbitmq.com/). For more information, see https://github.com/opensciencegrid/xrootd-monitoring-shoveler origincache * * * ##### Shoveler.MessageQueueProtocol Type:string Default:amqp Select which protocol to use in order to connect to the MQ. Options are amqp, stomp. For amqp, the following configurations are required: - URL: amqps://username:password@example.com/vhost - Topic: mytopic - AMQPExchange: shoveled-xrd - AMQPTokenLocation: /etc/pelican/xrootd-monitoring-shoveler-token For stomp, the following configurations are required: - URL: messagebroker.org:port - Topic: mytopic - StompUsername: username - PasswordLocation: path/to/password/file - StompCert: path/to/cert/file - StompCertKey: path/to/certkey/file origincache * * * ##### Shoveler.URL Type:url Default:none For amqp and stomp. The URL to connect to the shoveler. origincache * * * ##### Shoveler.Topic Type:string Default:none For amqp and stomp. The topic of the messages. For stomp, it defaults to xrootd.shoveler. origincache * * * ##### Shoveler.AMQPExchange Type:string Default:shoveled-xrd For amqp only. The exchange to shovel messages. origincache * * * ##### Shoveler.AMQPTokenLocation Type:filename Default:$ConfigBase/shoveler-token Root Default:/etc/pelican/shoveler-token For amqp only. A filepath to the location of the JWT used for authenticating amqp connection. origincache * * * ##### Shoveler.StompUsername Type:string Default:none For stomp only. Username for authentication. origincache * * * ##### Shoveler.PasswordLocation Type:filename Default:none For stomp only. Password file location for authentication. origincache * * * ##### Shoveler.StompCert Type:filename Default:none For stomp only. A filepath to the location of the TLS certificate. origincache * * * ##### Shoveler.StompCertKey Type:filename Default:none For stomp only. A filepath to the location of the private key associated with the certificate. origincache * * * ##### Shoveler.PortLower Type:int Default:9930 The lower end of a range of Shoveler ports for Shoveler to set up UDP server. origin * * * ##### Shoveler.PortHigher Type:int Default:9999 The lower end of a range of Shoveler ports for Shoveler to set up UDP server. origin * * * ##### Shoveler.OutputDestinations Type:stringSlice Default: A list of IP:Port destinations to forward XRootD monitoring packet to. origincache * * * ##### Shoveler.VerifyHeader Type:bool Default:false Whether to verify the header of the packet matches XRootD's monitoring packet format. origincache * * * ##### Shoveler.QueueDirectory Type:filename Default:$ConfigBase/shoveler/queue Root Default:/var/spool/pelican/shoveler/queue Directory to store overflow of queue onto disk. The queue keeps 100 messages in memory. If the shoveler is disconnected from the message bus, it will store messages over the 100 in memory onto disk into this directory. Once the connection has been re-established the queue will be emptied. The queue on disk is persistent between restarts, so a persistent directory should be used. origincache * * * ##### Shoveler.IPMapping Type:object Default:none IP Mapping for remote IP addresses in forwarding to the destinations. You may either pass one IP address to map all messages to the configured origin, or a list of key-value pairs for one-to-one mapping. One-to-all mapping: ``` IPMapping: - All: "172.0.0.4" ``` If a packet comes in with the private ip address of 192.168.0.4, the packet origin will be changed to 172.0.0.4 The port is always preserved. One-to-one mapping: ``` IPMapping: - Source: "192.168.0.5" Dest: "172.0.0.5" - Source: "192.168.0.6" Dest: "129.93.10.7" ``` origincache * * * #### Plugin * * * ##### Plugin.DirectorDecisionPercentage Type:int Default:20 The percentage of transfers for which the plugin will request director decision information and include it in the transfer ad's DeveloperData. A value of 0 disables this feature; a value of 100 requests it for every transfer. The director decision information describes how the director chose and ranked the servers for the transfer. plugin * * * ##### Plugin.Token Type:string Default:none The specified token for pelican plugin staging. plugin * * * #### StagePlugin * * * ##### StagePlugin.Hook Type:bool Default:false Flag to specify HTCondor hook behavior. plugin * * * ##### StagePlugin.MountPrefix Type:string Default:none Prefix corresponding to the local mount point of the origin. plugin * * * ##### StagePlugin.OriginPrefix Type:string Default:none Prefix corresponding to the local origin. plugin * * * ##### StagePlugin.ShadowOriginPrefix Type:string Default:none Prefix corresponding to the shadow origin. plugin * * * #### Lotman * * * ##### Lotman.LotHome Type:filename Default:$ConfigBase Root Default:/var/lib/lotman The prefix indicating where LotMan should store its lot database. Configured with path ``, the database will be stored at `/.lot/lotman_cpp.sqlite`. cache * * * ##### Lotman.DbLocation Type:filename Default:$ConfigBase Root Default:/var/lib/lotman \[Deprecated\] `Lotman.DbLocation` is deprecated and will be removed in a future release. Please migrate to use `Lotman.LotHome` instead. The prefix indicating where LotMan should store its lot database. For the provided path, the database will be stored at /.lot/lotman\_cpp.sqlite. cache * * * ##### Lotman.LibLocation Type:filename Default:none The location of the system's installed LotMan library (libLotMan.so). When unset, the system will attempt to find Lotman at these fallback paths: - /usr/lib64/libLotMan.so - /usr/local/lib64/libLotMan.so - /opt/local/lib64/libLotMan.so cache * * * ##### Lotman.EnableAPI Type:bool Default:false Whether Lotman should enable its CRUD web endpoints. If true, administrators with an appropriately-signed token can interface with Lotman via HTTP. Otherwise, lots are only configurable via the Pelican configuration file at the cache. cache * * * ##### Lotman.PolicyDefinitions Type:object Default:\[object Object\] A list of named Lotman purge policy definitions that may be enabled by the cache administrator through setting the `Lotman.EnabledPolicy` configuration. Each policy definition is an object with the following fields: - `PolicyName`: The name of the policy. This is used to identify the policy in the `Lotman.EnabledPolicy` configuration. - `PurgeOrder`: An ordered list of strings indicating the order in which lots should be purged. The strings should be one of the following: - `del`: Purge lots that have passed their deletion time. - `exp`: Purge lots that have passed their expiration time. - `opp`: Purge lots that have passed their opportunistic storage quota. - `ded`: Purge lots that have passed their dedicated storage quota. - `DiscoverPrefixes`: A boolean indicating whether Lotman should automatically discover prefixes from the Director. If true, Lotman will attempt to create lots for all discovered federation prefixes. Locally-defined lots will take precedence over discovered lots if the two have the same name. - `MergeLocalWithDiscovered`: A boolean indicating whether Lotman should merge locally-defined lot configurations with discovered namespaces. Most Lot configuration fields will take precedence from local configuration, but the `Paths` and `Parents` fields are additive. - `DivideUnallocated`: A boolean indicating whether Lotman should attempt to make intelligent decisions regarding management policy attributes for lots that have not provided explicit values. These decisions are based on the cache's total storage capacity and the number of lots that have been explicitly configured, and are intended to maximize potential cache utilization. This should be set to "true" in most cases. - `Lots`: A list of lot objects, each of which describes a "lot". Every lot can be defined with the following: - `LotName`: REQUIRED. The name of the lot. This is used to identify the lot in the LotMan database. - `Owner`: REQUIRED. A string identifying the owner of the lot's data (as opposed to someone who can modify the lot itself). The Owner field should generally be set to the issue for the lot's namespace path. For example, if the lot tracks namespace `/foo/bar`, the owner might be set to `https://registry.com/api/v1.0/registry/foo/bar`. - `Paths`: OPTIONAL. A list of path objects, each of which describes a path that should be managed by the lot. - `Path`: REQUIRED. The path to be managed by the lot. - `Recursive`: REQUIRED. A boolean indicating whether the path should be managed recursively. If true, the lot will manage all files and directories under the specified path. - `ManagementPolicyAttrs`: REQUIRED. The lot's management policy attributes object. This contains information about resources the lot should be allocated, and how it should be managed. - `DedicatedGB`: REQUIRED. The amount of storage, in GB, that should be dedicated to the lot. This means the lot can assume it always has access to this quantity. - `OpportunisticGB`: REQUIRED. The amount of opportunistic storage, in GB, the lot should have access to, when storage is available. - `MaxNumObjects`: REQUIRED. The maximum number of objects a lot is allowed to store. - `CreationTime`: REQUIRED. A unix timestamp indicating when the lot should begin being considered valid. Times in the future indicate the lot should not be considered valid until that time. - `ExpirationTime`: REQUIRED. A unix timestamp indicating when the lot expires. Lots may continue to function after expiration, but lot data owners should recognize the storage is at-will and may be preempted at any time. - `DeletionTime`: REQUIRED. A unix timestamp indicating when the lot and its associated data should be deleted. For example, Lotman could be configured with the "my-policy" policy with the following: ``` Lotman: EnabledPolicy: "my-policy" PolicyDefinitions: - PolicyName: "my-policy" DivideUnallocated: true PurgeOrder: ["del", "exp", "opp", "ded"] DiscoverPrefixes: true MergeLocalWithDiscovered: true Lots: - LotName: "/foo/bar" Owner: "https://registry.com/api/v1.0/registry/foo/bar" Paths: Path: "/foo/bar" Recursive: true ManagementPolicyAttrs: DedicatedGB: 100 OpportunisticGB: 100 MaxNumObjects: 1000 CreationTime: 1614556800 ExpirationTime: 1614556800 DeletionTime: 1614556800 - LotName ... ``` Additional example configurations can be found in lotman/resources/lots-config.yaml For more information about LotMan configuration, see: [https://github.com/pelicanplatform/lotman](https://github.com/pelicanplatform/lotman) cache * * * ##### Lotman.EnabledPolicy Type:string Default:fairshare The name of the policy to use with Lotman's purge logic. Policy names are defined in the Lotman.PolicyDefinitions list object. If unset, the "fairshare" policy is used, which evenly divides the cache's space amongst all top-level namespaces discoverable through the Director and purges data according in order of lots past deletion, lots past expiration, lots past opportunistic storage, and lots past dedicated storage. The "fairshare" policy is defined as follows: ``` Lotman: EnabledPolicy: "fairshare" DefaultLotExpirationLifetime: "2016h" DefaultLotDeletionLifetime: "4032h" PolicyDefinitions: - PolicyName: "fairshare" DivideUnallocated: true PurgeOrder: ["del", "exp", "opp", "ded"] DiscoverPrefixes: true MergeLocalWithDiscovered: false ``` cache * * * ##### Lotman.DefaultLotExpirationLifetime Type:duration Default:2016h The default expiration lifetime for lots that have not provided an explicit expiration time. Valid time units are: - ns for nanoseconds - us (or µs) for microseconds - ms for milliseconds - s for seconds - m for minutes - h for hours This value is fed to Lotman as a unix timestamp in microseconds, adjusted from the current time. cache * * * ##### Lotman.DefaultLotDeletionLifetime Type:duration Default:4032h The default deletion lifetime for lots that have not provided an explicit deletion time. Valid time units are: - ns for nanoseconds - us (or µs) for microseconds - ms for milliseconds - s for seconds - m for minutes - h for hours This value is fed to Lotman as a unix timestamp in microseconds, adjusted from the current time. cache * * * --- Source: https://docs.pelicanplatform.org/getting-started # Getting Started This page is intended to help users and administrators find the right documentation to accomplish what they need to. Each section header describes the type of actions that section is intended to help with. ## Getting Objects From An Existing Federation Objects are referenced in Pelican using `pelican://`\-schemed URLs, which help Pelican clients identify the correct federation and routing information they need to find the source of any object. Whether you know the `pelican://`\-url name of your object or not, you can find a quick tutorial for interacting with data using the Pelican command line client at [Accessing Data](./getting-started/accessing-data). For a more complete walkthrough of Pelican’s clients, see [Getting Data with Pelican](./getting-data-with-pelican), which discusses each of Pelican’s clients and explains how to find detailed documentation for each. This page also discusses the general methods for providing authorization to access “protected” objects through a federation. ## Adding Your Data To an Existing Federation A goal of Pelican is to make data easily accessible regardless of how it’s stored. If you have data and you want to add it to a federation, you have two options: **a)** Use a Pelican client to write the data from its existing storage location to a location already federated via a Pelican Origin **b)** Connect your own Pelican Origin to the data wherever it’s already stored Choosing the option that’s best for you depends on a few factors, like how much data you have and whether you have previous experience administering servers. Pelican aims to make the process of running an Origin as simple as setting up a new home wifi router, but until we’ve realized that goal you may be better off working with an existing Origin administrator. If you plan to add this data to the OSDF, you can get help by emailing [support@osg-htc.org](mailto:support@osg-htc.org). ### Use a Pelican Client to Write Data via an Origin You can use Pelican clients to write data to a location within an existing federation. Such write (or PUT) actions using a Pelican client require that you provide some form of authorization. For more information on the authorization process, see [Getting Data with Pelican](./getting-data-with-pelican). > **NOTE:** Make sure you use the authorization method required by the maintainer of the namespace within the federation (which may not be the same as the maintainer of the federation). ### Serving An Origin If you maintain the storage for the data you want to share, you can connect your storage system to an existing federation. Integrating your data with a Pelican federation starts by serving an Origin in front of whatever service already holds the data. Origins are a critical component in Pelican federations because they act as the adapter plug that lets a broad variety of storage technologies (posix, S3, HTTP, Globus, etc) interact with Pelican’s clients and caching infrastructure. For a more complete discussion of Origins and how you can run your own, see [Federating Your Data](./federating-your-data). > **NOTE:** You should contact the federation maintainer to discuss their requirements/recommendations for setting up an Origin. ## Sharing Your Storage Resources If you administer storage resources and you want to share them with a Pelican federation, you can do so by: **a)** Serving a Cache ([see below](#serving-a-cache)), or **b)** [Serving an Origin](#connect-your-data-storage-to-an-existing-federation-through-serving-an-origin) Both options will benefit data consumers, but which you choose may depend on your particular hardware. For example, Pelican Caches work best when they’re run with low-latency storage like modern NVMe SSDs and high-bandwidth network connections. Origins, on the other hand, are assumed to be less performant. In either case, sharing your storage resources will require some coordination with the administrators for your federation of choice. The best place to start is by reaching out to them. If you know the hostname of your federation’s Director, you can usually find a contact email at: `< director hostname >/api/v1.0/director_ui/contact`. ### Sharing your storage resources with the OSDF Have storage but don’t know any federations to share it with? Consider contributing to the Pelican flagship [Open Science Data Federation (OSDF)](https://osdf.osg-htc.org/) . This well-established federation is used to distribute data within the [OSPool](https://osg-htc.org/services/open_science_pool.html) , a national distributed computing system that is accessible to any researcher affiliated with a US academic institution. The OSDF is connected directly with network backbones like ESNet and Internet2, enabling researchers across the country to incorporate their data in their OSPool workloads. > **Note:** If you or your campus have a storage allocation granted through something like the NSF’s [CC\*](https://new.nsf.gov/funding/opportunities/cc-campus-cyberinfrastructure)  program, contributing storage to the OSDF can help satisfy some grant requirements. ### Serving a Cache Pelican federations are built from the ground up to take advantage of distributed object caching. Pelican Caches enable a federation to distribute data - especially frequently reused data - even more efficiently. This means that even if you don’t have data to share in a federation, you can help a federation run more efficiently by serving a cache that utilizes your existing storage resources. If you are interested in contributing a cache to an existing federation, see [Operating a Federation/Cache](./operating-a-federation/cache) for the general process. ## Starting Your Own Federation The Pelican flagship [Open Science Data Federation (OSDF)](https://osdf.osg-htc.org/)  is a highly-performant, well-established federation connected directly with network backbones like ESNet and Internet2. In the majority of cases, we recommend you join the OSDF to gain access to this infrastructure. However, if the OSDF does not do not meet the requirements for your data distribution needs, you can still leverage the power of Pelican by launching and maintaining your own Pelican federation. The process of running/managing a Pelican object federation involves setting up two primary services: a Registry and a Director. These services are called the federation’s “Central Services”, and together they handle the registration/verification of Caches and Origins (Registry), and maintaining an understanding of where clients should be sent to interact with objects (Director). You can find more information about operating these services in the [Operating a Federation](./operating-a-federation) section. Information specific to Directors can be found in the sub page titled [Director](./operating-a-federation/director) while information for Registries can be found in the sub page titled [Registry](./operating-a-federation/registry). --- Source: https://docs.pelicanplatform.org/getting-started/accessing-data # Accessing Data The developers of the Pelican Platform believe data is the lifeblood of science. That’s why Pelican provides a suite of tools that make it easy for researchers to federate their data, giving them the ability to share objects from a broad range of data repositories and allowing them to connect their data with distributed computational capacity, such as the OSPool. For more information on Pelican, visit the [About Pelican](../about-pelican) page. ## Before Starting ### Assumptions Before using the Pelican client to interact with objects, this guide makes several assumptions: - You are on a computer where you have access to a terminal. The Pelican client is a command line tool. - You’ve already installed the version of Pelican appropriate for your system, and Pelican is accessible via your path. To test this on Linux, you can run ``` which pelican ``` which should output a path to the executable. If there is no output to this command, refer to the [Pelican installation docs](../install) to acquire a working installation. ### Useful Terminology To learn our terminology and about Pelican’s architecture, we recommend you read through the [About Pelican](../about-pelican) page and [Core Concepts and Terminology](../about-pelican/core-concepts) page to learn how Pelican works and how it can be useful to you. ## Your First Pelican Client Command You can easily access public objects from a Pelican federation with the use of the Pelican client commands. Here we will be using the `pelican object get` command to get an object from a federation. This command is formatted like so: ``` pelican object get pelican:/// ``` We can break down the components of this command before you try it yourself: - `pelican object get`: This is the client command we are utilizing. `object` because we want to interact with Pelican objects and `get` because we want to GET the file from our federation. - `pelican://`: This is the URL scheme for accessing this object. Pelican accepts different URL schemes which are detailed in the [Pelican Client usage](./client_usage#the-different-pelican-url-schemes) page. - ``: The URL indicating which federation you would like to interact with. - ``: The path to the object we would like to get. - ``: The local destination where we want the file to download to (can be a local or absolute path) ### Try The Command Yourself Now it is time for you to get your first object from a federation. Before getting your first object, be sure you are in a directory/location where you want to download this test file. To get the object, run the command below: ``` pelican object get pelican://osg-htc.org/pelicanplatform/test/hello-world.txt downloaded-test.txt ``` > **Note:** with pelican:// URLs like the one above, it is important to include the correct number of slashes (`/`). This is because pelican URLs rely on the hostname being the federation URL therefore, they must only contain two slashes after `pelican`. For more information, visit the [Pelican Client Usage](../getting-data-with-pelican/client#the-different-pelican-url-schemes) page. You should see a progress bar output and eventually a file named `downloaded-test.txt` within your local directory: ``` $ pelican object get pelican://osg-htc.org/pelicanplatform/test/hello-world.txt downloaded-test.txt downloaded-test.txt 27.00 b / 27.00 b [==============================================================================] Done! $ ls downloaded-test.txt $ cat downloaded-test.txt If you are seeing this message, getting an object from OSDF was successful. ``` ## Where to Go from Here **For More Info on Pelican:** - Go to our website located at [https://pelicanplatform.org](https://pelicanplatform.org)  **For Users:** - Learn more advanced client-usage from the [Pelican Client Usage](../getting-data-with-pelican/client) page. - Learn how to federate your data to a Pelican federation with the [Pelican Origin](../federating-your-data) **For Admins:** - Learn how to operate your own [Pelican Federation](../operating-a-federation) --- Source: https://docs.pelicanplatform.org/getting-data-with-pelican # Getting Data With Pelican Pelican is built on top of HTTP and uses the most common [HTTP verbs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods)  to interact with data: downloading objects happen with an HTTP “GET”, uploading an object is an HTTP “PUT”, and discovering data uses a combination of HTTP’s “HEAD” and WebDav’s “PROPFIND”. This architecture means any tool that already speaks HTTP _can_ integrate with Pelican. However, Pelican’s official clients — any of our tools designed for interacting with remote objects — make intelligent use of special information provided by federation services like the Director to deliver the best experience. They also have built-in optimizations that help interact with data more efficiently, like multi-worker object streaming, automatic retry policies, and tools for packing/unpacking objects while they’re in flight. One of the Pelican Platform’s core goals is enabling data access wherever it needs to happen — whether that’s from the the command line, from a browser, in an HTC workflow, a PyTorch training loop, or a Jupyter Notebook. To that end, we’ve been working hard to develop and maintain a wide range of clients that meet our users’ diverse needs. Information about each of our clients can be found in this section and are laid out by client type. ## Which Client Is Right For You Picking a client starts with understanding what you want to accomplish and where you want to accomplish it. ### Pelican’s Command Line Client Pelican’s command line client (also referred to as the Pelican CLI) excels at broad object manipulation and management tasks, including writing/reading large collections of objects, syncing data between local and remote resources, and discovering data that’s accessible through a given namespace/federation prefix. For details about Pelican’s CLI, see [Getting Data With Pelican/Command Line Client](./getting-data-with-pelican/client). ### Pelican’s Python Filesystem Specification If your goal is to integrate Pelican with Python, you’re looking for our [Pelican Filesystem Specification](https://github.com/PelicanPlatform/pelicanfs) , or “FSSpec” for short. This client lets you interact with Pelican objects at any level of your code, including by plugging Pelican directly into popular Python libraries like [xarray](https://github.com/PelicanPlatform/pelicanfs/tree/main/examples/xarray)  and [PyTorch data loaders](https://github.com/PelicanPlatform/pelicanfs/tree/main/examples/pytorch) . ### Pelican’s HTCondor Plugin Pelican maintains a plugin that acts as the preferred transfer tool for [HTCondor](https://htcondor.org/) , a software suite used in many distributed/clustered compute federations, such as the [OSPool](https://osg-htc.org/services/open_science_pool.html) . This plugin leverages HTCondor’s existing plugin architecture, enabling HTCondor to manage all `pelican://` and `osdf://` file transfers as part of distributed, high-throughput computing workflows. For more information on how file transfer plugins work in HTCondor, see [HTCondor’s documentation](https://htcondor.readthedocs.io/en/latest/users-manual/file-transfer.html#file-transfer-using-a-url) . --- Source: https://docs.pelicanplatform.org/getting-data-with-pelican/client # Pelican’s Command Line Client It is recommended that you start with our [Quick Start Guide](../getting-started/accessing-data) where you can learn the basics of Pelican Client commands and learn some useful terminology. ## Before Starting ### Assumptions Before using the Pelican client to interact with objects from your federation, this guide makes several assumptions: - You are on a computer where you have access to a terminal. The Pelican client is a command line tool. - You’ve already installed the version of Pelican appropriate for your system, and Pelican is accessible via your path. To test this on Linux, you can run ``` which pelican ``` which should output a path to the executable. If there is no output to this command, refer to the [Pelican installation docs](../install) to acquire a working installation. - You are roughly familiar with Pelican client commands and terminology as demonstrated in the [Quick Start Guide](../getting-started/accessing-data). ### Note on Federations All object paths in a federation begin with a preceding `/`, and no relative paths are allowed. Here is the example URL we used in the [Quick Start Guide](../getting-started/accessing-data#federations): ``` /pelicanplatform/test/hello-world.txt ``` This is the full object path that we will be using in examples below. ### Tokens and JWT Some namespace prefixes are public, like `/pelicanplatform/`, while others are protected (i.e. they require authorization). Objects in public namespaces can be downloaded by anybody, but downloading objects from protected namespaces requires you prove to the origin supporting that namespace that you are allowed to access the object. In Pelican, this is done using signed JSON Web Tokens, or _JWT_s for short. In many cases, these tokens can be generated automatically. ### The Different Pelican URL Schemes When running client commands with a request URL, there are two different URL schemes a user can use: `pelican` and `osdf`. This page uses examples with the `pelican://` URL scheme. Below is a description of the two URL schemes and what to expect when they are used: #### The `pelican://` URL Scheme URLs that begin with `pelican://` allow you to interact with objects from any federation simply by knowing the hostname of the federation’s discovery URL and the name of that object in that federation. These URLs take the form: ``` pelican:/// ``` When a Pelican client encounters a Pelican URL, it uses the hostname in the federation discovery portion of the URL to find other services in the specified federation that will help the client discover the object’s location > **Note**: Federation discovery URLs can also be specified with the `-f` flag rather than the hostname. Just be sure not to include the URL in both the `-f` flag and in the hostname of the object path, as this will lead to errors. > **Note:** Federation discovery URLs cannot contain a path component (e.g. [https://my-federation.com/path/component](https://my-federation.com/path/component) ). Clients that encounter a discovery URL with a path component will incorrectly assume the path is part of the namespace prefix. #### The `osdf:///` URL Scheme: When using the `osdf:///` scheme, Pelican will load in the federation metadata automatically using the defaults for accessing the Open Science Data Federation (OSDF). This scheme should only be used if your federation or data can be accessed through the OSDF. When using this scheme, you should not specify the federation URL at all, as Pelican will handle this automatically. For example: ``` pelican object get osdf:/// ``` > This scheme has three slashes (`///`) after the `osdf` because the hostname is left empty to be automatically populated therefore, just start the URL with the namespace prefix. Pelican currently recognizes the `osdf://` scheme (with two slashes) in case the user forgets to pass the third slash, but this is not recommended. ## Get a Public Object from your Federation To use the pelican client to get public objects from a federation, use Pelican’s `object get` sub-command ### Getting an Object Using the `pelican://` URL Scheme: To use the `pelican://` URL scheme, you need to specify the federation URL within the request URL. To do this, you would format the `object get` command like so: ``` pelican object get pelican:// ``` You can try this yourself by getting the public file that was mentioned earlier from the OSDF. Using the `object get` sub command, and providing the federation URL for the OSDF: ``` pelican object get pelican://osg-htc.org/pelicanplatform/test/hello-world.txt downloaded-testfile.txt ``` This command will download the object `test/hello-world.txt` from the OSDF’s `/pelicanplatform/` namespace and save it in your current working directory with the name `downloaded-testfile.txt`. More specifically, it breaks into these components: - Federation hostname: `osg-htc.org` - Namespace: `/pelicanplatform/` - Object name: `test/hello-world.txt` - Destination: `downloaded-testfile.txt` ## Get A Protected Object From Your Federation Protected namespaces require that a Pelican client prove it is allowed to access objects from the namespace before the object can be downloaded. In many cases, Pelican clients can guide users through the process of acquiring a token by initiating an OpenID-Connect (OIDC) flow that uses an external log-in service (such as [CILogon](https://www.cilogon.org) ) through your browser. In other cases, a token must be provided to the Pelican client manually. ### For Issuers That Support CILogon Code Flow Some origins support authentication with CILogon’s OIDC client. In these cases, the Pelican client is capable of guiding users through authentication with the origin, once you logged in via CILogon. To download protected objects from origins that support CILogon, run the same command as for downloading a public object: ``` pelican object get pelican:// ``` If you’re doing this for the very first time, Pelican will create an encrypted token wallet on your system and you will be required to provide Pelican with a password for the wallet. If this isn’t your first time, you will be asked to provide your already-configured password to unlock the token wallet. The output looks like so: ``` Please create a new password to protect your local credentials: ``` Enter in a password and press `enter` to continue and you should see a response like so: ``` To approve credentials for this operation, please navigate to the following URL and approve the request: ``` Pelican will display a URL in your terminal and indicate that you should visit the URL in your browser. After copying/pasting the URL to your browser, follow all the instructions there for logging in with CILogon. Once everything is successful, you should see a page like this: ![Screenshot of CILogon success page](/pelican/optimized_images/cilogon-opt-1920.WEBP) Finally, if the login is successful, Pelican will automatically fetch the token from the CILogon service and continue with the download. ### For Issuers Without CILogon Support Some origins do not support authentication though CILogon. In this case, users must supply their own JWT that’s signed by the origin. Contact your federation administrator to get a token for downloading a protected object. For Origin and federation admins, follow the documentation to [generate an object access token](../federating-your-data/generating-tokens) for your users. For OSDF users, contact [support@osg-htc.org](mailto:support@osg-htc.org) for help. Once you have the token, copy and paste it into a file on your machine, and pass the path to the file containing the token to the Pelican client with `-t` flag. ``` # Pass the path to the file containing the token using -t flag pelican object get pelican:// -t ``` For example, if a token is saved in a file named `my-token`, it can be used to get the object `/ospool/PROTECTED/auth-test.txt` by running: ``` # Get the object with the path to token file provided via -t flag $ pelican object get -f https://osg-htc.org /ospool/PROTECTED/auth-test.txt downloaded-auth-test.txt -t my-token ``` ## PUT an Object to a Data Repository via the Federation Another powerful Pelican client command is the `pelican object put` command. This command does a simple PUT request to add your object to a data repository via the federation, and putting files into a data repository always requires a token. For the example, we will need a token to perform these requests (see the [previous section](#get-a-protected-object-from-your-federation) for more information). Here is how you can use `pelican object put`: ``` pelican object put pelican:// -t ``` > **Note:** you can also specify the federation url here with the `-f` flag, just be sure not to include it in the request URL as the host name if you decide to do so. ## Pelican Object Copy > **Note**: We are phasing out the `object copy` command and we recommend user use `object get` and `object put` command instead. As well with Pelican’s `object get` and `object put` commands, there is also an older command called `pelican object copy`. This functions the same as the `pelican object get/put` except that it works for both gets and puts. For example, to do an `object get`: ``` pelican object copy pelican:// ``` and to do an `object put`: ``` pelican object copy pelican:// -t ``` ## Utilizing Queries with your URL The Pelican client allows users to modify the behavior of requests by passing URL query parameters in the remote path of an object. Currently supported queries include: `?pack`, `?recursive`, and `?directread`. ### Packing Objects with the `?pack` Query The Pelican client has the ability to automatically compress/unpack tarballs when uploading and downloading to and from a federation. The user just needs to supply their request URL with a query. For example, to automatically unpack a tarball on downloading, run: ``` pelican object get pelican://?pack=tar.gz ``` To upload, specify a directory and Pelican will compress it for you: ``` pelican object put pelican://?pack=tar.gz ``` Pelican accepts the following values for the `pack` query: - `pack=auto`: - For downloading, auto-detect the file format and unpack (throws error if it is not any detected format). - For uploading, compress using `.tar.xz`. - `pack=tar`, `pack=tar.gz`, `pack=tar.xz`, `pack=zip` : - For downloading, throws an error if the specified object is not in the specified format (`tar`, `tar.gz`, `tar.xz`, `zip`, respectively). - For uploading, create the object in the specified format (`tar`, `tar.gz`, `tar.xz`, `zip`, respectively). ### Recursive Downloads and Uploads with the `?recursive` Query The `?recursive` query can be utilized if the desired remote object is a collection. When this query is enabled, it indicates to Pelican that all sub paths at the level of the provided namespace should be copied recursively. To use this query, run: ``` pelican object get pelican://?recursive ``` To upload, you can run something similar but with an `object put`: ``` pelican object put pelican://?recursive ``` > **Note:** This query functions the same as specifying the `-r` flag described below. ### Bypass Caches for Downloads with the `?directread` Query The `?directread` query is used if you would like to download an object without utilizing a cache. This way, the object will come directly from the origin each time and never use the cache. To use this query, you can run: ``` pelican object get pelican://?directread ``` This query does not make sense to use for uploads since uploads go directly to the origin anyway. If you use this query by mistake, you should not run into any issues and your upload will function as normal. ### Note about Queries The `?recursive` and the `?directread` queries do **not** require any sort of values assigned to them (e.g. pelican://some/object?recursive=true). If a value is assigned to these queries, that value will be ignored and Pelican will act as if there was no value assigned to that query (e.g. `pelican://some/object?recursive=false` acts the same as `pelican://some/object?recursive` meaning the recursive query **will** be set even if the value is set to false). ## Additional Flags The Pelican client supports a variety of command line flags that modify the client’s behavior: ### Global Flags: - **`-h` or `--help`:** Takes no argument and can be used with any Pelican sub command for more information about the sub command and additional supported flags. - **`-f` or `--federation`:** Takes a URL that indicates to Pelican which federation the request should be made to. - **`-d` or `--debug`:** Takes no argument, but runs Pelican in debug mode, which provides verbose output for debugging purposes. - **`--config`:** Takes a filepath and indicates to Pelican the location of the Pelican configuration file. - **`--json`:** Takes no argument and outputs results in JSON format. - **`-l` or `--log`:** Takes a string that specifies a file location to output the pelican logs to rather than the stderr. - **`--version`:** Prints the version of Pelican and exits. ### Flags For `object get/put/copy`: - **`-c` or `--cache`:** Takes a cache URL and indicates to Pelican that only the specified cache should be used. When used, Pelican will not attempt to use other caches if the provided cache cannot provide the file. - **`--caches`:** Takes the path to a JSON file containing a list of caches. Similar to the `-c` flag, Pelican will attempt to use only these caches in the order they are listed. - **`-h` or `--help`:** Gives additional information on how to use the command as well as lists these flags with short descriptions for the `object copy` command. - **`--methods`:** Takes a comma separated list of methods to try for downloads/uploads, the default is just http. - **`-r` or `--recursive`:** Takes no argument and indicates to Pelican that all sub paths at the level of the provided namespace should be copied recursively. This option is only supported if the origin supports the WebDav protocol. - **`-t` or `--token`:** Takes a path to a file containing a signed JWT, and is used to download protected objects. ## Binary Aliases Both Pelican binaries inspect their own name at startup and change behavior accordingly. ### `pelican` Client Aliases The `pelican` client binary supports the following name-based modes: #### Prefixing The Binary Name With `osdf` When the name of the `pelican` binary begins with `osdf`, Pelican will assume that all objects are coming from the OSDF which allows it to make several assumptions. The most immediate effect for users is that you no longer need to specify a federation URL, ONLY with no URL scheme or an `osdf:///` URL scheme. The command to download a public file from above can then be simplified to: ``` osdf object copy /pelicanplatform/test/hello-world.txt downloaded-testfile.txt ``` > **Note:** When using client commands with the OSDF binary, be careful when using it with `pelican://` URL schemes. When using the `pelican://` URL scheme, you are still required to provide a federation URL no matter what binary name you are using. #### Naming The Binary `stashcp` Or `stash_plugin` The Pelican Platform grew out of a command line tool called `stashcp` with an associated HTCondor plugin called `stash_plugin`, which were also used for interacting with objects in the OSDF. To support these legacy tools, Pelican has been built to behave similarly as `stashcp` and `stash_plugin` did whenever the Pelican binary is renamed to match the names of these tools. ### `pelican-server` Server Alias (Docker only) In the official Pelican Docker images (e.g. `hub.opensciencegrid.org/pelican_platform/osdf-origin`), a symlink named `osdf-server` is created that points to `pelican-server`. When the server binary is invoked as `osdf-server`, it automatically applies OSDF defaults (equivalent to setting `Federation.DiscoveryUrl: https://osg-htc.org`) so OSDF container deployments do not need to specify the federation URL explicitly. The `osdf-server` symlink is used as the `ENTRYPOINT` in the `osdf-origin`, `osdf-cache`, `osdf-director`, and `osdf-registry` Docker image targets. It is a Docker-only convenience and is not installed by the RPM packages. --- Source: https://docs.pelicanplatform.org/getting-data-with-pelican/fsspec # Pelican’s Python FSSpec ## What is FSSpec? [FSSpec](https://filesystem-spec.readthedocs.io/en/latest/) , short for Filesystem Specification, is a Python library that provides a unified interface for interacting with various filesystems. It abstracts the complexities of different storage backends, allowing users to work with local files, cloud storage, and other remote filesystems using a consistent API. Pelican chose to develop an FSSpec implementation because it delivers maximum reward with less development effort by providing: - **Unified Interface**: A consistent API for different filesystems, meaning many users already interact with FSSpec in their projects. - **Extensibility**: Robust support for custom filesystem implementations Because Pelican is built on top of HTTP, our FSSpec is built on top of the core HTTP FSSpec. - **Compatibility**: FSSpec is used widely in the Python ecosystem, and already integrates seamlessly with tools like [xarray](https://github.com/PelicanPlatform/pelicanfs/tree/main/examples/xarray)  and [PyTorch data loaders](https://github.com/PelicanPlatform/pelicanfs/tree/main/examples/pytorch) . While we’re careful to say Pelican is a platform of tools for working with _objects_, because many of Pelican’s storage backends don’t have a true filesystem interface[1](#user-content-fn-1), FSSpec lets users interact with Pelican objects in a more familiar manner by translating common file operations such as reading, writing, listing directories, and more, without worrying about the underlying storage technology. ## How to Get it Instructions for downloading and working with Pelican’s FSSpec library can be found on our [FSSpec Github repository](https://github.com/pelicanplatform/pelicanfs) . If you have further questions or feature requests, we welcome new issues, contributions and discussions there! ## Footnotes 1. For more information about the fundamental differences between **_objects_** and **_files/filesystems_**, checkout [this publication from RedHat](https://www.redhat.com/en/topics/data-storage/file-block-object-storage) . [↩](#user-content-fnref-1) --- Source: https://docs.pelicanplatform.org/getting-data-with-pelican/auth # Working with Protected Data Whenever a new namespace joins a Pelican Federation, the Origins that serve the namespace tell the Federation about their data access policies. These policies say something about _who’s_ allowed to access the data and _how_ they’re allowed to access it. For example, a namespace’s policy might declare that all its data is publicly-readable, meaning anyone can access its objects via Pelican without any extra steps. In this case, anyone could use the Pelican CLI to download the data with a simple `pelican object get `. However, some namespaces use policies that impose authorization limits on their data. This is the case whenever a namespace allows writes (the ability to upload data is always protected) or non-public reads (see “[Origin and Namespace Capabilities](../../federating-your-data/origin#origin-and-namespace-capabilities)”). In these settings, users must prove to the Origin that they’re authorized to perform an operation (read/GET or write/PUT) on an object in the namespace. This document describes how Pelican CLI Client users can interact with data that requires some form of access authorization. It will gloss over most technical terms, preferring instead to provide only what’s needed for a client user to work with protected data. For a more technical walkthrough of Pelican’s authorization framework including how “authorization” differs from “authentication”, see “[Advanced Usage/Pelican’s Authorization System](../../advanced-concepts/auth)”. ## Background Most people are familiar with using “username & password” authentication to protect data, but this setup rarely scales well in federated services whose components cross administrative boundaries, which is how the OSDF functions. Instead, Pelican uses [authorization tokens](../../advanced-concepts/auth#tokens) to decide whether to grant or deny object access. The simplest description of authorization tokens is that they function much like concert tickets: - whoever has the ticket is allowed to sit in a specific seat at the concert venue - if you don’t have a ticket with the right paper, hologram or barcode, the ticket isn’t valid - you’re not allowed to sit in a seat other than what your ticket permits - your ticket gets you into the concert on Saturday night, but not the concerts on Friday or Sunday In Pelican, tokens act like digital concert tickets by granting the ability to perform a specific operation (e.g. GET or PUT) on a specific part of a namespace or object to anyone who possesses the token. Whenever a user tries to work with a protected resource, the success of their request depends on having the right token. While Pelican needs these tokens to enforce access policies, they can be very confusing for anyone who doesn’t regularly work with them. That’s why, in most cases, Pelican Client users won’t need to explicitly interact with tokens; Pelican Clients work hard to deal with them under the hood while exposing users to more familiar access methods, such as the ability to log into a “Single Sign-On” service like [CILogon](https://www.cilogon.org/home)  that generates tokens on users’ behalf. The section of this page titled “[Automatic Token Generation](./#automatic-token-generation)” describes the various setups that let users access protected data without needing to explicitly interact with tokens. If your namespace or local environment are not configured using one of these options, you may need to create tokens manually. See this page’s “[Explicit Token Creation/Management with Pelican](./#explicit-token-creation-&-management-with-pelican)” for instructions in that case. ## Automatic Token Generation Wherever possible, Pelican services should be set up in a way that does not require users to interact with tokens directly. The two methods that accomplish this apply to different Origin/namespace setups and are intended to serve different user audiences. In both cases, client user who runs a command like `pelican object ` will be walked through the steps to complete the transfer. The first of these methods — [OAuth2/OIDC integration](#automatic-tokens-for-namespacesorigins-that-support-oauth2oidc) — should be the standard case for the majority of users. This option most often applies to multi-tenant namespaces whose protected data is meant to be accessed by a variety of people. The second method is for users that own a namespace or otherwise possess the secret “private signing key” used to create the namespace at the Federation’s Registry service. If the last sentence didn’t mean anything to you, you probably don’t fall in this category! ### Automatic Tokens for Namespaces/Origins that Support OAuth2/OIDC This is the setup that should apply to most users; when you execute the Pelican CLI, it’ll provide you with a link that can be copy-pasted into the browser to complete the action. This section does not cover how Origin administrators can set up their services to function in this way; it’s meant only to describe client access to Origins/namespaces that already support it. For more information on Origin configuration, see “[Federating Your Data](../../federating-your-data)”. Documentation specific to Origin Issuer configuration is under development. For additional help with this in the meantime, reach out to [help@pelicanplatform.org](mailto:help@pelicanplatform.org). Here’s what it looks like in practice (note that this example is not copy-pastable — it’s for demonstration purposes only): **Example:** Download a protected object to the local directory ``` $ pelican object get pelican://osg-htc.org/protected-namespace/foo.txt ./The OSDF client configuration is encrypted. Enter your password for the local OSDF client configuration file: To approve credentials for this operation, please navigate to the following URL and approve the request:https://osdf-example-issuer.com/device?user_code=ABC-123-XYZ ``` After copy-pasting the URL into a browser and logging into the portal, the Client should automatically proceed to download the object In this example, the user is asked to enter a password _before_ being given the URL. This password unlocks a local, encrypted “token cache” that may already contain tokens to fulfill the request. When such a token exists, the URL will not be provided and the download will proceed with the cached token. If you ever forget the password for this local token cache, you can only reset it by deleting the file located at: - (for non-root users) `~/.config/pelican/credentials/client-credentials.pem` - (for root users) `/etc/pelican/credentials/client-credentials.pem` Deleting this file is generally safe, but it may force you to re-generate some tokens using the procedure described above. ### Automatic Token Generation for Clients that Possess an Issuer Signing Key When an Origin/namespace owner possesses the private key for their own token issuer, this key can be connected to Pelican Clients to enable the Client to generate its own tokens. This is accomplished by dropping the private signing key into one of the following directories: - (for non-root users) `~/.config/pelican/issuer-keys` - (for root users) `/etc/pelican/issuer-keys` When this is done, running a command like `pelican object ` should automatically generate the needed token without additional input. ## Explicit Token Creation & Management with Pelican The Pelican CLI provides a token creation tool for cases where automatic token generation is not preferred or does not succeed. This command lives under `pelican token create [flags]`, and it lets anyone who possesses a namespace issuer’s private signing key create tokens on behalf of the namespace. By default, it will look for these signing keys under `~/.config/pelican/issuer-keys` for non-root users or `/etc/pelican/issuer-keys` for root users. Before continuing, it’s recommended that you have an overview understanding of the concepts discussed in “[Advanced Usage/Pelican’s Authorization System](../../advanced-concepts/auth)” because this command lets you build arbitrary data access tokens. Generating tokens without understanding what they permit could result in unintentionally exposing your data to the wrong people. It’s _**highly**_ recommended that you always limit your tokens with fine-grained access scopes and minimal lifetimes as opposed to long-lived tokens that can access data from anywhere in the namespace. ### Flags for Access Permissions The `pelican token create` command uses a set capability flags to apply various access permissions to the generated token for a namespaced resource. These flags are: - `-r, --read`: adds the ability to _read_ the specified resource. All tokens needed for a `pelican object get` should apply this flag. - `-w, --write`: adds the ability to _write_ or _create_ the specified resource. Note that this **does not** grant permission to _overwrite_ or _delete_ the resource. All tokens needed for a `pelican object put` should apply this flag. - `-m, --modify`: adds the ability to _modify_ or _delete_ the specified resource, but will also grant the ability to _write_ or _create_. All tokens needed for a `pelican object delete` should apply this flag. **Example:** To create a token that lets Client users **read/get** or **write/put** anything under the `/foobar` namespace in the `osg-htc.org` federation (OSDF), run ``` pelican token create --read --write pelican://osg-htc.org/foobar ``` (Note that this assumes the private signing key of the `/foobar` issuer resides in your `~/.config/pelican/issuer-keys` or `/etc/pelican/issuer-keys` directory.) ### Specifying Token Issuers Tokens contain information about “who” created/issued them, and this information is required for token authorization to work at whichever Origin or Cache that examines the token. In most cases, Pelican can figure out what values to set in your token by comparing what’s known about the specified namespace with the private key you’re using to sign the token. The `pelican token create` command uses this information to warn you when it thinks there’s an error that will prevent your token from functioning. These errors typically take one of several forms: - the command determines the valid issuer(s), but the key you’re signing with does not match any of the public keys those issuers advertise - the command cannot determine valid issuer(s) via the federation’s Director, and thus needs to be told explicitly what value to use **Example:** In this example, the command determines your signing key isn’t expected to work for the indicated namespace ``` $ pelican token create --read pelican://osg-htc.org/my-prefix --private-key wrong-key.pem Error: unable to determine issuer for resource pelican://osg-htc.org/my-prefix; you may need to re-run with '--issuer ' to specify an issuer: none of the issuers discovered at the director match your signing key; issuers that were checked: https://correct-issuer.com ``` If you encounter this error, it means you either don’t have the correct signing key to create tokens, or the namespace’s issuer is misconfigured. Fixing this error usually requires help from the namespace’s Origin administrator. **Example:** In this example, the command discovers an issuer from the Director that is not reachable on the web (`https://issuer-does-not-exist.com`). command cannot determine which issuer(s) are valid for the indicated namespace. ``` $ pelican token create --read pelican://osg-htc.org/my-prefix WARNING[2025-10-03T15:38:47Z] Unable to get JWKS from issuer URL https://issuer-does-not-exist.com: Error getting JWKS URL from issuer URL: failed to lookup openid-configuration for issuer https://issuer-does-not-exist.com: Get "https://issuer-does-not-exist.com": dial tcp: lookup https://issuer-does-not-exist.com on 192.168.65.7:53: no such host; skipping Error: unable to determine issuer for resource pelican://osg-htc.org/my-prefix; you may need to re-run with '--issuer ' to specify an issuer: none of the issuers discovered at the director match your signing key; issuers that were checked: https://issuer-does-not-exist.com ``` This usually points to misconfiguration at the namespace’s Origin service (the configured issuer does not exist or is not reachable) and fixing it likely requires help from the Origin’s administrator. If you encounter these errors or any related to issuers, you can still force the creation your token by manually specifying your issuer. Before you do, double check your federation’s Director to see which “Token Issuer” is configured for the namespace by clicking on the relevant namespace in the “Namespaces” dropdown. Then re-run the command and set the `--issuer` flag to the desired value: **Example:** Manually set an issuer for your token ``` pelican token create --read --write pelican://osg-htc.org/my-prefix --issuer https://my-prefix-issuer.com ``` --- Source: https://docs.pelicanplatform.org/federating-your-data # Federating Your Data ## The Role of Origins in Pelican Federations Data is made accessible via Pelican through Origins — the service that acts like a universal adapter plug allowing Pelican clients to interact with a wide variety of underlying storage technologies. From posix filesystems to S3 cloud storage, Origins translate Pelican client commands into requests that work natively with whatever holds the actual objects/files. It’s important to note that Origins often do not store the objects themselves, which is why we say data is access _via_ Pelican and not _from_ Pelican. Not only do Origins translate requests to/from the data repository, but they’re also the component responsible for telling the federation they have data from some namespace in the federation. That is, you may have one Origin that ties the S3 bucket `foo` to the namespace `/my-origin/foo` while also tying the bucket `bar` to the federation prefix `/my-origin/bar`. When this Origin talks to the Director, it tells the Director which namespaces it supports so that requests for `/my-origin/foo` are forwarded by the Director to the correct Origin. For those who want to make their data accessible via Pelican, the Origin is the service they’ll get to know most intimately. It’s the place where data owners can define what data is federated, craft fine-grained access policies to describe who/how the objects are accessed, and monitor how users are interacting with the data. Generally speaking, an Origin’s configuration has a few key elements (not including stuff like TLS configuration, which is needed by all Pelican servers). These include: - **Storage Type**: The underlying storage technology the Origin will be translating on behalf of. Valid options include (but may not be limited to) POSIX, S3, HTTP, Globus and XRootD. - **Exports**: The actual sections of the storage instance that are going to be made available through the Origin. For each export, some namespace prefix is tied to some portion of the underlying storage. For example, in posix, each export generally points to some distinct directory tree, allowing the origin admin to map unique namespaces to certain pieces of the filesystem. For S3, exports usually point to individual buckets. - **Capabilities (per export)**: The access policy imposed on users when accessing objects from this part of the namespace. Understanding how these pieces fit together and what each one enables allow Origin administrators to realize a vast array of conceivable configurations. Documents in this section should help guide Origin administrators, both new and experienced, through the steps needed to add their data to their Pelican federation of choice. ## Namespaces and their Relationship with Origins In general, the origin-namespace relationship should be thought of as many-to-many. One Origin may export multiple prefixes, as in the previous example, but one prefix may also be [striped](https://en.wikipedia.org/wiki/Data_striping)  across multiple Origins. Pelican developers are still working on full support for situations where one prefix is spread across multiple Origins. At this time, certain client commands, like those that involve object listing, may not behave entirely as expected in multi-origin, single-namespace setups. ## Contributing to the OSDF If you are planning to make your data available via the OSDF, we encourage you to consult the OSDF documentation for system administrators here: [osg-htc.org/docs/data/osdf/overview/](https://osg-htc.org/docs/data/osdf/overview/) . --- Source: https://docs.pelicanplatform.org/federating-your-data/choosing-namespaces # Namespace Prefixes and How To Choose One In “Romeo & Juliet”, William Shakespeare writes: > What’s in a name? That which we call a rose > By any other name would smell as sweet. Unfortunately for us, Shakespeare’s wisdom doesn’t hold true in environments where names carry relational information or otherwise convey information about the named entity. That’s why picking a good federation prefix for Pelican is so important — it’s the data owner’s opportunity to _say something_ about their data and to frame the way data consumers will think about the objects they access. The process of organizing and naming data can be difficult. Even the Dewey Decimal System has gone through 23 major revisions over ~150 years and now comprises two entire volumes[\[1\]](https://en.wikipedia.org/wiki/Dewey_Decimal_Classification). Luckily, there are some best practices that can lead to reasonable choices in picking a namespace. The following sections contain advice about our best practices for organizing your data in Pelican, and understanding each section will help you pick namespaces that maximize the FAIR principles of your data. ## Data Provenance and Taxonomy One of the the best ways to start picking namespaces is to think about how your objects might be taxonomically structured. Taxonomy is the practice of organizing things into hierarchical categories based on their characteristics, relationships, and differences. Consider the following example: ``` CoolScienceOrg/ └── WhaleTrackingProject/ ├── Atlantic/ │ ├── 2023/ │ │ ├── Jan/ │ │ │ ├── image1.jpg │ │ │ └── ... │ │ ├── Feb │ │ └── ... │ └── 2024 ├── Pacific └── ... ``` If a user understands how to find one object under this namespace, they’ll likely understand how to find _any_ object that fits the same naming scheme under this namespace by changing predictable keywords like ocean names, years and months. Choosing these keywords is often unique to each project and highly dependent on the underlying data. However, one common technique to discover reasonable choices for keywords is to think about the terms you’d use to Google for certain objects or collections of your objects. In the previous example, one might imagine the user Googling something similar to: > whale sightings over time in the atlantic and pacific This expresses some of the relationships inherent between individual objects in this data repository — data relates to whale sightings and is split over time and by ocean. It’s also important to note that this example includes information highlighting the data’s _provenance_ or where it came from and who’s responsible for it. More often than not, it’s a good idea that your namespace starts with something signifying your organization, group, or project. This is especially useful because Pelican namespaces are hierarchical and can be further subdivided (see the section on [namespace delegation](#prefixes-as-a-means-of-delegated-management) below for more information). In the previous example, a namespace beginning with `/CoolScienceOrg/WhaleProject` lets users quickly attribute any data coming from this section of the federation to your organization and project. If data consumers have questions about some object, this namespace prefix is likely enough for them to figure out who to ask for more information. Lastly, this type of naming scheme makes the objects very interoperable for machine/workflow access because a simple nested `for` loop could be written to fetch every individual image. The ability to create predictable naming conventions that integrate with large-scale computing workflows is crucial for data re-use. ### Namespaces versus Object Names The full “name” of any Pelican object comes from its Pelican URL, e.g. > `pelican://osg-htc.com/this/is/an/object` which tells Pelican clients what federation to work with and how to find the actual object. Because Origins map namespace/federation prefixes to sections of the underlying data repository, it can be confusing to understand where the namespace ends and the object begins in a path like `/this/is/an/object`. Information about how to determine this split given a Pelican URL can be found in [Core Concepts](../about-pelican/core-concepts#differentiating-namespace-prefixes-and-object-names). However, this presents a unique challenge to Origin administrators, because they have to decide which parts of a Pelican object’s name should come from the federation prefix, and which should come from the underlying object name as understood by the storage resource. Consider a posix filesystem with the following directory tree: ``` physics/ ├── detector1/ │ ├── 2023/ │ │ ├── raw/ │ │ │ └── event.csv │ │ └── processed/ │ │ └── event.csv │ └── 2024/ │ └── ... └── detector2/ └── ... ``` The administrator who wishes to federate this data has several choices to make. For example, they could create federation prefixes with the following prefix-to-directory mapping: > /my-namespace/detector1 —> /physics/detector1 > /my-namespace/detector2 —> /physics/detector2 which would expose detector1’s `2023/` and `2024/` directories under the prefix `/my-namespace/detector1` and detector2’s directories under `/my-namespace/detector2`. This is completely valid. However, they may also choose to scope things differently, creating the simple mapping: > /my-namespace —> /physics where the entire contents of the `/physics` directory are exposed through the Pelican namespace prefix of `/my-namespace`. Choosing the right scoping level depends in part on the administrators broader goals and needs. #### Prefixes as a Means of Reorganization Because namespace prefixes essentially map some path in a Pelican federation to underlying storage, they can be used for minor re-organizational tasks. For example, this filesystem has a series of poorly-named and poorly-organized directories: ``` /my-data/ ├── grad-school-stuff/ │ ├── first-detector/ │ └── detector2/ └── jan2021/ └── detector-three/ ``` In this case, namespacing through Pelican may allow the Origin administrator to correct these issues without touching the underlying filesystem through the creation of a mapping like: > /particle-physics/detector1 —> /my-data/grad-school-stuff/first-detector > /particle-physics/detector2 —> /my-data/grad-school-stuff/detector2 > /particle-physics/detector3 —> /my-data/jan2021/detector-three To users, each event from these detectors will now follow a predictable access pattern when accessed via Pelican. #### Prefixes as a Means of Delegated Management Because prefixes are hierarchical entities in Pelican, they can be further subdivided by anyone who has the appropriate cryptographic keys. This means the “owner” of a namespace can delegate sub-namespaces to other people in their group, even if the namespace itself isn’t exported through any origin. Because prefix registration can be done through the Pelican independent of any origin, the head of “ProjectXYZ” may choose to create the top-level `/project-xyz` namespace with one set of keys. From here, they may use their ownership of the top-level namespace to create a nested namespace like `/project-xyz/johnson-lab` tied to a separate set of keys that they give to the leader of the Johnson lab. Johnson may then use his/her ownership of the sub-namespace to create further sub-namespaces for lab students. In this setup, each prefix owner has control over their section of the namespace without having control over the parent namespaces. > **Warning**: One caveat here is that matching a namespace prefix will take precedence over any potential object names. That is, if the prefix `/project-xyz/johnson-lab/foo` is registered, then the origin exporting `/project-xyz/johnson-lab` will not be able to export objects that begin with `foo/`. ### Prefixes as a Means of Aggregating Disparate Repositories One additional benefit of namespaces and the fact that they can be exported by multiple Origins is that they may allow data owners to aggregate data from multiple sources under a common name. For example, consider a case where you have two telescopes taking snapshots of the night sky. Due to each telescope’s technology, one saves its images in S3 and the other stores its images in posix. Namespacing allows the data owner to hide this fact from data consumers by exporting both the S3 bucket and the posix filesystem under the same namespace: ![Objects from multiple storage types being accessed under the same namespace](/pelican/optimized_images/aggregated-objects-opt-1920.WEBP) ## Namespace Prefix Restrictions There are two categories of characters and character combinations that Pelican prohibits in namespace prefixes: those that are special in POSIX filepaths and those that are special inside URLs. In both cases, Pelican disallows these characters as a matter of design because they may cause issues with the way Pelican uses HTTP, or they may have unintended consequences when interacting with XRootD, where object names are treated like POSIX paths. For example, the character sequence `../` in POSIX means “up one directory”, such that a filepath like `/foo/bar/../baz` is actually expanded to `/foo/baz`. This makes it potentially unsafe to allow a prefix like `/my-prefix/../`, which when interpreted as a filepath is just `/`. Because of the special meaning given to these characters, and because their use can lead to certain forms of computer attacks when handled incorrectly, Pelican disallows some characters and character combinations in prefixes. Other top-level prefixes are restricted because Pelican uses them internally for things like origin/cache registration, or in monitoring. The following table lists the characters that Pelican does not allow when defining a namespace prefix. ### POSIX Characters - `//` - `./` - `..` - `~` - `$` - `*` - `\` ### URL Characters - `?` - `#` - `%` ### Top-Level Prefixes - `/cache` - `/origin` - `/pelican` - `/view` If you attempt to set up a namespace prefix using one of these restricted paths/characters, Pelican will fail to start with a warning that lets you know why the prefix is disallowed. --- Source: https://docs.pelicanplatform.org/federating-your-data/origin # Federating Your Data via a Pelican Origin Pelican users who want to share data within a Pelican federation do so via an [_**Origin**_](../about-pelican/core-concepts#origins). Origins are a crucial component of Pelican’s architecture for several reasons: they act as an adapter between various storage backends and Pelican federations, they provide fine-grained access controls for that data, and they act as a circuit breaker that protects the underlying data repository from large volumes of data movement. That is, they figure out how to take data from wherever it lives (such as a POSIX filesystem, S3 buckets, HTTPS servers, etc.) and transform it into a format that the federation can utilize, all while respecting your data access requirements and protecting the storage they make accessible. An important distinction between Origins and data backends is that, generally speaking, Origins do **NOT** store any data themselves; their primary function is to facilitate data accessibility. Are you setting up an Origin to connect to the OSDF? If so, please read the OSDF documentation for system administrators here: [osg-htc.org/docs/data/osdf/overview/](https://osg-htc.org/docs/data/osdf/overview/) . ## Before Starting This guide assumes you’ve already installed Pelican. If you haven’t, see [Installing Pelican](../install) for more information. There are three important prerequisites to adding your data to a federation through serving a new Origin. These include: - Making sure your Origin can connect to internet-accessible ports so it can receive/send internet traffic - Acquiring a domain name with TLS credentials so that other Pelican services know how to find your Origin on the internet - Picking a federation to join and the namespace prefixes you want to claim in that federation If it is not possible for you to expose any ports through the firewall (e.g. you’re on a local network or behind a NAT) or you can’t get a domain name, Pelican has a special feature called a _Connection Broker_ that allows you to serve Origins without publicly-accessible ports or TLS credentials. However, this is an experimental feature and requires the Pelican federation you are joining to be compatible. If you are interested in learning more about the Connection Broker, please contact [help@pelicanplatform.org](mailto:help@pelicanplatform.org) for further instructions. ### Enable Time Synchronization Pelican servers rely on time-sensitive tokens for authorization. If your server’s clock is not synchronized, tokens may be rejected and your Origin may fail to join the federation. Please ensure NTP is enabled on your server. See [Server Prerequisites](../install#enable-time-synchronization) for setup instructions. ### Origin Port Requirements Pelican Origins have two major components — one is a data transfer endpoint powered by [XRootD](https://xrootd.org/) , and the other is a browser interface that lets you configure, monitor, and troubleshoot your Origin and its interactions with a federation. By design, these two components are hosted behind two separate ports, each dedicated a distinct function. Pelican has chosen ports 8443 for data transfers and 8444 for the browser interface as defaults, but you may change these port numbers through your Origin’s [configuration file](../parameters) with parameters [`Server.WebPort`](../parameters#Server-WebPort) and [`Origin.Port`](../parameters#Origin-Port), respectively. In order for Pelican Origins to work properly, **both** of these ports need to be accessible by the federation, which in most cases means they need to be open to the internet. If your server host has a firewall policy in place, please open these two ports for both incoming and outgoing TCP requests. ### Preparing TLS Credentials Data transfers in Pelican rely on HTTPS, the web encryption scheme used by everyone from banks to instagram that’s responsible for securely transmitting data between internet-connected computers. To configure the Origin with HTTPS, you’ll first need to acquire three things: - A valid Transport Layer Security (TLS) certificate - The private key associated with the certificate - The Intermediate Certificate or the chain file, that establishes the trust chain to a root certificate You need to contact a Certificate Authority (CA) who owns the root certificate for getting these credentials. One popular CA that provides free TLS certificates is [Let’s Encrypt](https://letsencrypt.org/) . You may follow [their guide](https://letsencrypt.org/getting-started/)  to obtain the credentials listed above. **Note that you need to have a valid domain before proceeding.** Once you go through the process, locate your credential files and set the following parameters to the file locations or copy the credential files to the default locations of the parameters: - `Server.TLSCertificateChain` - The certificate chain file from Let’s Encrypt (or another CA), usually named example.com.crt or example.com.pem, containing the host certificate followed by any required intermediate certificates. - Default location: `~/.config/pelican/certificates/tls.crt` for non-root users or `/etc/pelican/certificates/tls.crt` for root users. - `Server.TLSKey` - The private key corresponding to the TLS certificate, usually named as `example.com.key`. - Default location: `~/.config/pelican/certificates/tls.key` for non-root users or `/etc/pelican/certificates/tls.key` for root users. - `Server.TLSCACertificateFile` - The intermediate certificate from Let’s Encrypt to establish the trust chain to a root certificate, usually named as `letsencrypt-intermediate.pem` or `chain.pem` - Default location: `~/.config/pelican/certificates/tlsca.pem` for non-root users or `/etc/pelican/certificates/tlsca.pem` for root users. Since your TLS certificate is associated with your domain name, you will need to change the default hostname of Pelican server to be consistent. Set `Server.Hostname` to your domain name (e.g. `example.com`). ### Picking a Federation and your Namespace Prefix(es) Before serving an Origin, you need to decide which [_**federation**_](../about-pelican/core-concepts#federations) your data will be accessed through. For example, the Open Science Data Federation (OSDF) is Pelican’s flagship federation, and if you are interested in serving an OSDF Origin, you can refer to the [OSDF website](https://osg-htc.org/services/osdf.html)  for details about how to join. If you’re unsure about which federation to join and aren’t ready to run your own federation, this is a good place to start. All federations are uniquely identified by their URL. For example, the OSDF’s URL is `https://osg-htc.org` and Pelican command line client commands that interact with objects from this federation would indicate this by using Pelican URLs like `pelican://osg-htc.org/some/namespace/path`. Once you’ve picked a federation, you should think about the namespace prefix(es) you’ll want to tie your data to. Namespace prefixes map data from Origins into something resembling a “file path” within their federation. For example, an S3 bucket with data about whale sightings may be mapped to the namespace prefix `/whales`, such that an object named `2025-sightings.csv` in the bucket would be referred to as `/whales/2025-sightings.csv`. Its fully-qualified name, scoped to the federation, would then be `pelican:///whales/2025-sitings.csv`. While it’s convenient to think of these prefixes as file paths, it should be noted the comparison is only logical — there isn’t necessarily a `/whales` directory anywhere. For more information about how to choose prefixes, see [Choosing a Namespace Prefix](./choosing-namespaces) ## Origin Configuration Before an Origin can serve data within a Pelican federation, it must be configured with key settings that define its identity, storage exports, network access, and security policies. These settings ensure that the Origin integrates properly with the federation and enforces your site’s operational and access control requirements. The configuration is organized into several areas, each covering different aspects of Origin behavior. The following sections describe these configurable components in detail and guide you through the choices needed to operate a functional and secure Origin. ### Configurable Components Origin functionality is affected by any configuration option in the [Configurable Parameters](../parameters#configurable-parameters) that lists `*` or `origin` as a component. These primarily consist of: - [Logging configuration](../parameters#logging): Used to set the logging verbosity of various Pelican/XRootD processes - [Federation configuration](../parameters#Federation): Used to point the Origin at a specific federation - [Origin configuration](../parameters#Origin): Used to specify exports, set up Origin access to the object store, and configure access policies and throttling - [Server configuration](../parameters#Server): Used to modify hostname/port setup, configure the server’s TLS credentials, and control other aspects of the server runtime - [Issuer configuration](../parameters#Issuer): Used for setting up a local [OA4MP token issuer](https://oa4mp.org/)  on behalf of the Origin - [XRootD configuration](../parameters#Xrootd): Used to control the Origin’s underlying XRootD instance - [Monitoring configuration](../parameters#Monitoring): Used to tune the Origin’s embedded Prometheus instance and the way it records Origin metrics While each of these sections of configuration may be used to tailor the Origin to a specific use case, several will be required by almost all Origins. These include the `Federation`, `Server` and `Origin`\-level configurations. Before continuing, it may be worth exploring these sections of documentation to understand which options you may need when setting up your Origin. See [Example Configurations](./#example-origin-configurations) for additional examples covering how to combine these into a coherent Origin configuration. ### Origin and Namespace Capabilities Origins and namespaces can be configured with a set of _capabilities_, which are the configuration options used to define data access controls. These capabilities are most often configured in the [`Exports`](../parameters#Origin-Exports) section of the `Origin` configuration block, e.g.: ``` Origin: Exports: - FederationPrefix: /my/prefix StoragePrefix: /some/directory # Capabilities define how your data can be accessed Capabilities: ["PublicReads", "Writes"] ``` This list of capabilities can be used for further control of what types of operations the namespace is willing to support. Available capabilities include: - `Reads`: When included, objects from the namespace can be read with a valid authorization token. - `PublicReads`: When set, objects from the namespace become public and require no authorization to read. - `Writes`: When included, objects can be written back to the storage backend by Pelican. Write operations _always_ require a valid authorization token. - `DirectReads`: When included, a namespace indicates that it is willing to serve clients directly and does not require data to be pulled through a cache. Disabling this feature may be useful in cases where the Origin isn’t very performant or has to pay egress costs when data moves through it. Note that this is respected by federation central services, but may not be respected by all clients. - `Listings`: When included, the namespace indicates it permits object discovery. Authorization requirements for listing objects through an Origin are tied to the values of `Reads` and `PublicReads`. If your namespace sets `Reads`, object discovery will require a valid token, while prefixes with `PublicReads` will not require tokens. This capability is _**required**_ if your Origin must support any recursive operations, such as downloading entire directories or object prefixes. Most Origins should have either `Reads` or `PublicReads` enabled. If neither is set, the Origin won’t export any data. There is an important distinction between _Origin_ capabilities and _namespace_ capabilities. While it’s sometimes easy to treat Origins and namespaces as the same thing, Pelican must distinguish between them because two separate Origins may export portions of the same namespace, and a single Origin may export two disparate prefixes. The only exception to this rule is when a single Origin serves a single namespace, or the Origin exports multiple prefixes that should all have the same capabilities. To configure _Origin_ capabilities, you can set top-level options for the Origin: - `Origin.EnableReads`: When true, the Origin supports reads that are accompanied by a valid authorization token. - `Origin.EnablePublicReads`: When true, the Origin supports reads by anyone without an authorization token. - `Origin.EnableWrites`: When true, objects can be written back to the storage backend through the Origin. Writes always require a valid authorization token. - `Origin.EnableDirectReads`: When true, the Origin indicates it’s willing to serve clients directly, potentially without caching data. Note that this is respected by federation central services, but may not be respected by all clients. - `Origin.EnableListings`: When true, the Origin will allow object discovery. If no `Origin.Exports` block is provided to Pelican, these values will also be applied to your federation prefix. Pelican tries to resolve differences between Origin and namespace configurations by respecting the more restrictive of the two. If you serve an Origin that enables public reads, but the underlying prefix it exports disables all reads, you won’t be able to read from that namespace. ### Origin with GitHub OAuth2 for Protected Namespaces For Origins that require token-based authentication (i.e., namespaces with `Reads` instead of `PublicReads`), Pelican can be configured to authenticate users via GitHub OAuth2. This allows you to restrict data access to members of specific GitHub organizations. GitHub uses OAuth2, not full OIDC (OpenID Connect). This means you need to explicitly configure the OAuth endpoints and specify which claims to use from GitHub’s user info response (see the configuration example below). #### Prerequisites 1. **Create a GitHub OAuth App**: Go to [GitHub Developer Settings](https://github.com/settings/developers)  and create a new OAuth App. 2. **Set the callback URL**: Use `https:///api/v1.0/auth/oauth/callback` as the authorization callback URL. 3. **Note your credentials**: Save your Client ID and generate a Client Secret. 4. **Grant organization access**: If you want to restrict access based on GitHub organization membership, you must authorize the OAuth App to access those organizations. #### Configuration Add the following to your Origin’s configuration file to enable GitHub OAuth2: pelican.yaml ``` # GitHub OAuth2 endpoints (required since GitHub doesn't support OIDC discovery) OIDC: Issuer: https://github.com AuthorizationEndpoint: https://github.com/login/oauth/authorize TokenEndpoint: https://github.com/login/oauth/access_token UserInfoEndpoint: https://api.github.com/user DeviceAuthEndpoint: https://github.com/login/device/code # Your GitHub OAuth App credentials ClientID: ClientSecretFile: /path/to/client-secret-file # GitHub-specific scopes Scopes: - user - read:org # Required for organization-based access control Origin: EnableIssuer: true EnableOIDC: true # Note: Pelican's built-in OA4MP token issuer currently supports only one # protected (non-public) export per Origin. This limitation applies to all # authentication providers, including GitHub and CILogon. However, the Origin # can still export any number of fully-public namespaces. Exports: - FederationPrefix: /my-origin/protected-data StoragePrefix: /data # No PublicReads - this is a protected namespace Capabilities: ["Writes", "DirectReads", "Reads", "Listings"] Issuer: # GitHub returns "login" for username instead of OIDC standard "sub" OIDCAuthenticationUserClaim: login # GitHub returns "id" (numeric) instead of OIDC standard "sub" OIDCSubjectClaim: id # Use GitHub organization membership as groups GroupSource: github # Require users to be members of at least one of these GitHub orgs GroupRequirements: - "my-github-org" # Define authorization based on GitHub org membership # The `prefix` field specifies which path under the FederationPrefix to authorize. # Use "/" to grant access to the entire namespace, or a subpath like "/subdir" for finer control. AuthorizationTemplates: - actions: ["read", "create", "modify"] prefix: "/" groups: ["my-github-org"] Server: ExternalWebUrl: https://your-origin.example.com # Grant admin access to specific GitHub users UIAdminUsers: [""] ``` #### User Authentication Flow When a user attempts to access data from a protected namespace, they will go through the following device authorization flow: 1. **Run the Pelican client command**: ``` pelican object get pelican:///my-origin/protected-data/file.txt . ``` 2. **Enter the local client password** (if configured). 3. **Follow the authorization URL**: The client will display a URL like: ``` https:///api/v1.0/issuer/device?user_code=XXXX-XXXX ``` 4. **Authenticate with GitHub**: The user is redirected to GitHub’s login page. 5. **Grant consent**: After GitHub authentication, the user sees a Pelican consent page. 6. **Access granted or denied**: - If the user is a member of a required GitHub organization (e.g., `my-github-org`), access is granted. - If the user is not a member, they will see an authorization error: ``` Authorization Error: Error code 4000: acquired token not valid for /my-origin/protected-data/file.txt (missing scope or namespace/base-path mismatch) ``` For organization-based access control to work, the GitHub OAuth App must have permission to access the organization. Users must grant this permission when authorizing the app. ### Multi-Export Origins Origins can export more than one prefix, and these are referred to as “multi-export” Origins. This is accomplished by enumerating the exports in the Origin’s `Origin.Exports` configuration block, e.g.: ``` Origin: Exports: - FederationPrefix: /some/prefix-1 StoragePrefix: /first/directory Capabilities: ["PublicReads", "Writes"] - FederationPrefix: /some/prefix-2 StoragePrefix: /second/directory Capabilities: ["PublicReads", "Writes"] ``` This kind of setup lets you to control exactly which storage resources are federated via the Origin so that only the data you want to make public is accessible. In addition to exporting multiple storage resources (directories, buckets, etc.), the capabilities defined for each export can be distinct, letting you define access control policies per export: ``` Origin: Exports: # This export is publicly readable - FederationPrefix: /my/read-only/public/data StoragePrefix: /first/directory Capabilities: ["PublicReads"] # This export is readable & writeable, but only by those with an # appropriate access token - FederationPrefix: /my/protected/data StoragePrefix: /second/directory Capabilities: ["Reads", "Writes"] ``` While Pelican can handle multiple export directories for POSIX Origins, these directories must not be nested within one another. As a result, the following configuration is illegal because one namespace prefix maps to a subdirectory of another namespace prefix: ``` Origin: Exports: - FederationPrefix: /some/prefix-1 StoragePrefix: /first/directory Capabilities: ["PublicReads", "Writes"] - FederationPrefix: /some/prefix-2 StoragePrefix: /first/directory/subdirectory Capabilities: ["PublicReads", "Writes"] ``` Multiple namespaces can be exported by the same Origin but they must all have the same underlying storage type. That is, if the Origin serves files from POSIX, it can _only_ serve files from POSIX and not objects from S3. However, separate Origins can serve files from POSIX and objects from S3 under the same namespace prefix, allowing the Origin administrators to aggregate data under a unified namespace. One current limitation to be aware of is that failure to advertise any of the prefixes in a multi-export Origin will prevent the entire Origin from functioning. For example, if your federation requires an administrator to pre-approve namespaces (as does the OSDF) but only a subset of the namespaces from the Origin are approved at the Registry, this will prevent the entire Origin from joining the federation. See [Federation Namespace Prefix Registration](#federation-namespace-prefix-registration) for more details. ### Example Origin Configurations This section contains several example configurations for Pelican Origins from minimal to complex. It is not intended to cover all possible Origin configurations, but rather give a brief overview of what several valid configurations look like and how you might start developing your own. These examples cover use cases where data comes from a POSIX backend, the filesystem used by Unix/Linux computers. However, Pelican aims to support a variety of backends and currently supports serving objects from S3 and Globus in addition to POSIX. Configuration for these backends is mostly similar to configuration for POSIX file systems, but with a few important differences. For details, refer to the [S3 Backend](./s3-backend) and [Globus Backend](./globus-backend) pages. #### OSDF Origin, Single Export, Publicly-Readable This example makes `/my/directory` from a POSIX filesystem publicly accessible via the OSDF as `/my/namespace`. pelican.yaml ``` # Tell Pelican which federation you're joining Federation: DiscoveryUrl: https://osg-htc.org Server: # Specify which port the server's Web UI/Browser interface can be # accessed on. The default is 8444, but this config sets it to 443. WebPort: 443 Hostname: my-origin.com # Configure your Origin Origin: # Specify which port data will be accessed on. 8443 is the default and # can be omitted. Port: 8443 # POSIX is the default storage type for Pelican Origins # and can be omitted. StorageType: "posix" # Define which directories the Origin exports and how they should # map into the Federation's namespace. Exports: - StoragePrefix: "/my/directory" FederationPrefix: "/my/namespace" # Explicitly state what capabilities you want this prefix to have # Note this doesn't set Reads -- it's inferred by setting "PublicReads" Capabilities: ["PublicReads", "Listings"] ``` The data from this Origin’s underlying filesystem is designated as publicly readable, meaning anyone could issue Pelican client commands to interact with it. For example, if there were a file called `/my/directory/super-cool-data.csv`, it could be downloaded as an object with the following command: ``` pelican object get pelican://osg-htc.org/my/namespace/super-cool-data.csv downloaded.txt ``` #### OSDF Origin, Multiple Protected Exports This example demonstrates a multi-export OSDF Origin that requires token auth for reads/writes: pelican.yaml ``` Federation: DiscoveryUrl: https://osg-htc.org Server: WebPort: 443 Hostname: my-origin.com Origin: # Example of changing the data access port to meet local firewall requirements Port: 1234 StorageType: posix Exports: - StoragePrefix: /first/path FederationPrefix: /prefix-1 Capabilities: ["Reads", "Writes", "Listings", "DirectReads"] - StoragePrefix: /second/path FederationPrefix: /prefix-2 Capabilities: ["Reads", "DirectReads"] ``` #### UWDF Origin, Multiple Exports, Custom Issuer with Multiuser Feature This example is more complex than the previous examples because it configures the Origin’s [“Multiuser”](../parameters#Origin-Multiuser) feature and makes use of a non-local token issuer. This feature is useful for multi-tenant Origins because it lets the Origin’s XRootD daemon switch the user it acts as when it reads/writes from the filesystem. XRootD determines which user to select by examining the auth token’s `sub` field. pelican.yaml ``` Federation: DiscoveryUrl: https://chtc.wisc.edu # Configure log levels and log file location # This is useful for long-running servers whose logs may need to be monitored # over time for debugging and reliability Logging: LogLocation: /var/log/pelican/origin.log Level: "debug" Server: WebPort: 443 Hostname: chtc-origin.org Origin: StorageType: posix # Turn on Multiuser in the Origin and configure scitokens subject mapping Multiuser: true ScitokensMapSubject: true Exports: - StoragePrefix: /first/path FederationPrefix: /prefix-1 Capabilities: ["Reads", "Writes", "Listings", "DirectReads"] IssuerUrls: ["https://chtc.cs.wisc.edu"] - StoragePrefix: /my/data/private FederationPrefix: /my/prefix/private Capabilities: ["Reads", "DirectReads"] IssuerUrls: ["https://chtc.cs.wisc.edu"] # Specify a human readable name for the Origin, which shows up in the Director's UI. # Without this specification, the Origin would show up in the Director under its hostname Xrootd: Sitename: UWDF-CLUSTER-1 ``` ### Federation Namespace Prefix Registration Registering a federation namespace prefix is the process of claiming the prefix with the federation’s [_**Registry**_](../about-pelican/core-concepts#registry). This asserts your ownership over the namespace and gives you the ability to further subdivide the prefix by tying it to a public/private key pair you posses. For more information about how to choose these prefixes, see [Choosing a Namespace Prefix](./choosing-namespaces) Generally this process is a pre-requisite to setting up an functional Origin, but it’s not included in this page’s “Before Starting” section because Origins attempt to do this automatically on server startup. However, there are some cases where you may not wish to rely on this automatic feature. These may include: - Your federation requires namespace registration approval and you want to complete this ahead of time - You want to claim a namespace before you’re ready to serve an Origin - You want to delegate running your Origin to someone else while maintaining control over the Origin’s issuer keys - (OSDF Only) Your desired prefix needs to be added in both the Registry _and_ Topology This approach requires generating a keypair ahead of time because you’ll need your public key to complete registration. For more information about key generation, see [Advanced Usage/Server](../advanced-concepts/server#pre-generate-credentials). To register your prefix through the Registry’s browser interface, visit the Registry for your federation (e.g. [https://osdf-registry.osg-htc.org](https://osdf-registry.osg-htc.org)  for the OSDF) and locate the namespace registration button. ![Where to find namespace registration button](/pelican/federating-your-data/optimized_images/registration-button-opt-1920.WEBP) After clicking on the ”+” icon, you may be asked to log in with an OIDC provider, such as CILogon. In such cases, you’ll be asked to “Log in with OAuth”. While you shouldn’t need a pre-configured account with the Registry to add new registrations, some federations tie identifiers from services like CILogon to registrations for bookkeeping purposes. After proceeding to the registration page, you should fill out fields as needed to register your the prefix. ![User-fillable fields for namespace registration](/pelican/federating-your-data/optimized_images/browser-registration-opt-1920.WEBP) Some versions of the Pelican CLI (`v7.13.0 - v7.14.0`) contain a bug that generates invalid public keys. If your public key is missing the `alg` or `kid` fields as demonstrated in this example registration, you’ll should regenerate the keypairs with a compatible version of Pelican. Finally, submit the registration, and if your federation requires namespace approval, contact the federation’s administrators to inquire about next steps. In the meantime, store your private key someplace safe — once you’re ready to start your Origin, you’ll configure it to use the private key using the [`IssuerKeysDirectory`](../parameters#IssuerKeysDirectory) configuration option. Once your registration is complete/approved and your keys are hooked up to the Origin, your Origin should have control over your new prefix. ## Serving & Administering Your Origin Once you’ve drafted your Origin’s configuration and handled the pre-requisites from the [Before Starting](#before-starting) section, you’re ready to start serving data. This section covers starting the server, finishing setup through your browser, and basic commands to test the Origin’s data access functionality. ### Starting the Origin Once your Origin’s configuration is set, you can serve it immediately with ``` pelican-server origin serve ``` If you installed the `pelican-server` executable from an RPM package, you can daemonize your Origin to always run in the background with `systemctl enable --now pelican-origin`. This assumes you understand the basics of `systemd` and is only recommended for advanced users. When you do this, Pelican will read the config file and apply it to your Origin. The first time the Origin is started, you will see something that looks like the following: ``` $ pelican-server origin serve Pelican admin interface is not initialized To initialize, login at https://localhost:8444/view/initialization/code/ with the following code: 551220 ``` See [Logging in to the Origin’s Admin Page](#logging-in-to-the-origins-admin-page) for more information about initializing your Origin’s admin website. ### Additional Command Line Arguments for Origins This section documents additional arguments you can pass via the command line when serving Origins. - **`-h` or `--help`**: Output documentation on the `serve` command and its arguments. - **`-m` or `--mode`**: Set the mode for the Origin service (‘posix’|‘s3, default to ‘posix’). - **`-p` or `--port`**: Set the port at which the Pelican admin website should be accessible. - **`--writeable`**: A boolean value to allow or disable writing to the Origin (default is true). - **`-v`**: A shortcut for configuring docker-style volume mounts/namespace prefixes for the Origin (POSIX only). For example, `-v /local/path:/federation/prefix` will bind a directory `/local/path` to the namespace prefix `/federation/prefix`. Use of configuration yaml is strongly preferred over this method because config passed with this flag cannot be picked up by tools like `pelican config summary`. - **`--config`**: Set the location of the configuration file. - **`-d` or `--debug`**: Enable debugging mode, which greatly increases the Pelican’s logging verbosity - **`-l` or `--log`**: Set the location of a file that will capture Pelican logs. Setting this will prevent logging output from printing to your terminal. For more information about available yaml configuration options, refer to the [Parameters page](../parameters). ### Logging in to the Origin’s Admin Page After your Origin is running, the next step is to initialize its admin website, which can be used by administrators for monitoring and further configuration. To initialize this interface, go to the URL specified in the terminal. By default, it should point to `https://localhost:8444/view/initialization/code/` You will be directed to the page to activate the website with a one-time passcode. Copy the passcode from the terminal where you launch Pelican Origin and paste to the website to finish activation. ![Screenshot of Pelican website activation page](/pelican/federating-your-data/optimized_images/origin-otp-opt-1920.WEBP) The example terminal from “Starting Your Origin” shows `551220`, but your one-time passcode will be different. These one-time passcodes will be refreshed every few minutes. Find the latest passcode in the terminal before proceeding. ### Set up password for the admin After activating the website, you will be redirected to set up the password for the admin account. Enter your desired password and store the password in a safe location. ![Screenshot of Pelican website reset password page](/pelican/federating-your-data/optimized_images/origin-setup-password-opt-1920.WEBP) ### Visit the Origin’s Dashboard Page Once the password is confirmed, you will be redirected to the Origin’s dashboard page. ![Screenshot of Pelican Origin website dashboard page](/pelican/federating-your-data/optimized_images/origin-web-dashboard-opt-1920.WEBP) The Origin dashboard page consists of the following panels: - Status - Data Exports - Federation Overview - Transfer Rate The **Status** panel shows information about the health status of the Origin, delineated by the following components: - **CMSD** indicates whether the [Cluster Management Service Distributed (CMSD)](https://xrootd.web.cern.ch/doc/dev57/cms_config.pdf)  daemon is running. - **Director** indicates whether the Origin can advertise itself to its federation director, which is required for other members in the federation to discover your Origin’s existence and how to access objects from it. - **Federation** indicates whether the Origin can fetch from the federation discovery URL, which contains locations of central services (Director and Registry). - **Registry** indicates whether the Origin successfully registered the federation prefixes it exports at the Registry. - **Web UI** indicates whether the admin website is configured and running. - **XRootD** indicates whether Pelican’s underlying file transfer software is functioning as expected. ![Screenshot of the Status panel on Pelican Origin website dashboard page](/pelican/federating-your-data/optimized_images/origin-dashboard-status-opt-1920.WEBP) The **Data Exports** panel lists information about the federation prefixes that are currently being exported by the Origin. It also shows each prefix’s advertised capabilities, like “reads”, “writes”, or “listings”. If any of the namespace prefix registrations is missing required information, there will be a warning message and a button to “Complete Registration”. Note that each namespace prefix is registered separately, and the Origin server itself also registers at the registry. **It is advised that the Origin admin completes the registration as soon as they can**. ![Screenshot of the Data Exports panel on Pelican Origin website dashboard page](/pelican/federating-your-data/optimized_images/origin-dashboard-data-exports-opt-1920.WEBP) The **Federation Overview** panel lists links to various federation services (director, registry, etc.). Note that the link to the **Discovery** item is the federation discovery URL where the metadata of a federation is located. ![Screenshot of the Data Exports panel on Pelican Origin website dashboard page](/pelican/federating-your-data/optimized_images/origin-dashboard-fed-overview-opt-1920.WEBP) The **Transfer Rate** graph visualizes object transfer metrics like transfer **speed** for both receiving (rx) and transmitting (tx) data. You may change the time range of the graph by changing the **Reporting Period** and **Graph Settings**. ![Screenshot of the graph panel on Pelican Origin website dashboard page](/pelican/federating-your-data/optimized_images/origin-dashboard-graph-opt-1920.WEBP) This graph may be empty when the Origin first starts, as it takes several minutes to collect enough data for the display. Try refreshing the page after the Origin has been running for ~5 minutes and you you should see data being aggregated. ### Test Origin Functionality Once you have your Origin set up, follow the steps below to test if your Origin can serve a file through a Pelican federation. It’s best to test your Origin while it’s serving public data to minimize the risk that any test tokens you generate may be malformed and the reason objects can’t be pulled through the Origin. 1. Create a test file under the directory on your host machine that binds to a Pelican namespace. Assuming your directory is `/tmp/demo`, run the following command to create a test file named `testfile.txt` under `/tmp/demo` ``` echo "This is a test file." > /tmp/demo/testfile.txt ``` 2. In a **separate terminal**, run the following command to get the data from your Origin through the Pelican federation ``` $ cd ~ $ pelican object get pelican:////testfile.txt . ``` For example, if you registered the `/demo` namespace in the OSDF, you would see output like the following: ``` $ pelican object get pelican://osg-htc.org/demo/testfile.txt . testfile.txt 36.00 b / 36.00 b [=============================================================================================] Done! ``` 3. Confirm the file content by running: ``` $ cat testfile.txt This is a test file. ``` Congratulations! Your have finished setting up and running your Origin. --- Source: https://docs.pelicanplatform.org/federating-your-data/s3-backend # S3 Storage Backend ## What is S3? S3, or “Simple Storage Service” is a type of object store introduced by Amazon Web Services (AWS) in 2006. Since then, the term S3 has grown to represent both the _service_ offered by Amazon as well as the _protocol_ used both by Amazon and many other providers who have no AWS affiliation. In general, Pelican works with any S3 provider and is not limited to what’s offered by AWS. References to “S3” in Pelican documentation should be interpreted as “S3 the protocol.” Unlike POSIX, which uses “files” organized into hierarchical directories with associated owners/permissions and a host of other metadata that are packaged together to act as a fundamental unit, S3 works with “objects” stored in “buckets”. Typically, objects consist of data, metadata, and a unique identifier and they are stored in a flat address space referred to as a bucket. Because of this, there is no inherent hierarchy or nesting like there would be in a file system. One goal of Pelican is to obfuscate the underlying differences between storage backends like these so that users can enjoy a common interface for all there data, wherever it may happen to come from. ## Launch the Origin with S3 Backend Serving S3 origins with Pelican is similar to serving POSIX origins, but with several key differences. The first is that Pelican must be configured to host an S3 backend, using the configuration option `Origin.StorageType = s3`. To make your work with S3, it needs to know at least four additional things: - The URL you use to access objects from S3, also known as the _S3 Service URL_ - The _region_ that your S3 instance is hosted out of (almost always `us-east-1` unless you’re actually using S3 from Amazon) - The name of the _bucket_ your objects are stored in - The type of bucket hosting used at the S3 service URL, which can be either _path_ or _virtual_. This determines whether objects are normally accessed like `https:////` (path-style hosting) or `https://./` (virtual-style hosting), but does not change the way you access objects through Pelican. In many cases, it’s safe to assume _path_\-style hosting, and this is set to Pelican’s default. For more information about different hosting styles in S3, see the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) . > **NOTE:** Pelican has a special mode where no bucket information is provided that allows you to export objects from all public buckets at a given service URL. This is covered in further detail [later in this section](#exporting-an-entire-s3-endpoint). The service URL, region, and hosting style can be configured using the Pelican config variables `Origin.S3ServiceUrl`, `Origin.S3Region`, and `Origin.S3UrlStyle`. Additionally, some buckets might require credentials that prove you’re allowed to access the objects they contain. In S3, these credentials are called the _access_ key and _secret_ key (In some cases the access key may also be referred to as the _API_ key). Essentially, they can be treated like a username and password, where the access/API key is your username and the secret key is your password. When a bucket you’d like to export requires authentication, you’ll need to pass these values to Pelican by putting your keys in separate files and telling Pelican where those files can be found via either the `Origin.S3AccessKeyfile` variable or the `Origin.Exports.S3AccessKeyfile`. See below for examples of S3 origin configurations that use these values, along with an explanation of how to choose which one is right for you. ## Configuration Examples Origins can be configured with multiple exports by using the `Origin.Exports` block of your configuration: pelican.yaml ``` Origin: # Things that configure the origin itself # Tell the origin it will be serving objects from S3 StorageType: "s3" S3ServiceUrl: "https://my-s3.com" S3Region: "us-east-1" S3UrlStyle: "virtual" # The actual namespaces we export. Each export is defined # via its own export block Exports: - S3Bucket: "first-bucket" FederationPrefix: /first/namespace Capabilities: ["PublicReads", "Writes", "Listings", "DirectReads"] - S3Bucket: "second-bucket" S3AccessKeyfile: "/path/to/second/access.key" S3SecretKeyfile: "/path/to/second/secret.key" FederationPrefix: /second/namespace # Notice we designate "Reads" and not "PublicReads" for this bucket # because we assume that if the bucket requires credentials to access, # the origin should, too. Capabilities: ["Reads", "Writes"] ``` In this example, the object `foo` from the bucket `first-bucket` would be accessible without any token authorization at the namespace path `/first/namespace/foo`. Getting the object `bar` from `second-bucket` would require a valid access token, and would be accessed via `/second/namespace/bar`. In this example, the actual bucket names hosting `foo` and `bar` are elided from a Pelican user’s perspective, because they are accessed through the namespace. If you’d like make users aware of the underlying bucket name, you can use the bucket name as your `FederationPrefix`. Alternatively, if your origin only exports a single bucket, the origin can be configured with top-level config variables (which could also be configured with their equivalent environment variables): pelican.yaml ``` Origin: StorageType: "s3" S3ServiceUrl: "https://my-s3.com" S3Region: "us-west-2" S3UrlStyle: "path" FederationPrefix: /my/namespace S3Bucket: "my-bucket" S3AccessKeyfile: "/path/to/access.key" S3SecretKeyfile: "/path/to/secret.key" # Set up origin capabilities that are also applied to the bucket EnableWrites: false EnableReads: true EnableListings: false EnableDirectReads: true ``` ### Exporting An Entire S3 Endpoint In some cases, it may be infeasible to set up an origin that exports every bucket you’d like to make accessible via a Pelican federation. For example, [Amazon’s Open Data program](https://aws.amazon.com/opendata)  hosts many terabytes of public data across thousands of buckets and a handful of regions. Manually enumerating all of these buckets in an origin config would quickly become intractable. Instead, Pelican provides a mechanism that allows you to export all the public buckets from an S3 endpoint. This is accomplished by omitting the bucket field when you set up the export. The following example could be used to set up an origin that exports AWS public data from the `us-east-1` region. pelican.yaml ``` Origin: # Things that configure the origin itself # Tell the origin it will be serving objects from S3 StorageType: "s3" S3ServiceUrl: "https://s3.us-east-1.amazonaws.com" S3Region: "us-east-1" S3UrlStyle: "virtual" # The actual namespaces we export. Each export is defined # via its own export block Exports: - FederationPrefix: /aws-public Capabilities: ["PublicReads", "Listings", "DirectReads"] ``` In this configuration, users who wish to fetch objects from the origin will still need to know the name of the bucket that hosts those objects. For example, the AWS public bucket `noaa-wod-pds` has an object called `MD5SUMS`, and with this configuration the object can be fetched at `/aws-public/noaa-wod-pds/MD5SUMS`. --- Source: https://docs.pelicanplatform.org/federating-your-data/globus-backend # Globus Storage Backend Serving a Globus-backed Origin in Pelican allows you to export data from a Globus Collection into a Pelican federation namespace. This page walks through setup and activation end-to-end, with extra operational details for administrators. ## Current Behavior and Limitations - The Globus backend currently supports exporting one collection per Origin process. - The full object operation suite is supported, including reads, writes, and listings (subject to your configured namespace capabilities). ## Prerequisites Before configuring Pelican, ensure you have: - A Globus Collection you can access ## 1\. Collect Required Information from Globus ### Collection UUID Each Globus Collection has a UUID shown on its Collection overview page. You will use this value `Origin.GlobusCollectionID` for single-export config. This can be found by going to the “File Manager - Collection Search” tab in the Globus web UI ([https://app.globus.org/file-manager/collections](https://app.globus.org/file-manager/collections) ), searching for your collection, and going to the collection overview by clicking on the three dots to the right: ![Globus File Manager Collection Search page showing how to locate a collection](/pelican/federating-your-data/optimized_images/origin-globus-uuid1-opt-1920.WEBP) After that, scroll to the bottom of the overview and copy the UUID. ![Globus Collection overview page showing where the collection UUID is displayed](/pelican/federating-your-data/optimized_images/origin-globus-uuid2-opt-1920.WEBP) ### OAuth Client ID and Secret Register a confidential client in Globus: - Globus web UI (direct link to confidential client registration): - [https://app.globus.org/settings/developers/registration/confidential\_client/select-project](https://app.globus.org/settings/developers/registration/confidential_client/select-project)  - Globus developer documentation (step-by-step app registration guide): - [https://docs.globus.org/api/auth/developer-guide/#register-app](https://docs.globus.org/api/auth/developer-guide/#register-app)  When registering the client: - Choose or create a Globus project - Use a descriptive application name - Add a redirect URL matching your Origin web UI callback path. - If you set URL fields from separate host and port settings, use `https://${Server.Hostname}:${Server.WebPort}/view/origin/globus/callback`. - If you set URL fields from one external URL setting, use `https://${Server.ExternalWebUrl}/view/origin/globus/callback`. - Relevant Pelican parameters: - [`Server.Hostname`](https://docs.pelicanplatform.org/parameters#Server-Hostname) - [`Server.WebPort`](https://docs.pelicanplatform.org/parameters#Server-WebPort) - [`Server.ExternalWebUrl`](https://docs.pelicanplatform.org/parameters#Server-ExternalWebUrl) - Example: `https://my-osdf-globus-origin:8444/view/origin/globus/callback` Then: - Copy the Globus client UUID to a local file - Create a client secret in Globus and copy the secret to a local file ## 2\. Configure Pelican for Globus Set `Origin.StorageType: "globus"` and configure one export with your collection details. pelican.yaml ``` Origin: StorageType: "globus" GlobusClientIDFile: "/path/to/globus-client-uuid" GlobusClientSecretFile: "/path/to/globus-client-secret" Exports: - StoragePrefix: "/" FederationPrefix: "/globus/test" GlobusCollectionID: "66429a34-0a52-47b7-8699-d10804a1b75b" GlobusCollectionName: "CHTC Projects" Capabilities: ["PublicReads", "DirectReads", "Writes", "Listings"] ``` Notes: - `Origin.GlobusClientIDFile` and `Origin.GlobusClientSecretFile` are the Globus client UUID and the client secret from step 1 respectively - The `StoragePrefix` field must match the path in Globus that you wish to expose via Pelican. For example, setting the field to `"/"` exposes the root of the collection, whereas setting it to `"/foo/bar"` limits access to that directory subpath. - `GlobusCollectionName` is an arbitrary, human-readable label shown in the Pelican Web UI for this export. - `GlobusCollectionName` does not need to match the confidential client application name from step 1. ## 3\. Start the Origin and Activate the Collection Unlike other backend types, Globus-backed Origins **require** additional setup through the Web UI because of OAuth2. Serve the Origin using your normal deployment flow before continuing. For startup and admin website initialization guidance, see [Serving & Administering Your Origin](../origin/page#serving--administering-your-origin) and [Logging in to the Origin’s Admin Page](../origin/page#logging-in-to-the-origins-admin-page). Then in the Pelican web UI: 1. Open the Origin dashboard. 2. Go to Globus exports via the “Globus Configurations” tab on the left side. 3. Click **Activate** for the collection. 4. Complete Globus login and consent. 5. Return to Pelican and confirm the collection shows as active. Activation authorizes the Pelican Origin’s client to access the collection on behalf of whoever is doing the setup. ## 4\. Verify Data Access Note that it may take a few minutes for the Pelican Origin to reboot after activation. After this occurs, test core operations from the exported namespace using a Pelican client: ``` pelican object get pelican://// . ``` ``` pelican object put ./local-file.txt pelican:////local-file.txt ``` ``` pelican object ls pelican://// ``` ## Operational Details (How It Works) For operators troubleshooting production setup, this is the backend flow: - Pelican authenticates the admin through Globus OAuth and stores tokens for collection access. - Pelican persists token material under `${Origin.GlobusConfigLocation}/tokens`. - Pelican configures XRootD with the Globus HTTPS endpoint and token files so backend requests carry proper authorization. - Pelican keeps access tokens fresh by periodically refreshing via refresh token (currently on an approximately 5 minute interval). At startup, Pelican also checks whether the collection was previously activated and restores state using persisted data. ### OAuth Scope and Callback Notes During activation, Pelican requests: - Standard Globus auth scopes - A collection HTTPS scope: `https://auth.globus.org/scopes//https` - A collection data-access scope: `https://auth.globus.org/scopes//data_access` - Transfer API scope access for transfer metadata lookups The callback handler exchanges the authorization code for tokens, then: - Obtains a transfer token for `transfer.api.globus.org` to query collection endpoint details - Obtains a collection token scoped to the specific collection for data access - Persists refresh/access token state and token files used by backend reads ## Troubleshooting Checklist - Callback mismatch: - Ensure the registered Globus redirect exactly matches `https://${Server.Hostname}:${Server.WebPort}/view/origin/globus/callback` or `https://${Server.ExternalWebUrl}/view/origin/globus/callback`. - Inactive collection after restart: - Verify `Origin.GlobusConfigLocation` is writable and persistent enough for runtime tokens. - OAuth credential issues: - Confirm client ID/secret files are readable by the Pelican process. - Collection not reachable: - Re-check the UUID and that the authenticated Globus account can access that collection. ## Useful Globus References - Globus Auth API reference: - [https://docs.globus.org/api/auth/reference/#client\_authentication](https://docs.globus.org/api/auth/reference/#client_authentication)  - Globus Transfer collection lookup: - [https://docs.globus.org/api/transfer/endpoints\_and\_collections/#get\_endpoint\_or\_collection\_by\_id](https://docs.globus.org/api/transfer/endpoints_and_collections/#get_endpoint_or_collection_by_id) --- Source: https://docs.pelicanplatform.org/federating-your-data/ssh-backend # SSH Storage Backend ## What is the SSH Backend? The SSH backend allows a Pelican Origin to serve data from a remote POSIX filesystem over an SSH connection. Instead of running Pelican directly on the storage host, you run the Origin on a separate machine (or Kubernetes cluster) and it reaches the data by SSH-ing into the storage server. Pelican automatically transfers a helper binary to the remote host, starts it, and forwards the requests to the helper. The helper uses a separate connection from the SSH host to the origin (outgoing connectivity, not incoming) to move data by default. This is useful when: - The storage server cannot run Pelican directly (e.g., it is a shared HPC login node or a managed appliance). - The storage server has no inbound network connectivity (firewalled, behind NAT) but can accept SSH connections. - You want to separate the Origin service from the data host for operational or security reasons. The SSH backend does not require any software to be pre-installed on the remote host beyond a standard SSH server. Pelican detects the remote platform, transfers a compatible helper binary, and manages its lifecycle automatically. ## How It Works When the Origin starts with `Origin.StorageType: ssh`, the following happens: 1. Pelican opens an SSH connection to the configured remote host. 2. It detects the remote OS and architecture. 3. It transfers a platform-appropriate Pelican helper binary to the remote host. 4. It starts the helper process, which translates the origin’s HTTP requests into local filesystem operations on the remote host. 5. Data requests arriving at the Origin are forwarded to the helper over the SSH connection. The Origin monitors the SSH connection and helper process with keepalives and will automatically reconnect if the connection drops. ### Connection Modes The SSH backend supports two connection modes: - **Broker mode** (default): The helper process polls the Origin over HTTPS to pick up pending requests. This requires the remote host to be able to reach the Origin’s external URL. - **Tunnel mode** (`Origin.SSH.TunnelCallback: true`): The Origin opens an SSH tunnel (remote port forward) and the helper communicates through it. This works even when the remote host has no outbound connectivity to the Origin but throughput is limited by SSH’s performance (typically, less than 1Gbps). ## Before Starting This guide assumes: - You have already [installed Pelican](../install). - You have SSH access to the remote storage host (password, public key, or ssh-agent). SSH does not need to be installed on the origin. - The remote host runs Linux or macOS on amd64 or arm64. ## Minimal Configuration The simplest SSH Origin configuration uses public-key authentication: pelican.yaml ``` Origin: StorageType: ssh FederationPrefix: /my-data StoragePrefix: /data/exports SSH: Host: storage.example.com User: pelican AuthMethods: ["publickey"] PrivateKeyFile: /etc/pelican/ssh/id_ed25519 ``` This exports the directory `/data/exports` on `storage.example.com` under the federation prefix `/my-data`. Make sure the SSH user has read access (and write access, if you enable writes) to the `StoragePrefix` directory on the remote host. ## Configuration Examples ### Multi-Export with Public Key pelican.yaml ``` Origin: StorageType: ssh Exports: - FederationPrefix: /project-a StoragePrefix: /data/project-a Capabilities: ["PublicReads", "Listings", "DirectReads"] - FederationPrefix: /project-b StoragePrefix: /data/project-b Capabilities: ["Reads", "Writes", "Listings"] SSH: Host: hpc-login.example.com User: pelican-svc AuthMethods: ["publickey"] PrivateKeyFile: /etc/pelican/ssh/id_ed25519 KnownHostsFile: /etc/pelican/ssh/known_hosts ``` ### SSH Agent Authentication To use keys managed by an SSH agent (including hardware keys like YubiKeys): pelican.yaml ``` Origin: StorageType: ssh FederationPrefix: /my-data StoragePrefix: /data/exports SSH: Host: storage.example.com User: pelican AuthMethods: ["agent"] ``` Make sure the `SSH_AUTH_SOCK` environment variable is set in the process that runs Pelican. For hardware keys that require a touch confirmation, the Origin will log a message prompting you to touch the key. ## Testing the Connection Before running a full Origin, you can test SSH connectivity with: ``` pelican-server origin ssh-auth test storage.example.com ``` Or with explicit options: ``` pelican-server origin ssh-auth test pelican@storage.example.com -i ~/.ssh/id_ed25519 ``` This will: 1. Connect to the remote host via SSH. 2. Detect the remote platform. 3. Transfer the helper binary. 4. Start and stop the helper process. 5. Clean up the remote binary. If the test succeeds, your Origin configuration should work. ### Checking Connection Status Once an Origin is running, check the SSH backend status with: ``` pelican-server origin ssh-auth status ``` This shows the current connection state, helper status, last keepalive time, and authentication method. ## Operational Notes ### Host Key Verification By default, the remote host’s SSH key must already be present in the known hosts file (`~/.ssh/known_hosts` or the path specified by `Origin.SSH.KnownHostsFile`). If the key is not found, the connection will fail. For test or development environments, you can set `Origin.SSH.AutoAddHostKey: true` to automatically accept and save unknown host keys. Do not enable `AutoAddHostKey` in production. It disables protection against man-in-the-middle attacks. ### Automatic Reconnection If the SSH connection drops or the helper process exits, the Origin automatically reconnects with exponential backoff. The maximum number of consecutive retries is controlled by `Origin.SSH.MaxRetries` (default: 5). After the connection is re-established, the Origin resumes serving data without manual intervention. ### Health Monitoring The SSH backend reports its health status through Pelican’s standard component health system: - **OK**: SSH connected, helper running, keepalives succeeding. - **Warning**: Initializing, reconnecting, or keepalives degraded. - **Critical**: Keepalives have failed beyond the timeout threshold. ### Session Establishment Timeout The `Origin.SSH.SessionEstablishTimeout` (default 5m) bounds the entire time to establish a working session: connecting, authenticating, detecting the platform, transferring the binary, and starting the helper. If this timeout is exceeded, the attempt is aborted and retried. This is particularly relevant when using keyboard-interactive authentication, where a human must complete the challenge within this window. ## Advanced Configuration ### Password Authentication If key-based authentication is not available, you can use a password stored in a file: pelican.yaml ``` Origin: StorageType: ssh FederationPrefix: /my-data StoragePrefix: /data/exports Capabilities: ["PublicReads", "Listings"] SSH: Host: storage.example.com User: pelican AuthMethods: ["password"] PasswordFile: /etc/pelican/ssh/password ``` The password file should contain only the password and have restricted permissions (`chmod 0600`). ### Keyboard-Interactive Authentication (2FA / OTP) Some hosts require keyboard-interactive authentication (e.g., for two-factor authentication or one-time passwords). In this mode, when the remote host presents an authentication challenge, the Origin logs a message telling you to run the interactive login command: pelican.yaml ``` Origin: StorageType: ssh FederationPrefix: /my-data StoragePrefix: /data/exports SSH: Host: secure-host.example.com User: pelican AuthMethods: ["keyboard-interactive"] ChallengeTimeout: 2m ``` When the Origin starts, it will wait for you to complete authentication. Open a second terminal and run: ``` pelican-server origin ssh-auth login ``` This connects to the Origin and presents the SSH authentication prompts in your terminal. Once you complete the challenge, the Origin finishes establishing the connection and begins serving data. While extremely flexible — this can complete 2FA challenges — be careful with this mode as Pelican will be unable to restart the origin without a human present. You can also specify a remote origin URL explicitly: ``` pelican-server origin ssh-auth login --origin https://my-origin.example.com ``` ### Tunnel Mode (No Outbound Connectivity from Remote Host) If the remote host cannot reach the Origin’s HTTPS port (e.g., it is behind a strict firewall), enable tunnel mode: pelican.yaml ``` Origin: StorageType: ssh FederationPrefix: /my-data StoragePrefix: /data/exports SSH: Host: firewalled-host.example.com User: pelican AuthMethods: ["publickey"] PrivateKeyFile: /etc/pelican/ssh/id_ed25519 TunnelCallback: true ``` In tunnel mode, the Origin sets up an SSH tunnel (using Unix domain socket forwarding) so the helper communicates back through the SSH connection itself. No additional network ports or firewall rules are needed on the remote host. ### ProxyJump (Bastion Host) To reach a storage host through a jump host, use `ProxyJump`: pelican.yaml ``` Origin: StorageType: ssh FederationPrefix: /my-data StoragePrefix: /data/exports SSH: Host: internal-storage.local User: pelican AuthMethods: ["publickey"] PrivateKeyFile: /etc/pelican/ssh/id_ed25519 ProxyJump: bastion.example.com ``` Chained jumps are supported with comma-separated hosts: `bastion1.example.com,bastion2.example.com`. ### Pre-Installed Helper Binary If the remote host already has a compatible Pelican binary (or you want to control which binary is used), you can skip the automatic transfer: pelican.yaml ``` Origin: StorageType: ssh FederationPrefix: /my-data StoragePrefix: /data/exports SSH: Host: storage.example.com User: pelican AuthMethods: ["publickey"] PrivateKeyFile: /etc/pelican/ssh/id_ed25519 RemotePelicanBinaryOverrides: - "linux/amd64=/usr/local/bin/pelican" - "linux/arm64=/usr/local/bin/pelican-arm64" ``` ## Configuration Reference All SSH backend parameters are documented in the [parameter reference](../parameters#Origin-SSH). Key parameters are summarized below: | Parameter | Type | Default | Description | | --- | --- | --- | --- | | `Origin.SSH.Host` | string | _(required)_ | Remote SSH hostname or IP | | `Origin.SSH.Port` | int | `22` | SSH port | | `Origin.SSH.User` | string | current OS user | SSH username | | `Origin.SSH.AuthMethods` | string list | `["publickey", "agent", "keyboard-interactive", "password"]` | Authentication methods to try, in order | | `Origin.SSH.PrivateKeyFile` | path | — | SSH private key file | | `Origin.SSH.PasswordFile` | path | — | File containing the SSH password | | `Origin.SSH.KnownHostsFile` | path | `~/.ssh/known_hosts` | SSH known hosts file | | `Origin.SSH.AutoAddHostKey` | bool | `false` | Accept unknown host keys automatically | | `Origin.SSH.ProxyJump` | string | — | Jump host(s) for SSH ProxyJump | | `Origin.SSH.TunnelCallback` | bool | `false` | Use SSH tunnel instead of direct callback | | `Origin.SSH.MaxRetries` | int | `5` | Max consecutive reconnection attempts | | `Origin.SSH.ConnectTimeout` | duration | `30s` | SSH connection timeout | | `Origin.SSH.SessionEstablishTimeout` | duration | `5m` | End-to-end session establishment timeout | | `Origin.SSH.KeepaliveInterval` | duration | `5s` | SSH keepalive interval | | `Origin.SSH.KeepaliveTimeout` | duration | `20s` | Keepalive failure threshold | | `Origin.SSH.ChallengeTimeout` | duration | `1m` | Timeout for a single auth challenge | | `Origin.SSH.RemotePelicanBinaryOverrides` | string list | `[]` | Platform-specific binary paths on remote host | --- Source: https://docs.pelicanplatform.org/federating-your-data/generating-tokens # Generating Tokens for Accessing Protected Objects The content in this page is outdated and describes use of a deprecated command line tool. For modern guidelines describing token generation, see “[Getting Data with Pelican/Working with Protected Data](../../getting-data-with-pelican/auth)”. # Generating Tokens for Accessing Protected Objects To access a protected object via a Pelican data federation, the client must present a token signed by the Origin where the object is hosted on. The token then is used by Pelican servers to verify the object access permissions granted to the user, along with other security checks to ensure the token is valid and not tampered by malicious attackers. Pelican takes advantage of JSON Web Token (JWT) to embed permission, user and server identity to the token. To allow user access the protected objects, the token must be generated by the Pelican Origin using its private key. Pelican Origin can be configured to use a third-party OIDC token issuer, in which case the client will be prompted to get a token from the issuer. By default, however, the Pelican Origin needs to generate the token by its built-in issuer and pass it to the client. This page documents different ways to generate the token to access the protected objects. ## Pelican CLI The content in this section is outdated and describes use of a deprecated command line tool. For modern guidelines describing token generation, see “[Getting Data with Pelican/Working with Protected Data](../../getting-data-with-pelican/auth)”. The Pelican binary comes with a command `pelican-server origin token create` to generate the token. To generate a valid token, this command **MUST** be run on the same server where the Pelican Origin hosting the target object runs. On the Origin server, run: ``` pelican-server origin token create \ --scope "storage.read:/" \ --issuer https://localhost:8444 \ --claim "wlcg.ver=1.0" \ --subject origin \ --audience https://wlcg.cern.ch/jwt/v1/any ``` where: - `--scope` defines the access permission of the token. It follows [WLCG claim semantics](https://github.com/WLCG-AuthZ-WG/common-jwt-profile/blob/master/profile.md#capability-based-authorization-scope) . There are three scopes that are most frequently used: - `storage.read`: Read data. - `storage.create`: Upload data. This capability includes the creation of directories and subdirectories at the specified path, and the creation of any non-existent directories required to create the path itself. This authorization does **not** permit overwriting or deletion of stored data. - `storage.modify`: Change the data, including creating new files and writing data. This permission includes overwriting or replacing stored data in addition to deleting or truncating data. This is a strict superset of storage.create. One can control the fine-grained permission based on object prefixes by appending the allowed path to the scope. For example, a `storage.read:/foo/bar` scope only has `read` access to the `/foo/bar` path and the child paths. In this example, the token has `read` permission to the `/` prefix, meaning it can be used to read any objects on this Origin. For other scope options, refer to [WLCG GitHub website](https://github.com/WLCG-AuthZ-WG/common-jwt-profile/blob/master/profile.md#capability-based-authorization-scope) . - `--issuer` tells Pelican the issuer URL it should embed in the token, which will be used by other Pelican servers to verify the integrity of the token. The issuer should be the publicly accessible URL of the Origin server. The port number should also be included if the issuer URL contains a port number other than `443`. - `--claim` encodes an additional JWT claim, `wlcg.ver` with value `1.0` to the token. `wlcg.ver` is a special claim to indicate the version of the token profile. Pelican Origin accepts both `SciToken` token profile and `WLCG` token profile. Here we set the token profile to `WLCG`. See details [here](https://github.com/WLCG-AuthZ-WG/common-jwt-profile/blob/master/profile.md#common-claims) . - `--subject` encodes the entity that issues the token. - `--audience` encodes the entities who requested the token. In most of cases, Pelican uses the special `any` audience. ### Additional Command Line Arguments The content in this section is outdated and describes use of a deprecated command line tool. For modern guidelines describing token generation, see “[Getting Data with Pelican/Working with Protected Data](../../getting-data-with-pelican/auth)”. - `--private-key` encodes the path to the private key used to sign the token. By default it uses the private key defined by the Origin server that the command runs on. - `--lifetime` encodes the duration in seconds where the token is valid. By default it’s 1200 seconds, or 20 minutes. --- Source: https://docs.pelicanplatform.org/operating-a-federation # Operating a Pelican Federation A federation in Pelican refers to an instance of a data sharing ecosystem where members can share and access data as objects. To allow such an exchange of data to work, Pelican has two top-level _central service components_ to coordinate origins and caches in the federation, direct client requests to the proper storage server, and manage namespace registration. They are the _Director_ and the _Registry_. A Pelican federation requires both a _Director_ and a _Registry_ to be set up first before any origin or cache server can join the federation or any client can access objects in the federation. ## Before Starting ### Install Pelican If you haven’t installed Pelican, follow the instructions to [install Pelican](/install). ### Prepare TLS Credentials Pelican servers use `https` for serving its web UI, handling objects transfer requests and internal http requests. `https` requires a set of credential files in place to work. For details on how to set up the TLS credentials, refer to [Federating Your Data](./federating-your-data#prepare-tls-credentials) page. ## Next Steps Follow the instructions below for serving a specific Pelican federation component. - [Serve a Pelican Registry](./operating-a-federation/registry) - [Serve a Pelican Director](./operating-a-federation/director) - [Serve a Pelican Cache](./operating-a-federation/cache) --- Source: https://docs.pelicanplatform.org/operating-a-federation/registry # Serving a Pelican Registry The Pelican [_Registry_](../about-pelican/core-concepts#registry-service) manages namespace registration and public key distribution. An Origin or Cache will automatically attempt to register their namespace when they are started. A federation user may also register a namespace without running a storage server by visiting the registry website through `https://`. ## Before Starting ### Enable Time Synchronization Pelican servers rely on time-sensitive tokens for authorization. If your server’s clock is not synchronized, tokens may be rejected and your Registry may fail to operate correctly. Please ensure NTP is enabled on your server. See [Server Prerequisites](../install#enable-time-synchronization) for setup instructions. ### Get OIDC Credential (Required) The Pelican registry follows [OIDC](https://openid.net/developers/how-connect-works/)  for user authentication, and [CILogon](https://www.cilogon.org/)  is our OpenID Provider by default, which enables single sign-on for users with an account associated with an institution that joins CILogon. (Check out [this page](https://cilogon.org)  for institutions in CILogon) For Pelican registry to work, you need to obtain a client id and associated client secret from CILogon. [This page](https://www.cilogon.org/oidc#h.p_ID_38)  details how you can request a client credential. You will need to register your client at [https://cilogon.org/oauth2/register](https://cilogon.org/oauth2/register)  and wait for approval before proceeding. Below is a guidance on how to fill in the registration form for CILogon. - **`Client Name`**: a human-readable name of your service. For example: Pelican Data Federation - **`Callback URLs`**: fill in `https://:/api/v1.0/auth/oauth/callback` where `:` is your server’s public endpoint. For example, `https://example-origin.org:8444/api/v1.0/auth/oauth/callback` - **`Client Type`**: select `Confidential` - **`Scopes`**: select `email`, `openid`, `org.cilogon.userinfo`, and `profile` - **`Refresh Tokens`**: select `No` Once approved, you will get your `client_id` and `client_secret` from CILogon. Pass them as configuration parameters and configuration files to Pelican. - Set the `OIDC.ClientID` config parameter to your `` value. - Create a file named `/etc/pelican/oidc-client-secret` ``` touch /etc/pelican/oidc-client-secret ``` - Copy and paste your `client_secret` into the file you just created. Please don’t share your `client_secret`. If you prefer to store your `client_secret` file in a path different from the default file path, change `OIDC.ClientSecretFile` to your desired file location. ### Bypass Required Credential Check To run the registry without CILogon credentials, create two files named `/etc/pelican/oidc-client-id` and `/etc/pelican/oidc-client-secret` and populate the files with non-empty content. Pelican checks if the files exist and are non-empty. It won’t check the content of the file. Note that, by doing this, when users attempt to use `CILogon` to login to the registry they will get an error page from `CILogon` and it’s only possible to login using admin password. ## Launch the Registry That’s it! Now you can launch the registry by running the following command: ``` pelican-server registry serve ``` and you will see the following message: ``` Pelican admin interface is not initialized To initialize, login at https://:8444/view/initialization/code/ with the following code: 865309 ``` > Note that your code will be different from what is being shown here. By default, a registry runs on port `8444`. You may change the port number by passing `-p ` when serving the registry or by setting `Server.WebPort` in the configuration file. Now you want to initialize the admin account for your registry web UI. You may refer to [Login to Admin Website](/federating-your-data#login-to-admin-website) for details. Once you finish the initialization, you should be able to see the following page as an admin user (lists are expected to be empty at a fresh start): ![Image registry home page in public view](/pelican/optimized_images/registry-admin-opt-1920.WEBP) You may view and manage namespaces registered in your federation. Pending registrations are shown as the first list and you can click each namespace strip for more detailed information. You can click the check button to approve a namespace registration or the cross button to deny the registration. As an admin user, you can also edit the registration by clicking the pencil icon. For approved namespaces, you can download the public key associated with the namespace by clicking the downward-arrow button. The homepage of the registry web UI is also publicly accessible, meaning users can see a list of _approved_ namespaces without logging into the registry. Denied and pending registrations are hidden from the public view. The image below shows what the public view looks like. ![Image of registry homepage in public view](/pelican/optimized_images/registry-public-opt-1920.WEBP) ## Useful Configurations for the Registry There are a couple of configuration parameters you could use to customize the behavior of your registry. Here we highlight the ones that are most frequently set for an admin. You may refer to the full set of registry parameters in the [Parameters page](./parameters#Registry-DbLocation). ### `Server.UIAdminUsers` > `Registry.AdminUsers` is deprecated in Pelican `v7.7.0`. Use `Server.UIAdminUsers` instead. By default, Pelican registry only has one user with admin privilege, which is whoever starts the registry service and initializes the web UI with the admin password. However, Pelican also allows you to pass a list of CILogon user identities and grant them the admin privilege. This requires you have your CILogon `client_id` and `client_secret` set up. The user identity of each admin user can be found on their [https://cilogon.org/](https://cilogon.org/)  user page, under “User Attributes”, `CILogon User Identifier`, which looks something like `http://cilogon.org/serverA/users/123456`. Once you have the identifiers, simply pass them as an array to `Registry.AdminUsers`: ``` Server: UIAdminUsers: ["http://cilogon.org/serverA/users/123456"] ``` ### `Registry.RequireOriginApproval` By default, a Pelican registry allows any origin to join the federation and serve objects (not the case if running in OSDF mode). You can set `Registry.RequireOriginApproval` if you want to review registration details of a new origin before allowing it to join your federation. With `Registry.RequireOriginApproval == true`, a new origin can successfully _register_ against your registry, but the director will not be able to direct object requests to the unapproved origin. A registry admin may edit, review, approve, or deny a new origin registration in your registry Web UI. ### `Registry.RequireCacheApproval` Similarly, Pelican by default allows any cache to join the federation. You can set `Registry.RequireCacheApproval` to `true` in order for caches to require approval before they can serve an object. ### `Registry.RequireKeyChaining` By default, Pelican requires namespaces requesting registration to possess a key matching any already-registered super/sub namespaces. For example, if a namespace `/foo/bar` is already registered, then registration of `/foo` or `/foo/bar/baz` can only be done using the public key registered to `/foo/bar`. ### `Registry.Institutions` and `Registry.InstitutionsUrl` When a user wants to register a namespace in the registry web UI, they must specify which institution this namespace is for. This is a list of options the Registry admin needs to provide. To do so you may either feed a list of `name` and `id` pairs of available institutions to register to [`Registry.Institutions`](../parameters#Registry-Institutions) or, if you already have a web endpoint to serve such data, you may pass the URL to [`Registry.InstitutionsUrl`](../parameters#Registry-InstitutionsUrl). --- Source: https://docs.pelicanplatform.org/operating-a-federation/director # Serving a Pelican Director The Pelican [_Director_](../about-pelican/core-concepts#director-service) handles data distribution in a Pelican federation. It directs object requests from Pelican Clients to the proper object provider (which can be an Origin or a Cache). It also maintains a collection of actively running Origin/Cache servers in the federation. ## Before Starting ### Enable Time Synchronization Pelican servers rely on time-sensitive tokens for authorization. If your server’s clock is not synchronized, tokens may be rejected and your Director may fail to operate correctly. Please ensure NTP is enabled on your server. See [Server Prerequisites](../install#enable-time-synchronization) for setup instructions. ### Get MaxMind License key (required) A Pelican director uses the [MaxMind GeoLite City database](https://dev.maxmind.com/geoip/docs/databases/city-and-country)  to determine which cache is nearest to a client’s IP address. The database, if not already found, will be downloaded automatically when a director is served and a valid key is present. The database is free to use, but you need a GeoLite2 account and provide a license key to Pelican so that we can download the database for you. Please follow the instructions [here](https://dev.maxmind.com/geoip/docs/databases/city-and-country)  to [register a free account](https://www.maxmind.com/en/geolite2/signup)  and [get your license key](https://www.maxmind.com/en/accounts/current/license-key/) . Once you have the key, copy and paste it to a desired location on your server. Then pass the file location to `Director.MaxMindKeyFile`. ### Set Registry URL (required) A Pelican director needs to talk to the registry to get the public keys for namespaces. You will need to set `Federation.RegistryUrl` to the public endpoint of your registry for your Pelican director to work. An example value looks like `https://your-registry.org:9999` ### Director as a Federation For Pelican origin, cache, and client, they need to know where the _federation_ they belong to so that they can make themselves known to other services or a access objects. This can be done by either manually passing configuration values (e.g. `Federation.DirectorUrl`, `Federation.RegistryUrl`, etc) or by setting the `Federation.DiscoveryUrl` and Pelican will fetch the metadata from the endpoint automatically. Pelican uses `Federation.DiscoveryUrl` to discover where a federation’s central services live, including the registry and director. By default, Pelican director hosts the federation discovery endpoint. Therefore, for origins and caches in your federation, as well as Pelican clients that want to access objects in your federation, their `Federation.DiscoveryUrl` value can be set to the director’s public URL, i.e. `https://:` or `https://` if the port number is simply `443`. Make sure your director is up and running before starting your origins and caches. ### Set Support Contact Information (Recommended) > `Director.SupportContact` is only available for Pelican >=7.7.0 The Pelican director is your federation’s entrypoint, and it is where other services in the federation can learn about each other. Because of this, it is also used to host the contact information for federation administrators. Setting values for the configuration options `Director.SupportContactEmail` and `Director.SupportContactUrl` causes the director to display the information through its website so that users know who and how to contact you for support. See [Configuration](./parameters#Director-SupportContact) page for details. pelican.yaml ``` Director: SupportContactEmail: SupportContactUrl: ``` Note that either `SupportContactEmail` or `SupportContactUrl` are optional. You may leave any of them empty is not needed. ## Launch the Director That’s it! Now you can launch the director by running the following command: ``` pelican-server director serve ``` and you will see the following message: ``` Pelican admin interface is not initialized To initialize, login at https://:8444/view/initialization/code/ with the following code: 865309 ``` By default, a director runs on port `8444`. You may change the port number by passing `-p ` when serving the director or by setting `Server.WebPort` in the configuration file. To finish setting up the web UI for the director, please refer to [Login to Admin Website](./serving_an_origin#login-to-admin-website) for details. Once you have finished the initialization and login with your admin password, you should be able to see the following page as an admin user (lists are expected to be empty at a fresh start): ![Image of director homepage in admin view](/pelican/optimized_images/director-homepage-opt-1920.WEBP) The “Origins” table and “Caches” table show the _active_ origins and caches in the federation. ## Useful Configurations for Director There are a couple of configuration parameters you could use to customize the behavior of your registry. Here we highlight the ones that are most frequently set for an admin. You may refer to the full set of director parameters in the [Parameters page](../parameters#Director-DefaultResponse). ### `Director.DefaultResponse` A Pelican director by default redirects object request from clients to the geo-spatially nearest cache in the federation to improve data transfer efficiency. If a director is hosted at [https://director.com](https://director.com) , then a `GET` request to [https://director.com/foo/bar.txt](https://director.com/foo/bar.txt)  will be redirected to the nearest cache for namespace `/foo`. You may change the default response type of a redirect for director by setting `Director.DefaultResponse`. It can be either `cache` or `origin`. If set to `origin`, director will redirect the request to the origin, bypassing caches. ### `GeoIPOverrides` The MaxMind database may fail to return a valid longitude/latitude pair given a server IP. For example, the database does not support IPV6 addresses. In such cases, you may pass a list of IP addresses to `GeoIPOverrides` parameter whose GeoIP resolution should be overridden with the supplied Lat/Long coordinates (in decimal form). This affects both server ads (for determining the location of origins and caches) and incoming client requests (for determining where a client request is coming from). ### `Director.CacheResponseHostnames` and `Director.OriginResponseHostnames` You may allow the director to redirect client traffic to both caches and origins. You can do it by adding virtual hostnames to `Director.CacheResponseHostnames` for a cache response or `Director.OriginResponseHostnames` for an origin response. If a request is sent by the client to one of these hostnames, the director assumes it should respond with a redirect to a cache/origin. If present, the hostname is taken from the `X-Forwarded-Host` header in the request. Otherwise, Host is used. --- Source: https://docs.pelicanplatform.org/operating-a-federation/cache # Serving a Pelican Cache The Pelican [_Cache_](../about-pelican/core-concepts#caches) creates a cache that connects to a Pelican data federation to allow faster data access. It stores data that is accessed via a Pelican origin until it gets evicted. The Cache has the potential to both be faster and physically closer to your location than the Pelican Origin you access the data from. This allows for faster data access when running complex workflows. This document contains instructions on how to serve a Pelican Cache. Are you setting up a Cache to connect to the OSDF? If so, please read the OSDF documentation for system administrators here: [osg-htc.org/docs/data/osdf/overview/](https://osg-htc.org/docs/data/osdf/overview/) . ## Before Starting ### Note on Installing Pelican Cache Server If you haven’t installed Pelican, follow the instructions to [install pelican](../install). For _Linux_ users, it is recommended to install Pelican using one of the package managers (RPM, APK, Deb, etc.) so that Pelican dependencies are automatically handled. You may also run a [Pelican Docker image](../install/docker) to serve a Pelican cache. If you prefer to install Pelican as a standalone binary, you need to follow [additional instructions](https://osg-htc.org/docs/data/xrootd/install-standalone/#install-xrootd-standalone)  to install dependencies for the Pelican cache. > Note that serving a Pelican cache with a standalone Pelican binary is possible, but not recommended or supported. For _macOS_ and _Windows_ users who want to serve a Pelican cache, please use [Pelican Docker image](../install/docker). ### Enable Time Synchronization Pelican servers rely on time-sensitive tokens for authorization. If your server’s clock is not synchronized, tokens may be rejected and your Cache may fail to join the federation. Please ensure NTP is enabled on your server. See [Server Prerequisites](../install#enable-time-synchronization) for setup instructions. ### Open Firewall Port for Pelican Cache The Pelican cache listens to two TCP ports for file transfers and Web UI. By default, the file transfer port is at `8442` and the Web UI and APIs port is at `8444`. If your server has firewall policy in place, please open the two ports for both incoming the outgoing TCP requests to allow the Pelican cache to function as expected. You may change the port numbers through the [configuration file](../parameters) with parameter [`Cache.Port`](../parameters#Cache-Port) and [`Server.WebPort`](../parameters#Server-WebPort) respectively. ### Find a federation to join Before serving a cache, you need to find a Pelican federation to join in. If you are unfamiliar with the term **federation**, refer to [Core Concepts and Terminology](../about-pelican/core-concepts#federations) before proceeding. If you don’t have a federation in mind, the Open Science Data Federation (OSDF) is an example Pelican federation that you can join in for testing purposes. If you are interesting in serving an OSDF cache, refer to the [OSDF website](https://osg-htc.org/)  for details. The federation discovery URL for OSDF is `osg-htc.org`. You may use this as your `` argument in the next section when launching your cache. ## Launch the Cache To launch a pelican cache, run: ``` pelican-server cache serve -f ``` Where: - `` is the URL to the federation the cache will be joining This will start a Pelican cache as a daemon process. ### Additional arguments to launch a Cache This section documents the additional arguments you can pass to the command above to run the cache. - **`-h` or `--help`**: Output documentation on the `serve` command and its arguments. - **`-p` or `--port`**: Set the port at which the Pelican admin website should be accessible. - **`--config`**: Set the location of the configuration file. - **`-d` or `--debug`**: Enable the debugging mode, allowing for more verbose log - **`-l` or `--log`**: Set the location of the file where log messages should be redirected to and not output to the console. There are other configurations available to modify via the configuration file. Refer to the [Parameters](../parameters) page for details. ## Test Cache Functionality Once you have your cache set up, follow the steps below to test if your cache can access a file through a Pelican federation. 1. Have data available and accessible via an origin. Since a cache only works with data available via an origin that’s in the same Pelican federation, the following assumes that you have a an accessible namespace within the same Pelican federation. For more information on how to set that up, refer to [Federating Your Data](../federating-your-data) page. The following also assume that the object to test is public file. To set up a public file through a Pelican origin, refer to the [Parameters](../parameters#Origin-Exports) page. 2. Curl the director for the test file and ensure that the url used is the cache that you set up. Assuming your directory is `/tmp/demo`, run the following command to curl your known public file from the cache. ``` $ curl -v https:/// ``` Where: - `` is the of your cache - `` is the namespace and the path of the known public file - Check that the curl output contains the data from the public file. Congratulations! You have finished setting up and running your cache. --- Source: https://docs.pelicanplatform.org/managing-your-server/managing-downtime # Managing Server Downtime Origin and Cache server administrators can declare **downtime periods** to signal to the federation that their server will be temporarily unavailable. The Pelican Director uses this information to route data requests away from affected servers during the declared period, minimizing disruptions for end users. Pelican can also set downtime automatically during a graceful shutdown. Federation administrators can also create or modify downtime for any server in the federation. If you need a downtime set but lack admin access to your own server, see [Method 3](#method-3-contact-the-federation-administrator) below. To see a consolidated view of all active and upcoming downtimes across the federation, visit the **Downtime** page in the **Director** web UI. ## Graceful Shutdown (Automatic Downtime) Typically, server administrators set a downtime just before shutting down or restarting a server. Instead of doing this manually, you can rely on Pelican’s “graceful shutdown” feature. It automatically establishes a downtime window under the hood to gracefully drain traffic, minimizing disruption to users’ ongoing transfers. When Pelican process receives a `SIGTERM` signal (e.g. via `kill -TERM `), or when the “Restart Server” button on web UI is clicked: 1. Pelican immediately sends a shutdown advertisement to the Director indicating that it is shutting down. 2. The Director applies an indefinite downtime window to the server, stopping it from redirecting new clients to this Origin/Cache. 3. Existing in-flight transfers are allowed to continue until `Xrootd.ShutdownTimeout` (default: 1m) expires and Pelican fully exits. 4. The downtime remains in effect indefinitely to prevent routing to offline servers, and is automatically cleared whenever the server restarts and sends a regular server ad. This automatic process does not affect any manually configured downtimes set by server or federation administrators. If you need to schedule downtime for the future, or require a downtime window while the server remains running, use the methods below. ## Method 1: Web UI (Recommended) The web interface provides the easiest way to manage downtime. Log in to your server’s web UI with your administrator credentials, then navigate to **Downtime** (the calendar icon) in the left sidebar. ### Add a Downtime 1. On the **Downtime** page, click the **“CREATE DOWNTIME”** button. 2. Fill in the form: - **Start Time** and **End Time** — use the date/time pickers. Toggle **Unknown Endtime** if the end time is unknown. - **Description** — briefly explain why the server is going down. - **Severity** — describes the impact level of the downtime. This field is informational only and does not currently affect how the Director routes traffic; leaving it at the default (Outage) is fine. 3. Click **SUBMIT** to save. ### Edit a Downtime Click the **pencil icon** on an existing downtime card to open the edit form, make your changes, then click **SUBMIT**. ### Delete a Downtime Click the **pencil icon** on an existing downtime card to open the edit form, then click the **DELETE DOWNTIME** button to remove it. The Director will stop routing around that downtime on the next advertisement cycle. * * * ## Method 2: CLI (`pelican downtime`) The `pelican-server downtime` CLI commands interact directly with your server’s admin API and require either local admin access or a valid admin token. All `pelican-server downtime` subcommands accept the following persistent flags: | Flag | Required | Description | | --- | --- | --- | | `-s` / `--server` | Yes | Web URL of your Pelican server (e.g. `https://my-origin.example.com:8444`) | | `-t` / `--token` | No | Path to a token file with downtime scopes (`pelican.downtime_create`, `pelican.downtime_modify`, `pelican.downtime_delete`). Omit to generate automatically (requires access to the server’s issuer key). | ### List Downtimes ``` # Show active and future downtimes (default) pelican-server downtime list --server https://my-server.example.com:8444 # Show all downtimes including past ones pelican-server downtime list --server https://my-server.example.com:8444 --status all ``` ### Create a Downtime The `create` subcommand is **interactive** — it will prompt you for each field: ``` pelican-server downtime create --server https://my-server.example.com:8444 ``` Example session: ``` Select downtime Class: 1. SCHEDULED 2. UNSCHEDULED Enter choice number: 1 Enter downtime Description: Weekly maintenance window Select downtime Severity: 1. Outage (completely inaccessible) 2. Severe (most services down) 3. Intermittent Outage (may be up for some of the time) 4. No Significant Outage Expected (you shouldn't notice) Enter number to choose: 1 Enter start time in UTC (YYYY-MM-DD HH:MM:SS): 2026-03-10 02:00:00 Enter end time in UTC (YYYY-MM-DD HH:MM:SS) or '-1' for indefinite: 2026-03-10 06:00:00 Downtime created successfully: ... ``` ### Update a Downtime The `update` subcommand is also interactive. Press **Enter** without typing anything to leave a field unchanged: ``` pelican-server downtime update --server https://my-server.example.com:8444 ``` ### Delete a Downtime ``` pelican-server downtime delete --server https://my-server.example.com:8444 ``` * * * ## Method 3: Contact the Federation Administrator If you do not have admin access to your server’s web UI or CLI, you can ask your federation administrator to set a downtime. Federation admins can manage downtime for any registered server in the federation from the **Registry** web interface. Provide your federation admin with: - Your server’s **name** - The expected **start** and **end** times (UTC) - A brief **description** --- Source: https://docs.pelicanplatform.org/monitoring-pelican-services/prometheus # Query Pelican Server Metrics via Prometheus > For Pelican `>= 7.16.0`. Older version of Pelican may not include all of the metrics listed. Pelican servers have [Prometheus](https://prometheus.io/)  embedded by default and provide a handful of Prometheus metrics to monitor server status. You can access the metrics endpoint at `https://:/metrics` to see all the available metrics and their current values. By default, `/metrics` is a protected endpoint and you are required to login and get authenticated to view the page. You can change `Monitoring.MetricAuthorization` to `false` in config to turn off the authentication. Pelican also exposes Prometheus [PromQL query engine](https://prometheus.io/docs/prometheus/latest/querying/basics/)  at `https://:/api/v1.0/prometheus` where you can query the metrics against Prometheus powerful query language. Example: `https://:/api/v1.0/prometheus/query?query=pelican_component_health_status[10m]` queries the `pelican_component_health_status` metric and shows data collected in past 10 min. However, Pelican does not support Prometheus native `/graph` endpoint nor other Prometheus native web services other than the two above. For custom data visualizations, [Grafana](https://grafana.com/)  is one of the popular software to use. Pelican included metrics from built-in [gin](https://gin-gonic.com/)  web server, as well as Go runtime. For all metrics available, visit `https://:/api/v1.0/prometheus/label/__name__/values`. Pelican also has a set of built-in metrics to monitor Pelican server’s status, listed below. ### Counter Metrics Many metrics in this documentation are **counters** (typically identified by names ending in `_total` or `_count`). Counter metrics are monotonically increasing values that accumulate over time. Important notes about counters: - **Counters reset on restart**: When a Pelican server restarts, all counter values reset to 0. The counter then begins accumulating from 0 again. - **Time ranges are required**: To get meaningful data from counters, you must use PromQL functions that calculate changes over time, such as: - `rate()` - calculates the per-second average rate of increase - `irate()` - calculates the per-second instant rate of increase - `increase()` - calculates the increase over a time range - **Example usage**: Instead of querying `xrootd_server_bytes_total` directly, use `rate(xrootd_server_bytes_total[5m])` to get bytes per second, or `increase(xrootd_server_bytes_total[1h])` to get total bytes transferred in the last hour. ### Gauge Metrics Many metrics in this documentation are **gauges**. Gauge metrics represent a single numerical value that can go up or down over time. Important notes about gauges: - **Gauges represent current state**: Unlike counters, gauges show the current value of something at a point in time (e.g., number of active connections, current CPU usage, available disk space). - **Gauges can be queried directly**: You can query gauge metrics directly without time-range functions like `rate()` or `increase()`. The value represents the current state. - **Gauges persist across restarts**: When a server restarts, gauge values reset to their initial state (often 0), but they don’t accumulate like counters. The gauge will reflect the new current state after restart. - **Example usage**: Query `xrootd_server_io_active` directly to see the current number of active IO operations, or use `avg_over_time(xrootd_server_io_active[5m])` to see the average over the last 5 minutes. ## All Servers All of the Pelican servers have the following metrics: ### `process_start_time_seconds` The UNIX epoch time in seconds when the Pelican process started. To get the duration of the Pelican server running time, use the following PromQL: PromQL ``` time() - process_start_time_seconds ``` This yields the duration in seconds. ### `pelican_component_health_status` The health status of Pelican server components. The metric value can be converted into following status: > **Note**: This is a gauge metric representing the current health status. Query directly to see the current state of each component. ``` 1: Critical 2: Warning 3: OK 4: Unknown ``` #### Label: `component` | Label Value | Description | Availability | | --- | --- | --- | | `web-ui` | Admin website | All servers | | `xrootd` | XRootD process | Origin and cache servers | | `cmsd` | CMSD process | Origin and cache servers | | `federation` | Advertisement to the Director | Origin and cache servers | | `registry` | Namespace registration at the Registry | Origin and cache servers | | `director` | Object transfer tests from the Director | Origin and cache servers | | `topology` | Data fetch from the OSDF topology server | All servers (OSDF mode only) | | `IO-concurrency` | Health status indicating whether the average concurrent IO operations exceed the configured concurrency limit, used by the Director to determine if redirects should be reduced | Origin and cache servers | | `prometheus` | Health status of the embedded Prometheus server. Critical indicates Prometheus failed to start or the server is not ready to receive web requests (metrics unavailable). OK indicates Prometheus started successfully and is ready | All servers | | `config-updates` | Health status of XRootD configuration file updates (authfile and scitokens.cfg). Critical indicates files are stale beyond the configured timeout and may trigger auto-shutdown if enabled. Warning indicates update failures observed but within timeout. OK indicates both files updated successfully | Origin and cache servers | ### `pelican_component_health_status_last_update` The timestamp of last update of health status of Pelican server components. The value is UNIX time in seconds. It shares the same label as `pelican_component_health_status` > **Note**: This is a gauge metric representing the last update timestamp. Query directly to see when each component’s health status was last updated. ### `pelican_server_xrootd_last_crash` The timestamp (seconds) of the last crash of the XRootD server. > **Note**: This is a gauge metric representing the timestamp of the last crash. Query directly to see when XRootD last crashed. A value of 0 indicates no crashes have occurred since the server started. ## Registry ### `pelican_registry_federation_namespaces` The number of namespace registrations in the registry. > **Note**: This is a gauge metric representing the current number of namespace registrations. Query directly to see the current count. #### Label: `status` | Label Values | Description | | --- | --- | | `ok` | The number of namespaces that are have a valid registration. | | `error` | The number of namespaces that have an error in their registration. | ### `pelican_osdf_institution_count` Total number of contributing institutions. This is only available when running in OSDF mode. > **Note**: This is a gauge metric representing the current number of institutions. Query directly to see the current count. ## Storage Servers (Origin and Cache) ### `xrootd_monitoring_packets_received_total` The total number of [XRootD monitoring](https://xrootd.web.cern.ch/doc/dev6/xrd_monitoring.htm)  UDP packets received. > **Note**: This is a counter metric. Use `rate(xrootd_monitoring_packets_received_total[5m])` to get packets per second, or `increase(xrootd_monitoring_packets_received_total[1h])` to get total packets in the last hour. ### `xrootd_sched_thread_count` The number of XRootD scheduler threads. Ref: [https://xrootd.web.cern.ch/doc/dev6/xrd\_monitoring.htm#\_Toc204013493](https://xrootd.web.cern.ch/doc/dev6/xrd_monitoring.htm#_Toc204013493)  > **Note**: This is a gauge metric representing the current number of threads. Query directly or use `avg_over_time(xrootd_sched_thread_count[5m])` to see the average over a time range. #### Label: `state` | Label Value | Description | | --- | --- | | `idle` | Scheduler threads waiting for work | | `running` | Scheduler threads running | ### `xrootd_sched_thread_creations` Number of scheduler thread creations. > **Note**: This is a counter metric. Use `rate(xrootd_sched_thread_creations[5m])` to get thread creations per second, or `increase(xrootd_sched_thread_creations[1h])` to get total thread creations in the last hour. ### `xrootd_sched_thread_destructions` Number of scheduler thread destructions. > **Note**: This is a counter metric. Use `rate(xrootd_sched_thread_destructions[5m])` to get thread destructions per second, or `increase(xrootd_sched_thread_destructions[1h])` to get total thread destructions in the last hour. ### `xrootd_sched_thread_limit_reached` Number of times the scheduler thread limit has been reached. > **Note**: This is a counter metric. Use `rate(xrootd_sched_thread_limit_reached[5m])` to get limit hits per second, or `increase(xrootd_sched_thread_limit_reached[1h])` to get total limit hits in the last hour. ### `xrootd_sched_jobs` Number of scheduler jobs requiring a thread. > **Note**: This is a gauge metric representing the current number of jobs. Query directly or use `avg_over_time(xrootd_sched_jobs[5m])` to see the average over a time range. ### `xrootd_sched_queue_longest_length` Length of the longest run-queue. > **Note**: This is a gauge metric representing the current longest queue length. Query directly or use `avg_over_time(xrootd_sched_queue_longest_length[5m])` to see the average over a time range. ### `xrootd_sched_queued` Number of jobs queued. > **Note**: This is a gauge metric representing the current number of queued jobs. Query directly or use `avg_over_time(xrootd_sched_queued[5m])` to see the average over a time range. ### `xrootd_server_bytes_total` The total number of bytes XRootD sent/received. Ref: [https://xrootd.web.cern.ch/doc/dev6/xrd\_monitoring.htm#\_Toc204013487](https://xrootd.web.cern.ch/doc/dev6/xrd_monitoring.htm#_Toc204013487)  (See `link.in` and `link.out`) > **Note**: This is a counter metric. Use `rate(xrootd_server_bytes_total[5m])` to get bytes per second, or `increase(xrootd_server_bytes_total[1h])` to get total bytes in the last hour. #### Label: `direction` | Label Values | Description | | --- | --- | | `tx` | Bytes sent | | `rx` | Bytes received | ### `xrootd_server_connections_total` The total number of server connections to XRootD. > **Note**: This is a counter metric. Use `rate(xrootd_server_connections_total[5m])` to get connections per second, or `increase(xrootd_server_connections_total[1h])` to get total connections in the last hour. ### `xrootd_storage_volume_bytes` The storage volume usage on the storage server. > **Note**: This is a gauge metric representing the current storage volume. Query directly or use `avg_over_time(xrootd_storage_volume_bytes[5m])` to see the average over a time range. #### Label: `type` | Label Values | Description | | --- | --- | | `total` | Total bytes visible on the storage server | | `free` | Available bytes to use | #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | #### Label: `ns` The top-level namespace the XRootD is serving for. Example: `/foo` ### `xrootd_transfer_bytes` The bytes of transfers for individual object. Ref: [https://xrootd.web.cern.ch/doc/dev6/xrd\_monitoring.htm#\_Toc204013508](https://xrootd.web.cern.ch/doc/dev6/xrd_monitoring.htm#_Toc204013508)  (See XrdXrootdMonStatXFR) > **Note**: This is a counter metric. Use `rate(xrootd_transfer_bytes[5m])` to get bytes per second, or `increase(xrootd_transfer_bytes[1h])` to get total bytes in the last hour. #### Label: `path` The path to the object (filename). #### Label: `ap` Authentication protocol name used to authenticate the client. Default is `https` #### Label: `dn` Client’s distinguished name as reported by ap. If no name is present, the variable data is null. #### Label: `role` Client’s role name as reported by prot. If no role name is present, the variable data is null. #### Label: `org` Client’s group names in a space-separated list. If no groups are present, the tag variable data is null. #### Label: `proj` Client’s `User-Agent` header when requesting the file. This is used to label the project name that accesses the file. #### Label: `type` | Label Values | Description | | --- | --- | | `read` | Bytes read from file using read() | | `readv` | Bytes read from file using readv() | | `write` | Bytes written to file | ### `xrootd_transfer_operations_total` The number of transfer operations performed for individual object. The labels for this metric is the same as the ones in `xrootd_transfer_bytes` > **Note**: This is a counter metric. Use `rate(xrootd_transfer_operations_total[5m])` to get operations per second, or `increase(xrootd_transfer_operations_total[1h])` to get total operations in the last hour. ### `xrootd_transfer_readv_segments_total` The number of segments in readv operations for individual object. The labels for this metric is the same as the ones in `xrootd_transfer_bytes` except that `type` label isn’t available in this metric. > **Note**: This is a counter metric. Use `rate(xrootd_transfer_readv_segments_total[5m])` to get segments per second, or `increase(xrootd_transfer_readv_segments_total[1h])` to get total segments in the last hour. ### `xrootd_cache_access_bytes` Number of bytes the data requested is in the cache or not. > **Note**: This is a gauge metric representing the current cache access state. Query directly or use `avg_over_time(xrootd_cache_access_bytes[5m])` to see the average over a time range. #### Label: `path` The path to the object (filename). #### Label: `type` | Label Values | Description | | --- | --- | | `hit` | Bytes served from cache. | | `miss` | Bytes missed in cache. | | `bypass` | Bytes that bypassed the cache. | ### `xrootd_server_io_total` Total storage operations in origin/cache server. > **Note**: This is a counter metric. Use `rate(xrootd_server_io_total[5m])` to get operations per second, or `increase(xrootd_server_io_total[1h])` to get total operations in the last hour. ### `xrootd_server_io_active` Number of ongoing storage operations in origin/cache server. > **Note**: This is a gauge metric representing the current number of active IO operations. Query directly or use `avg_over_time(xrootd_server_io_active[5m])` to see the average over a time range. ### `xrootd_server_io_wait_seconds_total` The aggregate time spent in storage operations in origin/cache server. > **Note**: This is a counter metric. Use `rate(xrootd_server_io_wait_seconds_total[5m])` to get average wait time per second, or `increase(xrootd_server_io_wait_seconds_total[1h])` to get total wait time in the last hour. ### `xrootd_cpu_utilization` CPU utilization of the XRootD server, represented as the average number of CPU cores utilized (e.g., 1.0 = one full core, 2.5 = two and a half cores). > **Note**: This is a gauge metric representing the current CPU utilization. Query directly to see the current utilization, or use `avg_over_time(xrootd_cpu_utilization[5m])` to see the average over a time range. ### OSS Layer Metrics The following metrics are available from the XRootD OSS layer. > **Note**: All metrics ending in `_total` are counter metrics. Use `rate()` or `increase()` with a time range to query them. All metrics ending in `_time_seconds` are histogram metrics that track operation duration distributions. | Metric Name | Description | | --- | --- | | `xrootd_oss_reads_total` | The total number of read operations on the OSS. | | `xrootd_oss_writes_total` | The total number of write operations on the OSS. | | `xrootd_oss_stats_total` | The total number of stat operations on the OSS. | | `xrootd_oss_pgreads_total` | The total number of page read operations on the OSS. | | `xrootd_oss_pgwrites_total` | The total number of page write operations on the OSS. | | `xrootd_oss_readv_total` | The total number of readv operations on the OSS. | | `xrootd_oss_readv_segments_total` | The total number of segments in readv operations on the OSS. | | `xrootd_oss_dirlists_total` | The total number of directory list operations on the OSS. | | `xrootd_oss_dirlist_entries_total` | The total number of directory list entries on the OSS. | | `xrootd_oss_truncates_total` | The total number of truncate operations on the OSS. | | `xrootd_oss_unlinks_total` | The total number of unlink operations on the OSS. | | `xrootd_oss_chmods_total` | The total number of chmod operations on the OSS. | | `xrootd_oss_opens_total` | The total number of open operations on the OSS. | | `xrootd_oss_renames_total` | The total number of rename operations on the OSS. | | `xrootd_oss_slow_reads_total` | The total number of slow read operations on the OSS. | | `xrootd_oss_slow_writes_total` | The total number of slow write operations on the OSS. | | `xrootd_oss_slow_stats_total` | The total number of slow stat operations on the OSS. | | `xrootd_oss_slow_pgreads_total` | The total number of slow page read operations on the OSS. | | `xrootd_oss_slow_pgwrites_total` | The total number of slow page write operations on the OSS. | | `xrootd_oss_slow_readv_total` | The total number of slow readv operations on the OSS. | | `xrootd_oss_slow_readv_segments_total` | The total number of segments in slow readv operations on the OSS. | | `xrootd_oss_slow_dirlists_total` | The total number of slow directory list operations on the OSS. | | `xrootd_oss_slow_dirlist_entries_total` | The total number of slow directory list entries on the OSS. | | `xrootd_oss_slow_truncates_total` | The total number of slow truncate operations on the OSS. | | `xrootd_oss_slow_unlinks_total` | The total number of slow unlink operations on the OSS. | | `xrootd_oss_slow_chmods_total` | The total number of slow chmod operations on the OSS. | | `xrootd_oss_slow_opens_total` | The total number of slow open operations on the OSS. | | `xrootd_oss_slow_renames_total` | The total number of slow rename operations on the OSS. | | `xrootd_oss_open_time_seconds` | The time taken for open operations on the OSS. | | `xrootd_oss_read_time_seconds` | The time taken for read operations on the OSS. | | `xrootd_oss_readv_time_seconds` | The time taken for readv operations on the OSS. | | `xrootd_oss_pgread_time_seconds` | The time taken for page read operations on the OSS. | | `xrootd_oss_write_time_seconds` | The time taken for write operations on the OSS. | | `xrootd_oss_pgwrite_time_seconds` | The time taken for page write operations on the OSS. | | `xrootd_oss_dirlist_time_seconds` | The time taken for directory list operations on the OSS. | | `xrootd_oss_stat_time_seconds` | The time taken for stat operations on the OSS. | | `xrootd_oss_truncate_time_seconds` | The time taken for truncate operations on the OSS. | | `xrootd_oss_unlink_time_seconds` | The time taken for unlink operations on the OSS. | | `xrootd_oss_rename_time_seconds` | The time taken for rename operations on the OSS. | | `xrootd_oss_chmod_time_seconds` | The time taken for chmod operations on the OSS. | | `xrootd_oss_slow_open_time_seconds` | The time taken for slow open operations on the OSS. | | `xrootd_oss_slow_read_time_seconds` | The time taken for slow read operations on the OSS. | | `xrootd_oss_slow_readv_time_seconds` | The time taken for slow readv operations on the OSS. | | `xrootd_oss_slow_pgread_time_seconds` | The time taken for slow page read operations on the OSS. | | `xrootd_oss_slow_write_time_seconds` | The time taken for slow write operations on the OSS. | | `xrootd_oss_slow_pgwrite_time_seconds` | The time taken for slow page write operations on the OSS. | | `xrootd_oss_slow_dirlist_time_seconds` | The time taken for slow directory list operations on the OSS. | | `xrootd_oss_slow_stat_time_seconds` | The time taken for slow stat operations on the OSS. | | `xrootd_oss_slow_truncate_time_seconds` | The time taken for slow truncate operations on the OSS. | | `xrootd_oss_slow_unlink_time_seconds` | The time taken for slow unlink operations on the OSS. | | `xrootd_oss_slow_rename_time_seconds` | The time taken for slow rename operations on the OSS. | | `xrootd_oss_slow_chmod_time_seconds` | The time taken for slow chmod operations on the OSS. | ### S3 Cache Plugin Metrics The following metrics are available from the XRootD S3 cache plugin. > **Note**: All metrics ending in `_total` are counter metrics. Use `rate()` or `increase()` with a time range to query them. | Metric Name | Description | Labels | | --- | --- | --- | | `xrootd_s3_cache_bytes_total` | Bytes transferred by the S3 cache plugin. | `type`: `hit`, `miss`, `bypass`, `fetch`, `unused`, `prefetch` | | `xrootd_s3_cache_hits_total` | Number of cache hits, partial hits, or misses. | `type`: `full`, `partial`, `miss` | | `xrootd_s3_cache_requests_total` | Number of cache requests. | `type`: `bypass`, `fetch`, `prefetch` | | `xrootd_s3_cache_errors_total` | Number of errors encountered by the S3 cache plugin. | | | `xrootd_s3_cache_request_seconds_total` | Total time spent in S3 requests. | `type`: `bypass`, `fetch` | ### XrdCl Client Metrics The following metrics are available from the XrdCl client. > **Note**: All metrics ending in `_total` are counter metrics. Use `rate()` or `increase()` with a time range to query them. Metrics ending in `_timestamp_seconds` are gauge metrics representing timestamps and can be queried directly. `xrootd_xrdcl_queue_pending` is a gauge metric representing the current queue size. | Metric Name | Description | Labels | | --- | --- | --- | | `xrootd_xrdcl_prefetch_count_total` | Total number of prefetches started. | | | `xrootd_xrdcl_prefetch_expired_total` | Total number of prefetches that expired. | | | `xrootd_xrdcl_prefetch_failed_total` | Total number of prefetches that failed. | | | `xrootd_xrdcl_prefetch_reads_hit_total` | Total number of successful reads from prefetch buffer. | | | `xrootd_xrdcl_prefetch_reads_miss_total` | Total number of reads that missed the prefetch buffer. | | | `xrootd_xrdcl_prefetch_bytes_used_total` | Total number of bytes served from prefetch. | | | `xrootd_xrdcl_queue_produced_total` | Total number of HTTP requests placed into the queue. | | | `xrootd_xrdcl_queue_consumed_total` | Total number of HTTP requests read from the queue. | | | `xrootd_xrdcl_queue_pending` | Number of pending HTTP requests in the queue. | | | `xrootd_xrdcl_queue_rejected_total` | Total number of HTTP requests rejected due to overload. | | | `xrootd_xrdcl_worker_oldest_op_timestamp_seconds` | Timestamp of the oldest operation in any of the worker threads. | | | `xrootd_xrdcl_worker_oldest_cycle_timestamp_seconds` | Timestamp of the oldest event loop completion in any of the worker threads. | | | `xrootd_xrdcl_http_requests_total` | Statistics about HTTP requests. | `verb`, `status`, `type` | | `xrootd_xrdcl_http_request_duration_seconds_total` | Total duration of HTTP requests. | `verb`, `status`, `type` | | `xrootd_xrdcl_http_bytes_total` | Bytes transferred for HTTP requests. | `verb`, `status` | | `xrootd_xrdcl_conncall_total` | Statistics about connection calls. | `type` | ### Cache Eviction Metrics The following metrics are available from the XRootD cache eviction process. > **Note**: All cache eviction metrics are gauge metrics representing current state. Query them directly or use `avg_over_time()` to see averages over a time range. | Metric Name | Description | Labels | | --- | --- | --- | | `xrootd_cache_eviction_last_update_time_seconds` | The last time xrootd cache eviction metrics were updated. | | | `xrootd_cache_eviction_disk_usage_bytes` | The disk usage of the xrootd cache. | | | `xrootd_cache_eviction_snapshot_stats_reset_time_seconds` | The time when the snapshot statistics were last reset. | | | `xrootd_cache_eviction_disk_total_bytes` | The total disk space available for the cache. | | | `xrootd_cache_eviction_file_usage_bytes` | The file usage of the xrootd cache. | | | `xrootd_cache_eviction_meta_total_bytes` | The total metadata storage available for the cache. | | | `xrootd_cache_eviction_meta_used_bytes` | The used metadata storage for the cache. | | | `xrootd_cache_eviction_dir_num_ios` | Number of I/Os per directory. | `dir_name` | | `xrootd_cache_eviction_dir_duration` | Duration of I/Os per directory. | `dir_name` | | `xrootd_cache_eviction_dir_bytes` | Bytes transferred per directory. | `dir_name`, `type`: `hit`, `missed`, `bypassed`, `written` | | `xrootd_cache_eviction_dir_st_block_bytes` | Bytes from storage blocks per directory. | `dir_name`, `type`: `added`, `removed` | | `xrootd_cache_eviction_dir_n_cksum_errors` | Number of checksum errors per directory. | `dir_name` | | `xrootd_cache_eviction_dir_files_count` | File operations per directory (opened, closed, created, removed). | `dir_name`, `type`: `opened`, `closed`, `created`, `removed` | | `xrootd_cache_eviction_dir_directories_count` | Directory operations (created, removed) per directory. | `dir_name`, `type`: `created`, `removed` | | `xrootd_cache_eviction_dir_last_access_time_seconds` | Last access time per directory. | `dir_name`, `type`: `open`, `close` | | `xrootd_cache_eviction_dir_st_blocks_usage_count` | Storage blocks usage per directory. | `dir_name` | | `xrootd_cache_eviction_dir_n_files_open_count` | Number of open files per directory. | `dir_name` | | `xrootd_cache_eviction_dir_n_files_count` | Number of files per directory. | `dir_name` | | `xrootd_cache_eviction_dir_n_directories_count` | Number of directories per directory. | `dir_name` | ## Director ### `up` The Pelican director scrapes Prometheus metrics from all origins and cache servers that successfully advertise to the director. This metric reflects the Pelican origin or cache servers that are scraped by the director. #### Label: `server_name` The name of the storage server. By default it’s the hostname. #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | #### Label: `server_url` The storage server XRootD url. #### Label: `server_web_url` The storage server web url. #### Label: `server_auth_url` The storage server authentication url. #### Label: `server_lat` The storage server latitude. #### Label: `server_long` The storage server longitude. ### `# of Active Origins and Caches` With the `up` metric, it is possible to count number of active origin and cache servers in the federation by a simple Prometheus query: `count(up{server_type=<"Origin">})` for counting origin servers, or `count(up{server_type=<"Cache">})` for counting cache servers. ### `pelican_director_advertisements_received_total` The accumulated number of origin/cache advertisements to the director. This metric shows if an origin/cache server successfully joins the federation or not. For origin servers, it also shows if each federation namespace prefix it exports passed director verification. > **Note**: This is a counter metric. Use `rate(pelican_director_advertisements_received_total[5m])` to get advertisements per second, or `increase(pelican_director_advertisements_received_total[1h])` to get total advertisements in the last hour. #### Label: `server_name` The name of the storage server. By default it’s the hostname. #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | #### Label: `server_web_url` The storage server web url. ### Label: `namespace_prefix` The federation namespace prefix the storage server exported. ### Label: `status_code` The status code of the director’s response. The most useful value is `403`, which means the server advertisement didn’t pass director’s verification. | Label Values | Description | | --- | --- | | `200` | Advertisement succeeded | | `403` | Advertisement verification failed | | `500` | Director has errors when verifying or saving the advertisement | ### `pelican_director_stat_total` The accumulated number of `stat` query the director made to origin/cache servers to check for object availability. Only available when `Director.EnableStat` is set to true. This metric is a good indicator of object availability and origin/cache service quality. > **Note**: This is a gauge metric representing the current accumulated count. Query directly to see the current total. Note that this metric accumulates but is a gauge (not a counter), so it may reset on restart. #### Label: `server_name` The name of the storage server. By default it’s the hostname. #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | #### Label: `server_url` The storage server XRootD url. #### Label: `result` The `stat` query result. | Label Values | Description | | --- | --- | | `Succeeded` | The object requested is on the server | | `NotFound` | The requested object could not be found on the server | | `Timeout` | The query exceeded the allotted time and was not completed. | | `Cancelled` | The query is cancelled as maximum number of responses has been reached | | `Forbidden` | The object request was denied due to lack of permissions or missing token | | `UnknownErr` | An unexpected error occurred. Typically when the server refused to connect | #### Label: `cached_result` Whether the result was cached. ### `pelican_director_stat_active` The ongoing `stat` queries at the server. Note that Prometheus samples the metric value per 15s, and each `stat` request only takes ~10-100ms to finish. The value of this metric can’t capture per-second transient requests. > **Note**: This is a gauge metric representing the current number of active stat queries. Query directly to see the current count. #### Label: `server_name` The name of the storage server. By default it’s the hostname. #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | #### Label: `server_url` The storage server XRootD url. ### `pelican_director_total_ftx_test_suite` The number of file transfer test suite the director issued. In Pelican, director creates a test file and sent to origin servers to as a health test. It issues such test suite when it receives the registration from the origin server. In a test suite, a timer was set to run a cycle of uploading, getting, and deleting the test file every 15 seconds. Such cycle is called a “test run”. In theory, director should issue only one test for each origin servers; however, since the registration information was stored in a TTL cache in director, and it expires after certain period of time, and the test suite issued will be cancelled. A new test suite is issued with the new registration. Thus, director _can_ issue multiple test suites to an origin server. > **Note**: This is a counter metric. Use `rate(pelican_director_total_ftx_test_suite[5m])` to get test suites per second, or `increase(pelican_director_total_ftx_test_suite[1h])` to get total test suites in the last hour. #### Label: `server_name` The name of the storage server. By default it’s the hostname. #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | #### Label: `server_web_url` The storage server web url. ### `pelican_director_active_ftx_test_suite` The number of _active_ director file transfer test suite. As mentioned in previous metric, the test suites are individual tasks running independently from the main program logic. This can cause race condition in some condition where an expired test suite was not cleared but a new test suite is issued for the same origin. This metric records such condition for debugging and monitoring. The value of the metric _should_ be 1 for all the time. This metric shares the same label as `pelican_director_total_ftx_test_suite` > **Note**: This is a gauge metric representing the current number of active test suites. Query directly to see the current count. ### `pelican_director_total_ftx_test_runs` The number of file transfer test runs the director issued. A “test run” is a set of upload/get/delete of test files to a origin. It executes in a cycle of 15s (by default). > **Note**: This is a counter metric. Use `rate(pelican_director_total_ftx_test_runs[5m])` to get test runs per second, or `increase(pelican_director_total_ftx_test_runs[1h])` to get total test runs in the last hour. This metric shares the same label as `pelican_director_total_ftx_test_suite`, with two additions: #### Label: `status` | Label Values | Description | | --- | --- | | `Success` | The test run succeeded | | `Failed` | The test run failed | #### Label: `report_status` | Label Values | Description | | --- | --- | | `Success` | The reporting to the origin of test run status succeeded | | `Failed` | The reporting to the origin of test run status failed | ### `pelican_director_map_items_total` The total number of map items in the director, by the name of the map. > **Note**: This is a gauge metric representing the current number of map items. Query directly to see the current count. #### Label: `name` The name of the map. One of `healthTestUtils`, `filteredServers`, `serverStatUtils`, `serverStatEntries`. ### `pelican_director_ttl_cache` The statistics of various TTL caches. > **Note**: This is a gauge metric representing the current TTL cache statistics. Query directly to see the current values. #### Label: `name` The name of the cache. One of `serverAds`, `jwks`. #### Label: `type` The type of the statistic. One of `evictions`, `insertions`, `hits`, `misses`, `total`. ### `pelican_director_server_count` The number of servers currently recognized by the Director, delineated by pelican/non-pelican and origin/cache. > **Note**: This is a gauge metric representing the current number of servers. Query directly to see the current count. #### Label: `server_name` The name of the server. #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | #### Label: `from_topology` Whether the server was discovered from the topology. ### `pelican_director_client_requests_total` The total number of requests from clients. > **Note**: This is a counter metric. Use `rate(pelican_director_client_requests_total[5m])` to get requests per second, or `increase(pelican_director_client_requests_total[1h])` to get total requests in the last hour. #### Label: `version` The client version. #### Label: `service` The service that received the request. ### `pelican_director_redirects_total` The total number of redirects the director issued. > **Note**: This is a counter metric. Use `rate(pelican_director_redirects_total[5m])` to get redirects per second, or `increase(pelican_director_redirects_total[1h])` to get total redirects in the last hour. #### Label: `destination` The destination of the redirect. #### Label: `status_code` The status code of the redirect. #### Label: `version` The client version. #### Label: `network` The network of the client. ### `pelican_director_maxmind_server_errors_total` The total number of errors encountered trying to resolve server coordinates using the GeoIP MaxMind database. > **Note**: This is a counter metric. Use `rate(pelican_director_maxmind_server_errors_total[5m])` to get errors per second, or `increase(pelican_director_maxmind_server_errors_total[1h])` to get total errors in the last hour. #### Label: `network` The network address that was being resolved. #### Label: `server_name` The name of the server that was being resolved. ### `pelican_director_maxmind_client_errors_total` The total number of errors encountered trying to resolve client coordinates using the GeoIP MaxMind database. > **Note**: This is a counter metric. Use `rate(pelican_director_maxmind_client_errors_total[5m])` to get errors per second, or `increase(pelican_director_maxmind_client_errors_total[1h])` to get total errors in the last hour. #### Label: `network` The network address that was being resolved. #### Label: `project` The project of the client that was being resolved. ### `pelican_director_rejected_advertisements` The total number of advertisements rejected by the director. > **Note**: This is a counter metric. Use `rate(pelican_director_rejected_advertisements[5m])` to get rejections per second, or `increase(pelican_director_rejected_advertisements[1h])` to get total rejections in the last hour. #### Label: `hostname` The hostname of the server that sent the advertisement. ### `pelican_director_server_statusweight` The EWMA-smoothed status weight generated by the Director for each server. > **Note**: This is a gauge metric representing the current status weight. Query directly to see the current weight value. #### Label: `server_name` The name of the server. #### Label: `server_url` The URL of the server. #### Label: `server_type` | Label Values | Description | | --- | --- | | `Origin` | Origin server | | `Cache` | Cache server | ### Deprecated Metrics The following metrics are deprecated and will be removed in a future release. - `pelican_director_geoip_errors`: \[Deprecated — split into separate client/server metrics (pelican\_director\_maxmind\_{server,client}\_errors\_total)\] The total number of errors encountered trying to resolve coordinates using the GeoIP MaxMind database. - `xrootd_monitoring_packets_received`: Renamed to `xrootd_monitoring_packets_received_total`. - `xrootd_transfer_readv_segments_count`: Renamed to `xrootd_transfer_readv_segments_total`. - `xrootd_transfer_operations_count`: Renamed to `xrootd_transfer_operations_total`. - `xrootd_server_connection_count`: Renamed to `xrootd_server_connections_total`. - `xrootd_server_bytes`: Renamed to `xrootd_server_bytes_total`. - `xrootd_server_io_wait_time`: Renamed to `xrootd_server_io_wait_seconds_total`. --- Source: https://docs.pelicanplatform.org/monitoring-pelican-services/grafana # Visualize Pelican Server Metrics with Grafana > Pelican added official support for Grafana in `v7.6.0`. Older versions of Pelican are not guaranteed to work with Grafana. Pelican servers have [Prometheus](./prometheus) embedded. In the Web UI of the origin server, Pelican has a visualization of data transfer metrics collected. There are many other metrics that Pelican collects but are hidden from UI. You may unleash the potential of the hidden metrics by building a customized Pelican server monitoring dashboard. To do this, we recommend using [Grafana](https://grafana.com/) , an open-source analytics and interactive visualization software. This documentation will get you started on connecting Grafana to Pelican. Before starting, make sure you have set up and run a Pelican server. For how to serve a Pelican server (origin, cache, registry, or director), please refer to previous sections. If you already have Grafana installed, skip to [Connect Grafana to Pelican](#connect-grafana-to-pelican). ## Set up Grafana Grafana can be accessed via Grafana Cloud or as a self-managed software. This documentation will focus on running Grafana locally. If you are interested in using Grafana Cloud, visit this [link](https://grafana.com/get/?pg=graf&plcmt=hero-btn-1)  for details. This section briefly introduces downloading, installing, and starting Grafana on a Linux system. For detailed instructions, refer to [Grafana Documentation](https://grafana.com/docs/grafana/latest/) . ### Install Grafana Follow the [link](https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1)  to download Grafana based on your operating system then follow [Grafana documentation](https://grafana.com/docs/grafana/latest/)  to install Grafana. This documentation uses Grafana Enterprise edition with version `10.3.3`. ### Start Grafana On a Linux system, you can start Grafana with systemd. For starting Grafana on other operating systems, refer to [Start Restart Grafana](https://grafana.com/docs/grafana/latest/setup-grafana/start-restart-grafana/) . 1. To start the service, run the following commands: ``` sudo systemctl daemon-reload && \ sudo systemctl start grafana-server ``` 2. To verify that the service is running, run the following command: ``` sudo systemctl status grafana-server ``` ### Initialize Grafana To sign in to Grafana for the first time, follow these steps: 1. Open your web browser and go to the root URL. Unless you have configured Grafana differently, it is set to use [http://localhost:3000](http://localhost:3000)  by default. 2. On the sign-in page, enter `admin` for username and password. 3. Click Sign in. 4. If successful, you will see a prompt to change the password. 5. Click OK on the prompt and change your password. > **Note**: We strongly recommend that you change the default administrator password. ## Connect Grafana to Pelican This section describes how to connect Grafana to the Pelican Prometheus instance. Grafana has built-in support for many data sources and you will need to add a new Prometheus data source first before configuring it for Pelican. [What are Grafana data sources?](https://grafana.com/docs/grafana/latest/datasources/)  ### Before starting 1. If you haven’t installed Pelican, follow the instructions to [install Pelican](../install). 2. Run your Pelican Origin server. You may run other Pelican servers, but in this documentation we will use the Origin server for demonstration. For details about serving a Pelican Origin, refer to [Federating Your Data](../federating-your-data) page. ``` pelican-server origin serve -v /tmp/stash:/test-namespace -f "osg-htc.org" ``` 3. Go to the URL for Web UI and confirm it’s working. The default URL is `https://localhost:8444`. You should be directed to the Pelican login page. ### Add a data source 1. Click **Connections** in the left-side menu. If there is no menu on the left-side, click the icon of stacked bars to toggle the menu. This is located on the top-left side of the page next to **Home** ![Screenshot of Grafana page to locate the menu](/pelican/optimized_images/grafana-menu-location-opt-1920.WEBP) 2. Enter **Prometheus** in the search dialog ![Screenshot of Grafana page to search for Prometheus](/pelican/optimized_images/grafana-search-prometheus-opt-1920.WEBP) 3. Click Prometheus and click **Add new data source** ![Screenshot of Grafana page to add Prometheus data source](/pelican/optimized_images/grafana-add-data-source-opt-1920.WEBP) 4. Configure the data source following instructions in the next section ### Configure the data source Follow the instructions below to configure your Prometheus data source to work with Pelican. Each subsection is equivalent to the subsection on the Grafana data source configuration page. #### Connection - **Name**: Change the name to `Pelican Prometheus`, or any name to distinguish it from other data sources #### Authentication - **Prometheus server URL**: `https://:/api/v1.0/prometheus` Where `:` is your Pelican server Web URL that Grafana can access. By default, it’s [https://localhost:8444](https://localhost:8444)  - **TLS settings**: Check “Skip TLS certificate validation” if you don’t have a TLS certificate for your _Pelican_ server, or if you are in local development environment ![Screenshot of Grafana page to configure the data source](/pelican/optimized_images/grafana-data-source-configuration-opt-1920.WEBP) #### Advanced settings - **Other**: Select `GET` for “HTTP Method” Before finishing the configuration, you need to set up authentication credentials for Grafana to access Prometheus instance on Pelican or change Pelican configuration to disable authentication. Follow the next section for instructions. ![Screenshot of Grafana page to configure advanced settings of the data source](/pelican/optimized_images/grafana-data-source-configuration-advanced-opt-1920.WEBP) ## Configure authentication This section describes how to configure Grafana to have proper authentication credentials to access a Prometheus instance on Pelican or how to turn off Pelican authentication for Prometheus. Depending on your deployment, you may prefer one way over the other. ### Authenticated Prometheus By default, Pelican protects the Prometheus endpoint by asking for a [JWT](https://jwt.io/introduction)  to verify the user accessing the endpoint. It reads `Bearer ` from `Authorization` header during any HTTPS request to `/api/v1.0/prometheus` and its subpath. You need to create a JWT using the Pelican CLI and pass it to Grafana. 1. Run the following command - Replace `` with the URL to your Pelican server Web UI. By default, it’s `https://localhost:8444`. - Replace `--lifetime 604800` with your desired lifetime for the token. The value is in seconds. The example below generates a token with a lifetime of 7 days. ``` $ pelican-server origin token create --scope monitoring.query --claim wlcg.ver=1.0 --issuer --subject grafana --audience --lifetime 604800 eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJodHRwczovL2V4YW1wbGUuY29tIiwiaWF0IjoxNzA5NjUwMjMyLCJleHAiOjE3NDExODYyMzIsImF1ZCI6Imh0dHBzOi8vZXhhbXBsZS5jb20iLCJzdWIiOiJncmFmYW5hIiwic2NvcGUiOiJtb25pdG9yaW5nLnF1ZXJ5Iiwid2xjZy52ZXIiOiIxLjAifQ.cK5QyYJinn3QCPa6YB10Wn37MEEKzt_kyDMR9meN1Ec ``` 2. Copy the generated token. Note that your token will be different from the example above. 3. In the Grafana data source configuration page, go to the **Authentication** section, expand the **HTTP Headers** subsection, and click **Add header** 4. Fill in `Authorization` for the **Header** field and `Bearer ` for the **Value** field with your generated token. Note that there is a whitespace between `Bearer` and your token. That’s it! Now you can scroll to the bottom of the configuration page and click **Save & Test**. There should be green alert with a title **Successfully queried the Prometheus API.** ### Unauthenticated Prometheus If for any reason it’s not possible to attach an authorization token for Grafana, Pelican also allows you to turn off Prometheus authentication. You can toggle the configuration value `Monitoring.PromQLAuthorization` and set it to `false` to allow unauthenticated access to Prometheus. You can either set the value by the configuration file or by the environment variable. #### Set through configuration file The Pelican configuration file is located at `/etc/pelican/pelican.yaml` for root users, or `~/.config/pelican/pelican.yaml` for non-root users Append the following lines to your `pelican.yaml` file: pelican.yaml ``` # Other configurations ... Monitoring: PromQLAuthorization: false ``` > Note that you need to restart the Pelican server to apply the changes. #### Set through environment variable The environment variable for the configuration is `PELICAN_MONITORING_PROMQLAUTHORIZATION`. You can either `export` this environment variable or append it before your command to run Pelican. ``` export PELICAN_MONITORING_PROMQLAUTHORIZATION=false ``` or ``` PELICAN_MONITORING_PROMQLAUTHORIZATION=false pelican-server origin serve ``` Once you configured the authentication either in Grafana or in Pelican, click **Save & Test** to see if the connection can be established. ## Build Monitoring Dashboard With Grafana connected to the Pelican Prometheus instance, you are able to build visualization with the metrics Pelican collects. This section describes how to get started with a template dashboard configuration for a _Pelican origin server_. For a detailed guide on how to create a Grafana dashboard, refer to the [Grafana documentation](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/create-dashboard/) . ### Before start 1. Make sure your Pelican origin server is running 2. Download the template dashboard file from [here](https://github.com/PelicanPlatform/pelican/blob/main/docs/public/origin-dashboard-template.json)  ### Modify dashboard template There are a couple of placeholders that need to be changed before you can import the dashboard template to Grafana. To do a batch replace, it is recommended that you use a text editor to find and replace placeholders to corresponding values. - Replace `` to the id of your Prometheus data source. You can find the ID on the data source configuration page. 1. Navigate to the configuration page 2. Copy the ID from your browser’s URL box, which should have a value similar to ``` http://localhost:3000/connections/datasources/edit/fc2115e0-ede3-4fdb-9056-ecefa1b3160a ``` Where `fc2115e0-ede3-4fdb-9056-ecefa1b3160a` is the ID you want to copy. Note that the ID in the example above will be different from your data source ID 3. Batch replace `` in the template dashboard file with the copied value - Batch replace `` with the web URL of your Pelican origin server, without the `https://` portion. Example: `localhost:8444` - Save the changes to your template dashboard file. ### Import template file to Grafana 1. Go to the home page of Grafana 2. Click **Dashboards** in the left-side menu. If there is no menu on the left-side, click the icon of stacked bars to toggle the menu, which is located on the top-left side of the page next to **Home** 3. Click the **New** button on the top-right side. Select the **Import** option in the dropdown menu. 4. Drag and drop the template JSON file from your computer to the upload section of the page. It should look like the screenshot below. ![Screenshot of Grafana page to import dashboard](/pelican/optimized_images/import-dashboard-opt-1920.WEBP) 5. Click **Import** Grafana should then redirect you to the page of the import dashboard, which should look like the screenshot below. ![Screenshot of Grafana page to import dashboard](/pelican/optimized_images/grafana-dashboard-origin-opt-1920.WEBP) You may notice that some panels have a `No data` warning. This should go away once you have run your server for a couple of minutes and refreshed the page. It takes about 15s for Pelican to collect the initial data points and there could be delay in visualizing the data. That’s it! Feel free to modify the dashboard to tailor to your needs. For a list of available Prometheus metrics to query, please refer to the [previous section](./prometheus). --- Source: https://docs.pelicanplatform.org/advanced-concepts/server # Managing Administrator Credentials Pelican services maintain multiple types of credentials: - The _admin password_ provides access to the server’s web interface. - The _namespace private key_ can create arbitrary new credentials (user tokens) for a portion of the federation’s namespace. This page covers various topics in managing these credentials. ## Pre-generate Credentials ### Namespace Private Key (Issuer Key) > **Note**: The namespace private key is a **powerful secret** and should never be shared with others. With the private key, one can create user credentials that can **arbitrarily read or write into your namespace**. Pelican services rely on a private/public key pair for creating (signing) and verifying the user credentials (tokens) used to read or write objects. When a new prefix in the data federation is created, an associated public key must be provided; the corresponding private key can generate any arbitrary permissions (including delete!) within the namespace. For more information about namespace registration, see details [here](../federating-your-data#obtain-the-public-key-of-the-origin). When a Pelican origin or other service is started, the server automatically generates a new private key if one is not present. However, if someone is running the origin on your behalf, you will need to generate the keypair on your own and send the public portion to the origin operator. If you’re doing the deployment, it can also be convenient to generate the key-pair in advance and mount the private key file to the Pelican server via configuration variable, especially in a container environment. To generate a private/public key pair, run the following command: ``` pelican-server key create ``` Resulting in public and private keys generated under the current working directory: ``` Successfully generated keys at: Private key: /path/to/private-key.pem Public Key: /path/to/issuer-pub.jwks ``` You can change the location of the generated keys by passing in arguments `private-key` and/or `public-key`: ``` pelican-server key create --private-key /tmp/foo --public-key /tmp/bar ``` You can also use this command to derive the public key of an existing private key: ``` pelican-server key create --private-key /path/to/existing/private/key ``` To configure a Pelican server to use the newly-generated _private key_, move it into the directory specified by the `IssuerKeysDirectory` parameter in the configuration `yaml` file: pelican.yaml ``` IssuerKeysDirectory: /path/to/issuer-keys ``` This directory supports multiple private keys. A token or payload signature will be considered valid if it can be verified by any key found either within `IssuerKeysDirectory` or at `IssuerKey`. If there is a remote operator for your origin, email them the **public key** portion. Never send anyone your private key. A sample public key looks like the following: ``` $ cat issuer-pub.jwks { "keys": [ { "alg": "ES256", "crv": "P-256", "kid": "N3Tks9H791dG7bdzjjlQzV-fwwCrnXd_mcd_2D5ZvYE", "kty": "EC", "x": "9IIkGEzyWywecBsIisrJsH-_XqMal-U25dthHewUC5w", "y": "h_1DUotM2qnLPPRia6hGXNKZdpi92VJ5u_NSrZ-pd0c" } ] } ``` A sample private key has the following PEM format: ``` $ cat private-key.pem -----BEGIN PRIVATE KEY----- M0G+AgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgzeomq6yJvek0fg2Z g+NY6qklGz4ZjVlrRgTDT9hk5EShRANCAAT0giQYTPJbLB5wGwiKysmwf79eoxqX 5Tbl22Ed7BQLnIf9Q1KLTNqpyzz0YmuoRlzSmXaYvdlSebvzUq2fqXdH -----END PRIVATE KEY----- ``` ### Web Interface Administrator Password By default, Pelican’s administrator web interface requires a username/password to be setup to access. This password can be enabled by a one-time passcode the server writes into the log file. However, by pre-generating the credential, Pelican server administrators can skip the initial setup and automate installation. The `pelican-server generate` command generates the [htpasswd](https://httpd.apache.org/docs/2.4/programs/htpasswd.html) \-formatted password file for storing the web interface administrator login password. To generate a password file, run: ``` pelican-server generate password ``` and Pelican will prompt you to enter and confirm the password: ``` Enter your password: Confirm the password: Successfully generated the admin password file at: /path/to/server-web-passwd ``` You can change the location of the generated htpasswd file by the `-o` or `--output` argument. You can also provide the file that contains your password through the `-p` or `--password` argument. ``` pelican-server generate password -o /path/to/generated-htpasswd-file -p /path/to/password ``` The generated htpasswd file can then be passed to the Pelican server via `Server.UIPasswordFile` configuration parameter. pelican.yaml ``` Server: UIPasswordFile: /path/to/generated-htpasswd-file ``` ## Key Auto-Generation, Multiple Keys, and Key Replacement ### Key Auto-Generation Starting with Pelican v7.13.0, if no pre-generated private key is found at either `IssuerKey` or within `IssuerKeysDirectory`, Pelican will automatically generate a new private key file in PEM format inside `IssuerKeysDirectory`. ``` IssuerKeysDirectory/ ├── pelican_generated__.pem └── ... ``` > The [Privacy-Enhanced Mail (PEM) format](https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail)  is a standardized format for sending and storing cryptographic data. ### Multiple Keys Since v7.13.0, Pelican has supported multiple issuer keys. Server administrator can place additional `.pem` file into `IssuerKeysDirectory`, and any valid private key in this directory or at `IssuerKey` can be used to verify token or payload signatures. When selecting a key for signing, Pelican automatically chooses the private key with the lexicographically smallest filename. ``` ConfigBase ├──IssuerKey └──IssuerKeysDirectory/ ├── pelican_generated__.pem ├── .pem └── ... ``` Pelican scans the above directory every 60 seconds to detect newly added or removed private keys. When [retrieving public keys](/federating-your-data/origin#obtain-the-public-key-of-the-origin), Pelican dynamically generate them from the current set of private keys on-the-fly. If `Server.IssuerJwks` is specified, Pelican also includes public keys from that `.jwks` file. ### Key Replacement With support for Multiple Keys, Pelican allows the administrator to replace private keys at anytime. Follow these steps to replace an existing key: 1. Generate a new private/public keypair by using the `pelican-server key create` command, see details [here](#namespace-private-key). 2. Ensure that the Pelican server is running. Copy the new private key file to [IssuerKeysDirectory](#multiple-keys). 3. Visit your federation’s Registry in your browser (you can find it in your Pelican server’s web interface) and download the latest registered public key set for your server under “Origins” section. Ensure the new public key is included. If it isn’t, wait for 60 seconds and retry. 4. Remove the old private key from [IssuerKeysDirectory](#multiple-keys) 5. Wait for 60s, then go back to the Registry, download the latest registered public key set again. Verify the old public key has been removed. > If you are using a [Credmon](https://htcondor.readthedocs.io/en/main/admin-manual/file-and-cred-transfer.html#enabling-the-fetching-and-use-of-credentials)  in HTCondor, increase the wait time in the above steps to the credmon’s configured token lifetime (15 minutes by default). ## PKCS#11 Enhanced TLS Security By default, the XRootD subprocess requires read access to the TLS private key file. This introduces a potential security risk: if an attacker could exploit XRootD to read arbitrary files (for example, through a symlink attack), they could potentially extract the private key. Starting with Pelican v7.23.0, Pelican can use [PKCS#11](https://en.wikipedia.org/wiki/PKCS_11)  to protect the TLS private key used by the XRootD subprocess. When enabled, the TLS private key never leaves the Pelican process memory. XRootD performs TLS signing operations via a PKCS#11 interface without directly reading the key file. ### How It Works When PKCS#11 is enabled, Pelican starts an internal `p11proxy` server that implements the PKCS#11 protocol over a Unix socket. The architecture works as follows: 1. **Pelican** starts a PKCS#11 RPC server (via `go-p11-kit`) that holds the TLS private key in memory 2. **XRootD** is configured to use a PKCS#11 URI instead of a direct key file path 3. When XRootD needs to perform TLS signing, **OpenSSL** calls out to the PKCS#11 provider/engine 4. The **p11-kit-client** library forwards the signing request over a Unix socket to Pelican 5. **Pelican** signs the request and returns only the signature—never the key itself ### Prerequisites Before enabling PKCS#11, you must install the required system packages. The packages differ based on your OpenSSL version: If you are using the official Pelican Origin container image, these OpenSSL 3.x packages are already included. **For OpenSSL 3.x** (e.g., AlmaLinux 9, Fedora 37+, Ubuntu 22.04+): ``` dnf install -y pkcs11-provider p11-kit-server ``` **For OpenSSL 1.x** (e.g., CentOS 7, RHEL 8): ``` dnf install -y openssl-pkcs11 p11-kit-server ``` On Debian/Ubuntu systems, use `apt` instead: ``` # OpenSSL 3.x apt install -y pkcs11-provider p11-kit # OpenSSL 1.x apt install -y libengine-pkcs11-openssl p11-kit ``` > **Note**: Pelican automatically detects your OpenSSL version and uses the appropriate mode (Provider API for OpenSSL 3.x, ENGINE API for OpenSSL 1.x). ### Enabling PKCS#11 To enable PKCS#11 TLS key protection, add the following to your `pelican.yaml`: pelican.yaml ``` Server: EnablePKCS11: true ``` Once enabled, Pelican will: 1. Start the internal p11proxy server 2. Generate a PKCS#11-compatible OpenSSL configuration 3. Configure XRootD to use the PKCS#11 URI for TLS operations ### Verifying PKCS#11 is Active When PKCS#11 is enabled and working correctly, you will see log messages similar to: ``` p11proxy: accepted connection ... ``` You can also run Pelican with verbose p11-kit debugging to see detailed RPC activity: ``` P11_KIT_DEBUG=rpc pelican-server origin serve ``` This will produce detailed output showing the PKCS#11 communication: ``` (p11-kit:PID) rpc_unix_init: initialized rpc socket: ... (p11-kit:PID) rpc_C_GetSlotList: C_GetSlotList: enter ... ``` ### Restricting Key File Permissions With PKCS#11 enabled, you can further harden your deployment by making the TLS key unreadable to the XRootD process: ``` chmod 600 /etc/pelican/certificates/tls.key chown pelican:pelican /etc/pelican/certificates/tls.key ``` ### Troubleshooting If PKCS#11 fails to initialize, Pelican will log a warning and fall back to the traditional method where XRootD reads the key file directly. Common issues include: | Issue | Solution | | --- | --- | | Missing packages | Install the required packages for your OpenSSL version (see [Prerequisites](#prerequisites)) | | Socket permission errors | Ensure the XRootD user can access the p11-kit socket directory | | OpenSSL version mismatch | Pelican auto-detects the version; ensure correct packages are installed | > **Note**: If you explicitly set `Server.EnablePKCS11: true` but the required dependencies are missing, Pelican will log a warning. If the parameter is not set (defaulting to `false`), Pelican will silently skip PKCS#11 setup. --- Source: https://docs.pelicanplatform.org/advanced-concepts/plugin # HTCondor Plugin ## Configure the Plugin in an HTCondor Job [HTCondor](https://htcondor.readthedocs.io/en/latest/)  has tight integration with Pelican for managing data transfers as part of running jobs on a High Throughput Computing (HTC) system. The user simply lists the Pelican address for the object(s) they want transferred as part of the HTCondor job, and HTCondor is responsible for transferring the object(s) at the appropriate points in the job lifecycle. For example, including the following line within the HTCondor job submit file will cause HTCondor to transfer the corresponding object to the job’s scratch directory before execution: ``` transfer_input_files = pelican://osg-htc.org/pelicanplatform/test/hello-world.txt ``` For more information about data transfers in HTCondor, see the HTCondor manual page about its [File Transfer Mechanism](https://htcondor.readthedocs.io/en/latest/users-manual/file-transfer.html) , especially the section on [File Transfer Using a URL](https://htcondor.readthedocs.io/en/latest/users-manual/file-transfer.html#file-transfer-using-a-url) . Pelican object transfers as part of an HTCondor job occur at the Execution Point (EP) - the machine where the job is executing. The transfers use the Pelican Plugin at the EP as configured by the EP owner. Currently, there is no way to pass additional arguments to the `pelican object` command-equivalent used at the EP. There is, however, a way to modify the environment variables that are set prior to the object transfer command. That means as long as the `pelican object` command respects the specific environment variable(s), the user submitting the HTCondor job can modify the Plugin’s transfers as desired. ### Pelican environment variables Configuration options listed in the [configuration parameters table](/parameters) are typically set via the `pelican.yaml` file. Alternatively, the configuration options can be set using environment variables. We recommend reading the relevant section of the configuration page before proceeding: [Environment Variable Configuration](/parameters#environment-variable-configuration). In the following section, we’ll leverage this functionality to pass configuration options to affect the file transfers involving the HTCondor Pelican Plugin. ### Setting environment variables for the Plugin as part of HTCondor job To set a Pelican environment variable to affect the Plugin transfers in an HTCondor job, declare the environment variables using the `environment` option in the submit file. ``` environment = "PELICAN_OPTION1=value1 PELICAN_OPTION2=value2" ``` > Note that there are a variety of subtleties in how the `environment` value is interpreted; it is recommended that you read the corresponding section in the [HTCondor manual](https://htcondor.readthedocs.io/en/latest/man-pages/condor_submit.html#environment)  for all but the simplest definitions. For example, to configure the Plugin to transfer objects only via a specific cache, you would use the following line: ``` environment = "PELICAN_CLIENT_PREFERREDCACHES=https://address.to.desired.cache:port" ``` > **Note:** Plugins older than v7.16.0 should use the `PELICAN_NEAREST_CACHE` environment variable, as `PELICAN_CLIENT_PREFERREDCACHES` was introduced in v7.16.0 with the intention of deprecating `PELICAN_NEAREST_CACHE`. > This particular example would be useful for testing that a specific cache is operational and accessible from the EP the job is running at. Use of `PELICAN_NEAREST_CACHE`, though, bypasses the Director and its logic that helps prevent any one Pelican component in the Federation from being overwhelmed. > > For production, it would be better to use the `PELICAN_PREFERRED_CACHE` configuration. The cache specified using this option is always tried first, but if there is an issue the Plugin will fall back to the Director for a list of other caches to try. A couple of things to note about this methodology for configuring the Plugin: - The configuration is overridden for _all_ transfers involving the Plugin within the job. This could be especially problematic if transferring objects via different federations within the same job. - The configuration for the Plugin set by the EP owner is likely optimized for that EP. Overriding that configuration could result in performance or network issues. - Environment variables set using the `environment` option will persist into the job execution environment. If additional Pelican transfers are run within the job’s executable script, the environment variables will affect those transfers unless unset by the executable script. --- Source: https://docs.pelicanplatform.org/advanced-concepts/auth # Pelican’s Authorization System At its core, the goal of Pelican’s authorization system is simple: give the right people access to the right data, and protect the data from everyone else. Whether data is private, public, or shared with a specific collaboration, Pelican ensures that only authorized users can access it. To achieve this in a distributed environment without sacrificing performance, Pelican relies on a modern, token-based architecture that separates the responsibility of _verifying identity_ from the responsibility of _granting access_. ## Authentication vs Authorization (AuthZ vs AuthN) While often used interchangeably in casual conversation, **Authentication** and **Authorization** (often abbreviated **AuthN** and **AuthZ**) are distinct concepts in computer security. - **Authentication (AuthN)** is the process of verifying _who you are_. It’s used to confirm your identity and typically involves providing some kind of secret or unique thing that _only_ you possess. Common authentication methods include passwords, certificates, or federated identity providers like [CILogon](https://www.cilogon.org/home) . **Example:** - Showing your passport to airport security proves you are the person named on your ticket — only you possess the face shown in the passport. - _In Computing:_ Logging in with a username and password, or using multi-factor authentication — only you possess the password and phone number associated with the account. - **Authorization (Authz)** answers the question _what you are allowed to do_. It determines your permissions and is typically managed through policies, roles, or access tokens. **Example:** - Your boarding pass allows you to board a specific flight and sit in a specific seat, but it doesn’t let you sit in someone else’s seat, fly the plane or enter the cockpit. - _In Computing:_ Having read-only access to a file, or administrative privileges on a server. Part of what makes these concepts confusing is that in order to determine _what_ you’re allowed to do, we usually first have to determine _who_ you are; before granting authorization, you have to be authenticated. This is true, but Pelican draws a clear line between the two by leaving it up to external **identity providers** to handle the “who” (Authentication), while Pelican focuses exclusively on the “what” (Authorization). Pelican does not manage usernames, passwords, or user accounts. Instead, it relies on the **[OpenID Connect (OIDC)](https://openid.net/developers/how-connect-works/)**  standard to integrate with trusted third-party identity providers like CILogon, university logins, or Google. ### The Challenge of Federated Identity In a distributed environment like the OSDF, **Federated Authentication** is notoriously difficult. It requires that every service in the federation trusts and understands the identity providers of every user. If a user from University A wants to access data at Laboratory B, Laboratory B must be able to verify University A’s credentials directly. Scaling this “mesh of trust” across hundreds of institutions is complex and fragile. Pelican addresses this by relying on **Federated Authorization**. Instead of passing user identities around, we pass **tokens with capabilities**. When a user authenticates with their home institution, that institution’s identity provider gives information about _who_ the user is to an _authorization server_. The [authorization server](#issuers) then decides _what_ the user is allowed to do and creates a token explaining those permissions. Importantly, this token doesn’t say “This is User X”; it says “The bearer of this token is allowed to read /data/project-y”. The services in the federation (Origins and Caches) do not need to know _who_ the user is; they only need to trust the authorization server identified by the `iss` claim in the token and verify that the token grants the necessary permissions. This decouples the identity verification from resource access, allowing Pelican to protect data without ever needing to store or manage sensitive user credentials. ## Tokens In the context of Pelican and modern web security, an **Authorization Token** is a portable, digital credential that grants access to specific resources. While tokens _can_ be used for authentication, Pelican uses them exclusively for authorization to specify what the token holder is allowed to do. Sometimes these tokens are also referred to as _Bearer Tokens_ because we don’t make any assertions about _who_ is presenting the tokens, only about _what_ the token says the bearer/presenter is allowed to do. This is a sure sign that we’re working with an authorization framework, _not_ an authentication framework. Since there is a distinction between authentication tokens and authorization tokens, whenever these docs refer simply to “tokens”, we’re talking about “authorization” tokens. A good analogy to understand how tokens work is to think of them like concert tickets: ### Concert Tickets ### Tokens **1.** You go to the ticket booth and prove you’re the one who paid **1.** You go to an identity provider and log into your account to prove who you are **2.** The receptionist at the ticket booth double checks which seat you paid for and then _prints/issues_ you a ticket that says “whoever holds this ticket can enter the venue today between 7-8pm and is allowed to sit in seat D12”. The ticket has a special hologram on it that _only_ this printer can create **2.** The identity provider hands various _identifiers_ to an **authorization server** that uses those identifiers—combined with its own policy—to decide what you’re allowed to do. It then creates a token that says “whoever presents this token is allowed to read and write anywhere under the `/foo` namespace for the next 20 minutes”. The token is cryptographically signed with a private key belonging to the authorization server (or a service it has delegated signing to) **3.** At 7pm, you present the ticket to a security guard who compares its hologram against an example on his clipboard. The guard confirms the hologram is authentic because he **trusts** that whoever handed him the clipboard works for the same company. **3.** Within the 20m timeframe, you present the token to an Origin along with a request to download an object called `/foo/picture.jpg`. The Origin fetches the issuer’s public key (from online or a local cache) and verifies the token’s signature is authentic because it **trusts** that the issuer is allowed to speak on behalf of the requested resources **4.** You enter the venue and are escorted by an usher to your seat **4.** Finally, the Origin let’s you proceed with the download ![Tokens as tickets analogy](/pelican/advanced-concepts/optimized_images/tickets-tokens-opt-1920.WEBP) Importantly, if you tried to use a ticket for a different band or you scribbled all over the token (tampered with it), you’d be denied entry by the security guard. These same principles apply to tokens, which is why they work well for guarding protected resources. While tokens cannot be revoked the way certificates can be, their short lifetimes minimize the potential for leaking data by building in an automatic expiration. Ultimately, the security guarantees of tokens come from _public/private key cryptography_, which lets the **authorization server** _sign_ tokens using a private key while the corresponding public key is openly available for anyone to _verify_ the signature on the token. In practice, the service that _signs_ the token and the service that hosts the public keys for _verification_ are often split. For more information about this, see the section on [“Issuers”](#issuers). ### JSON Web Tokens (JWTs) So far we’ve discussed what role tokens play and how they work at a high level, but we haven’t yet discussed what a real token looks like. Pelican uses a specific type of token called a **JSON Web Token (JWT)**. A JWT is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. JWTs consist of three parts: 1. **Header:** Describes _how_ the token was signed (e.g., “We used the RS256 algorithm”). 2. **Payload:** Contains the “Claims” — the actual data/permissions. This includes: - `iss` (Issuer): An identifier (URL) for the authorization server that issued this token - `aud` (Audience): Who do we expect this token to be presented to? - `exp` (Expiration): When does this token expire? - `iat` (Issued At): When was this token created? - `nbf` (Not Before): When does the lifetime of this token start? - `scope` (Scope): Which actions can be performed on which resources, i.e. what is this token allowed to do? - `jti` (JWT Identifier): A unique identifier that is specific to this token. - Other fields can be defined according to specific token _profiles_ (see [“Token Profiles”](#token-profiles) for more information) 3. **Signature:** The cryptographic proof that ensures the token was created by the right issuer and hasn’t been altered. Signatures do not decode to JSON **Example:** Below are JSON objects that represent an example JWT header and payload: ``` { "alg": "RS256", "kid": "key-rs256", "typ": "JWT" } { "aud": "https://demo.scitokens.org", "iss": "https://demo.scitokens.org", "exp": 1764179999, "iat": 1764179399, "nbf": 1764179399, "jti": "f84a5cb6-8e1b-46b4-a205-0c44503a3d27", "scope": "read:/foo", "ver": "scitoken:2.0" } ``` In order to pass these tokens around, each part of the token is base64-encoded and strung together using periods (`.`), i.e. `Header.Payload.Signature`. **Example:** The previous token JSON (along with a made up signature) would be encoded as: ``` eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS1yczI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJodHRwczovL2RlbW8uc2NpdG9rZW5zLm9yZyIsImlzcyI6Imh0dHBzOi8vZGVtby5zY2l0b2tlbnMub3JnIiwiZXhwIjoxNzY0MTgwNjI2LCJpYXQiOjE3NjQxODAwMjYsIm5iZiI6MTc2NDE4MDAyNiwianRpIjoiZjg0YTVjYjYtOGUxYi00NmI0LWEyMDUtMGM0NDUwM2EzZDI3Iiwic2NvcGUiOiJyZWFkOi9mb28iLCJ2ZXIiOiJzY2l0b2tlbjoyLjAifQ.RmtEjRW63s2v2HuhulWhB0FHltza9eyUVDKhYs-jGHQYsEL65gIMMHb6IbB1BL9YBzdMCJd4J8n8sPt8rN5YJxC8SJ1nQHjzJOBDXZhhQ52W5Rj5KQQINwISuU6QA39RMZ3hOWexijW9OEcmvCZLpDPqRCyc7xAcW3DgCzWqiopgS3ff-Cj9AR8RFa90V5VEfXYwvyVsmE4sUy3FqrDpyG4OuNkVIpuuF7l7XlqnSC4b2QFFmlbkxw6B4q3s2_mHPIcCMIhwo8mCNuXVaeSCIh6EtJ7eALQpbagWvdqFhL6dGRNFsHPDgfas98nkRfRv4gVP5_Qj7qVcUSp_1std5Q ``` You should be _**extremely careful**_ about sharing token signatures; treat this like sharing a password! You can safely share the first two sections (everything **up to but not after** the second period) of your tokens because they don’t contain secrets. **Encoded vs. Encrypted** It is crucial to understand that the **Header** and **Payload** of a JWT are merely **Base64Url encoded**, not encrypted. This means **anyone who intercepts the token can read the information inside it** (like the namespace paths you are accessing). However, they cannot _change_ that information. Changing the payload would invalidate the **Signature**, causing the token to be rejected by Pelican. #### JWT Tools If you’re working with JWTs, it’s useful to know about a few tools that help you decode them for interpreting what they say. Both the [SciTokens demo site](https://demo.scitokens.org/)  and [jwt.io](https://jwt.io)  are great web-based portals for converting tokens to/from their encoded and decoded forms. In particular, jwt.io is nice because you can hover your mouse over the token’s timestamps to see them in human-readable form. Both websites operate purely in your browser, so any tokens you input won’t be sent anywhere — this lets you use the sites without worrying about anyone stealing your secrets! In general, most of the JWTs you’ll need to create to work with Pelican can be created using [Pelican’s command line token tools](../../getting-data-with-pelican/auth). However, other command line libraries like the ones provided at [https://demo.scitokens.org/](https://demo.scitokens.org/)  and the [htgettoken](https://github.com/fermitools/htgettoken?tab=readme-ov-file)  tool can be used to create/decode generic JWTs. ### Token Profiles JWT is a generic framework for packing information into a string that’s easy to pass between web services. While some of the core JSON keys like `iss`, `exp`, etc. are present in every JWT, some people have extended the keys/semantics of JWTs to communicate specific information relevant in their ecosystems. Specifications that describe the extra contents of a JWT are called _profiles_, and they typically describe which “scopes” are recognized and which other fields are mandatory. The two profiles used in the Pelican ecosystem are the **WLCG Profile** and the **SciTokens** profile. #### **WLCG Profile** The [Worldwide LHC Computing Grid](https://wlcg.web.cern.ch/)  (WLCG) auth group maintains the WLCG token profile. Some of the WLCG scopes/capabilities Pelican uses include: - `storage.read:/path/to/resource`: grants the ability to _read_ the specified resource - `storage.create:/path/to/resource`: grants the ability to _create_ but **not** modify the specified resource - `storage.modify:/path/to/resource`: a superset of `storage.create`, grants the ability to _modify and delete_ the specified resource See [WLCG’s Token Profile documentation](https://github.com/WLCG-AuthZ-WG/common-jwt-profile/blob/master/profile.md)  for more information about other WLCG token requirements and available capabilities/scopes. #### **SciTokens Profile** The SciTokens Profile is another option Pelican understands how to work with, although Pelican prefers the WLCG profile. Some of the SciTokens scopes/capabilities Pelican uses include: - `read:/path/to/resource`: grants the ability to _read_ the specified resource - `write:/path/to/resource`: grants the ability to create/modify the specified resource See the [SciTokens site](https://scitokens.org/)  for generic documentation and the [SciTokens Claims specification](https://scitokens.org/technical_docs/Claims)  for more information about other SciTokens requirements and available capabilities/scopes. Be careful using tokens that let you modify object contents after the object has been written — Pelican objects should be treated as immutable, so these tokens can get you in trouble! ## Issuers In the Pelican and OAuth2 ecosystems, people often refer to **issuers**, and it can be difficult to separate the concept of issuers from **authorization servers**. Put simply, the term “issuer” refers to an identifier (i.e. a URL) for an authorization server that is used to indicate where the authorization server’s public keys can be obtained. It is important to understand that an issuer is just a string: it doesn’t perform actions, make decisions, or create tokens on its own. The entity that actually does things is the **authorization server**. It is the service responsible for creating and cryptographically signing tokens. When an Origin or Cache receives a token, it reads the `iss` claim as a key to look up the corresponding authorization server’s public keys and verify the token’s signature. Depending on what the authorization server advertises in its metadata, Pelican may also be able to use it to _obtain_ tokens (via standard OAuth2 flows) on behalf of a user. ### OIDC Metadata Discovery The issuer URL serves as a _lookup key_ for finding public keys — it is **not necessarily the address of the service that signed the token**. The signing may have been done by a completely different service (see [Delegated Signing](#how-authorization-servers-sign-tokens) below); the issuer URL is simply where consumers go to discover the keys needed to verify that signature. Because Pelican follows the [OpenID Connect (OIDC)](https://openid.net/developers/how-connect-works/)  specification, those public keys can be found by: 1. Appending `/.well-known/openid-configuration` to the issuer URL and fetching the JSON at that address. For example, for the issuer `https://osg-htc.org`, fetch `https://osg-htc.org/.well-known/openid-configuration`. 2. Parsing the resulting metadata document for a `"jwks_uri"` key. Its value is a URL pointing to a JSON Web Key Set (JWKS) containing the public keys trusted for this issuer. 3. Fetching those keys and using them to verify that a given token’s signature is authentic. If the metadata document also contains a `"token_endpoint"`, Pelican can use it to request tokens via an OAuth2 flow. ### How Authorization Servers Sign Tokens There are two common patterns for how an authorization server handles signing: 1. **Co-located signing:** The signing service runs at (or is proxied through) the issuer URL, so OIDC discovery on the issuer URL directly returns the signing keys. This is how Pelican’s built-in authorization works: [OA4MP (OAuth for Many People)](https://oa4mp.org/)  performs the signing, but because it is embedded within and proxied through the Pelican Origin process, the `iss` claim is simply set to the Pelican Origin’s own URL (e.g. `https://my-origin.com:8443`). Token consumers run OIDC discovery on that URL and find the keys without any indirection. 2. **Decoupled issuer identity:** The signing service runs at a different address than the `iss` URL. The `iss` claim is set to some other URL — for example, a lab or project URL — and the party controlling that URL is responsible for publishing the signer’s public keys there via OIDC discovery. Token consumers look up keys at the `iss` URL without needing to know where the signing actually occurred or which service performed it. Services such as HTCondor’s [CredMon](https://github.com/htcondor/scitokens-credmon)  commonly use this pattern. The Origin must be configured to trust an authorization server for each exported namespace by providing the issuer URL — the URL where OIDC key discovery will be performed, which in the decoupled case may be entirely different from the address of the service that performed the signing. **Issuers are not Identity Providers.** An identity provider handles _authentication_, i.e. verifying who a user is. It then passes identifiers (like group membership) to an authorization server, which handles _authorization_: deciding what the user is allowed to do and, if appropriate, creating a signed token. The token an authorization server creates is the intersection of what it is configured to grant and what the user has requested. ### Trust Relationships At the end of the day, the user provides a token to an Origin or Cache along with a request to perform an action on a resource (`read /foo`). But it’s the Origin/Cache that has to decide whether it will fulfill the request and let the user do what it wants. Because this process involves stringing together multiple services, it’s worth taking a moment to analyze how trust is bootstrapped between each of them. 1. **The Storage Provider Trusts the Origin and the Identity Provider:** The entire chain of trust starts with the owner of some underlying storage trusting the Origin to enforce whatever access policies it provides. Crucially, because the Storage Provider defines the policies that map a user’s identity to their permissions (e.g. “Bob can read /foo”), they must fundamentally trust the Identity Provider to accurately verify that identity. If the Identity Provider cannot be trusted to say “This is Bob”, nobody can safely grant “Bob” access. 2. **The Authorization Server Trusts the Identity Provider:** When users authenticate with the Identity Provider, the authorization server trusts the identifiers it is handed. This trust is typically established when the authorization server is registered as a client with the Identity Provider. The authorization server trusts that the Identity Provider has rigorously verified the user’s credentials (password, MFA) before asserting their identity and attributes. 3. **Origins Trust their Authorization Servers and Federation Central Services (Director/Registry):** The Origin is explicitly configured with which authorization servers it trusts for which namespaces. This is done by providing an issuer URL for each namespace in the Origin’s exports; Pelican uses OIDC discovery on these URLs to fetch the public keys needed to verify tokens. The Origin trusts that if a token bears the signature of a trusted authorization server, the permissions inside that token are valid. Furthermore, Origins send information about their namespaces and their issuers to the Federation’s Director, which the Origin also trusts. 4. **The Director/Registry Can Verify Origins/Caches:** When Origins and Caches join a federation, they register their identities with the Registry and provide a public key corresponding to a private key they possess. Origins/Caches advertise who they are and what they do to the Director service, which uses the Registry to verify their identities. The Director can hand out the information in these advertisements, and anyone who’s part of the federation trusts the information because they trust the Director. Because Origins trust Central Services and Central Services trust Caches, Origins transitively trust that Caches will respect their access policies. 5. **Caches Trust the Director/Registry:** Because Caches can grant access to copies of namespaced data, they must know which authorization servers are trusted for which namespaces. Each Origin/namespace and their issuers are advertised to the Director by the Origin. Because both the Cache and the Origin trust the Director, the Cache trusts each namespace authorization server it learns about via the Director. By chaining these trust relationships together transitively, we create a system where a Cache can serve an object to a user without ever knowing who the user is or directly contacting the user’s home institution. ![Bootstrapping Trust between Pelican Services](/pelican/advanced-concepts/optimized_images/trust-diagram-opt-1920.WEBP) ## Anatomy of an Origin’s/Cache’s Authorization Configuration While a Pelican Origin or Cache is typically referred to as a single entity, it’s actually made up of two separate processes. The first is the Pelican process that manages everything else going on in the server, including handling server configuration, interacting with federations, and registering namespaces. The second is an [XRootD](https://xrootd.web.cern.ch/xrootd/)  process that actually hands out data, receives data access tokens and decides whether the token is sufficient to permit the request. This is why Origins and Caches are typically associated with two separate ports; each process binds to its own port. XRootD manages data access authorization using two frameworks, the [“Authorization Database File”](https://xrootd.web.cern.ch/doc/dev56/sec_config.htm#_Toc119617472)  (often abbreviated as “authfile”) and the [XRootD-Scitokens plugin](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens) . Note that SciTokens plugin works with multiple token formats (WLCG, SciTokens). When Origins/Caches start up, they parse the policies provided in their configuration (Origins) or discovered via the Director (Caches) to generate configuration files for both of these frameworks. Because Caches usually serve multiple federation namespaces, a Cache’s authfile/SciTokens configuration is a union over the policies of the federation’s namespaces as presented by Origins. Additionally, these generated configuration files may be merged with admin-supplied files by specifying the [`Xrootd.Authfile`](../parameters#Xrootd-Authfile) and [`Xrootd.ScitokensConfig`](../parameters#Xrootd-ScitokensConfig) config parameters. However, any extra information provided in custom authfile or SciTokens configuration _**does not**_ propagate through the rest of the federation and improper configuration can result in _**unintentional data exposure**_. Use these at your own risk! If you’re setting up an Origin to serve protected data and you can use tokens to download directly via the Origin but not via Caches, double check that you’re not relying on admin-supplied authfile/SciTokens configuration! Whenever an Origin/Cache receives a request, it first consults the SciTokens configuration and then falls back to the authfile if access cannot be granted via the SciTokens plugin. ### SciTokens Configuration File The SciTokens config file’s basic contents include a list of Issuer URLs along with which namespaces (base paths) those Issuers should be used for. **Example:** Below is a sample of a generated Origin SciTokens configuration: ``` # # Copyright (C) 2024, Pelican Project, Morgridge Institute for Research # # Licensed under the Apache License, Version 2.0 (the "License"); you # may not use this file except in compliance with the License. You may # obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # # This is a generated configuration file -- DO NOT HAND EDIT. # It will be overwritten on the next startup of pelican. # [Global] audience_json = ["https://my-origin.com:8443"] [Issuer Origin https://my-origin:8440 and Built-in Monitoring] issuer = https://my-origin:8440 base_path = /my-namespace, /pelican/monitoring [Issuer Federation-based Monitoring] issuer = https://osg-htc.org base_path = /pelican/monitoring default_user = xrootd # End of config ``` For more information about XRootD’s SciTokens plugin configuration, see [https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens](https://github.com/xrootd/xrootd/tree/master/src/XrdSciTokens) . ### Authfile Configuration Each line in an authfile at Origins/Caches maps some kind of identifier to a list of path:privilege pairs. While admin-supplied authfiles can be quite complicated, a Pelican-generated authfile that hasn’t been merged with anything will only use the `l` (list) and `r` (read) privileges for a given path. A `-` is used before each set of privileges to subtract those privileges, while the absence of a `-` means the privileges are granted. Pelican-generated authfiles are never used to grant write privileges — this is always done via the SciTokens configuration. **Example:** Below is a sample of a generated Origin authfile configuration: ``` u * /my-prefix-auth -lr /.well-known lr /my-prefix lr ``` This authfile allows _any_ user to list/read the contents of the `/my-prefix` namespace with `/my-prefix lr`, but prevents _all_ users from listing/reading the contents of `/my-prefix-auth` with `/my-prefix/auth -lr`. The `/.well-known` path for the Origin’s public keys is also exposed via XRootD. It is **very important** to get the ordering of each authfile line and each privilege in a line correct. XRootD’s authorization library parses top to bottom, left to right and uses basic string matching to determine access privileges. The following authline is incorrect because `/my-prefix-auth` will match the policy for `/my-prefix` before the `/my-prefix-auth` policy is checked, accidentally granting privileges: ``` u * /my-prefix lr /my-prefix-auth -lr ``` For more information about XRootD Authfiles, see [https://xrootd.web.cern.ch/doc/dev56/sec\_config.htm#\_Toc119617472](https://xrootd.web.cern.ch/doc/dev56/sec_config.htm#_Toc119617472) --- Source: https://docs.pelicanplatform.org/getting-help # Getting Help For a general overview of the Pelican Platform, check out the main website at [pelicanplatform.org](https://pelicanplatform.org) . ## Documentation If you’re looking for documentation about Pelican, you’re in the right place! The documentation is split into different sections, navigable using the **left-hand side bar**. Search for keywords using the search bar in the **top right corner**. A table of contents for the current page is given in the **right-hand side bar**. We also provide a few specific guides to assist users: - [FAQs](/getting-help/faqs) - answers to frequently asked questions. - [Troubleshooting](/getting-help/troubleshooting) - common errors and how to troubleshoot them. If you have feedback about the documentation, use one of the following links at the bottom of the table of contents (right-hand side): - The link `Question? Give us feedback` will open a new GitHub issue for the current page. - The link `Edit this page on GitHub` will open the GitHub editor for the current page. ## GitHub Pelican Platform software is published on GitHub under the [PelicanPlatform organization](https://github.com/PelicanPlatform) . The majority of the functionality is published in the [PelicanPlatform/pelican repository](https://github.com/PelicanPlatform/pelican) . The software is open source under an [Apache 2.0 license](https://github.com/PelicanPlatform/pelican?tab=Apache-2.0-1-ov-file) . ### Bug reports You can report bugs that you encounter via GitHub. - For a **specific** bug, create a new bug report in the corresponding repository. In most cases, that will typically be the `PelicanPlatform/pelican` repository: [Open Bug Report](https://github.com/PelicanPlatform/pelican/issues/new?template=bug_report.md) . - For **general** issues, or if you’re not sure where to file your bug report, you can start a [new discussion in the PelicanPlatform organization](https://github.com/orgs/PelicanPlatform/discussions/new/choose) . Existing discussions can be found at [github.com/orgs/PelicanPlatform/discussions](https://github.com/orgs/PelicanPlatform/discussions) . Before reporting a bug, we recommend that you 1. Check the [FAQs](/getting-help/faqs) and [Troubleshooting](/getting-help/troubleshooting) pages to see if the behavior is expected. 2. Search the existing issues (typically in [PelicanPlatform/pelican](https://github.com/PelicanPlatform/pelican/issues) ) to see if we are already aware of the problem. ### Feature requests Features can be requested in a similar manner to reporting bugs. - To request a **specific feature**, open a new issue in the corresponding repository. - To request a **general** feature or functionality, or if you aren’t sure where to request one, you can open a [new discussion in the PelicanPlatform organization](https://github.com/orgs/PelicanPlatform/discussions/new/choose) . ## Contact us Generally, the best way to get in touch with the development team is to open a discussion or issue on GitHub, [as described above](#github). This is a public forum, however. For email support, see [pelicanplatform.org/contact](https://pelicanplatform.org/contact) . For security issues, see our [security policy on GitHub](https://github.com/PelicanPlatform/pelican?tab=security-ov-file#security-policy) . ## OSDF For assistance with the OSDF, from downloading data to integrating with your storage, please see the main website at [osg-htc.org/services/osdf](https://osg-htc.org/services/osdf) . --- Source: https://docs.pelicanplatform.org/getting-help/faqs # Frequently Asked Questions ## Meta ### Is it “Pelican” or “Pelican Platform”? “Pelican Platform” is the proper name for the software described in these documents, as there are other pre-existing tech projects that use the name “Pelican”. That being said, `pelican` is the name of the command for interfacing with components of the Pelican Project, and we frequently refer to said components as “Pelican \_\_\_\_\_” in our documentation as a convenient shorthand. ### How is the Pelican Platform related to the HTCondor Software Suite (HTCSS)? Both the Pelican Platform and [HTCSS](https://htcondor.org/)  are open-source software technologies whose development is led by the [Center for High Throughput Computing (CHTC)](https://chtc.cs.wisc.edu)  at the University of Wisconsin-Madison, whose mission is “to bring the power of High Throughput Computing to all fields of research, and to allow the future of HTC to be shaped by insight from all fields.” While the funding for developing these technologies comes from different sources, there is a lot of overlap of the staff that are involved in the development of these technologies. The flagship instance of HTCSS is the [OSPool](https://osg-htc.org/services/ospool/) , a nation-wide scientific and research computing platform based in the U.S. The need for managing the efficient movement of large data within the OSPool led to the creation of the [OSDF](https://osg-htc.org/services/osdf) . As adoption of the OSDF became more widespread, it became clear that a separate software platform was needed for deploying and managing the OSDF and other data federations like it, and that software is the Pelican Platform. Now, the Pelican Plugin Client ships with HTCSS, meaning any computational work that is managed by HTCondor can access data via a Pelican Federation. ## Data Access Model ### Can I use Pelican Platform to search for the data I need? When a namespace is configured to allow listings, Pelican clients let users search for objects by prefix, similar to the way you might check for files on a computer with the terminal command `ls /foo/bar/`. However, this kind of searching is limited in that it can only tell you the names of objects under a prefix as defined by the prefix owner – it does not necessarily tell you anything about the contents of objects, and it does not guarantee the data owner has chosen descriptive, reasonable names. This is why Pelican encourages leaving breadcrumbs of data provenance in namespace prefixes, because it gives users information about who to contact for more information. See [Choosing a Namespace Prefix](/federating-your-data/choosing-namespaces) for more information. ### Can Pelican be used to satisfy “FAIR” data requirements? Supporting [FAIR data](https://www.go-fair.org/fair-principles/)  – that is, making data “Findable”, “Accessible”, “Interoperable” and “Reusable” – is one of Pelican’s main goals, because we believe that FAIR data is the bedrock of robust, open science. However, at this time Pelican’s primary targets are data accessibility and interoperability, which Pelican supports through its federated approach to data management (accessibility) along with its tight integration with HTCSS and caching mechanisms (interoperability). While Pelican lets users discover object names by prefix/namespace, Pelican does not meet the full set of requirements for “Findability” because it has no native integration with data cataloging technologies, and it provides no way for the generic researcher to answer “where can I get data related to ABC that looks like XYZ”. However, Pelican has partnered with the [National Data Platform](https://www.nationaldataplatform.org/)  to explore these concepts further. Finally, Pelican does not address data “Reusability” because it has no archival features, and its “object immutability” rules are to prevent undefined behavior, not to guarantee that object contents never change. All this being said, researchers who use Pelican can still help address “Findability” and “Reusability” through practicing good data hygiene, such as by choosing a good namespace prefix and structuring object/file names according to best practices. See [Choosing a Namespace Prefix](/federating-your-data/choosing-namespaces) for more information. ## Using Pelican Platform ### I am using \_\_\_\_\_ computer with \_\_\_\_\_ operating system - can I still use Pelican to download objects? Yes! The Pelican Platform provides several ways of accessing objects via a Pelican Federation using the Pelican Client. The most commonly used is the Pelican Command-Line Interface (aka Pelican CLI), which is a standalone program that should be compatible with most modern computers, and it does not require admin permissions to install or use. As long as you have an internet connection and know how to access the object via a Pelican Federation, you just need to download the Pelican CLI! Visit [Getting Started - Accessing Data](/getting-started/accessing-data) to get started with the Pelican CLI. To learn more about the Pelican Clients, visit [Getting Data With Pelican](/getting-data-with-pelican). ### Why do `osdf:///` URLs use 3 slashes instead of two slashes like regular URLs? > **TL;DR:** `osdf:///` URLs use three slashes because the `osdf` scheme inherently specifies the networked system, making the federation hostname redundant. This is similar to how `file:///` URLs work, where the triple slash indicates the resource is on the local machine, eliminating the need for a hostname. Most Pelican clients should detect and handle `osdf://`\-schemed URLs if they’re missing the third slash, but this is technically an abuse of the well-defined URL structure. URLs, or **Uniform Resource Locators**, play a crucial role in the way computers are able to discover, locate and access digital resources. Since their broad adoption in the mid 1990s, their structure has become a well-defined internet standard[1](#triple-slash-fn1). To quote their definition: > “Uniform Resource Locators” (URLs), in addition to identifying a resource, provide a means of locating the resource by describing its primary access mechanism (e.g., its network “location”).[2](#triple-slash-fn2) Whether you’re trying to access a remote PDF or watch your favorite playlist of YouTube cats, the URL you give your browser conveys important information about _what_ it’s supposed to find, _where_ it can look, and _how_ it should be accessed. Understanding a URLs basic components will help answer why `osdf:///` URLs typically require triple slashes. For our intents and purposes, URLs contain three main parts — a _scheme_, a _hostname_ and a _path_, where these pieces can be loosely defined as follows [3](#triple-slash-fn3): 1. **scheme**: A URL’s _scheme_ tells the computer _how_ something should be accessed. In most cases, this specifies a protocol like `https`, `ftp`, or in our case `pelican` and `osdf`. Essentially, this tells your computer what “language” it needs to speak to interact with the resource. 2. **hostname**: The URL’s _hostname_ gives the computer information about who/what remote resource might be able to fulfill your request. 3. **path**: The _path_ component of a URL specifies the name of a requested resource from the requested hostname. Typically this is something like a specific web page or file. These components are stitched together in a predictable fashion: ``` :/// ``` For example, when you visit `https://docs.pelicanplatform.org/parameters`, you’ve defined the URL scheme as `https`, the hostname as `docs.pelicanplatform.org` and the path as `parameters`. Together, these pieces tell your computer to use HTTPS to access the `parameters` page from Pelican’s `docs.pelicanplatform.org` documentation website. The `pelican`\-schemed URLs you use to access objects from Pelican federations follow the same setup, leading to URLs like: ``` pelican://osg-htc.org/some/object ``` Here, you’ve indicated you want to use the `pelican` protocol to interact with `some/object` from the `osg-htc.org` federation. However, some URL schemes are inherent to a specific location and don’t need a hostname. If you’ve ever used a browser to open a PDF on your personal computer, you’ve likely seen a URL like `file:///some/path/to/file.pdf`. The triple slash after the `file` scheme happens because `file` already pre-supposes that the browser needs to get a file from the local machine, so the hostname information isn’t needed. It is equally valid to use the URL `file://localhost/some/path/to/file.pdf`, but that’s more to type! Instead, we wind up cutting out the redundant information to yield > file://~localhost~/some/path/to/file.pdf —> file:///some/path/to/file.pdf Similarly, the `osdf` URL scheme already encodes two pieces of information — that you’re speaking Pelican _and_ you’re talking to the OSDF, whose hostname is `osg-htc.org`. By using `osdf` URLs, you’ve indicated the object you’re interacting with is part of a specific networked system that should already be understood. The hostname in the previous `pelican`\-schemed URL matches the OSDF’s hostname, so it can be rewritten using an `osdf` url: > pelican://osg-htc.org/some/object —> osdf://~osg-htc.org~/some/object = osdf:///some/object On the other hand, construction of a URL like `osdf://some/object` has the potential to confuse many clients that aren’t aware of the `osdf://` protocol. That is because now there is part of the object’s name (“`some`”) where such clients might be expecting the federation’s hostname. More information about `pelican` and `osdf` URLs can be found in our [client usage docs](./getting-data-with-pelican/client#the-different-pelican-url-schemes). > **1**: For more information about the structure of URLs, see [RFC 1738](https://www.rfc-editor.org/rfc/rfc1738) . > **2**: For more information on the difference between URIs, URLs and URNs, see [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986#section-1.1.3) . > **3**: URLs can also contain things like ports, query parameters and “fragments,” and while Pelican makes use of these, they aren’t as crucial to understanding the question at hand. ### Why isn’t Pelican using the closest cache(s) when I download objects? Whenever a Pelican client tries to download an object, one of its first steps is to talk to the appropriate federation’s Director, where the Directors job is to match the client’s request to some service(s) that can best fulfill the request. This usually means giving the client an ordered list of caches that the Director thinks either have the object or that are capable of delivering the object quickly. By default, Directors order this list by trying to determine the physical distance between the client and any caches in the federation with closer caches being assigned higher priority[1](#geoloc-fn1). This troubleshooting guide assumes the Director is configured for distance-based sorting. There are several ways this process can break. #### Client Resolution First, the Director uses the IP address of the incoming client request to generate a lat/long pair and confidence range for the client. It does this by running the client’s IP address through a local database[2](#geoloc-fn2). Issues that can occur at this stage include: - The IP address reported by the client is invalid, or in a private range (e.g. 192.168.0.12 for IPv4) - The IP address is valid, but the database doesn’t have an entry for it - The IP address is valid and has an entry, but the database reports a confidence range greater than 900km. In any of these cases, the Director will decide it can’t reasonably determine where the client is, and it will assign a temporary lat/long pair by picking a coordinate somewhere in the continental US. This coordinate is cached for a short time (~20 minutes), so subsequent requests from the same client will resolve to the same spot. If the list of caches you see being tried look like they have a geographic center, but not the _correct_ geographic center, you might try determining the IP address the Director sees when the client contacts it. This can be done by running: ``` curl ifconfig.me ``` and running the resulting IP address through [MaxMind’s GeoLite City demo](https://www.maxmind.com/en/geoip-demo) . If the location it determines is incorrect, has a large accuracy radius, or appears to be otherwise invalid, that’s likely causing a problem. > **NOTE**: The database used by this demo is not exactly the same database used by the Director. If you see a problem here, there’s definitely an issue, but if this step yields the expected results, there may still be issues with the Director’s database. If the list of caches tried by your client(s) appear to have a geographic center that’s incorrect, contact your federation administrators to ask if they can create a manual override for your IP range. #### Cache/Origin Resolution Alternatively, the client’s location may be known by the Director, but locations for some caches (or even origins) in the list can’t be determined. While the Director can use a client’s IP address directly for geo-location, it uses a DNS lookup against cache/origin hostnames to determine IP addresses. Failure to produce an IP address in this step means something more fundamental is wrong with the cache/origin, and that it should be fixed before receiving any requests. However, it’s still possible that the resolved IP address is incorrect, or has the same types of issues client IPs might have with the MaxMind database. When this happens, the server should be sorted to the end of the potential list of servers. Errors can compound if both of these issues (client _and_ cache/origin geo-location failures) occur. If the cache list you see from the Director has no discernible geographic center, you should contact your federation administrators for help debugging. Finally, some Pelican clients may allow you to select a cache manually if you have a strong preference for which cache to use. For example, the Pelican CLI lets you do this by specifying the cache’s hostname/port with the `-c` flag: ``` pelican object get -c https://my-chosen-cache:8443 pelican://osg-htc.org/some/object ``` > **NOTE**: Specifying caches in this manner should only be done if you’re positive the Director is having sorting troubles — the Director’s sorting algorithms are no longer tied explicitly to geo-location, and it may be trying to make better decisions based on object availability and the detected load of various caches. > **1**: Directors may implement more intelligent cache selection schemes. For a full list of options, see the documentation for the Director’s [`Director.CacheSortMethod`](./parameters#Director-CacheSortMethod) config parameter. > **2**: In particular, the Director uses the [MaxMind GeoLite City database](https://www.maxmind.com/en/geolite-free-ip-geolocation-data) , which it updates twice weekly on Wednesdays and Fridays (shortly after the databases are updated upstream by MaxMind). ### How can I tell what services my Pelican client will talk to _before_ I try to get/put objects? Pelican is built on HTTP, so any client that speaks HTTP can determine this. For more information about Pelican’s use of HTTP verbs, see [Getting Data With Pelican](./getting-data-with-pelican#getting-data-with-pelican) These instructions are for using `curl` on the command line. Broadly speaking, they require: 1. Finding the hostname of your federation’s Director 2. Determining whether you want the Director to provide a list of caches or origins 3. Creating the correct curl command 4. Interpreting the Director’s response headers If you don’t have a working understanding of how Pelican finds and uses Directors in a federation, see [About Pelican/A First Look Under The Hood](./about-pelican#making-bytes-accessible-and-moving-them----a-first-look-under-the-hood) for a quick recap. #### Discovering your Director If you have a `pelican`\-schemed URL and a terminal with `curl`, you have everything you need to get started. First, get the federation discovery URL from your `pelican` url, e.g. `pelican://osg-htc.org/some/object` results in the discovery URL of `osg-htc.org` Next, curl the discovery URL at the `/.well-known/pelican-configuration` path: ``` $ curl https://osg-htc.org/.well-known/pelican-configuration { "director_endpoint": "https://osdf-director.osg-htc.org", "namespace_registration_endpoint": "https://osdf-registry.osg-htc.org", "jwks_uri": "https://osg-htc.org/osdf/public_signing_key.jwks" } ``` If successful, this should return a JSON. Your federation’s Director is the URL provided by the `director_endpoint` key, e.g. `https://osdf-director.osg-htc.org` #### Deciding Whether to Ask About Caches or Origins Next, you should determine whether you expect your client to talk to a cache or an origin. The answer is usually a cache unless you are trying to run a) GET operation using the “direct read” flags/URL query parameters, or b) PUT operation to write an object via the origin. The following steps are very similar in both cases, but require minor adjustments in the way you interact with the Director. ##### Discovering Caches If you’re expecting your client to manipulate objects through a cache, you’ll point `curl` at the Director’s “object” discovery API, which uses the path `/api/v1.0/director/object/` path prefix. For the previous `pelican` URL of `pelican://osg-htc.org/some/object`, you’d then construct the following curl command pointed at the Director with the correct path: ``` $ curl -v https://osdf-director.osg-htc.org/api/v1.0/director/object/some/object ``` If successful, the Director’s response will be a series of HTTP headers with information you can use to answer a variety of questions. In particular, the list of caches you can expect your client to try are included in the `link` header, which will look something like: ``` link: ; rel="duplicate"; pri=1; depth=2, ; rel="duplicate"; pri=2; depth=2, ; rel="duplicate"; pri=3; depth=2, ; rel="duplicate"; pri=4; depth=2, ; rel="duplicate"; pri=5; depth=2, ; rel="duplicate"; pri=6; depth=2 ``` ##### Discovering Origins If you’re expecting your client to manipulate objects through an origin, you’ll point `curl` at the Director’s “origin” discovery API, which uses the path `/api/v1.0/director/origin/` path prefix. Using the previous `pelican`\-schemed URL, you’d then construct the following `curl` command: ``` $ curl -v https://osdf-director.osg-htc.org/api/v1.0/director/origin/some/object ``` Once again, the `link` header will include an ordered list of Origins that your client expects to try when looking for `/some/object`: ``` < link: ; rel="duplicate"; pri=1; depth=1 ``` This list is usually much shorter and likely contains a single Origin. ## Federating Data ### Can I run a Pelican Origin from my laptop? You can, but it is not recommended. While it is technically possible to run a Pelican Origin for data on your laptop, there are several reasons why you should not. - Whenever your laptop is closed, runs out of battery, or simply isn’t connected to the internet, the Origin would no longer be available to any researcher who wishes to access that data. - You would need to restart the Origin each time your laptop shut down. - There is the potential for high network usage on your laptop if a large amount of the data is requested at once. ### How do I get the public JWKS key from the private key that I have? Pelican uses cryptographic key pairs for creating and verifying user credentials (tokens). Private keys are stored in PEM format (`.pem` files), while public keys are stored in [JWKS (JSON Web Key Set)](https://datatracker.ietf.org/doc/html/rfc7517)  format (`.jwks` files). If you have a Pelican-formatted private key and need to derive its corresponding public key, follow these steps: #### Step 1: Locate your private key file Pelican private keys are stored in PEM format. A typical private key file looks like: ``` -----BEGIN PRIVATE KEY----- M0G+AgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgzeomq6yJvek0fg2Z g+NY6qklGz4ZjVlrRgTDT9hk5EShRANCAAT0giQYTPJbLB5wGwiKysmwf79eoxqX 5Tbl22Ed7BQLnIf9Q1KLTNqpyzz0YmuoRlzSmXaYvdlSebvzUq2fqXdH -----END PRIVATE KEY----- ``` Private keys are located in the directory specified by the `IssuerKeysDirectory` configuration parameter. #### Step 2: Use the Pelican CLI to derive the public key Run the following command to generate the public key from your existing private key: ``` pelican-server key create --private-key /path/to/your/private-key.pem ``` Replace `/path/to/your/private-key.pem` with the actual path to your private key file. #### Step 3: Retrieve the generated public key By default, the public key will be generated in the same directory as the private key with the filename `issuer-pub.jwks`. To specify a custom output location, use the `--public-key` flag: ``` pelican-server key create --private-key /path/to/your/private-key.pem --public-key /path/to/output/public-key.jwks ``` #### Step 4: Verify the public key format The generated public key file should be in JWKS format: ``` { "keys": [ { "alg": "ES256", "crv": "P-256", "kid": "N3Tks9H791dG7bdzjjlQzV-fwwCrnXd_mcd_2D5ZvYE", "kty": "EC", "x": "9IIkGEzyWywecBsIisrJsH-_XqMal-U25dthHewUC5w", "y": "h_1DUotM2qnLPPRia6hGXNKZdpi92VJ5u_NSrZ-pd0c" } ] } ``` For more information about managing Pelican keys, see [Managing Administrator Credentials](/advanced-concepts/server). --- Source: https://docs.pelicanplatform.org/getting-help/troubleshooting # Troubleshooting This guide details the common errors that occur when working with Pelican, and how to troubleshoot them. These sections are intended for **end users** who are using Pelican to transfer data: - [General client issues](#general-client-issues): problems that apply to any implementation of a Pelican client. - [Pelican CLI](#pelican-cli): problems using the [Pelican CLI](/getting-data-with-pelican/client) client, specifically. - [Pelican HTCondor Plugin](#pelican-htcondor-plugin): problems using [Pelican’s HTCondor file transfer plugin](/getting-data-with-pelican#pelicans-htcondor-plugin), specifically. These sections are intended for **system administrators** deploying Pelican services: - [Federating data](#federating-data): problems integrating a data store with an existing Pelican Federation - [Operating a Federation](#operating-a-federation): problems operating Pelican services used to run a Pelican Federation ## General client issues Regardless of how a Pelican client is implemented, it will be interacting with the Pelican Federation using a common set of operations, including: - Connecting to Federation services - Getting information about a namespace - Transferring data among other things. (For more information, see [About Pelican](/about-pelican) and [Getting Data with Pelican](/getting-data-with-pelican).) Furthermore, these operations are conducted over the internet and so common networking problems can impact these operations. These issues can range from slow internet or an interrupted connection to more obscure problems with finding or connecting to web servers. The error message you encounter should describe where in the chain of services the issue occurred. When troubleshooting issues, you should always consider upgrading your client to the latest version! ### Understanding Pelican client error messages The Pelican client you are using will return an error if something goes wrong. Ideally, you can use this error to understand the problem and what needs to be done. Most errors are structured as a single line in the form `ERROR[] Message`. One error may raise additional errors for actions dependent on the original action that failed. In that case, you’ll see multiple such `ERROR` lines, but the last one should summarize the failure including the sequence of events. ### Issues connecting to Federation services A Pelican Federation relies on several central services for validating and directing transfer requests. If these services are offline or your Pelican client cannot otherwise reach them, you will get an error. In most cases, the server running the services is restarting. Typically, you just need to wait a few minutes for the restart to complete and the error will go away on its own. If the services are under maintenance or otherwise experiencing an outage, then trying again will not fix the issue. Check with the Federation administrators if there are any outages. For example, the status of OSDF services is reported here: [status.osg-htc.org](https://status.osg-htc.org/) . If there are no outages, then it may be that your device or local internet network is unable to connect to the central services. Check if you are able to connect to the Federation from a different device or a different internet network (perhaps by activating a VPN). ### Issues finding a namespace When you request something from a Federation, the Director is responsible for redirecting the request to the appropriate service. The namespace you are trying to access has to be registered with the central services in order for the redirection to occur. If the Director is unable to redirect the request, you will get an error along the lines of `no sources found for the requested namespace`. This could happen for a couple of reasons. - The namespace does not exist in the Federation - The Origin service for the namespace is not online - The Origin service for the namespace is not registered with the Federation The most common cause is a typo in the namespace address that you are trying to access. Double check that everything is spelled correctly. If there is a typo in what you entered, then the namespace as written doesn’t actually exist in the Federation! You may be able to check if the namespace exists in the Federation by checking the Director website. The error message should report the Director’s web address and you should be able to navigate to that address in a web browser. If you are able to connect to the Director website, there should be a section called `Namespaces` where you can search for the namespaces registered with the Federation. Note that this only lists the **currently active** namespaces. If you have successfully used this command in the past, then it is likely that the Origin service is offline or otherwise unable to connect to the Director. In that case, the service may be restarting and you just need to try again later. If you are having persistent errors, then you should contact the Origin administrator for further assistance. ### Authentication issues For protected reads and rights, Pelican can integrate with authentication services such as CILogon. When you try to run a protected operation, you’ll be prompted by Pelican to authenticate, usually by going to a web address in your browser and signing in with the necessary identity. If the identity you authenticated with is not authorized to view a namespace, you will get an `Authorization error`. Here are some things to check to troubleshoot issues with authentication: - Make sure you are trying to access the correct namespace - Make sure that you are logging in with the correct identity - Install (or upgrade to) the latest version of the Pelican CLI and try again If you continue to have issues, contact the namespace administrator for assistance. It could be that the authentication integration is broken, or that you need to follow additional steps in order to authenticate. ### Issues downloading data via a Federation A common issue is that the data being requested does not exist. This could be because of a typo in the object name, or that the bytes do not exist on the connected storage. Double check that the object address does not have typos. Here, use of the `pelican object ls` command can help identify whether or not the object exists as written. Another issue is slow transfers. This could be because of an issue with the Origin serving the namespace, but usually it’s a problem with a Cache. This error typically results in a `transfer timeout` message. The problem usually resolves itself and you just need to try again. For persistent errors, you should contact the Federation administrator. #### Downloaded data is corrupted For the purposes of caching, Pelican assumes that the object is **immutable**, that is, **the content of an object does not change** once it has been fetched. If an object (as identified by its name) has its contents changed, this causes “undefined behavior”. More specifically, if an object has been cached but the content has changed at the original storage, the data downloaded by the client could be - the original version of the object - the new version of the object - **or some combination thereof (!)** To avoid this scenario, **always change the name of the object when you modify its contents**. ### Issues uploading data via a Federation The most common issue when uploading data via a Federation is that an object with the same name already exists. By default, Pelican will not attempt to modify or overwrite an object that already exists (to avoid the behavior described above). Changing the name of the object that you are trying to upload should avoid this problem. This issue frequently occurs because of an interrupted upload. Currently, the only workaround is to change the name of the object you are uploading. In the future, Pelican will not create an object until it is certain the upload was successful (e.g., not interrupted). ## Pelican CLI This section discusses common errors when working with the [Pelican CLI](/getting-data-with-pelican/client) client. ### Reset local client If the you’ve followed the other troubleshooting instructions but you continue to experience the issue, or if you are experiencing an issue not listed here, there may be a problem with the local configuration for your client. If you enter the password incorrectly for your local credentials file, you should see a message suggesting that you use the command `pelican credentials reset-local` to remove your local credentials. The next time Pelican fetches credentials for an action, you’ll be prompted to create a new password to save the local credentials. This is generally safe to do, unless you are actively transferring data in another process. You can reset your local credentials file at anytime by running ``` pelican credentials reset-local ``` As a last resort, you can try removing your local configuration for the Pelican client, as follows. Resetting your local configuration will remove any stored credentials or custom configuration. If you are using a shared server and the system administrator is the one who installed Pelican, you should ask the system administrator for assistance instead! To reset your local configuration on **Linux**/**MacOS**: 1. Move to the containing directory. ``` cd ~/.config ``` You should see a `pelican` directory when you run `ls`. 2. Remove the `pelican` directory. ``` rm -r ./pelican ``` Make sure you avoid any typos! This command will **recursively delete** whatever you provide to it. To reset your local configuration on **Windows**: 1. Move to the containing directory. ``` cd ~\.config ``` You should see a `pelican` directory when you run `dir`. 2. Remove the `pelican` directory. ``` rm .\pelican ``` If prompted to remove a file, double-check the path before confirming with `Y`. ## Pelican HTCondor Plugin If an HTCondor job encounters an error with the Pelican Plugin, the job will either (a) automatically retry, or (b) go on hold. In the former case, the user does not have to do anything. If the job goes on hold because of a Pelican Plugin error, the hold reason should explain the issue. The Plugin is susceptible to the general client issues described [above](#general-client-issues). For information on investigating held jobs in HTCondor, see [this manual page](https://htcondor.readthedocs.io/en/latest/users-manual/troubleshooting.html#how-to-debug-a-held-jobs) . If you are having persistent issues, check if you are able to perform the equivalent action using the Pelican CLI client. HTCondor Access Points (APs) should also have the `pelican` command installed. The namespace you are attempting to access may have specific authentication requirements involving the HTCondor Access Point, such that you may not be able to authenticate properly using the Pelican CLI client. If this is the case, you should contact your Access Point administrator for further assistance. ## Federating data When setting up an Origin to make your data available via a Pelican Federation, it’s good practice to test access using the same mechanism that your target audience will use. To make it easier to identify failure points, we recommend this ordering: 1. Test that `pelican object ls` works 2. Test that `pelican object get --direct` works 3. Test that `pelican object get` works In principle, once you’ve successfully completed a transfer this way, everything should be good to go. Keep in mind that general networking issues can complicate the testing process, so it’s also a good idea to test the transfer multiple times before jumping into troubleshooting your configuration. ### Unable to transfer data via Caches One issue you may face with some Origins is the case where you can download an object **directly** from the Origin (using the `pelican object get --direct` command) but **are not** able to download the object **via a Cache** (using the `pelican object get` command). This typically arises when the Origin restricts read access using additional authorization policies configured with parameters like `Xrootd.ScitokensConfig`, `Xrootd.Authfile` or even `Xrootd.ConfigFile`. Some Origin administrators may use these parameters to give special privileges to a subset of users, to side door the Origin for externally-integrated data services like Rucio, or to enable x509 authentication at the Origin. The problem with providing Origins with authorization configuration using these parameters is that this information doesn’t propagate to the rest of the federation because these parameters are local to the Origin and aren’t shared via the federation’s advertisement protocol. ### Troubleshooting Steps #### Checking the Director To troubleshoot this issue, start by visiting your federation’s Director and searching for the namespace in question. If the Director does not have your namespace, there is likely an issue with your Origin’s ability to advertise to the federation, and this can only be fixed by the Origin administrator or the federation’s central services operators. If the namespace **does** exist at the Director, click on it to see two important pieces of information: 1. The namespace capabilities explaining what operations the federation thinks is allowable for the namespace (e.g. “Reads”, “Writes”, “Listings”) 2. The namespace’s configured issuer(s) (use the “Token Issuer” display, not the “Token Generation” display) Double check that these are what you expect to see for the namespace. Any incorrect values should be addressed by modifying the Origin’s configuration. If you’re unsure, you may need to consult the namespace/Origin’s system administrator. #### Checking your token If there are no obvious issues there, it’s time to inspect any token you’re using to successfully access data via the Origin. There are two tools you can use to inspect the token: 1. The CLI called [`htgettoken`](https://pypi.org/project/htgettoken/) 2. The [jwt.io](jwt.io) interface The key things to look for in the token are the `iss` (issuer) and the `aud` (audience) fields. The token’s issuer **must** match one of the issuers configured in the Director, and the audience must be set to the correct “any” string for the token’s token profile (`"https://wlcg.cern.ch/jwt/v1/any"` for wlcg and `"ANY"` for scitokens). Caches will only approve tokens using the issuers it discovers from the Director, and they expect tokens with a broad audience so they can validate the request on behalf of any client. When these two things aren’t true, the token cannot work with Caches. Since the token works with the Origin, it implies special authorization polices were configured with one of the params listed above, and a different token is needed to access objects through Caches. ## Operating a Federation If you are deploying your own Pelican Federation, we recommend that you contact the Pelican team for assistance in getting started and troubleshooting issues: [pelicanplatform.org/contact](https://pelicanplatform.org/contact) . --- Source: https://docs.pelicanplatform.org/api-docs # API Documentation Pelican's HTTP API is documented as an OpenAPI (Swagger) specification. The rendered page is an interactive Swagger UI; the raw machine-readable spec is here: https://raw.githubusercontent.com/PelicanPlatform/pelican/main/swagger/pelican-swagger.yaml --- Source: https://docs.pelicanplatform.org/commands-reference # commands-reference ## Commands Reference Pelican is distributed as two separate executables: | Binary | Purpose | | --- | --- | | `pelican` | Client-side tools — transfer data, manage credentials, submit jobs | | `pelican-server` | Server-side tools — run origins, caches, directors, and registries | Both binaries share a common set of commands (`config`, `credentials`, `namespace`, `token`). * * * ### pelican — client binary The `pelican` binary is for users who need to **access data** from a federation or manage client-side credentials. [Browse the pelican command reference →](/commands-reference/pelican) Key subcommands: - [`pelican object`](/commands-reference/pelican/object) — get, put, copy, and delete objects - [`pelican job`](/commands-reference/pelican/job) — manage asynchronous transfer jobs - [`pelican plugin`](/commands-reference/pelican/plugin) — HTCSS plugin integration - [`pelican rclone`](/commands-reference/pelican/rclone) — rclone integration * * * ### pelican-server — server binary The `pelican-server` binary is for operators who need to **run federation services**. [Browse the pelican-server command reference →](/commands-reference/pelican-server) Key subcommands: - [`pelican-server origin`](/commands-reference/pelican-server/origin) — run an origin service - [`pelican-server cache`](/commands-reference/pelican-server/cache) — run a cache service - [`pelican-server director`](/commands-reference/pelican-server/director) — run a director service - [`pelican-server registry`](/commands-reference/pelican-server/registry) — run a registry service - [`pelican-server key`](/commands-reference/pelican-server/key) — manage issuer keys - [`pelican-server generate`](/commands-reference/pelican-server/generate) — generate server credentials --- Source: https://docs.pelicanplatform.org/commands-reference/pelican # commands-reference/pelican ## pelican Interact with data federations ### Synopsis The pelican software allows one to build and interact with data federations, enabling the sharing of objects and collections across multiple dataset providers. ### Options ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize -h, --help help for pelican --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican client-agent](/commands-reference/pelican/client-agent) - Manage the Pelican client agent server - [pelican config](/commands-reference/pelican/config) - View and search for configuration parameters - [pelican credentials](/commands-reference/pelican/credentials) - Interact with the credential configuration file - [pelican job](/commands-reference/pelican/job) - Manage asynchronous transfer jobs - [pelican namespace](/commands-reference/pelican/namespace) - Work with namespaces - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation - [pelican plugin](/commands-reference/pelican/plugin) - Plugin management for HTCSS - [pelican token](/commands-reference/pelican/token) - Interact with tokens used to interact with objects in Pelican --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/client-agent # commands-reference/pelican/client-agent ## pelican client-agent Manage the Pelican client agent server ### Synopsis The client-agent server provides a RESTful API for interacting with the Pelican client functionality over a Unix domain socket. This enables external applications to use Pelican transfer capabilities without directly invoking the CLI. ### Options ``` -h, --help help for client-agent --pid-file string Path to the PID file (default: ~/.pelican/client-agent.pid) --socket string Path to the Unix domain socket (default: ~/.pelican/client-agent.sock) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican client-agent start](/commands-reference/pelican/client-agent/start) - Start the client agent server - [pelican client-agent status](/commands-reference/pelican/client-agent/status) - Check the status of the client agent server - [pelican client-agent stop](/commands-reference/pelican/client-agent/stop) - Stop the client agent server --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/client-agent/start # commands-reference/pelican/client-agent/start ## pelican client-agent start Start the client agent server ### Synopsis Start the client agent server as a daemon process. The server will listen on a Unix domain socket and handle job-based transfer requests. ``` pelican client-agent start [flags] ``` ### Options ``` --database string Path to the SQLite database file for persistence (default: ~/.pelican/client-agent.db) --foreground Run in foreground instead of daemonizing (default: daemonize) -h, --help help for start --max-jobs int Maximum number of concurrent transfer jobs (default: uses ClientAgent.MaxConcurrentJobs parameter, or 5) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --pid-file string Path to the PID file (default: ~/.pelican/client-agent.pid) --socket string Path to the Unix domain socket (default: ~/.pelican/client-agent.sock) --version Print the version and exit ``` ### SEE ALSO - [pelican client-agent](/commands-reference/pelican/client-agent) - Manage the Pelican client agent server --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/client-agent/status # commands-reference/pelican/client-agent/status ## pelican client-agent status Check the status of the client agent server ### Synopsis Check if the client agent server is running. ``` pelican client-agent status [flags] ``` ### Options ``` -h, --help help for status ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --pid-file string Path to the PID file (default: ~/.pelican/client-agent.pid) --socket string Path to the Unix domain socket (default: ~/.pelican/client-agent.sock) --version Print the version and exit ``` ### SEE ALSO - [pelican client-agent](/commands-reference/pelican/client-agent) - Manage the Pelican client agent server --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/client-agent/stop # commands-reference/pelican/client-agent/stop ## pelican client-agent stop Stop the client agent server ### Synopsis Stop a running client agent server daemon. ``` pelican client-agent stop [flags] ``` ### Options ``` -h, --help help for stop ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --pid-file string Path to the PID file (default: ~/.pelican/client-agent.pid) --socket string Path to the Unix domain socket (default: ~/.pelican/client-agent.sock) --version Print the version and exit ``` ### SEE ALSO - [pelican client-agent](/commands-reference/pelican/client-agent) - Manage the Pelican client agent server --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/config # commands-reference/pelican/config ## pelican config View and search for configuration parameters ### Synopsis The ‘config’ command allows users to view, search, and see the documentation for various configuration parameters in the Pelican system. ### Options ``` -h, --help help for config ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican config describe](/commands-reference/pelican/config/describe) - Print documentation for the specified config parameter - [pelican config dump](/commands-reference/pelican/config/dump) - Dump all configuration parameters - [pelican config get](/commands-reference/pelican/config/get) - Retrieve config parameters that match any of the given arguments - [pelican config summary](/commands-reference/pelican/config/summary) - Print config parameters that differ from default values --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/config/describe # commands-reference/pelican/config/describe ## pelican config describe Print documentation for the specified config parameter ### Synopsis The ‘describe’ command prints detailed documentation for a specified configuration parameter, including its type, default value, description, related components, and whether it is deprecated or hidden. ``` pelican config describe [parameter] [flags] ``` ### Examples ``` # View documentation for the Server.WebPort parameter pelican config describe server.webPort ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican config](/commands-reference/pelican/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/config/dump # commands-reference/pelican/config/dump ## pelican config dump Dump all configuration parameters ### Synopsis The ‘dump’ command outputs all current configuration parameters and their values to the console. This includes default values that have not been explicitly set. ``` pelican config dump [flags] ``` ### Examples ``` # Dump all configuration parameters pelican config dump ``` ### Options ``` -o, --format string Output format (yaml or json) (default "yaml") -h, --help help for dump ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican config](/commands-reference/pelican/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/config/get # commands-reference/pelican/config/get ## pelican config get Retrieve config parameters that match any of the given arguments ### Synopsis The ‘get’ command retrieves and displays configuration parameters that contain any of the provided argument patterns in their name or value. The search space can be narrowed or expanded using available flags. The matching is case-insensitive. If no arguments are provided, all configuration parameters are retrieved. The command outputs the results in a flattened format from the nested configuration, making it grep-friendly for easier searching. ``` pelican config get [arguments] [flags] ``` ### Examples ``` # Retrieve parameters that have either 'log' or 'monitor' in their name or value, # and relate to either 'origin' or 'cache', including deprecated parameters in the search space pelican config get log monitor -m origin -m cache --include-deprecated ``` ### Options ``` --exact-match Match configuration parameter names exactly instead of using substring matching -h, --help help for get --include-deprecated Include deprecated configuration parameters --include-hidden Include hidden configuration parameters -m, --module config get Specify modules to filter the output of config get. The recognized modules are `client`, `registry`, `director`, `origin`, `cache`, and `localcache`. Multiple modules can be specified at the same time, for example: `config get -m cache -m origin`. If multiple modules are provided, parameters related to any of the modules will be retrieved. If no modules are specified, no module-based filter is applied to the search space. ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican config](/commands-reference/pelican/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/config/summary # commands-reference/pelican/config/summary ## pelican config summary Print config parameters that differ from default values ### Synopsis The ‘summary’ command outputs configuration parameters whose values differ from their default settings. ``` pelican config summary [flags] ``` ### Examples ``` # Show configuration parameters that are set differently from their default values pelican config summary ``` ### Options ``` -o, --format string Output format (yaml or json) (default "yaml") -h, --help help for summary ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican config](/commands-reference/pelican/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials # commands-reference/pelican/credentials ## pelican credentials Interact with the credential configuration file ### Options ``` -h, --help help for credentials ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican credentials prefix](/commands-reference/pelican/credentials/prefix) - Manage the prefix configuration - [pelican credentials print](/commands-reference/pelican/credentials/print) - Print the credential configuration file - [pelican credentials replace](/commands-reference/pelican/credentials/replace) - Replace the credential configuration file - [pelican credentials reset-local](/commands-reference/pelican/credentials/reset-local) - Delete all local credentials for the current user - [pelican credentials reset-password](/commands-reference/pelican/credentials/reset-password) - Reset the password for the current user - [pelican credentials token](/commands-reference/pelican/credentials/token) - Manage the available tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/prefix # commands-reference/pelican/credentials/prefix ## pelican credentials prefix Manage the prefix configuration ### Synopsis Manage the prefix configuration ### Options ``` -h, --help help for prefix ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials](/commands-reference/pelican/credentials) - Interact with the credential configuration file - [pelican credentials prefix add](/commands-reference/pelican/credentials/prefix/add) - Add a new oauth client - [pelican credentials prefix delete](/commands-reference/pelican/credentials/prefix/delete) - Delete the oauth client - [pelican credentials prefix print](/commands-reference/pelican/credentials/prefix/print) - Print the oauth client configuration file - [pelican credentials prefix set](/commands-reference/pelican/credentials/prefix/set) - Set the oauth client attributes --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/prefix/add # commands-reference/pelican/credentials/prefix/add ## pelican credentials prefix add Add a new oauth client ### Synopsis Add a new oauth client ``` pelican credentials prefix add [flags] ``` ### Options ``` -h, --help help for add ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials prefix](/commands-reference/pelican/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/prefix/delete # commands-reference/pelican/credentials/prefix/delete ## pelican credentials prefix delete Delete the oauth client ### Synopsis Delete the oauth client ``` pelican credentials prefix delete [flags] ``` ### Options ``` -h, --help help for delete ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials prefix](/commands-reference/pelican/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/prefix/print # commands-reference/pelican/credentials/prefix/print ## pelican credentials prefix print Print the oauth client configuration file ### Synopsis Print the oauth client configuration file ``` pelican credentials prefix print [flags] ``` ### Options ``` -h, --help help for print ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials prefix](/commands-reference/pelican/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/prefix/set # commands-reference/pelican/credentials/prefix/set ## pelican credentials prefix set Set the oauth client attributes ### Synopsis Set the oauth client attributes (client\_id or client\_secret) ``` pelican credentials prefix set [flags] ``` ### Options ``` -h, --help help for set ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials prefix](/commands-reference/pelican/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/print # commands-reference/pelican/credentials/print ## pelican credentials print Print the credential configuration file ### Synopsis Print the credential configuration file ``` pelican credentials print [flags] ``` ### Options ``` -h, --help help for print ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials](/commands-reference/pelican/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/replace # commands-reference/pelican/credentials/replace ## pelican credentials replace Replace the credential configuration file ### Synopsis Replace the credential configuration file ``` pelican credentials replace [flags] ``` ### Options ``` -h, --help help for replace ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials](/commands-reference/pelican/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/reset-local # commands-reference/pelican/credentials/reset-local ## pelican credentials reset-local Delete all local credentials for the current user ### Synopsis Delete all local credentials for the current user ``` pelican credentials reset-local [flags] ``` ### Options ``` -h, --help help for reset-local ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials](/commands-reference/pelican/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/reset-password # commands-reference/pelican/credentials/reset-password ## pelican credentials reset-password Reset the password for the current user ### Synopsis Reset the password for the current user ``` pelican credentials reset-password [flags] ``` ### Options ``` -h, --help help for reset-password ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials](/commands-reference/pelican/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/token # commands-reference/pelican/credentials/token ## pelican credentials token Manage the available tokens ### Synopsis Manage the available tokens ### Options ``` -h, --help help for token ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials](/commands-reference/pelican/credentials) - Interact with the credential configuration file - [pelican credentials token get](/commands-reference/pelican/credentials/token/get) - Get a new token for a given prefix - [pelican credentials token setup](/commands-reference/pelican/credentials/token/setup) - Set up a credential file containing tokens for a Pelican namespace --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/token/get # commands-reference/pelican/credentials/token/get ## pelican credentials token get Get a new token for a given prefix ### Synopsis Get a new token for a given prefix ``` pelican credentials token get [flags] ``` ### Options ``` -h, --help help for get ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials token](/commands-reference/pelican/credentials/token) - Manage the available tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/credentials/token/setup # Set up credentials for reading from a namespace ## pelican credentials token setup Set up a credential file containing tokens for a Pelican namespace ### Synopsis Acquire a token for the specified Pelican namespace and save it to a credential file on disk. The credential file contains the access token, refresh token, and OAuth2 client credentials needed to obtain fresh tokens later without re-authenticating. By default, the credential file is password-protected. Use —no-password to save the file without encryption, which is useful for non-interactive contexts where password prompts would fail. Use —credential-file to specify an alternative path for the credential file. Examples: # Set up credentials for reading from a namespace pelican credentials token setup —read pelican://federation.example.org/namespace/path # Set up credentials for reading and writing pelican credentials token setup —write pelican://federation.example.org/namespace/path # Set up credentials without password protection pelican credentials token setup —no-password —read pelican://federation.example.org/namespace/path # Set up credentials to a specific file pelican credentials token setup —credential-file /path/to/creds.pem —read pelican://federation.example.org/namespace/path ``` pelican credentials token setup [flags] ``` ### Options ``` --credential-file string Path to the credential file to write -h, --help help for setup --no-password Save the credential file without password protection -r, --read Request a read token -w, --write Request a write token (implies read) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican credentials token](/commands-reference/pelican/credentials/token) - Manage the available tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/job # commands-reference/pelican/job ## pelican job Manage asynchronous transfer jobs ### Synopsis Manage asynchronous transfer jobs created with the —async flag. Jobs can contain one or more file transfers and can be monitored, listed, and cancelled. ### Options ``` -h, --help help for job ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican job cancel](/commands-reference/pelican/job/cancel) - Cancel a transfer job - [pelican job list](/commands-reference/pelican/job/list) - List all transfer jobs - [pelican job status](/commands-reference/pelican/job/status) - Get the status of a transfer job --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/job/cancel # commands-reference/pelican/job/cancel ## pelican job cancel Cancel a transfer job ### Synopsis Cancel a transfer job and all its incomplete transfers. Completed transfers within the job are not affected. ``` pelican job cancel [flags] ``` ### Options ``` -h, --help help for cancel ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican job](/commands-reference/pelican/job) - Manage asynchronous transfer jobs --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/job/list # commands-reference/pelican/job/list ## pelican job list List all transfer jobs ### Synopsis List all transfer jobs, with optional filtering by status. Shows a summary of each job including completion status and transfer counts. ``` pelican job list [flags] ``` ### Options ``` -h, --help help for list -l, --limit int Maximum number of jobs to return (default 10) -o, --offset int Offset for pagination -s, --status string Filter by status (pending, running, completed, failed, cancelled) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican job](/commands-reference/pelican/job) - Manage asynchronous transfer jobs --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/job/status # commands-reference/pelican/job/status ## pelican job status Get the status of a transfer job ### Synopsis Get detailed status information about a transfer job, including the status of all transfers within the job and overall progress. ``` pelican job status [flags] ``` ### Options ``` -h, --help help for status -w, --watch Watch job status until completion ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican job](/commands-reference/pelican/job) - Manage asynchronous transfer jobs --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/namespace # commands-reference/pelican/namespace ## pelican namespace Work with namespaces ### Options ``` -h, --help help for namespace --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican namespace delete](/commands-reference/pelican/namespace/delete) - Delete a namespace - [pelican namespace list](/commands-reference/pelican/namespace/list) - List all namespaces - [pelican namespace register](/commands-reference/pelican/namespace/register) - Register a new namespace --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/namespace/delete # commands-reference/pelican/namespace/delete ## pelican namespace delete Delete a namespace ``` pelican namespace delete [flags] ``` ### Options ``` -h, --help help for delete --prefix string prefix for delete namespace ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key --version Print the version and exit ``` ### SEE ALSO - [pelican namespace](/commands-reference/pelican/namespace) - Work with namespaces --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/namespace/list # commands-reference/pelican/namespace/list ## pelican namespace list List all namespaces ``` pelican namespace list [flags] ``` ### Options ``` -h, --help help for list ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key --version Print the version and exit ``` ### SEE ALSO - [pelican namespace](/commands-reference/pelican/namespace) - Work with namespaces --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/namespace/register # commands-reference/pelican/namespace/register ## pelican namespace register Register a new namespace ``` pelican namespace register [flags] ``` ### Options ``` -h, --help help for register --prefix string prefix for registering namespace --with-identity Register a namespace with an identity ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key --version Print the version and exit ``` ### SEE ALSO - [pelican namespace](/commands-reference/pelican/namespace) - Work with namespaces --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object # commands-reference/pelican/object ## pelican object Interact with objects in the federation ### Options ``` -h, --help help for object ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican object copy](/commands-reference/pelican/object/copy) - Copy a file to/from a Pelican federation - [pelican object get](/commands-reference/pelican/object/get) - Get a file from a Pelican federation - [pelican object ls](/commands-reference/pelican/object/ls) - List objects in a namespace from a federation - [pelican object put](/commands-reference/pelican/object/put) - Send a file to a Pelican federation - [pelican object share](/commands-reference/pelican/object/share) - Generate a string for sharing access to a namespace. Note the sharing is based on prefixes; all object names matching the prefix will be accessible - [pelican object stat](/commands-reference/pelican/object/stat) - Stat objects in a namespace from a federation - [pelican object sync](/commands-reference/pelican/object/sync) - Sync a directory to or from a Pelican federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object/copy # commands-reference/pelican/object/copy ## pelican object copy Copy a file to/from a Pelican federation ``` pelican object copy {source ...} {destination} [flags] ``` ### Options ``` --async Run the transfer asynchronously through the client API server and return a job ID -c, --cache string A comma-separated list of preferred caches to try for the transfer, where a "+" in the list indicates the client should fallback to discovered caches if all preferred caches fail. --caches string A JSON file containing the list of caches -h, --help help for copy --methods string Comma separated list of methods to try, in order (default "http") -r, --recursive Recursively copy a collection. Forces methods to only be http to get the freshest collection contents -t, --token string Token file to use for transfer --wait When used with --async, wait for the job to complete before returning ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object/get # commands-reference/pelican/object/get ## pelican object get Get a file from a Pelican federation ``` pelican object get {source ...} {destination} [flags] ``` ### Options ``` --async Run the transfer asynchronously through the client API server and return a job ID -c, --cache string A comma-separated list of preferred caches to try for the transfer, where a "+" in the list indicates the client should fallback to discovered caches if all preferred caches fail. --caches string A JSON file containing the list of caches --direct Download directly from an origin, bypassing any caches (same as '?directread' query) --dry-run Show what would be downloaded without actually downloading -h, --help help for get --inplace Write files directly to destination (default: use temporary files) --pack string Package transfer using remote packing functionality (same as '?pack=' query). Options: auto, tar, tar.gz, tar.xz, zip. Default: auto when flag is provided without an explicit value -r, --recursive Recursively download a collection. Forces methods to only be http to get the freshest collection contents -t, --token string Token file to use for transfer --transfer-stats string A path to a file to write transfer statistics to --wait When used with --async, wait for the job to complete before returning ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object/ls # commands-reference/pelican/object/ls ## pelican object ls List objects in a namespace from a federation ``` pelican object ls {object} [flags] ``` ### Options ``` -C, --collection-only List collections only --collections-url string URL to use for collection listing, overriding the director's response --depth int Maximum depth to recurse (e.g., --depth 2 similar to find . -maxdepth 2). Use -1 for unlimited depth. (default -1) -h, --help help for ls -j, --json Print results in JSON format -l, --long Include extended information -O, --object-only List objects only -r, --recursive Recursively list all objects in subdirectories -1, --single-column Force output to be one entry per line -t, --token string Token file to use for transfer ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object/put # commands-reference/pelican/object/put ## pelican object put Send a file to a Pelican federation ``` pelican object put {source ...} {destination} [flags] ``` ### Options ``` --async Run the transfer asynchronously through the client API server and return a job ID --checksum-algorithm string Checksum algorithm to use for upload and validation --checksums string Verify files against a checksums manifest. The format is ALGORITHM:FILENAME --dry-run Show what would be uploaded without actually uploading -h, --help help for put --pack string Package transfer using remote packing functionality (same as '?pack=' query). Options: auto, tar, tar.gz, tar.xz, zip. Default: auto when flag is provided without an explicit value -r, --recursive Recursively upload a collection. Forces methods to only be http to get the freshest collection contents --require-checksum Require the server to return a checksum for the uploaded file (uses crc32c algorithm if no specific algorithm is specified) -t, --token string Token file to use for transfer --transfer-stats string File to write transfer stats to --wait When used with --async, wait for the job to complete before returning ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object/share # commands-reference/pelican/object/share ## pelican object share Generate a string for sharing access to a namespace. Note the sharing is based on prefixes; all object names matching the prefix will be accessible ``` pelican object share {URL} [flags] ``` ### Options ``` -h, --help help for share --write Allow writes to the target prefix ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object/stat # commands-reference/pelican/object/stat ## pelican object stat Stat objects in a namespace from a federation ``` pelican object stat {object} [flags] ``` ### Options ``` --checksums stringArray Checksums to request from the server. Known values are: md5, crc32c, crc32, sha -h, --help help for stat -j, --json Print results in JSON format -t, --token string Token file to use for transfer ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/object/sync # commands-reference/pelican/object/sync ## pelican object sync Sync a directory to or from a Pelican federation ``` pelican object sync {source ...} {destination} [flags] ``` ### Options ``` -c, --cache string A comma-separated list of preferred caches to try for the transfer, where a "+" in the list indicates the client should fallback to discovered caches if all preferred caches fail. --dry-run Show what would be synchronized without actually modifying the destination -h, --help help for sync --inplace Write files directly to destination (default: use temporary files) -t, --token string Token file to use for transfer ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican object](/commands-reference/pelican/object) - Interact with objects in the federation --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/plugin # commands-reference/pelican/plugin ## pelican plugin Plugin management for HTCSS ### Options ``` -h, --help help for plugin ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican plugin stage](/commands-reference/pelican/plugin/stage) - Run pelican CLI to stage files as a HTCSS shadow plugin - [pelican plugin transfer](/commands-reference/pelican/plugin/transfer) - Run pelican CLI in HTCSS file transfer plugin mode --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/plugin/stage # commands-reference/pelican/plugin/stage ## pelican plugin stage Run pelican CLI to stage files as a HTCSS shadow plugin ``` pelican plugin stage [flags] ``` ### Options ``` -h, --help help for stage --hook Implement the HTCondor hook behavior -m, --mount string Prefix corresponding to the local mount point of the origin -o, --origin-prefix string Prefix corresponding to the local origin -s, --shadow-prefix string Prefix corresponding to the shadow origin -t, --token string Token file to use for reading and/or writing ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican plugin](/commands-reference/pelican/plugin) - Plugin management for HTCSS --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/plugin/transfer # commands-reference/pelican/plugin/transfer ## pelican plugin transfer Run pelican CLI in HTCSS file transfer plugin mode ``` pelican plugin transfer [flags] ``` ### Options ``` -h, --help help for transfer ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican plugin](/commands-reference/pelican/plugin) - Plugin management for HTCSS --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/rclone # commands-reference/pelican/rclone ## pelican rclone Commands for integrating Pelican with rclone ### Synopsis The rclone subcommands help integrate Pelican with the rclone tool for syncing files to and from Pelican federations. The ‘setup’ command generates an rclone configuration and appends it to the rclone configuration file, including automatic token renewal via the bearer\_token\_command option. The ‘install’ command downloads and installs rclone if it is not already available on the system. ### Options ``` -h, --help help for rclone ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican rclone install](/commands-reference/pelican/rclone/install) - Download and install rclone - [pelican rclone setup](/commands-reference/pelican/rclone/setup) - Generate an rclone configuration for a Pelican namespace --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/rclone/install # commands-reference/pelican/rclone/install ## pelican rclone install Download and install rclone ### Synopsis Download and install the latest stable release of rclone from [https://downloads.rclone.org](https://downloads.rclone.org) . By default the binary is placed in ~/.local/bin (or ~/bin if it exists and ~/.local/bin does not). Use —prefix to choose a different location: pelican rclone install —prefix /usr/local/bin If rclone is already installed on your PATH, the command exits with a message showing the installed version. ``` pelican rclone install [flags] ``` ### Options ``` -h, --help help for install --prefix string Directory in which to place the rclone binary (default: ~/.local/bin) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/rclone/setup # Generate config for reading from a namespace ## pelican rclone setup Generate an rclone configuration for a Pelican namespace ### Synopsis Generate an rclone configuration that can be used to sync files to and from a Pelican namespace. The generated section is automatically appended to the rclone configuration file (typically ~/.config/rclone/rclone.conf), creating the file if it does not already exist. The configuration is also printed to stdout for reference. Use —output to write to a different file instead. The configuration includes a bearer\_token\_command that automatically fetches fresh tokens when needed. When rclone receives an HTTP 401 Unauthorized response (indicating token expiry), it re-runs the bearer\_token\_command to get a fresh token and retries the request. The Pelican Director inspects bearer tokens in client requests and returns HTTP 401 if a token has expired. This triggers rclone’s automatic token refresh, ensuring seamless operation with long-running sync jobs. Examples: # Generate config for reading from a namespace pelican rclone setup —read pelican://federation.example.org/namespace/path # Generate config for reading and writing to a namespace pelican rclone setup —write pelican://federation.example.org/namespace/path # Generate config with a custom remote name pelican rclone setup —name my-pelican —write pelican://federation.example.org/namespace/path If rclone is not installed, the command will suggest running ‘pelican rclone install’ to install it. ``` pelican rclone setup [flags] ``` ### Options ``` -h, --help help for setup -n, --name string Name for the rclone remote (default: derived from namespace) --no-password-file Do not create a separate passwordless credential file for rclone -o, --output string Output file for the configuration (default: stdout) -r, --read Configure for read-only access -w, --write Configure for write access (implies read) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/token # commands-reference/pelican/token ## pelican token Interact with tokens used to interact with objects in Pelican ### Options ``` -h, --help help for token ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican](/commands-reference/pelican) - Interact with data federations - [pelican token create](/commands-reference/pelican/token/create) - Create a token --- Source: https://docs.pelicanplatform.org/commands-reference/pelican/token/create # commands-reference/pelican/token/create ## pelican token create Create a token ### Synopsis Create a signed JWT for accessing Pelican resources. The generated token is a Bearer token. To authorize requests with it, pass it to other Pelican CLI commands via the —token flag or in the HTTP Authorization header. SCOPES Scopes control what the token permits. Use the flags below to set them, or pass arbitrary values with —raw-scope. Flag WLCG profile (default) SciTokens2 profile (—profile scitokens2) * * * —read storage.read: read: —write storage.create: write: —modify storage.modify: write: —stage storage.stage: write: The in each scope is the object path with the namespace prefix stripped. Use —scope-path to override it, or —raw-scope to supply scopes verbatim, e.g.: —raw-scope “storage.read:/ storage.create:/uploads” ISSUER The issuer (—issuer) is auto-discovered from the Director using the supplied pelican URL. Provide —issuer manually when: - Director discovery is unavailable - The namespace has multiple issuers and you need a specific one EXPIRATION Set the token lifetime with —lifetime (seconds, default 1200) or with —expiration (an absolute RFC3339 timestamp, e.g. 2026-12-31T23:59:59Z). These two flags are mutually exclusive. ``` pelican token create [flags] ``` ### Examples ``` # Read/write token for a path in OSDF (issuer auto-discovered): pelican token create --read --write pelican://osg-htc.org/some/namespace/path # Token expiring at a specific time (RFC3339: YYYY-MM-DDTHH:MM:SSZ): pelican token create --read --expiration 2026-06-30T00:00:00Z \ pelican://osg-htc.org/some/namespace/path # One-hour token with an explicit issuer: pelican token create --read --lifetime 3600 \ --issuer https://my-origin.com:8443 \ pelican://osg-htc.org/some/namespace/path # Token with custom raw scopes: pelican token create --raw-scope "storage.read:/ storage.create:/uploads" \ pelican://osg-htc.org/some/namespace/path ``` ### Options ``` -a, --audience string Specify the token's 'audience/aud' claim. If not provided, the equivalent 'any' audience for the selected profile will be used (e.g. 'https://wlcg.cern.ch/jwt/v1/any' for the 'wlcg' profile). --expiration string Set the token's expiration as an absolute RFC3339 timestamp (e.g., 2026-12-31T23:59:59Z). Mutually exclusive with --lifetime. -h, --help help for create -i, --issuer string Set the token's 'issuer/iss' claim. If not provided, the issuer will be discovered via the Director. -l, --lifetime int Set the token's lifetime in seconds. (default 1200) -m, --modify Indicate the requested token should provide the ability to modify/delete the specified resource. -k, --private-key string Path to the private key used to sign the token. If not provided, Pelican will look for the private key in the default location pointed to by the '{IssuerKeysDirectory}' config parameter. -p, --profile string Create a token with a specific JWT profile. Accepted values are scitokens2 and wlcg. (default "wlcg") --raw-claim stringArray Set claims to be added to the token. Format: =. --raw-scope stringArray Set non-typical values for the token's 'scope' claim. Scopes should be space-separated, e.g. 'storage.read:/ storage.create:/'. -r, --read Indicate the requested token should provide the ability to read the specified resource. --scope-path string Specify the path to use when creating the token's scopes. This should generally be the object path without the namespace prefix. -s, --stage Indicate the requested token should provide the ability to stage the specified resource. --subject string Set token's 'subject/sub' claim. If not provided, the current user will be used as the default subject. -w, --write Indicate the requested token should provide the ability to create/write the specified resource. Does not grant the ability to overwrite/modify existing resources. ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican token](/commands-reference/pelican/token) - Interact with tokens used to interact with objects in Pelican --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server # commands-reference/pelican-server ## pelican-server Interact with data federations ### Synopsis The pelican software allows one to build and interact with data federations, enabling the sharing of objects and collections across multiple dataset providers. ### Options ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize -h, --help help for pelican-server --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server apikey](/commands-reference/pelican-server/apikey) - Manage API keys for server operations - [pelican-server cache](/commands-reference/pelican-server/cache) - Operate a Pelican cache service - [pelican-server config](/commands-reference/pelican-server/config) - View and search for configuration parameters - [pelican-server credentials](/commands-reference/pelican-server/credentials) - Interact with the credential configuration file - [pelican-server director](/commands-reference/pelican-server/director) - Launch a Pelican Director - [pelican-server downtime](/commands-reference/pelican-server/downtime) - Manage server’s own downtime periods - [pelican-server generate](/commands-reference/pelican-server/generate) - Generate credentials for Pelican server - [pelican-server key](/commands-reference/pelican-server/key) - Manage Pelican issuer keys - [pelican-server namespace](/commands-reference/pelican-server/namespace) - Work with namespaces - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service - [pelican-server registry](/commands-reference/pelican-server/registry) - Interact with a Pelican registry service - [pelican-server server](/commands-reference/pelican-server/server) - Manage server operations - [pelican-server token](/commands-reference/pelican-server/token) - Interact with tokens used to interact with objects in Pelican --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/apikey # commands-reference/pelican-server/apikey ## pelican-server apikey Manage API keys for server operations ### Synopsis Provide commands to generate and manage API keys for Pelican servers. ### Options ``` -h, --help help for apikey -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server apikey generate](/commands-reference/pelican-server/apikey/generate) - Generate a new API key for the server --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/apikey/generate # commands-reference/pelican-server/apikey/generate ## pelican-server apikey generate Generate a new API key for the server ### Synopsis Generate a new API key with specified scopes. ``` pelican-server apikey generate [flags] ``` ### Options ``` --expiration string Expiration time in RFC3339 format -h, --help help for generate --name string Name for the API key (defaults to cli-generated-{timestamp}) --scopes string Comma-separated list of scopes (e.g., monitoring.query,monitoring.scrape) (required) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file --version Print the version and exit ``` ### SEE ALSO - [pelican-server apikey](/commands-reference/pelican-server/apikey) - Manage API keys for server operations --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/cache # commands-reference/pelican-server/cache ## pelican-server cache Operate a Pelican cache service ### Options ``` -h, --help help for cache ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server cache serve](/commands-reference/pelican-server/cache/serve) - Start the cache service --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/cache/serve # commands-reference/pelican-server/cache/serve ## pelican-server cache serve Start the cache service ``` pelican-server cache serve [flags] ``` ### Options ``` -h, --help help for serve -p, --port uint16 Set the port at which the web server should be accessible ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server cache](/commands-reference/pelican-server/cache) - Operate a Pelican cache service --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/config # commands-reference/pelican-server/config ## pelican-server config View and search for configuration parameters ### Synopsis The ‘config’ command allows users to view, search, and see the documentation for various configuration parameters in the Pelican system. ### Options ``` -h, --help help for config ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server config describe](/commands-reference/pelican-server/config/describe) - Print documentation for the specified config parameter - [pelican-server config dump](/commands-reference/pelican-server/config/dump) - Dump all configuration parameters - [pelican-server config get](/commands-reference/pelican-server/config/get) - Retrieve config parameters that match any of the given arguments - [pelican-server config summary](/commands-reference/pelican-server/config/summary) - Print config parameters that differ from default values --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/config/describe # commands-reference/pelican-server/config/describe ## pelican-server config describe Print documentation for the specified config parameter ### Synopsis The ‘describe’ command prints detailed documentation for a specified configuration parameter, including its type, default value, description, related components, and whether it is deprecated or hidden. ``` pelican-server config describe [parameter] [flags] ``` ### Examples ``` # View documentation for the Server.WebPort parameter pelican config describe server.webPort ``` ### Options ``` -h, --help help for describe ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server config](/commands-reference/pelican-server/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/config/dump # commands-reference/pelican-server/config/dump ## pelican-server config dump Dump all configuration parameters ### Synopsis The ‘dump’ command outputs all current configuration parameters and their values to the console. This includes default values that have not been explicitly set. ``` pelican-server config dump [flags] ``` ### Examples ``` # Dump all configuration parameters pelican config dump ``` ### Options ``` -o, --format string Output format (yaml or json) (default "yaml") -h, --help help for dump ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server config](/commands-reference/pelican-server/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/config/get # commands-reference/pelican-server/config/get ## pelican-server config get Retrieve config parameters that match any of the given arguments ### Synopsis The ‘get’ command retrieves and displays configuration parameters that contain any of the provided argument patterns in their name or value. The search space can be narrowed or expanded using available flags. The matching is case-insensitive. If no arguments are provided, all configuration parameters are retrieved. The command outputs the results in a flattened format from the nested configuration, making it grep-friendly for easier searching. ``` pelican-server config get [arguments] [flags] ``` ### Examples ``` # Retrieve parameters that have either 'log' or 'monitor' in their name or value, # and relate to either 'origin' or 'cache', including deprecated parameters in the search space pelican config get log monitor -m origin -m cache --include-deprecated ``` ### Options ``` --exact-match Match configuration parameter names exactly instead of using substring matching -h, --help help for get --include-deprecated Include deprecated configuration parameters --include-hidden Include hidden configuration parameters -m, --module config get Specify modules to filter the output of config get. The recognized modules are `client`, `registry`, `director`, `origin`, `cache`, and `localcache`. Multiple modules can be specified at the same time, for example: `config get -m cache -m origin`. If multiple modules are provided, parameters related to any of the modules will be retrieved. If no modules are specified, no module-based filter is applied to the search space. ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server config](/commands-reference/pelican-server/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/config/summary # commands-reference/pelican-server/config/summary ## pelican-server config summary Print config parameters that differ from default values ### Synopsis The ‘summary’ command outputs configuration parameters whose values differ from their default settings. ``` pelican-server config summary [flags] ``` ### Examples ``` # Show configuration parameters that are set differently from their default values pelican config summary ``` ### Options ``` -o, --format string Output format (yaml or json) (default "yaml") -h, --help help for summary ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server config](/commands-reference/pelican-server/config) - View and search for configuration parameters --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials # commands-reference/pelican-server/credentials ## pelican-server credentials Interact with the credential configuration file ### Options ``` -h, --help help for credentials ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server credentials prefix](/commands-reference/pelican-server/credentials/prefix) - Manage the prefix configuration - [pelican-server credentials print](/commands-reference/pelican-server/credentials/print) - Print the credential configuration file - [pelican-server credentials replace](/commands-reference/pelican-server/credentials/replace) - Replace the credential configuration file - [pelican-server credentials reset-local](/commands-reference/pelican-server/credentials/reset-local) - Delete all local credentials for the current user - [pelican-server credentials reset-password](/commands-reference/pelican-server/credentials/reset-password) - Reset the password for the current user - [pelican-server credentials token](/commands-reference/pelican-server/credentials/token) - Manage the available tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/prefix # commands-reference/pelican-server/credentials/prefix ## pelican-server credentials prefix Manage the prefix configuration ### Synopsis Manage the prefix configuration ### Options ``` -h, --help help for prefix ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials](/commands-reference/pelican-server/credentials) - Interact with the credential configuration file - [pelican-server credentials prefix add](/commands-reference/pelican-server/credentials/prefix/add) - Add a new oauth client - [pelican-server credentials prefix delete](/commands-reference/pelican-server/credentials/prefix/delete) - Delete the oauth client - [pelican-server credentials prefix print](/commands-reference/pelican-server/credentials/prefix/print) - Print the oauth client configuration file - [pelican-server credentials prefix set](/commands-reference/pelican-server/credentials/prefix/set) - Set the oauth client attributes --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/prefix/add # commands-reference/pelican-server/credentials/prefix/add ## pelican-server credentials prefix add Add a new oauth client ### Synopsis Add a new oauth client ``` pelican-server credentials prefix add [flags] ``` ### Options ``` -h, --help help for add ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials prefix](/commands-reference/pelican-server/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/prefix/delete # commands-reference/pelican-server/credentials/prefix/delete ## pelican-server credentials prefix delete Delete the oauth client ### Synopsis Delete the oauth client ``` pelican-server credentials prefix delete [flags] ``` ### Options ``` -h, --help help for delete ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials prefix](/commands-reference/pelican-server/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/prefix/print # commands-reference/pelican-server/credentials/prefix/print ## pelican-server credentials prefix print Print the oauth client configuration file ### Synopsis Print the oauth client configuration file ``` pelican-server credentials prefix print [flags] ``` ### Options ``` -h, --help help for print ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials prefix](/commands-reference/pelican-server/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/prefix/set # commands-reference/pelican-server/credentials/prefix/set ## pelican-server credentials prefix set Set the oauth client attributes ### Synopsis Set the oauth client attributes (client\_id or client\_secret) ``` pelican-server credentials prefix set [flags] ``` ### Options ``` -h, --help help for set ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials prefix](/commands-reference/pelican-server/credentials/prefix) - Manage the prefix configuration --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/print # commands-reference/pelican-server/credentials/print ## pelican-server credentials print Print the credential configuration file ### Synopsis Print the credential configuration file ``` pelican-server credentials print [flags] ``` ### Options ``` -h, --help help for print ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials](/commands-reference/pelican-server/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/replace # commands-reference/pelican-server/credentials/replace ## pelican-server credentials replace Replace the credential configuration file ### Synopsis Replace the credential configuration file ``` pelican-server credentials replace [flags] ``` ### Options ``` -h, --help help for replace ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials](/commands-reference/pelican-server/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/reset-local # commands-reference/pelican-server/credentials/reset-local ## pelican-server credentials reset-local Delete all local credentials for the current user ### Synopsis Delete all local credentials for the current user ``` pelican-server credentials reset-local [flags] ``` ### Options ``` -h, --help help for reset-local ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials](/commands-reference/pelican-server/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/reset-password # commands-reference/pelican-server/credentials/reset-password ## pelican-server credentials reset-password Reset the password for the current user ### Synopsis Reset the password for the current user ``` pelican-server credentials reset-password [flags] ``` ### Options ``` -h, --help help for reset-password ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials](/commands-reference/pelican-server/credentials) - Interact with the credential configuration file --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/token # commands-reference/pelican-server/credentials/token ## pelican-server credentials token Manage the available tokens ### Synopsis Manage the available tokens ### Options ``` -h, --help help for token ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials](/commands-reference/pelican-server/credentials) - Interact with the credential configuration file - [pelican-server credentials token get](/commands-reference/pelican-server/credentials/token/get) - Get a new token for a given prefix - [pelican-server credentials token setup](/commands-reference/pelican-server/credentials/token/setup) - Set up a credential file containing tokens for a Pelican namespace --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/token/get # commands-reference/pelican-server/credentials/token/get ## pelican-server credentials token get Get a new token for a given prefix ### Synopsis Get a new token for a given prefix ``` pelican-server credentials token get [flags] ``` ### Options ``` -h, --help help for get ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials token](/commands-reference/pelican-server/credentials/token) - Manage the available tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/credentials/token/setup # Set up credentials for reading from a namespace ## pelican-server credentials token setup Set up a credential file containing tokens for a Pelican namespace ### Synopsis Acquire a token for the specified Pelican namespace and save it to a credential file on disk. The credential file contains the access token, refresh token, and OAuth2 client credentials needed to obtain fresh tokens later without re-authenticating. By default, the credential file is password-protected. Use —no-password to save the file without encryption, which is useful for non-interactive contexts where password prompts would fail. Use —credential-file to specify an alternative path for the credential file. Examples: # Set up credentials for reading from a namespace pelican credentials token setup —read pelican://federation.example.org/namespace/path # Set up credentials for reading and writing pelican credentials token setup —write pelican://federation.example.org/namespace/path # Set up credentials without password protection pelican credentials token setup —no-password —read pelican://federation.example.org/namespace/path # Set up credentials to a specific file pelican credentials token setup —credential-file /path/to/creds.pem —read pelican://federation.example.org/namespace/path ``` pelican-server credentials token setup [flags] ``` ### Options ``` --credential-file string Path to the credential file to write -h, --help help for setup --no-password Save the credential file without password protection -r, --read Request a read token -w, --write Request a write token (implies read) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server credentials token](/commands-reference/pelican-server/credentials/token) - Manage the available tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/director # commands-reference/pelican-server/director ## pelican-server director Launch a Pelican Director ### Synopsis Launch a Pelican Director service: The Pelican Director is the primary mechanism by which clients/caches can discover the source of a requested resource. It has two endpoints at /api/v1.0/director/origin/ and /api/v1.0/director/object/, where the former redirects to the closest origin supporting the object and the latter redirects to the closest cache. As a shortcut, requests to the director at /foo/bar will be treated as a request for the object from cache. ### Options ``` -h, --help help for director ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server director serve](/commands-reference/pelican-server/director/serve) - serve the director service --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/director/serve # commands-reference/pelican-server/director/serve ## pelican-server director serve serve the director service ``` pelican-server director serve [flags] ``` ### Options ``` --default-response string Set whether the default endpoint should redirect clients to caches or origins -h, --help help for serve -p, --port uint16 Set the port at which the web server should be accessible ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server director](/commands-reference/pelican-server/director) - Launch a Pelican Director --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/downtime # commands-reference/pelican-server/downtime ## pelican-server downtime Manage server’s own downtime periods ### Synopsis Provide commands to list, create, update, and delete scheduled downtime periods for Pelican servers (Origins/Caches). These commands interact with the server’s administrative API endpoint. ### Options ``` -h, --help help for downtime -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server downtime create](/commands-reference/pelican-server/downtime/create) - Create a new downtime period for the server - [pelican-server downtime delete](/commands-reference/pelican-server/downtime/delete) - Delete a downtime period - [pelican-server downtime list](/commands-reference/pelican-server/downtime/list) - List server’s scheduled downtime periods - [pelican-server downtime update](/commands-reference/pelican-server/downtime/update) - Update an existing downtime period --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/downtime/create # commands-reference/pelican-server/downtime/create ## pelican-server downtime create Create a new downtime period for the server ### Synopsis Interactively prompt for downtime fields and send a POST request to create a new downtime period. ``` pelican-server downtime create [flags] ``` ### Options ``` -h, --help help for create ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file --version Print the version and exit ``` ### SEE ALSO - [pelican-server downtime](/commands-reference/pelican-server/downtime) - Manage server’s own downtime periods --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/downtime/delete # commands-reference/pelican-server/downtime/delete ## pelican-server downtime delete Delete a downtime period ### Synopsis Delete the specified downtime period by UUID. Sends a DELETE request to the downtime API. ``` pelican-server downtime delete [uuid] [flags] ``` ### Options ``` -h, --help help for delete ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file --version Print the version and exit ``` ### SEE ALSO - [pelican-server downtime](/commands-reference/pelican-server/downtime) - Manage server’s own downtime periods --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/downtime/list # commands-reference/pelican-server/downtime/list ## pelican-server downtime list List server’s scheduled downtime periods ### Synopsis List scheduled downtime periods for a Pelican server (Origin/Cache). Requires an administrative token for the server. Shows active and future downtimes (‘incomplete’) by default. ``` pelican-server downtime list [flags] ``` ### Options ``` -h, --help help for list --status string Filter downtimes by status ('incomplete' shows active/future, 'all' shows all history) (default "incomplete") ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file --version Print the version and exit ``` ### SEE ALSO - [pelican-server downtime](/commands-reference/pelican-server/downtime) - Manage server’s own downtime periods --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/downtime/update # commands-reference/pelican-server/downtime/update ## pelican-server downtime update Update an existing downtime period ### Synopsis Interactively prompt for downtime fields and send a PUT request to update the specified downtime period. Press Enter without typing anything to leave a field unchanged. ``` pelican-server downtime update [uuid] [flags] ``` ### Options ``` -h, --help help for update ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file --version Print the version and exit ``` ### SEE ALSO - [pelican-server downtime](/commands-reference/pelican-server/downtime) - Manage server’s own downtime periods --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/generate # commands-reference/pelican-server/generate ## pelican-server generate Generate credentials for Pelican server ### Options ``` -h, --help help for generate ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server generate password](/commands-reference/pelican-server/generate/password) - Generate a Pelican admin website password file (htpasswd) --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/generate/password # commands-reference/pelican-server/generate/password ## pelican-server generate password Generate a Pelican admin website password file (htpasswd) ### Synopsis Given a password for the admin website, generate the htpasswd file that Pelican server uses to store the password and authenticate the admin user. You may put the generated file under /etc/pelican with name “server-web-passwd”, or change Server.UIPasswordFile to the path to generated file to initialize the admin website. ``` pelican-server generate password [flags] ``` ### Options ``` -h, --help help for password -o, --output string The path to the generate htpasswd password file. Default: ./server-web-passwd -p, --password string The path to the file containing the password. Will take from terminal input if not provided ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server generate](/commands-reference/pelican-server/generate) - Generate credentials for Pelican server --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/key # commands-reference/pelican-server/key ## pelican-server key Manage Pelican issuer keys ### Options ``` -h, --help help for key ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server key create](/commands-reference/pelican-server/key/create) - Generate a public-private key-pair for Pelican server --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/key/create # commands-reference/pelican-server/key/create ## pelican-server key create Generate a public-private key-pair for Pelican server ### Synopsis Generate a public-private key-pair for a Pelican server. The private key is an ECDSA key with P256 curve. The corresponding public key is a JSON Web Key Set (JWKS), which can be used for JWT signature verification. ``` pelican-server key create [flags] ``` ### Options ``` -h, --help help for create --private-key string The file path where the generated private key will be saved. If a key already exists at the provided path, it will not be overwritten but will be used to derive a public key (default "./private-key.pem") --public-key string The file path where the generated public key (derived from the generated private key) will be saved. (default "./issuer-pub.jwks") ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server key](/commands-reference/pelican-server/key) - Manage Pelican issuer keys --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/namespace # commands-reference/pelican-server/namespace ## pelican-server namespace Work with namespaces ### Options ``` -h, --help help for namespace --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server namespace delete](/commands-reference/pelican-server/namespace/delete) - Delete a namespace - [pelican-server namespace list](/commands-reference/pelican-server/namespace/list) - List all namespaces - [pelican-server namespace register](/commands-reference/pelican-server/namespace/register) - Register a new namespace --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/namespace/delete # commands-reference/pelican-server/namespace/delete ## pelican-server namespace delete Delete a namespace ``` pelican-server namespace delete [flags] ``` ### Options ``` -h, --help help for delete --prefix string prefix for delete namespace ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key --version Print the version and exit ``` ### SEE ALSO - [pelican-server namespace](/commands-reference/pelican-server/namespace) - Work with namespaces --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/namespace/list # commands-reference/pelican-server/namespace/list ## pelican-server namespace list List all namespaces ``` pelican-server namespace list [flags] ``` ### Options ``` -h, --help help for list ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key --version Print the version and exit ``` ### SEE ALSO - [pelican-server namespace](/commands-reference/pelican-server/namespace) - Work with namespaces --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/namespace/register # commands-reference/pelican-server/namespace/register ## pelican-server namespace register Register a new namespace ``` pelican-server namespace register [flags] ``` ### Options ``` -h, --help help for register --prefix string prefix for registering namespace --with-identity Register a namespace with an identity ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace-url string Endpoint for the namespace registry --privkey string Path to the private key --pubkey string Path to the public key --version Print the version and exit ``` ### SEE ALSO - [pelican-server namespace](/commands-reference/pelican-server/namespace) - Work with namespaces --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin # commands-reference/pelican-server/origin ## pelican-server origin Operate a Pelican origin service ### Options ``` -h, --help help for origin ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin - [pelican-server origin config](/commands-reference/pelican-server/origin/config) - Launch the Pelican web service in configuration mode - [pelican-server origin issuer](/commands-reference/pelican-server/origin/issuer) - Manage the origin’s embedded OIDC token issuer - [pelican-server origin serve](/commands-reference/pelican-server/origin/serve) - Start the origin service - [pelican-server origin token](/commands-reference/pelican-server/origin/token) - Manage Pelican origin tokens - [pelican-server origin web-ui](/commands-reference/pelican-server/origin/web-ui) - Manage the Pelican origin web UI --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection # commands-reference/pelican-server/origin/collection ## pelican-server origin collection Manage collections on a Pelican origin ### Synopsis Manage collections on a Pelican origin server using OAuth2 authentication ### Options ``` -h, --help help for collection ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service - [pelican-server origin collection acl](/commands-reference/pelican-server/origin/collection/acl) - Manage collection ACLs - [pelican-server origin collection create](/commands-reference/pelican-server/origin/collection/create) - Create a new collection - [pelican-server origin collection delete](/commands-reference/pelican-server/origin/collection/delete) - Delete a collection - [pelican-server origin collection get](/commands-reference/pelican-server/origin/collection/get) - Get a collection by ID - [pelican-server origin collection list](/commands-reference/pelican-server/origin/collection/list) - List all collections - [pelican-server origin collection metadata](/commands-reference/pelican-server/origin/collection/metadata) - Manage collection metadata - [pelican-server origin collection update](/commands-reference/pelican-server/origin/collection/update) - Update a collection --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/acl # commands-reference/pelican-server/origin/collection/acl ## pelican-server origin collection acl Manage collection ACLs ### Options ``` -h, --help help for acl ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin - [pelican-server origin collection acl grant](/commands-reference/pelican-server/origin/collection/acl/grant) - Grant an ACL to a collection - [pelican-server origin collection acl list](/commands-reference/pelican-server/origin/collection/acl/list) - List all ACLs for a collection - [pelican-server origin collection acl revoke](/commands-reference/pelican-server/origin/collection/acl/revoke) - Revoke an ACL from a collection --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/acl/grant # commands-reference/pelican-server/origin/collection/acl/grant ## pelican-server origin collection acl grant Grant an ACL to a collection ``` pelican-server origin collection acl grant [flags] ``` ### Options ``` --expires-at string Expiration time in RFC3339 format (e.g., 2006-01-02T15:04:05Z07:00) --group-id string Group ID for the ACL (required) -h, --help help for grant --role string Role for the ACL (required) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection acl](/commands-reference/pelican-server/origin/collection/acl) - Manage collection ACLs --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/acl/list # commands-reference/pelican-server/origin/collection/acl/list ## pelican-server origin collection acl list List all ACLs for a collection ``` pelican-server origin collection acl list [flags] ``` ### Options ``` -h, --help help for list ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection acl](/commands-reference/pelican-server/origin/collection/acl) - Manage collection ACLs --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/acl/revoke # commands-reference/pelican-server/origin/collection/acl/revoke ## pelican-server origin collection acl revoke Revoke an ACL from a collection ``` pelican-server origin collection acl revoke [flags] ``` ### Options ``` --group-id string Group ID for the ACL (required) -h, --help help for revoke --role string Role for the ACL (required) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection acl](/commands-reference/pelican-server/origin/collection/acl) - Manage collection ACLs --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/create # commands-reference/pelican-server/origin/collection/create ## pelican-server origin collection create Create a new collection ``` pelican-server origin collection create [flags] ``` ### Options ``` --description string Collection description -h, --help help for create --metadata strings Metadata as key=value pairs (can be specified multiple times) --name string Collection name (required) --namespace string Collection namespace (required) --visibility string Collection visibility (private|public) (default "private") ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/delete # commands-reference/pelican-server/origin/collection/delete ## pelican-server origin collection delete Delete a collection ``` pelican-server origin collection delete [flags] ``` ### Options ``` -h, --help help for delete ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/get # commands-reference/pelican-server/origin/collection/get ## pelican-server origin collection get Get a collection by ID ``` pelican-server origin collection get [flags] ``` ### Options ``` -h, --help help for get ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/list # commands-reference/pelican-server/origin/collection/list ## pelican-server origin collection list List all collections ``` pelican-server origin collection list [flags] ``` ### Options ``` -h, --help help for list ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/metadata # commands-reference/pelican-server/origin/collection/metadata ## pelican-server origin collection metadata Manage collection metadata ### Options ``` -h, --help help for metadata ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin - [pelican-server origin collection metadata delete](/commands-reference/pelican-server/origin/collection/metadata/delete) - Delete a metadata key from a collection - [pelican-server origin collection metadata get](/commands-reference/pelican-server/origin/collection/metadata/get) - Get all metadata for a collection - [pelican-server origin collection metadata set](/commands-reference/pelican-server/origin/collection/metadata/set) - Set a metadata key-value pair for a collection --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/metadata/delete # commands-reference/pelican-server/origin/collection/metadata/delete ## pelican-server origin collection metadata delete Delete a metadata key from a collection ``` pelican-server origin collection metadata delete [flags] ``` ### Options ``` -h, --help help for delete ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection metadata](/commands-reference/pelican-server/origin/collection/metadata) - Manage collection metadata --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/metadata/get # commands-reference/pelican-server/origin/collection/metadata/get ## pelican-server origin collection metadata get Get all metadata for a collection ``` pelican-server origin collection metadata get [flags] ``` ### Options ``` -h, --help help for get ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection metadata](/commands-reference/pelican-server/origin/collection/metadata) - Manage collection metadata --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/metadata/set # commands-reference/pelican-server/origin/collection/metadata/set ## pelican-server origin collection metadata set Set a metadata key-value pair for a collection ``` pelican-server origin collection metadata set [flags] ``` ### Options ``` -h, --help help for set ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection metadata](/commands-reference/pelican-server/origin/collection/metadata) - Manage collection metadata --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/collection/update # commands-reference/pelican-server/origin/collection/update ## pelican-server origin collection update Update a collection ``` pelican-server origin collection update [flags] ``` ### Options ``` --description string New collection description -h, --help help for update --name string New collection name --visibility string New collection visibility (private|public) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin collection](/commands-reference/pelican-server/origin/collection) - Manage collections on a Pelican origin --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/config # commands-reference/pelican-server/origin/config ## pelican-server origin config Launch the Pelican web service in configuration mode ``` pelican-server origin config [flags] ``` ### Options ``` -h, --help help for config ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/issuer # commands-reference/pelican-server/origin/issuer ## pelican-server origin issuer Manage the origin’s embedded OIDC token issuer ### Options ``` -h, --help help for issuer ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service - [pelican-server origin issuer client](/commands-reference/pelican-server/origin/issuer/client) - Manage OIDC clients for the origin’s embedded issuer --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/issuer/client # commands-reference/pelican-server/origin/issuer/client ## pelican-server origin issuer client Manage OIDC clients for the origin’s embedded issuer ### Synopsis Provides commands to create, list, update, and delete OIDC clients configured on the origin’s embedded token issuer. ### Options ``` -h, --help help for client --namespace string Federation namespace prefix for the issuer (e.g. /data/analysis) (required) --server string Web URL of the Pelican origin server (e.g. https://my-origin:8447) --token string Path to a file containing an admin token (optional; generated automatically if omitted) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin issuer](/commands-reference/pelican-server/origin/issuer) - Manage the origin’s embedded OIDC token issuer - [pelican-server origin issuer client create](/commands-reference/pelican-server/origin/issuer/client/create) - Create a new OIDC client - [pelican-server origin issuer client delete](/commands-reference/pelican-server/origin/issuer/client/delete) - Delete an OIDC client - [pelican-server origin issuer client list](/commands-reference/pelican-server/origin/issuer/client/list) - List all OIDC clients - [pelican-server origin issuer client update](/commands-reference/pelican-server/origin/issuer/client/update) - Update an existing OIDC client --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/issuer/client/create # commands-reference/pelican-server/origin/issuer/client/create ## pelican-server origin issuer client create Create a new OIDC client ### Synopsis Create a new OIDC client on the origin’s embedded issuer. The —grant-types flag controls which OAuth2 flows the client can use. Accepted values (comma-separated): authorization\_code refresh\_token urn:ietf:params:oauth:grant-type:device\_code urn:ietf:params:oauth:grant-type:token-exchange Example — create a token-exchange client: pelican-server origin issuer client create —server [https://my-origin:8447](https://my-origin:8447)  —grant-types “urn:ietf:params:oauth:grant-type:token-exchange,refresh\_token” ``` pelican-server origin issuer client create [flags] ``` ### Options ``` --grant-types string Comma-separated list of grant types (required) -h, --help help for create --scopes string Comma-separated list of scopes (optional; defaults to common WLCG scopes) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace string Federation namespace prefix for the issuer (e.g. /data/analysis) (required) --server string Web URL of the Pelican origin server (e.g. https://my-origin:8447) --token string Path to a file containing an admin token (optional; generated automatically if omitted) --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin issuer client](/commands-reference/pelican-server/origin/issuer/client) - Manage OIDC clients for the origin’s embedded issuer --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/issuer/client/delete # commands-reference/pelican-server/origin/issuer/client/delete ## pelican-server origin issuer client delete Delete an OIDC client ``` pelican-server origin issuer client delete [flags] ``` ### Options ``` -h, --help help for delete --id string Client ID to delete (required) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace string Federation namespace prefix for the issuer (e.g. /data/analysis) (required) --server string Web URL of the Pelican origin server (e.g. https://my-origin:8447) --token string Path to a file containing an admin token (optional; generated automatically if omitted) --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin issuer client](/commands-reference/pelican-server/origin/issuer/client) - Manage OIDC clients for the origin’s embedded issuer --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/issuer/client/list # commands-reference/pelican-server/origin/issuer/client/list ## pelican-server origin issuer client list List all OIDC clients ``` pelican-server origin issuer client list [flags] ``` ### Options ``` -h, --help help for list ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace string Federation namespace prefix for the issuer (e.g. /data/analysis) (required) --server string Web URL of the Pelican origin server (e.g. https://my-origin:8447) --token string Path to a file containing an admin token (optional; generated automatically if omitted) --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin issuer client](/commands-reference/pelican-server/origin/issuer/client) - Manage OIDC clients for the origin’s embedded issuer --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/issuer/client/update # commands-reference/pelican-server/origin/issuer/client/update ## pelican-server origin issuer client update Update an existing OIDC client ### Synopsis Update the configuration of an existing OIDC client. Only the flags you provide are changed; omitted fields are left unchanged. Example — add token-exchange grant and narrow scopes: pelican-server origin issuer client update —server [https://my-origin:8447](https://my-origin:8447)  —id —grant-types “urn:ietf:params:oauth:grant-type:token-exchange,refresh\_token” —scopes “openid,storage.read:/“ ``` pelican-server origin issuer client update [flags] ``` ### Options ``` --grant-types string Comma-separated list of grant types -h, --help help for update --id string Client ID to update (required) --scopes string Comma-separated list of scopes ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --namespace string Federation namespace prefix for the issuer (e.g. /data/analysis) (required) --server string Web URL of the Pelican origin server (e.g. https://my-origin:8447) --token string Path to a file containing an admin token (optional; generated automatically if omitted) --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin issuer client](/commands-reference/pelican-server/origin/issuer/client) - Manage OIDC clients for the origin’s embedded issuer --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/serve # commands-reference/pelican-server/origin/serve ## pelican-server origin serve Start the origin service ``` pelican-server origin serve [flags] ``` ### Options ``` --bucket-access-keyfile string Specify a filepath to use for configuring the bucket's access key. --bucket-secret-keyfile string Specify a filepath to use for configuring the bucket's access key. -h, --help help for serve --http-service-url string Specify the http(s) service-url. Only used when an origin is launched in https/globus modes. --http-tokenfile string Specify a filepath to use for configuring the http(s) token. See documentation for details. -m, --mode string Set the mode for the origin service (default is 'posix'). Supported modes are 'posix', 's3, 'https', 'globus' and 'xroot'. (default "posix") -p, --port uint16 Set the port at which the web server should be accessible --region string Specify the S3 region. Only used when an origin is launched in S3 mode. --service-url string Specify the S3 service-url. Only used when an origin is launched in S3 mode. --url-style string Specify the S3 url-style. Only used when an origin is launched in S3 mode, and can be either 'path' (default) or 'virtual. -v, --volume strings Setting the volume to /SRC:/DEST will export the contents of /SRC as /DEST in the Pelican federation --writeable Allow/disable writing to the origin (default true) --xroot-service-url string When configured in xroot mode, specifies the hostname and port of the upstream xroot server (not to be mistaken with the current server's hostname). ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/ssh-auth # Interactive login via WebSocket (auto-detects local origin) ## pelican-server origin ssh-auth SSH authentication tools for the SSH backend ### Synopsis Tools for SSH backend authentication and testing. Sub-commands: login - Interactive keyboard-interactive authentication via WebSocket test - Test SSH connection, binary upload, and helper lifecycle status - Check SSH connection status For the ‘login’ and ‘status’ commands, if —origin is not specified, the command will auto-detect the origin URL from the pelican.addresses file (for local origins) or from the configuration file. Example: # Interactive login via WebSocket (auto-detects local origin) pelican-server origin ssh-auth login # Interactive login to a specific origin pelican-server origin ssh-auth login —origin [https://origin.example.com](https://origin.example.com)  # Check the SSH connection status (auto-detects local origin) pelican-server origin ssh-auth status # Test SSH connectivity (similar to ssh command) pelican-server origin ssh-auth test storage.example.com pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) -i ~/.ssh/id\_rsa ### Options ``` -h, --help help for ssh-auth ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service - [pelican-server origin ssh-auth login](/commands-reference/pelican-server/origin/ssh-auth/login) - Interactive keyboard-interactive authentication via WebSocket - [pelican-server origin ssh-auth status](/commands-reference/pelican-server/origin/ssh-auth/status) - Check SSH connection status of an origin - [pelican-server origin ssh-auth test](/commands-reference/pelican-server/origin/ssh-auth/test) - Test SSH POSIXv2 connection and helper lifecycle --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/ssh-auth/login # commands-reference/pelican-server/origin/ssh-auth/login ## pelican-server origin ssh-auth login Interactive keyboard-interactive authentication via WebSocket ### Synopsis Connect to an origin’s SSH backend via WebSocket to complete keyboard-interactive authentication challenges from your terminal. This is useful when the origin needs to authenticate to a remote SSH server that requires keyboard-interactive authentication (e.g., 2FA, OTP). If —origin is not specified, the command will try to determine the origin URL from the pelican.addresses file (for local origins) or the configuration. Example: pelican-server origin ssh-auth login pelican-server origin ssh-auth login —origin [https://origin.example.com](https://origin.example.com)  pelican-server origin ssh-auth login —origin [https://origin.example.com](https://origin.example.com)  —host storage.internal ``` pelican-server origin ssh-auth login [flags] ``` ### Options ``` -h, --help help for login --host string SSH host to authenticate (optional, uses default if not specified) --origin string Origin URL to connect to (auto-detected if not specified) --token string Path to a file containing an admin token (auto-generated if not specified) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/ssh-auth/status # commands-reference/pelican-server/origin/ssh-auth/status ## pelican-server origin ssh-auth status Check SSH connection status of an origin ### Synopsis Query the SSH connection status of an origin’s SSH backend. If —origin is not specified, the command will try to determine the origin URL from the pelican.addresses file (for local origins) or the configuration. Example: pelican-server origin ssh-auth status pelican-server origin ssh-auth status —origin [https://origin.example.com](https://origin.example.com)  ``` pelican-server origin ssh-auth status [flags] ``` ### Options ``` -h, --help help for status --origin string Origin URL to check (auto-detected if not specified) --token string Path to a file containing an admin token (auto-generated if not specified) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/ssh-auth/test # Test with default settings (uses SSH agent or default keys) ## pelican-server origin ssh-auth test Test SSH POSIXv2 connection and helper lifecycle ### Synopsis Test an SSH connection to a remote server, upload the Pelican binary, start the helper process, verify keepalives work, and demonstrate clean shutdown. This command allows testing the SSH POSIXv2 backend without running a full origin. It’s useful for verifying SSH connectivity and authentication before deploying. The destination can be specified as \[user@\]host, similar to the ssh command. If user is not specified, it defaults to the current OS username. The known\_hosts file defaults to ~/.ssh/known\_hosts. Example: # Test with default settings (uses SSH agent or default keys) pelican-server origin ssh-auth test storage.example.com # Test with explicit username pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) # Test with specific private key pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) —private-key ~/.ssh/id\_rsa # Test with password authentication pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) —password-file /path/to/password.txt # Test with keyboard-interactive authentication only (disable agent/keys) pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) —auth-methods keyboard-interactive # Test with specific auth methods in order pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) —auth-methods agent,keyboard-interactive # Connect through a jump host (ProxyJump) pelican-server origin ssh-auth test internal-server -J bastion.example.com # Connect through a jump host with explicit user pelican-server origin ssh-auth test pelican@internal-server -J [admin@bastion.example.com](mailto:admin@bastion.example.com) # Chained jump hosts pelican-server origin ssh-auth test pelican@internal-server -J jump1.example.com,jump2.example.com # Quick connectivity test without starting the helper pelican-server origin ssh-auth test [pelican@storage.example.com](mailto:pelican@storage.example.com) —connect-only ``` pelican-server origin ssh-auth test [user@]host [flags] ``` ### Options ``` --auth-method string Single authentication method (deprecated, use --auth-methods) --auth-methods string Comma-separated list of auth methods to try: agent,publickey,password,keyboard-interactive --connect-only Only test connectivity, don't start the helper -h, --help help for test -J, --jump string Jump host(s) for ProxyJump ([user@]host[:port], comma-separated for chaining) --keepalive-count int Number of keepalive cycles to verify before shutdown (default 3) --keepalive-interval duration Keepalive interval for testing (default 5s) -o, --known-hosts string Path to known_hosts file (default: ~/.ssh/known_hosts) --password-file string Path to file containing SSH password --pelican-binary string Path to local Pelican binary to upload (defaults to current binary) -p, --port int SSH port (default 22) -i, --private-key string Path to SSH private key --private-key-passphrase-file string Path to file containing private key passphrase --remote-binary string Path to pre-built binary for remote platform (os/arch=/path or just /path for auto-detect) --remote-dir string Remote directory for Pelican binary (default "/tmp/pelican-test") -l, --user string SSH username (overrides user@host) ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/token # commands-reference/pelican-server/origin/token ## pelican-server origin token Manage Pelican origin tokens ### Options ``` -h, --help help for token --profile string Passing a profile ensures the token adheres to the profile's requirements. Accepted values are scitokens2 and wlcg (default "wlcg") ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service - [pelican-server origin token create](/commands-reference/pelican-server/origin/token/create) - Create a Pelican origin token - [pelican-server origin token verify](/commands-reference/pelican-server/origin/token/verify) - Verify a Pelican origin token --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/token/create # commands-reference/pelican-server/origin/token/create ## pelican-server origin token create Create a Pelican origin token ### Synopsis Create a JSON web token (JWT) using the origin’s signing keys: Usage: pelican-server origin token create \[FLAGS\] claims E.g. pelican-server origin token create —profile scitokens2 aud=my-audience scope=“read:/storage” scope=“write:/storage” Pelican origins use JWTs as bearer tokens for authorizing specific requests, such as reading from or writing to the origin’s underlying storage, advertising to a director, etc. For more information about the makeup of a JWT, see [https://jwt.io/introduction](https://jwt.io/introduction) . Additional profiles that expand on JWT are supported. They include scitokens2 and wlcg. For more information about these profiles, see [https://scitokens.org/technical\_docs/Claims](https://scitokens.org/technical_docs/Claims)  and [https://github.com/WLCG-AuthZ-WG/common-jwt-profile/blob/master/profile.md](https://github.com/WLCG-AuthZ-WG/common-jwt-profile/blob/master/profile.md) , respectively ``` pelican-server origin token create [flags] ``` ### Options ``` --audience strings The token's intended audience. --claim strings Additional token claims. A claim must be of the form = -h, --help help for create --issuer string The URL of the token's issuer. If not provided, the tool will attempt to find one in the configuration file. --lifetime int The lifetime of the token, in seconds. (default 1200) --private-key string Filepath designating the location of the private key in PEM format to be used for signing, if different from the origin's default. --scope strings Scopes for granting fine-grained permissions to the token. --subject string The token's subject. ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --profile string Passing a profile ensures the token adheres to the profile's requirements. Accepted values are scitokens2 and wlcg (default "wlcg") --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin token](/commands-reference/pelican-server/origin/token) - Manage Pelican origin tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/token/verify # commands-reference/pelican-server/origin/token/verify ## pelican-server origin token verify Verify a Pelican origin token ``` pelican-server origin token verify [flags] ``` ### Options ``` -h, --help help for verify ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --profile string Passing a profile ensures the token adheres to the profile's requirements. Accepted values are scitokens2 and wlcg (default "wlcg") --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin token](/commands-reference/pelican-server/origin/token) - Manage Pelican origin tokens --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/web-ui # commands-reference/pelican-server/origin/web-ui ## pelican-server origin web-ui Manage the Pelican origin web UI ### Options ``` -h, --help help for web-ui ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin](/commands-reference/pelican-server/origin) - Operate a Pelican origin service - [pelican-server origin web-ui reset-password](/commands-reference/pelican-server/origin/web-ui/reset-password) - Reset the admin password for the web UI --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/origin/web-ui/reset-password # commands-reference/pelican-server/origin/web-ui/reset-password ## pelican-server origin web-ui reset-password Reset the admin password for the web UI ``` pelican-server origin web-ui reset-password [flags] ``` ### Options ``` -h, --help help for reset-password --stdin Read the password in from stdin. --user string The user whose password should be reset. (default "admin") ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server origin web-ui](/commands-reference/pelican-server/origin/web-ui) - Manage the Pelican origin web UI --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/registry # commands-reference/pelican-server/registry ## pelican-server registry Interact with a Pelican registry service ### Synopsis Interact with a Pelican registry service: The namespace registry lies at the core of Pelican’s security model by serving as the central point for clients to fetch the public keys associated with namespaced resources. When origins wish to claim a namespace prefix in their federation, they securely associate the public key of their issuer with the namespace registry (many origins may act as their own issuer). Sometimes origins will provide additional OIDC metadata if the origins wish to be accessible to the OSDF’s caching infrastructure. Services wishing to validate the authenticity of a token from an issuer can then reference the namespace registry’s listed public key for that origin and verify that it was signed by the correct private key. ### Options ``` -h, --help help for registry ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server registry serve](/commands-reference/pelican-server/registry/serve) - serve the registry --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/registry/serve # commands-reference/pelican-server/registry/serve ## pelican-server registry serve serve the registry ``` pelican-server registry serve [flags] ``` ### Options ``` -h, --help help for serve -p, --port uint16 Set the port at which the web server should be accessible ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server registry](/commands-reference/pelican-server/registry) - Interact with a Pelican registry service --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server # commands-reference/pelican-server/server ## pelican-server server Manage server operations ### Synopsis Provide commands to manage and interact with Pelican server operations. These commands allow administrators to interact with server administrative APIs. ### Options ``` -h, --help help for server ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server server database](/commands-reference/pelican-server/server/database) - Manage the Pelican server database - [pelican-server server set-logging-level](/commands-reference/pelican-server/server/set-logging-level) - Temporarily change the server’s log level --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/database # commands-reference/pelican-server/server/database ## pelican-server server database Manage the Pelican server database ### Synopsis Provide commands for managing the Pelican server’s SQLite database, including backup creation, listing, verification, and restoration. ### Options ``` -h, --help help for database ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server](/commands-reference/pelican-server/server) - Manage server operations - [pelican-server server database backup](/commands-reference/pelican-server/server/database/backup) - Manage database backups --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/database/backup # commands-reference/pelican-server/server/database/backup ## pelican-server server database backup Manage database backups ### Synopsis Provide commands for creating, listing, verifying, and restoring encrypted database backups. ### Options ``` -h, --help help for backup ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server database](/commands-reference/pelican-server/server/database) - Manage the Pelican server database - [pelican-server server database backup create](/commands-reference/pelican-server/server/database/backup/create) - Create a database backup now - [pelican-server server database backup info](/commands-reference/pelican-server/server/database/backup/info) - Show metadata for a backup file - [pelican-server server database backup list](/commands-reference/pelican-server/server/database/backup/list) - List available database backups - [pelican-server server database backup restore](/commands-reference/pelican-server/server/database/backup/restore) - Restore the database from a backup - [pelican-server server database backup verify](/commands-reference/pelican-server/server/database/backup/verify) - Verify a backup can be decrypted --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/database/backup/create # commands-reference/pelican-server/server/database/backup/create ## pelican-server server database backup create Create a database backup now ### Synopsis Create a compressed and encrypted backup of the server database. The backup is written to the configured backup directory (Server.DatabaseBackup.Location) and encrypted with all available issuer keys. ``` pelican-server server database backup create [flags] ``` ### Options ``` -h, --help help for create ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server database backup](/commands-reference/pelican-server/server/database/backup) - Manage database backups --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/database/backup/info # commands-reference/pelican-server/server/database/backup/info ## pelican-server server database backup info Show metadata for a backup file ### Synopsis Display the human-readable metadata stored in a backup file, including the hostname, username, Pelican version, server URL, and timestamp of when the backup was created. This information is stored unencrypted and can be read without issuer keys. If no file is specified, the most recent backup is used. ``` pelican-server server database backup info [backup-file] [flags] ``` ### Options ``` -h, --help help for info ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server database backup](/commands-reference/pelican-server/server/database/backup) - Manage database backups --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/database/backup/list # commands-reference/pelican-server/server/database/backup/list ## pelican-server server database backup list List available database backups ### Synopsis List all available database backup files in the configured backup directory, showing filename, size, and timestamp. Output can be formatted as JSON with the —json flag. ``` pelican-server server database backup list [flags] ``` ### Options ``` -h, --help help for list ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server database backup](/commands-reference/pelican-server/server/database/backup) - Manage database backups --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/database/backup/restore # commands-reference/pelican-server/server/database/backup/restore ## pelican-server server database backup restore Restore the database from a backup ### Synopsis Restore the server database from a specific backup file. If the database already exists, the —force flag must be specified; in that case the existing database is renamed with a .pre-restore suffix before restoring. If no backup file is specified by path, the most recent backup is used when —latest is provided. Use —output to restore to an alternate location (e.g. for inspection) instead of the configured database path. ``` pelican-server server database backup restore [backup-file] [flags] ``` ### Options ``` --force Overwrite an existing database (backs up the current one first) -h, --help help for restore --latest Restore the most recent backup instead of specifying a file -o, --output string Restore to this path instead of the configured database location ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server database backup](/commands-reference/pelican-server/server/database/backup) - Manage database backups --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/database/backup/verify # commands-reference/pelican-server/server/database/backup/verify ## pelican-server server database backup verify Verify a backup can be decrypted ### Synopsis Verify that a backup file can be successfully decrypted and decompressed using the available issuer keys, without actually restoring any data. If no file is specified, the most recent backup is verified. ``` pelican-server server database backup verify [backup-file] [flags] ``` ### Options ``` -h, --help help for verify ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server database backup](/commands-reference/pelican-server/server/database/backup) - Manage database backups --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/server/set-logging-level # commands-reference/pelican-server/server/set-logging-level ## pelican-server server set-logging-level Temporarily change the server’s log level ### Synopsis Temporarily change the server’s log level for a specified duration. The log level will automatically revert to the configured level after the duration expires. Valid log levels: debug, info, warn, error, fatal, panic Duration should be specified as a Go duration string (e.g., 5m, 1h30m, 300s) Examples: pelican-server server set-logging-level debug 5m -s [https://my-origin.com:8447](https://my-origin.com:8447)  pelican-server server set-logging-level info 30m -s [https://my-cache.com:8447](https://my-cache.com:8447)  -t /path/to/token pelican-server server set-logging-level debug 2m -s [https://my-origin.com:8447](https://my-origin.com:8447)  —param Logging.Origin.Xrootd ``` pelican-server server set-logging-level [flags] ``` ### Options ``` -h, --help help for set-logging-level -p, --param string Target parameter for the log level (e.g., Logging.Level, Logging.Origin.Xrootd, Logging.Cache.Xrootd) (default "Logging.Level") -s, --server string Web URL of the Pelican server (e.g. https://my-origin.com:8447) -t, --token string Path to the admin token file ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server server](/commands-reference/pelican-server/server) - Manage server operations --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/token # commands-reference/pelican-server/token ## pelican-server token Interact with tokens used to interact with objects in Pelican ### Options ``` -h, --help help for token ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server](/commands-reference/pelican-server) - Interact with data federations - [pelican-server token create](/commands-reference/pelican-server/token/create) - Create a token --- Source: https://docs.pelicanplatform.org/commands-reference/pelican-server/token/create # commands-reference/pelican-server/token/create ## pelican-server token create Create a token ### Synopsis Create a signed JWT for accessing Pelican resources. The generated token is a Bearer token. To authorize requests with it, pass it to other Pelican CLI commands via the —token flag or in the HTTP Authorization header. SCOPES Scopes control what the token permits. Use the flags below to set them, or pass arbitrary values with —raw-scope. Flag WLCG profile (default) SciTokens2 profile (—profile scitokens2) * * * —read storage.read: read: —write storage.create: write: —modify storage.modify: write: —stage storage.stage: write: The in each scope is the object path with the namespace prefix stripped. Use —scope-path to override it, or —raw-scope to supply scopes verbatim, e.g.: —raw-scope “storage.read:/ storage.create:/uploads” ISSUER The issuer (—issuer) is auto-discovered from the Director using the supplied pelican URL. Provide —issuer manually when: - Director discovery is unavailable - The namespace has multiple issuers and you need a specific one EXPIRATION Set the token lifetime with —lifetime (seconds, default 1200) or with —expiration (an absolute RFC3339 timestamp, e.g. 2026-12-31T23:59:59Z). These two flags are mutually exclusive. ``` pelican-server token create [flags] ``` ### Examples ``` # Read/write token for a path in OSDF (issuer auto-discovered): pelican token create --read --write pelican://osg-htc.org/some/namespace/path # Token expiring at a specific time (RFC3339: YYYY-MM-DDTHH:MM:SSZ): pelican token create --read --expiration 2026-06-30T00:00:00Z \ pelican://osg-htc.org/some/namespace/path # One-hour token with an explicit issuer: pelican token create --read --lifetime 3600 \ --issuer https://my-origin.com:8443 \ pelican://osg-htc.org/some/namespace/path # Token with custom raw scopes: pelican token create --raw-scope "storage.read:/ storage.create:/uploads" \ pelican://osg-htc.org/some/namespace/path ``` ### Options ``` -a, --audience string Specify the token's 'audience/aud' claim. If not provided, the equivalent 'any' audience for the selected profile will be used (e.g. 'https://wlcg.cern.ch/jwt/v1/any' for the 'wlcg' profile). --expiration string Set the token's expiration as an absolute RFC3339 timestamp (e.g., 2026-12-31T23:59:59Z). Mutually exclusive with --lifetime. -h, --help help for create -i, --issuer string Set the token's 'issuer/iss' claim. If not provided, the issuer will be discovered via the Director. -l, --lifetime int Set the token's lifetime in seconds. (default 1200) -m, --modify Indicate the requested token should provide the ability to modify/delete the specified resource. -k, --private-key string Path to the private key used to sign the token. If not provided, Pelican will look for the private key in the default location pointed to by the '{IssuerKeysDirectory}' config parameter. -p, --profile string Create a token with a specific JWT profile. Accepted values are scitokens2 and wlcg. (default "wlcg") --raw-claim stringArray Set claims to be added to the token. Format: =. --raw-scope stringArray Set non-typical values for the token's 'scope' claim. Scopes should be space-separated, e.g. 'storage.read:/ storage.create:/'. -r, --read Indicate the requested token should provide the ability to read the specified resource. --scope-path string Specify the path to use when creating the token's scopes. This should generally be the object path without the namespace prefix. -s, --stage Indicate the requested token should provide the ability to stage the specified resource. --subject string Set token's 'subject/sub' claim. If not provided, the current user will be used as the default subject. -w, --write Indicate the requested token should provide the ability to create/write the specified resource. Does not grant the ability to overwrite/modify existing resources. ``` ### Options inherited from parent commands ``` --config string config file (default is $HOME/.config/pelican/pelican.yaml) -d, --debug Enable debug log messages -f, --federation string Pelican federation to utilize --json output results in JSON format -L, --log string Specified log output file --version Print the version and exit ``` ### SEE ALSO - [pelican-server token](/commands-reference/pelican-server/token) - Interact with tokens used to interact with objects in Pelican --- Source: https://docs.pelicanplatform.org/about-pelican/core-concepts # Core Concepts and Terminology Pelican is a tool for building _**data federations**_, a model in which decentralized, autonomous data repositories work together to make their data broadly available to other members of the federation under a minimally-centralized structure. In this model, data is accessed through a unified namespace regardless of where the data comes from or what type of storage is used to host it — to a user, everything feels like it’s coming from the same source. ## Terminology ### Objects Pelican should be thought of as a tool that works with federated _objects_ as opposed to _files_. The reason for this is that calling something a file carries with it the connotation of a filesystem. File systems have extra features such as owners, permissions, etc., which Pelican does not implement. In addition, a _file_ gives the connotation that the file is mutable, i.e., its contents can change without requiring a new name. Objects in a Pelican federation, however, should be treated as **immutable**, especially in any case where objects are pulled through a cache (which will be the case for almost all files in the [Open Science Data Federation](https://osg-htc.org/services/osdf) , or OSDF). This is because the underlying cache mechanism, powered by [XRootD](https://xrootd.org/) , will deliver whatever object it already has access to; if an object’s contents change at the origin, the cache will remain unaware and continue to deliver the old object. In the worst case, when the cache only has a partial object, it may attempt to combine its stale version with whatever exists at the origin. Use object names wisely! ### Federations Objects in Pelican belong to _federations_, which are aggregations of data that are exposed to other individuals in the federation. Each Pelican federation constitutes its own global namespace of objects and each object within a federation has its own path, much like files on a computer. Fetching any object from a federation requires at minimum two pieces of information: the federation’s root (i.e. the discovery URL), and the path to the object within that federation (there is the potential that some objects require access tokens as well, but more on that later). For example, the OSDF’s federation hostname is `osg-htc.org`, an example path to an object in the federation is: ``` /pelicanplatform/test/hello-world.txt ``` Combining these two pieces of information, we say the the **object’s URL** is: ``` pelican://osg-htc.org/pelicanplatform/test/hello-world.txt ``` ### Namespace or Federation Prefixes Each origin supports one or more _namespace prefixes_, which are analogous to the folders or directories from your computer that you use to organize files. The terms “namespace prefix” and “federation prefix” may be used interchangeably throughout this documentation. Below is an example of a full Pelican object name: ``` # Note -- this is an example and the object is not expected to exist pelican://osg-htc.org/demo/testfile.txt ``` In this example, the full object name encodes the federation (osg-htc.org), and some path within that federation — `/demo/testfile.txt`. When new Pelican object servers are added to a federation (called Origins), they act on behalf of some subpath within the federation, called a namespace or federation prefix. In the above example, the origin providing access to the object likely acts on behalf of `/demo` namespace. These namespace prefixes are a way for data owners to carve out sections of a federation in a way that ties namespace ownership/management to a public-private keypair. In this way, the group that owns the `/demo` namespace can make decisions on its behalf, including the creation of further sub namespaces and the ability to make decisions about the read/write permissions of the namespace’s underlying objects. Using the “directory permissions” analogy, the owner of the `/demo` directory gets complete control over what happens in the directory, including the ability to delegate the creation of further subdirectories to other users who have access to `/demo`. Starting with a full Pelican URL object name can make it tricky to determine where the actual namespace prefix ends and the underlying object name (as understood by the Origin’s configured data repository) begins. For example, this test object has a deeply-nested path: ``` pelican://osg-htc.org/pelicanplatform/test/hello-world.txt ``` In this example, the namespace prefix happens to be /pelicanplatform/`, and this is hosted by a POSIX Origin that exports a directory tree containing` test/hello-world.txt\`. In most cases, users don’t need to know the namespace/object name split for the objects they’d like to interact with — knowing the full pelican URL is generally sufficient. However, understanding this becomes more important for those who want to federate their own data by hosting an Origin, or for some types of troubleshooting. Pelican provides several tools for determining which portion of the path constitutes the namespace prefix. #### Differentiating Namespace Prefixes and Object Names One way to determine the namespace prefix associated with an object is to access your federation’s registry (e.g. [https://osdf-registry.osg-htc.org/view/registry/](https://osdf-registry.osg-htc.org/view/registry/) ) which should list all prefixes registered in the federation. Using the search bar, you can start typing the path component of your object’s name, and matching prefixes will be filtered for you. Another way to get this information is by interacting with the federation Directly using an HTTP client like curl. The following command is an example of a valid request to and response from the OSDF Director for the `pelican://osg-htc.org/pelicanplatform/test/hello-world.txt` object: ``` $ curl -v https://osdf-director.osg-htc.org/pelicanplatform/test/hello-world.txt > GET /pelicanplatform/test/hello-world.txt HTTP/2 > Host: osdf-director.osg-htc.org > user-agent: curl/7.76.1 > accept: */* > < HTTP/2 307 < content-type: text/html; charset=utf-8 < date: Thu, 19 Dec 2024 21:12:35 GMT < link: ; rel="duplicate"; pri=1; depth=3 < location: https://osdf-uw-cache.svc.osg-htc.org:8443/pelicanplatform/test/hello-world.txt < x-pelican-authorization: issuer=https://osg-htc.org/ospool < x-pelican-namespace: namespace=/pelicanplatform, require-token=false, collections-url=https://pelicanplatform-origin.osdf-prod.chtc.io:8443 < x-pelican-token-generation: issuer=https://osg-htc.org/ospool, max-scope-depth=3, strategy=OAuth2 < x-pelican-jobid: d07c5925-8a39-4a2d-890c-404fca7a21dc < content-length: 115 < Temporary Redirect. ``` Note that the `X-Pelican-Namespace` header has a “namespace” attribute explaining which portion of the path is the registered namespace. ## Core Entities Pelican federations consist of 6 core entities: - [Clients](#clients) - [Data Repositories](#data-repository) - [Origin Servers](#origins) - [Cache Servers](#caches) - Central Services (the [Director](#director-service) and [Registry](#registry-service)) where each of these federation stakeholders represents a unique set of interests. One of Pelican’s core functionalities is balancing the sometimes-competing needs of each of its constituents. A description for each of these entities is provided below. ### Clients Pelican views itself as serving two types of users; data providers and data consumers. _Pelican Clients_ are the tools built around Pelican that support consumers, enabling them to download data via a federation. Pelican currently has three Clients, and more are under development. Existing Clients include the [Pelican CLI tool](install), the [Pelican FSSpec](https://github.com/pelicanplatform/pelicanfs)  for Python, and a file transfer plugin for [HTCondor](https://htcondor.readthedocs.io/en/latest/users-manual/file-transfer.html#file-transfer-using-a-url) . Pelican Clients are designed to work with `pelican://`\-style URLs, which defines a metadata lookup protocol on top of HTTP. For more information on this URL specification, see Pelican’s [client usage documentation](./getting-data-with-pelican/client). Lastly, because Pelican builds on top of HTTP, most HTTP clients (e.g. curl) can be modified to interact with Pelican federations. ### Data Repository Data can live in any number of places, from a hard drive with an associated POSIX filesystem, to buckets in S3. Pelican defines a _Data Repository_ as any instance of a storage backend. Data Repositories often have their own policies that are unique from federation policies, including things like authentication/access control and rate limiting. Pelican’s primary goal with respect to Data Repositories is to make the data they hold accessible to clients within a federation, without requiring that users know what type of repository the data comes from or how it works. ### Origins To make data from a Repository available through a Pelican federation, the data provider must serve an _Origin_ in front of the Repository. Origins are a crucial component of Pelican’s architecture for two reasons: they act as an adapter between various storage backends and Pelican federations, and they provide fine-grained access controls for that data. That is, they figure out how to take data from wherever it lives and transform it into a format that the clients from the federation can utilize while respecting the Repository’s data access requirements. This implies an inherent trust relationship between Origins and Data Repositories, as the Origin is responsible for enforcing the Repository’s needs and wishes within the rest of the federation. However, while the Origin is responsible for translating the Repository’s data access policies into something the federation can understand, Pelican is designed so that Origins never need to share secrets with their federation. Pelican Origins work by making their underlying Repository accessible under some namespace path via HTTPs, which is accomplished by building on top of [XRootD](https://xrootd.org) . The namespace path, also called the _federation prefix_, is the path at which data from the Origin can be accessed in the federation. For example, an Origin that exports the namespace path `/foo` might provide access to an object `bar` in the underlying Data Repository. The full path for this object in the federation would be `/foo/bar`. > **NOTE**: An important distinction between Origins and Data Repositories is that, generally speaking, Origins do **NOT** store any data themselves; their primary function is to facilitate data access _from_ the Repository, which may not coincide on the same machine. ![Pelican and OSDF](/pelican/optimized_images/pelican-bus-opt-1920.WEBP) Pelican Origins serve as a transport bus, connecting a variety of backend storage types to their federation ### Caches Pelican _Caches_ are responsible for storing copies of data inside the federation with the goal of providing more efficient access to reusable data. By default, requests to a Pelican federation for an object are proxied through a Cache, resulting in the federation storing a temporary copy of the object. Currently, objects are cleared from Caches based on a “least recently used” algorithm whenever the server begins running out of storage space, but more robust forms of cache management are in active development. Like Origins, Caches build on top of [XRootD’s “Proxy Storage Services.”](https://xrootd.web.cern.ch/doc/dev6/pss_config.htm)  Because Caches store copies of data for re-distribution in the federation, they must also respect the Origin’s data access policies. That is, the Origin should trust Caches to protect any data that isn’t marked as publicly accessible. Caches in a Pelican federation accomplish this by aggregating access policies from the Origins they support and following the same approval/denial rules the Origins themselves would follow. Generally, Caches are operated by the federation and placed close to computing clusters where data may be quickly re-used as part of High-Throughput Computing workflows, but this is not a requirement. ### Central Services It was mentioned that data federations operate under a minimally-centralized structure. In Pelican, this structure is made up of the _Central Services_, namely the _Director_ and the _Registry_. > **NOTE**: Pelican’s Central Services are responsible for connecting Data Repositories and data consumers, but a core part of Pelican’s architecture is that objects never pass through the Central Services. In fact, the federation’s Central Services are unable to access any authorization-protected objects via Origins unless the Origin mints a token granting that permission. In this way, Origins that don’t allow their data to be staged/cached in the federation need not trust the federation operators, because each Origin acts as its own token issuer and is solely responsible for deciding which requests to respect. This architecture also prevents the creation of centralized bottlenecks as a federation grows. #### Director Service Data access in a Pelican federation requires two fundamental pieces of information — the federation’s hostname (also called the _root_ of the federation), and the name of the object within the federation. Notably, the hostnames of any Origins that facilitate access to objects are absent from that list. Instead, the Pelican model uses the federation root to discover and route all Client requests for objects through its _Director_, an HTTP server whose job is determining the best location(s) at which to access a given object. In some cases, this is accomplished by redirecting clients to a nearby Cache that might already have a copy of the object, and in other cases the Director might send the client to an Origin that can provide direct access. Generally, the Director’s hostname is used as the federation’s hostname because it auto-populates and makes available the federation’s metadata. This information is hosted at the _discovery endpoint_, a URL obtained by appending `/.well-known/pelican-configuration` to the federation’s root. However, some federations may wish to set up the Director/Registry as subdomains of the federation’s hostname. For example, the OSDF breaks these two endpoints apart by providing federation metadata at osg-htc.org, which then points to `osdf-director.osg-htc.org` and `osdf-registry.osg-htc.org`, respectively. All Origins and Caches in a federation send periodic advertisements to the discovered Director at a default interval of 1 minute to let it know where they can be accessed, which namespace(s) they provide, and any information pertaining to data access policies (such as authorization schemes). In this way, the Director is the only service that has a nearly real-time view of all the Origins and Caches in the federation — if an Origin or Cache fails to re-advertise after the required period (15 minutes by default), it is assumed to be offline until another advertisement is received, and the Director will stop sending clients to that location. #### Registry Service Whenever a new Origin or Cache is created and added to a federation, its first step is to register itself with the _Registry_, which acts as the federation’s locus of trust. In the case of Origins, the process of registration entails sending the Registry the namespace prefix the Origin exports, along with the Origin’s public key and a variety of other bookkeeping information. After the Registry and the Origin have performed a handshake that proves the Origin owns the corresponding private key, the Registry stores the information in a persistent database. This process serves two purposes — first, whenever the Origin re-advertises with the federation’s Director, the Director can verify the authenticity of those advertisements through public/private key asymmetric cryptography by looking at the Registry’s stored public key for that Origin and namespace. Second, the Registry’s persistent database prevents other Origins from registering namespaces under an already-registered namespace without first proving they’re allowed to do so by the namespace owner (i.e. the entity that possesses the appropriate private key). --- Source: https://docs.pelicanplatform.org/operating-a-federation/cache/site-local-cache # Running a Site-Local Cache A **site-local Cache** is a Pelican Cache that operates independently from a federation’s central services. When [`Cache.EnableSiteLocalMode`](/parameters#Cache-EnableSiteLocalMode) is set to `true`, the Cache will **not** register with the federation’s Registry and will **not** advertise itself to the Director. This means the Cache is invisible to the federation, and clients must be explicitly configured to use it. ## Who Is This For? Site-local mode is intended for organizations that want a local caching layer for Pelican data without making that Cache discoverable by the broader federation. Common scenarios include: - **Campus or lab deployments** where a local Cache improves download speeds for researchers who repeatedly access the same datasets, but where the organization does not want (or need) the Cache to participate in the global federation. - **Testing and development** environments where operators want to experiment with Cache behavior without registering infrastructure with a production federation. ## How It Works When site-local mode is enabled, the following changes take effect compared to a normal federated Cache: | Behavior | Normal Cache | Site-Local Cache | | --- | --- | --- | | Registers namespace with the Registry | Yes | **No** | | Advertises to the Director | Yes | **No** | | Connection broker operations | Yes (if enabled) | **No** | | Director health tests | Yes | **No** | | Federation token management | Yes | **No** | | Discoverable by clients via the Director | Yes | **No** — clients must specify the Cache URL explicitly | Because the Cache does not advertise to the Director, the Director cannot route client requests to it. Clients must instead be configured to use the site-local Cache directly (see [Pointing Clients at a Site-Local Cache](#pointing-clients-at-a-site-local-cache) below). Even though the site-local Cache does not join the federation, it still requires a [`Federation.DiscoveryUrl`](/parameters#Federation-DiscoveryUrl) (or a `-f ` argument) so it can discover federation-managed Caches to pull objects from on a Cache miss. Note that site-local Cache attempt to fetch missing objects from federation Caches, not directly from Origins. ## Configuration To enable site-local mode, add the following to your Cache’s configuration file: ``` Cache: EnableSiteLocalMode: true ``` A minimal configuration file for a site-local Cache might look like: ``` Cache: EnableSiteLocalMode: true Port: 8442 StorageLocation: /mnt/pelican/cache Federation: DiscoveryUrl: https://osg-htc.org ``` Where: - [`Cache.Port`](/parameters#Cache-Port) is the port your Cache’s XRootD file transfer service listens on (default `8442`). - [`Cache.StorageLocation`](/parameters#Cache-StorageLocation) is the directory where cached data and metadata will be stored. - [`Federation.DiscoveryUrl`](/parameters#Federation-DiscoveryUrl) tells the Cache where to discover metadata for the broader federation Then launch the Cache as usual: ``` pelican-server cache serve --config /path/to/pelican.yaml ``` ### Restricting Namespaces Served By Cache By default, the site-local Cache will attempt to serve any namespace advertised by the federation’s Director. You can restrict this to a specific set of namespaces using [`Cache.PermittedNamespaces`](/parameters#Cache-PermittedNamespaces). When this list is non-empty, the Cache will only pull from the namespaces you specify and will refuse requests for any others. ``` Cache: EnableSiteLocalMode: true PermittedNamespaces: - /my-org/dataset-a - /my-org/dataset-b ``` Restricting namespaces is especially useful in site-local mode because the federation’s Director is not monitoring the Cache. Limiting the Cache to only the namespaces your users actually need reduces the data surface the Cache holds and simplifies reviewing its authorization configuration. ## Pointing Clients at a Site-Local Cache Because a site-local Cache is not known to the Director, the Director cannot redirect clients to it. You must configure clients to send requests to the Cache directly. To find the URL of your site-local Cache’s XRootD service, check the [`Cache.Url`](/parameters#Cache-Url) value in your configuration. The URL typically looks like `https://:`. ### Pelican CLI (Go Client) Use the `-c` (or `--cache`) flag to direct the Pelican client to your site-local Cache: ``` pelican object get pelican:///namespace/path/to/file ./local-file \ --cache https://my-site-local-cache.example.com:8442 ``` Or via the environment variable (space-separated): ``` export PELICAN_CLIENT_PREFERREDCACHES="https://my-site-local-cache.example.com:8442" ``` For persistent configuration, set [`Client.PreferredCaches`](/parameters#Client-PreferredCaches) in your Pelican configuration file or environment variable: ``` Client: PreferredCaches: - https://my-site-local-cache.example.com:8442 ``` #### Falling Back to Federation Caches Supplied by the Director If you want clients to _prefer_ the site-local Cache but still fall back to Director-discovered Caches when it is unavailable, add a `+` as the last element: ``` Client: PreferredCaches: - https://my-site-local-cache.example.com:8442 - "+" ``` Without the `+`, the client will **only** attempt the listed Caches and fail if none can serve the object. ### PelicanFS (Python FSSpec Client) When constructing a `PelicanFileSystem`, pass the `preferred_caches` parameter: ``` from pelicanfs.core import PelicanFileSystem pelfs = PelicanFileSystem( "pelican://federation.example.com", preferred_caches=["https://my-site-local-cache.example.com:8442"], ) # Open and read a file through the site-local cache with pelfs.open("/namespace/path/to/file") as f: data = f.read() ``` As with the Go client, appending `"+"` to the list enables fallback to Director-discovered Caches: ``` pelfs = PelicanFileSystem( "pelican://federation.example.com", preferred_caches=["https://my-site-local-cache.example.com:8442", "+"], ) ``` ## Important Considerations ### Security and Updates Because your federation’s Director is unaware of the site-local Cache, it cannot monitor the Cache’s health or software version. As an operator, you are responsible for: - **Tracking Pelican releases** for Cache-related security patches and applying them promptly. - **Monitoring the Cache’s health** independently, since the Director will not run health tests against it. **You are responsible for the access security of all data held by your site-local Cache.** Because the site-local Cache operates outside the federation’s oversight, no federation service enforces or audits access controls on its behalf. Ensure that your Cache’s derived namespace authorization policies remain in sync with the Director and that TLS configuration and network perimeter controls are correctly configured to prevent unauthorized access to cached data. Determining whether your site-local Cache’s authorization configuration is in sync with the broader federation is a two-step process: 1. For each namespace your Cache supports, check your federation’s Director for namespace capability information using the “Namespaces” search box. For example, a namespace might support “Reads” and “Listings”, but not “PublicReads”, “DirectReads” or “Writes”. 2. Inspect your Cache’s SciTokens and Authfile configurations to verify that these policies are implemented correctly. For more information on how to read SciTokens and Authfile configurations, see “[Anatomy of an Origin’s/Cache’s Authorization Configuration](/advanced-concepts/auth#anatomy-of-an-originscaches-authorization-configuration).” Because analyzing your Cache’s authorization configuration for each namespace can be a difficult and tedious task, consider [restricting the namespaces your Cache is willing to serve](#restricting-namespaces-served-by-cache) to only those you or your users intend to access. This lets you limit the amount of authorization policy you need to maintain to only the namespaces you care about. ### Metrics and Statistics Usage data that is normally collected by the Director (transfer statistics, availability metrics, etc.) will **not** include this Cache. If your organization relies on federation-level reporting, be aware that site-local Cache activity will not appear in those reports. ### No Incoming Connections Required from the Federation Because the Cache does not register or advertise, no federation service needs to reach the Cache. The Cache only makes _outbound_ connections: to the Director to discover which federation-managed Caches hold a requested object, and then to those federation Caches to fetch data. This can simplify firewall rules in environments where inbound connections from the federation are restricted.