Pelican's Python FSSpec

What is FSSpec?

FSSpec (opens in a new tab), short for Filesystem Specification, is a Python library that provides a unified interface for interacting with various filesystems. It abstracts the complexities of different storage backends, allowing users to work with local files, cloud storage, and other remote filesystems using a consistent API.

Pelican chose to develop an FSSpec implementation because it delivers maximum reward with less development effort by providing:

• Unified Interface: A consistent API for different filesystems, meaning many users already interact with FSSpec in their projects.
• Extensibility: Robust support for custom filesystem implementations Because Pelican is built on top of HTTP, our FSSpec is built on top of the core HTTP FSSpec.
• Compatibility: FSSpec is used widely in the Python ecosystem, and already integrates seamlessly with tools like xarray (opens in a new tab) and PyTorch data loaders (opens in a new tab).

While we're careful to say Pelican is a platform of tools for working with objects, because many of Pelican's storage backends don't have a true filesystem interface¹, FSSpec lets users interact with Pelican objects in a more familiar manner by translating common file operations such as reading, writing, listing directories, and more, without worrying about the underlying storage technology.

How to Get it

Instructions for downloading and working with Pelican's FSSpec library can be found on our FSSpec Github repository (opens in a new tab). If you have further questions or feature requests, we welcome new issues, contributions and discussions there!