Hierarchical Storage Management in IO-SEA
Written by Thomas Leibovici @ CEA
IO-SEA aims to implement a deep storage hierarchy by integrating various storage technologies from NVMe to tapes.
Some technologies provide high performance for application I/Os (NVMe, SSD…) while other provide more capacity at affordable price for long term storage (HDDs, tapes). By combining these various storage technologies, Hierarchical Storage Management (HSM) enables the design of storage systems that provide both high performance and large storage capacity at a reduced TCO.
In IO-SEA, we are designing a Hierarchical Storage mechanism for Exascale systems. This will enable automated data migration with smart and flexible data placement policies across a deep hierarchy of storage from NVMe to tapes.
Although this HSM mechanism aims to be transparent to users and applications, it will however provide a way to pass information on a top-down approach: the application will specify its needs in terms of I/O patterns and data life cycle. This information is then interpreted by lower layers of the system to locate the data to the most appropriate storage technology.
IO-SEA HSM implementation is based on 3 leading open-source software in the ecosystem of HPC and storage:
- Cortx-Motr[1]: a distributed object storage system with an innovative design to address the requirements of Exascale;
- Phobos[2]: a parallel object store specialized to manage storage on magnetic tapes, for a reduced TCO and lower carbon footprint;
- Robinhood policy engine[3]: a tool to query efficiently large storage systems meta-data, thus enabling the efficient application of multi-criteria policies in such systems.
These three components will be enhanced, completed and integrated to the rest of the IOSEA software stack to implement a complete IO stack for Exascale.
