Blog: Data Access and Storage Interface (DASI)
Authored By Metin Cakircali (ECMWF), James Hawkes (ECMWF) & Jenny Wong (ECMWF)
Upcoming exascale systems are facing various storage challenges, in order to meet exascale computing requirements. IO-SEA aims to implement Hierarchical Storage Management (HSM) to provide high performance and large capacity storage by combining various storage technologies. Hierarchical storage gives flexibility and performance, but adds complexity for application developers.
DASI {https://github.com/ecmwf-projects/dasi} provides an abstraction for scientific data handling, granting access to the underlying complex storage mechanisms, in a user-friendly manner, which is simple for application developers to adopt and understand.
The key feature of DASI is that it provides a “semantic interface” for data, which means the data is indexed and uniquely identified by set of scientifically-meaningful metadata keys. An example is shown in figure 1.
Figure 1 Example Semantic Key description of data.
DASI is built on top of a domain-specific object store library, FDB (Fields DataBase) {https://github.com/ecmwf/fdb} which has been developed by ECMWF for storing, indexing and retrieving data. DASI is modular and supports multiple backend technologies, such as POSIX, Ceph, NVRAM, and soon CORTX-Motr. In Figure 2, we show a diagram of DASI and its relationship to the user applications and backends.
Figure 2 DASI and its relationship to the user applications and backends.
In the current phase of the IO-SEA project, DASI has been implemented on top of FDB, and many of the IO-SEA use-case applications have adapted their code to use DASI. Currently, we are implementing the Cortx-Motr backend to DASI that will allow the user applications to run seamlessly on clusters with Cortx-Motr {https://github.com/Seagate/cortx-motr}. Cortx-Motr is a distributed object storage system designed to run on Exascale systems.
