Reflection & Use Cases

As the IO-SEA project winds down, it is rewarding to reflect on the results of the three years of the IO-SEA team’s efforts.

In the final year of the project, the IO-SEA storage environment has been deployed on the IO-SEA prototype hardware on the DEEP system at Jülich Supercomputing Centre.
We can now run the workflows of the five scientific use cases using three different ephemeral storage services, the SBB (Smart Burst Buffer), the NFS service and the DASI service.
We can also make use of a variety of features that are available through the Workflow Manager (WFM) version 1.6.

In the final months of the project the use case teams have been exploring these features and services to see what works well — or perhaps, not so well — in conjunction with the particular characteristics of each workflow.

The SBB and NFS services have proven similarly easy to use — one merely configures the application to use an I/O path directed through the service. Both offer a private storage system that is faster than networked storage and insulated from other user tasks slowing down the shared storage system.

The big difference is that the NFS stores the application data in a dataset on the datanodes,  where it can reap the benefits of the Hestia hierarchical storage management. The SBB, on the other hand synchs the data to the filesytem, where it can be accessed outside of a WFM session.

DASI has now been deployed as an ephemeral storage service. Within a workflow, data stored and organized with DASI semantic keys is now also stored in a dataset.

We regret that we do not have time to do more in-depth experiments in within the context of the project.
The past few months have a allowed a fascinating glimpse at the possibilities of a storage environment like the IO-SEA system.