Object store

Swift storage

Storage in the cloud is accessible at https://genostack-data.genouest.org

This storage is a cold storage facility, you may compare it to a Dropbox storage. This means that you cannot read/write files directly but you need to pull/push your files to access them. However, you can remotely access them via your browser and share them with temporary url with other users. Files can also be annoted with additional meta-data.

This service is hosted in our Openstack cloud (Openstack Swift, similar to Amazon S3), and data can be accessed from your cloud virtual machines, our cluster or any external location. Storage is linked to an openstack project. By default, you have a cloud project matching your genouest user identifier, and your identifiers are your genouest account ones.

You can upload/download files from a Web user interface, a python client, or via API to manage access via your scripts/services:

Command-line example:

swift --os-auth-url https://genostack-api-keystone.genouest.org/v3 --auth-version 3 --os-project-name my_genouest_user_identifier --os-project-domain-name Users --os-username my_genouest_user_identifier --os-user-domain-name Users --os-password my_genouest_user_password list

Default quota is 100Gb but can be extended on demand.

Herodote

Herodote is a "data to compute" serverless software.

When you push data (new file or update a file) to the Openstack object storage in a project (bucket), Herodote checks for hooks. If a hook is defined for this data, then a job is automatically submitted. The hook will download the file, execute the commands you defined and upload the results back to the storage server.

All is about automation, users focus on data, not on jobs.