Hello,
my name is Peter Amstutz and I'm a
developer on the Arvados open source
project.
Today I'm going to give a technical
overview of some of the features and
components of the Arvados platform.
This presentation covers Arvados
features up to the 2.1 release.
First I want to say a little bit about
the company behind Arvados,
Curii. Curii is an open source software
company
serving the life sciences. Our customers
include pharma,
small biotechs and academic labs. Curii
offers support and development services
for the Arvados platform,
as well as engaging in research and
development on machine learning for
genomics
and around open biomedical data and
standards.
What is Arvados? Arvados is an open
source platform for managing,
processing and sharing genomic and other
large scientific and
biomedical data. We'll talk about what
this means
in terms of specific features and
capabilities in a minute,
but first let me tell you about some of
our high level goals.
We wanted to build a platform that could
be deployed in a variety of environments
such as various cloud providers and
on-premise HPC
clusters. We also wanted to make it
possible
to access and use data located on
multiple clusters
that may be running in different
locations, environments or organizations.
Another high-level goal is to work at
the scale
of modern biomedical research.
Instruments such as sequencers and
microscopes can produce hundreds of
gigabytes of data in a single run,
and produce hundreds of terabytes of
data annually,
so Arvados needs to work at petabyte
scale
and manage thousands of cores to process
all that data.
Finally, we want a complete record of
everything done on the system
so we can determine what was done,
confirm or reproduce results,
track the origin of data and be able to
verify
that a data set has not been modified, or
have a
history of changes when it
has been modified.
The key capabilities that
Arvados gives you
are the ability to manage storage and
compute at scale
and the ability to integrate those
capabilities into your existing
infrastructure and applications
using Arvados APIs. The main
unit of data management is an Arvados
collection which is a self-contained set
of files and directories.
A collection can be identified by a name
assigned by the user, by a universally
unique database identifier,
or by a portable data hash, also called a
content address.
The portable data hash is an immutable
identifier computed
based on the content and structure of
the files making up the collection,
and if any of those files change you get
a different
portable data hash. Because the portable
data hash
is based on the content of the
collection, you can verify
that the actual data has the expected
hash
as well as easily determine if a copy of
the collection is the same as the
original.
Access to collections is controlled, and
collections
can be shared with users and groups at
different levels of read
only or write access.
Once data is loaded into the system, you
probably want to analyze it.
Arvados provides a complete workflow
engine that's closely integrated with
the storage system
to ensure efficient data movement, as
well as handling
the scheduling of compute jobs across
multiple nodes,
and keeps a record of everything that's
been done so it's easy to repeat
previous computations. Finally,
Arvados provides a complete API making
it possible to integrate with existing
systems
as well as build new applications on top
of Arvados.
The Arvados storage system is called "Keep".
As I mentioned, it organizes sets of
files
into a collection. A collection
can have additional user metadata
associated with it in the form of
searchable key value properties,
and records the history of changes made
to that collection.
To store data in a collection, the files
are broken up into a set of blocks
up to 64 megabytes in size which are
hashed
to get an identifier which is used to
store and retrieve the data block.
The identifier can be used to validate
the content of the block.
It also provides for deduplication as
the same data block
can be referenced by multiple
collections this means
that it is cheap to copy and modify
collections as only the block identifiers
need to be copied and not the actual
data.
On the back end, the actual data blocks can
be stored on a conventional file system
in S3 buckets or in Azure blob storage.
Using the block hash as the file name
On the front end, keep
provides a variety of access options:
you can use the Arvados SDK which uses
the block level API and reassembles
files on the client side,
but you can also use higher level access
methods including
a file system in user space or FUSE
mount
which lets you access Keep more or less
like a regular POSIX file system,
or over HTTP using WebDAV or using an
AWS S3 compatible API
where an Arvados collection acts like a
bucket and enables applications that
already support object storage
to access files in Arvados.
The Arvados compute layer is called
"Crunch".
Requests to run a compute job are
submitted through the Arvados API.
This includes the command line to run,
the container image to use,
required hardware resources, and what
input files are required by the job.
Crunch handles translating that job
request into an
HPC batch submission or on the cloud
by using cloud providers APIs to request
a new compute
instance on demand, running the job and
then
shutting down the compute instance when
it's no longer needed.
Crunch keeps a record of every job that
has been submitted
and all of the inputs, outputs, and
container images
are keep collections
identified by portable data hashes.
The same portable data hash means you
have the same
file structure and content.
If a job which is identical to a
previous job is submitted
meaning it was submitted with an
identical container image,
input files, and command line, Arvados
will recognize that it is identical to a
previous job
and instead of redundantly
re-running the job it will simply reuse
the result from the past run.
This is especially useful if you need to
stop and restart a workflow
as it will quickly move through all the
reused steps
that have previously been executed.
Crunch also collects complete logs,
information about the compute node, and
extensive metrics about the job,
such as moment-to-moment CPU usage, RAM
usage,
I/O and so forth enabling you to easily
diagnose
common problems such as out of memory
conditions
or optimize cost by determining the most
compact and least expensive node size
that fits a job.
The native workflow language of Arvados
is common workflow language.
CWL is an open standard for describing
computational data analysis workflows
which
is supported by a number of different
vendors and software platforms.
The Arvados project has been involved in
the development of CWL
since its inception, and offers robust
CWL support.
If you would like to learn more go to
commonwl.org.
Day-to-day use of Arvados typically
involves using the Arvados workbench web
application.
Workbench lets you search and browse
collections,
start and monitor the progress of
workflows, create projects,
upload data, share data with other users,
and a variety of other features.
there's also a suite of command line
tools
that are capable of doing everything
that can be done through the web UI.
Arvados offers software development kits
for several different languages,
currently Python, Go, R,
Ruby and Java. The SDKs
make it easy to access the underlying
REST APIs
as well as direct access to data stored
in Keep.
In addition, software can access files in
Keep
through WebDAV and S3 compatible APIs
offered by Arvados.
Arvados takes security and access control
very seriously.
Access to API endpoints requires a
client
to present an access token that
identifies the user.
All traffic is encrypted by default
using TLS
and Arvados can be easily configured for
data
to be encrypted at rest. Arvados
supports various single sign-on systems
including LDAP, OpenId Connect and
Google accounts.
Data upload to Arvados is private by
default
but can be shared with other users or
groups at different access levels.
A major feature of Arvados is federation.
Arvados clusters are able to communicate
with other clusters in a federation
in order to enable the user to log in
with a consistent identity and
credentials,
and search and access data across
multiple clusters in different
regions or organizations.
Federation enables you to use Arvados to
create a
data commons in which data can be both
shared widely among users in an
organization,
or between organizations, while still
having controlled, audited access.
In April 2020, a week-long online
BioHackathon was organized to see how
bioinformatics
could help in the fight against SARS-CoV-2.
One of the projects to emerge from this
was the public sequence resource PubSeq
located at covid19.genenetwork.org.
The vision of this resource was to
provide viral sequence data
in a place where, unlike other
repositories that supply data but no
compute capability,
scientists could easily run their own
custom batch analysis.
Using Arvados we were able to go from
concept to working prototype in five
days.
Today this resource has over thirty
thousand viral sequences available
along with metadata and a number of
workflows for processing the data.
To sum up, Arvados is a complete software
platform
for managing data and compute on that
data
without compromising on scale, security,
or control.
Curii has set up a demo instance at
playground.arvados.org
where you can get a feel for what it is
like to use Arvados
as a regular user. Follow the
documentation link on the arvados.org
front page to get to the Arvados
documentation,
which includes our user guide, and if you
think you would like to set up Arvados
for yourself,
our installation guide. Arvados is open
source and we welcome and encourage
community users.
Follow the community link on arvados.org
to get information about the Arvados
discussion forum,
chat, video calls, and other community
events.
Finally if you're interested in
professional support and development
services, please contact us at info@curii.com
for more information.
And with that, thank you for your time.