Skip to content
Key Points

Aggregated resource

Key Points

This page is generated from episode key points.

6 episodes covered
14 takeaways
Best for recap and publish checks

Episode 1

Introduction

3 takeaways collected from this episode.

  • Public cloud providers are companies that offer pay-as-you-go computing resources and services over the internet to multiple users or organizations.
  • Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications and their associated workflows across clusters of host machines.
  • Argo Workflows is an open-source tool for orchestrating sequential and parallel jobs on Kubernetes.

Episode 2

Storage for the output files

1 takeaways collected from this episode.

  • The storage cost depends on the volume stored and the duration of storage, and for this type of processing is very small.

Episode 3

Kubernetes cluster

2 takeaways collected from this episode.

  • Kubernetes clusters can be created from the Web GUI
  • kubectl is the tool to interact with the cluster.

Episode 4

Set up a workflow

2 takeaways collected from this episode.

  • Once the cluster is up, you will first deploy the Argo Workflows services using kubectl.
  • You will submit and monitor the workflow with argo.

Episode 5

Scaling up

3 takeaways collected from this episode.

  • The resource request should be set so that one job runs in one vCPU.
  • The optimal number of nodes in a cluster depends on the number of files in the dataset, and it should be chosen so that each job has the same number of files.
  • For the benchmarking task of processing an entire CMS dataset, Infomaniak resources were found to be remarkably less expensive than Google Cloud Platform: 14 CHF instead of 77 CHF.

Episode 6

Discussion

3 takeaways collected from this episode.

  • Technically, deploying the resources, setting up the workflow and running the processing on Infomaniak was very smooth.
  • The Infomaniak support team reacted within the expected delay.
  • Considering the ease of setup and very advantageous pricing, Infomaniak can be a very convenient choice for small-scale processing by individual researchers.