Episode

Introduction

This tutorial shows how to set up a CMS open data processing workflow using Kubernetes clusters from Infomaniak.

Using public cloud resources is an option for you if you do not have enough computing resources and want to run some heavy processing. In this tutorial, we use as an example processing of CMS open data MiniAOD to a “custom” NanoAOD, including more information than the standard NanoAOD but still in the same flat file format.

We assume that you would want to download the output files to your local area and analyse them with your own resources. Note that analysis using Infomaniak resources with your files stored on Infomaniak storage is also possible, but is not covered in this tutorial.

Infomaniak

Public cloud providers are companies that offers computing resources and services over the internet to multiple users or organizations. Infomaniak is one of them: it is a Swiss company, less known than its big competitors. You define and deploy the resources that you need and pay for what you use. As many other such resource providers (for example Google Cloud Platform, AWS, Azure, OHV), it offers some free getting-started “credits”.

You can create, manage and delete resources using the Infomaniak administrator console (a Web UI). Note that an account is needed to access this page.

In this tutorial, we use openstack commands to create the persistent storage for the output data, swift to inspect and download them, and Infomaniak Web UI to provision the Kubernetes cluster where the processing workflow will run.

OpenStack Swift

OpenStack is a set of software components that provide common services for cloud infrastructure. In this tutorial, we its Swift component to interact with the object storage.

Kubernetes

Kubernetes is a system to manage containerized workflows on computing clusters. kubectl is the command-line tool to interact with the cluster resources.

In this tutorial, we use kubectl commands to set up some services and to observe the status of the cluster and the data processing workflow.

Argo Workflows

The processing workflow consist of some sequential and parallel steps. We use Argo Workflows to define (or “orchestrate”) the workflow steps.

In this tutorial, the Argo Workflows services are set up using a kubectl command. We use argo, the command-line tool, to submit and manage the workflows.

Setup

If not yet done, prepare your environment as instructed in Setup.

Episode

Storage for the output files

The processing workflow writes the output files to a storage from which they can be downloaded afterwards.

For this tutorial, we use Infomaniak Object Storage.

Move to your local working environment and get access to the storage by sourcing the OpenStack RC file that you have downloaded previously. If you use a virtual environment, activate it.

cd <your_path>/infomaniak-cloud-test
source venv/bin/activate # if you created a virtual environment with name venv
source <path_to_rc_file_if_not_local/>PCU-XXXXXX-openrc.txt

Storage

Create

Create the storage - if not already done - with

openstack container create mystorage

You can list the existing storage containers with

openstack container list

Test

Before continuing, go through some basic operations for the storage container.

Write

Create a test file and write it into the storage

echo "Storage test" > test.txt
openstack object create mystorage test.txt

Inspect

You can list the contents with

$ openstack object list mystorage
+----------+
| Name     |
+----------+
| test.txt |
+----------+

or

$ swift list mystorage
test.txt

You can also login to the Infomaniak Horizon dashboard, reachable from a top-right button of the Cloud Computing dashboard or from the cloud computing project view.

You will find the storage container by choosing “Object Store” from the left bar.

Download

The file is now stored on the Infomaniak object storage. You can access it anywhere provided you have sourced the OpenStack RC file and have the Python tools (openstackand swift installed).

You can download it with (to a different file with the -o option, omit it to keep the same name):

$ swift download mystorage test.txt -o downloaded.txt
test.txt [auth 1.021s, headers 1.428s, total 1.430s, 0.000 MB/s]

or write to stdout with

$ swift download mystorage test.txt -o -
Storage test

A download option is also available from the Horizon dashboard

Delete

You can delete the test file with

swift delete mystorage  test.txt

Costs

Storage

The cost for the storage is currently 0.000013 €/GB/hour, i.e. 0.11 €/GB/year, 114 €/TB/year. In the context of this tutorial, the cost is very small, but keep it in mind if you plan to store large output files for a long time.

Data access

Uploading to the storage is free.

Downloading is free for the 10 first TB per month and then 0.00811 €/GB i.e 8.11 €/TB.

Episode

Kubernetes cluster

Create the cluster

Go to the project view from the Cloud Computing dashboard:

In the Kubernetes services box, click on “Create a cluster” and from the available options, choose “Cluster Dedicated 4”. This selection is for the control plane, i.e. for the part of the cluster that runs various software component to make the cluster work. The selection of the actual computing nodes comes after.

Give a name to the cluster, and keep the defaults for the other options. Click on “Order”. The cluster creation takes some time, 10 minutes or so. If you follow the link to the Control Panel, you will see that the creation is in progress, and once done, you see it marked as “Active”:

Once the cluster is created, download the Kubeconfig file. The file name is pck-XXXXXX-kubeconfig where pck-XXXXXX is the ID of your cluster. Copy it to your working directory.

Create the cluster worker nodes (also called the instance pool) by clicking on “Add a group of instances”. Give it a name, and keep the defaults for the other options.

For this tutorial, we will use nodes with 4 vCPUs, 16 GB of RAM and 80 GB of disk. You can use filter options to find them.

Select this option and click on Next. Keep the instance management “Manual”, and choose the number of nodes. For testing, a small cluster of two nodes is enough. Click on “Order”. You will see the cluster being scaled up on the Control panel:

This will take some time, typically less than one hour. While waiting you can follow the link on the Control panel read more about Kubernetes in their docs.

Connect to the cluster

In your working directory, set the KUBECONFIG variable so that you can connect to the cluster (replace XXXXXX with what you have in your downloaded file):

export KUBECONFIG=<path_to_config_file_if_not_local/>pck-XXXXXX-kubeconfig

Check the connection to your cluster with:

$ kubectl cluster-info
Kubernetes control plane is running at https://[IP]
CoreDNS is running at https://[IP]/api/v1/namespaces/kube-system/services/pck-XXXXXX-addon-coredns:udp-53/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

You can check the nodes, but while the cluster is still scaling up, no resources are found.

$ kubectl get nodes
No resources found

Once ready, you will see them:

$ kubectl get nodes
NAME                          STATUS   ROLES    AGE   VERSION
pck-uo8l8a8-p6x-zxhh7-7f4g7   Ready    <none>   16m   v1.35.1
pck-uo8l8a8-p6x-zxhh7-trtmj   Ready    <none>   16m   v1.35.1

Costs

Cluster management

There’s a cluster management fee that depends on the type of the control plane. For the one used in this tutorial it is about 0.04 €/hour, independent of the cluster size.

CPU and memory

For the type of nodes (4 vCPUs, 16 GB RAM, 80 GB disk) that we use for this tutorial, the cost of one computing node is about 0.03 €/hour.

Episode

Set up a workflow

Move to your local working environment and get access to the storage by sourcing the OpenStack RC file that you have downloaded previously. If you use a virtual environment, activate it. Make sure you have set the KUBECONFIG variable to get access to the cluster:

cd <your_path>/infomaniak-cloud-test
source venv/bin/activate # if you created a virtual environment with name venv
source <path_to_rc_file_if_not_local/>PCU-XXXXXX-openrc.txt
export KUBECONFIG=<path_to_config_file_if_not_local/>pck-XXXXXX-kubeconfig

Deploy Argo Workflows service

Create a namespace

kubectl create ns argo

Set up Argo workflows controller and server with

kubectl apply -n argo --server-side -f https://github.com/argoproj/argo-workflows/releases/download/v4.0.1/install.yaml

Deploy a service account and its roles and bindings with

kubectl apply -f manifests/

Check the resources with

kubectl get all -n argo

When ready, you will see an output similar to this:

NAME                                       READY   STATUS    RESTARTS   AGE
pod/argo-server-7895b75d9b-w7nhl           1/1     Running   0          35s
pod/workflow-controller-57f7bb576d-zd968   1/1     Running   0          35s

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/argo-server   ClusterIP   10.100.218.36   <none>        2746/TCP   35s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argo-server           1/1     1            1           36s
deployment.apps/workflow-controller   1/1     1            1           36s

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/argo-server-7895b75d9b           1         1         1       36s
replicaset.apps/workflow-controller-57f7bb576d   1         1         1       36s

Access to storage

Ensure that you have created the storage container:

$ openstack container list
+-----------+
| Name      |
+-----------+
| mystorage |
+-----------+

Create access credentials

openstack ec2 credentials create

You will need the values of access and secret fields in what follows.

In your terminal, create the cluster “secrets” to access the storage with

kubectl create secret generic s3-credentials \
  --from-literal=S3_ACCESS_KEY_ID='<the value of the access field>' \
  --from-literal=S3_SECRET_ACCESS_KEY='<the value of the secret field>' \
  -n argo

Simple test job

The example code repository has a simple Argo workflow configuration example to test the storage access. It uses the credentials that were just created as cluster secrets.

Submit the test job with

argo submit -n argo read-write-test-s3/simple-test-s3.yaml

Follow its progress with

argo get @latest -n argo

Once done (very quickly for this short job), you will see:

Name:                simple-s3-mr9p6
Namespace:           argo
ServiceAccount:      argo-service-account
Status:              Succeeded
Conditions:
 PodRunning          False
 Completed           True
Created:             Tue May 05 20:39:49 +0200 (37 seconds ago)
Started:             Tue May 05 20:39:49 +0200 (37 seconds ago)
Finished:            Tue May 05 20:40:20 +0200 (6 seconds ago)
Duration:            31 seconds
Progress:            1/1
ResourcesDuration:   0s*(1 cpu),6s*(100Mi memory)

STEP                TEMPLATE          PODNAME          DURATION  MESSAGE
 ✔ simple-s3-mr9p6  write-and-upload  simple-s3-mr9p6  26s

If all went as expected, you will find the test output in the storage container:

$ swift download mystorage simple.txt
simple.txt [auth 0.295s, headers 0.364s, total 0.431s, 0.000 MB/s]
$ cat simple.txt
Hello from Argo + Infomaniak S3!!!

What happened?

Have a look in the Argo workflow definition file (read-write-test-s3/simple-test-s3.yaml). The workflow has a single job step which uses an alpine container and writes some text in a file:

    container:
      image: alpine:3.18
      command: [sh, -c]
      args:
        - |
          # Create the file in /tmp
          echo "Hello from Argo + Infomaniak S3!!!" > /tmp/simple.txt
          ls -la /tmp/simple.txt

The file is written to the object storage through the output definition:

    outputs:
      artifacts:
      - name: simple-file
        path: /tmp/simple.txt
        archive:
          none: {}           # no compression
        s3:
          endpoint: s3.pub1.infomaniak.cloud # from https://docs.infomaniak.cloud/object_storage/s3/
          bucket: mystorage  # created with: openstack container create mystorage
          region: us-east-1  # placeholder, Infomaniak storage is in Switzerland
          accessKeySecret:   # created with: openstack ec2 credentials create (take access, secret)
            name: s3-credentials
            key: S3_ACCESS_KEY_ID
          secretKeySecret:
            name: s3-credentials
            key: S3_SECRET_ACCESS_KEY
          insecure: false
          key: simple.txt  # the file name for the storage

It uses an artifact of type S3, and it reads the access credentials from the secret that we just created, and writes the new file to the storage container.

It is a good practice to delete the job resources after the job has finished. You can delete the latest workflow with:

argo delete @latest -n argo

CMS open data workflow

Workflow structure

The CMS open data processing workflow example is slightly more complex. It consists of four tasks.

Run the job

Change the default parameters

The default input values are for a 45-node cluster and to process all 353 files of the CMS “MuonEG” Open dataset. Change the input values of the workflow file as follows:

  arguments:
    parameters:
    - name: nEvents
      #FIXME
      # Number of events in the dataset to be processed (-1 is all)
      value: 8000
    - name: recid
      #FIXME
      # Record id of the dataset to be processed
      value: 30511
    - name: nJobs
      #FIXME
      # Number of jobs the processing workflow should be split into
      # Matches number of files(and defines the cluster size, 4 x nodes if 4 vCPU)
      value: 8
    - name: bucket
      #FIXME
      # Name of cloud storage bucket for storing outputs
      value: mystorage
    - name: nNodes
      #FIXME
      # Number of nodes for prepull
      value: 2

This will share the full file list in eight parts, pass them to the processing step which will then process 1000 first events in the respective files.

Submit the job

Submit the job as before:

$ argo submit -n argo cms-pfnano-process/run-pfnano-s3.yaml
Name:                pfnano-process-949g2
Namespace:           argo
ServiceAccount:      argo-service-account
Status:              Pending
Created:             Tue May 05 21:52:05 +0200 (1 second ago)
Progress:
Parameters:
  nEvents:           8000
  recid:             30511
  nJobs:             8
  bucket:            mystorage
  nNodes:            2

Follow the progress

Observe the progress - you will see that the first two steps go fast but the prepull step takes longer:

$ argo get @latest -n argo
Name:                pfnano-process-949g2
Namespace:           argo
ServiceAccount:      argo-service-account
Status:              Running
Conditions:
 PodRunning          False
Created:             Tue May 05 21:52:05 +0200 (5 minutes ago)
Started:             Tue May 05 21:52:05 +0200 (5 minutes ago)
Duration:            5 minutes 25 seconds
Progress:            2/4
ResourcesDuration:   17s*(100Mi memory),0s*(1 cpu)
Parameters:
  nEvents:           8000
  recid:             30511
  nJobs:             8
  bucket:            mystorage
  nNodes:            2

STEP                            TEMPLATE               PODNAME                                                DURATION  MESSAGE
 ● pfnano-process-949g2         cms-od-example
 ├─✔ get-metadata               get-metadata-template  pfnano-process-949g2-get-metadata-template-3541722543  12s
 ├─✔ joblist                    joblist-template       pfnano-process-949g2-joblist-template-2567872736       9s
 ├─◷ prepull-pfnano-image(0:1)  prepull-pfnano-image   pfnano-process-949g2-prepull-pfnano-image-647290456    4m        PodInitializing
 └─◷ prepull-pfnano-image(1:2)  prepull-pfnano-image   pfnano-process-949g2-prepull-pfnano-image-2409759730   4m        PodInitializing

The prepull takes approximately 15 minutes. In some cases, the image pull fails but it restarts automatically. If you are interested to inspect what happens, you can find the pod names with:

$ kubectl get pods -n argo
NAME                                                    READY   STATUS            RESTARTS   AGE
argo-server-7895b75d9b-w7nhl                            1/1     Running           0          114m
pfnano-process-949g2-get-metadata-template-3541722543   0/2     Completed         0          7m49s
pfnano-process-949g2-joblist-template-2567872736        0/2     Completed         0          7m19s
pfnano-process-949g2-prepull-pfnano-image-2409759730    0/2     PodInitializing   0          6m59s
pfnano-process-949g2-prepull-pfnano-image-647290456     0/2     PodInitializing   0          6m59s
workflow-controller-57f7bb576d-zd968                    1/1     Running           0          114m

and you then read the events with kubectl describe pod <pod name> -n argo.

Once the processing starts, you will see the jobs running in parallel.

STEP                                                                TEMPLATE               PODNAME                                                DURATION  MESSAGE
 ● pfnano-process-949g2                                             cms-od-example
 ├─✔ get-metadata                                                   get-metadata-template  pfnano-process-949g2-get-metadata-template-3541722543  12s
 ├─✔ joblist                                                        joblist-template       pfnano-process-949g2-joblist-template-2567872736       9s
 ├─✔ prepull-pfnano-image(0:1)                                      prepull-pfnano-image   pfnano-process-949g2-prepull-pfnano-image-647290456    12m
 ├─✔ prepull-pfnano-image(1:2)                                      prepull-pfnano-image   pfnano-process-949g2-prepull-pfnano-image-2409759730   15m
 ├─● runpfnano(0:eventsinjob:1000,firstfile:1,it:1,lastfile:45)     runpfnano-template     pfnano-process-949g2-runpfnano-template-2847614267     2m
 ├─● runpfnano(1:eventsinjob:1000,firstfile:46,it:2,lastfile:89)    runpfnano-template     pfnano-process-949g2-runpfnano-template-3575107822     2m
 ├─● runpfnano(2:eventsinjob:1000,firstfile:90,it:3,lastfile:133)   runpfnano-template     pfnano-process-949g2-runpfnano-template-3476365621     2m
 ├─● runpfnano(3:eventsinjob:1000,firstfile:134,it:4,lastfile:177)  runpfnano-template     pfnano-process-949g2-runpfnano-template-2488933712     2m
 ├─● runpfnano(4:eventsinjob:1000,firstfile:178,it:5,lastfile:221)  runpfnano-template     pfnano-process-949g2-runpfnano-template-3276409680     2m
 ├─● runpfnano(5:eventsinjob:1000,firstfile:222,it:6,lastfile:265)  runpfnano-template     pfnano-process-949g2-runpfnano-template-2485157448     2m
 ├─● runpfnano(6:eventsinjob:1000,firstfile:266,it:7,lastfile:309)  runpfnano-template     pfnano-process-949g2-runpfnano-template-2352203977     2m
 └─● runpfnano(7:eventsinjob:1000,firstfile:310,it:8,lastfile:353)  runpfnano-template     pfnano-process-949g2-runpfnano-template-1215130620     2m

You can observe the progress of each job from the logs with kubectl logs <pod name> -n argo.

Find the output

Once finished, you can find the output files in the storage container:

 $ swift list --lh mystorage
7.8K 2026-05-05 19:52:17         application/gzip pfnano/30511/files_30511.txt
 11K 2026-05-05 20:11:05 application/octet-stream pfnano/30511/logs/1.logs
 11K 2026-05-05 20:11:13 application/octet-stream pfnano/30511/logs/2.logs
 11K 2026-05-05 20:11:13 application/octet-stream pfnano/30511/logs/3.logs
 11K 2026-05-05 20:11:15 application/octet-stream pfnano/30511/logs/4.logs
 11K 2026-05-05 20:11:19 application/octet-stream pfnano/30511/logs/5.logs
 11K 2026-05-05 20:11:20 application/octet-stream pfnano/30511/logs/6.logs
 11K 2026-05-05 20:11:11 application/octet-stream pfnano/30511/logs/7.logs
 11K 2026-05-05 20:11:20 application/octet-stream pfnano/30511/logs/8.logs
 25M 2026-05-05 20:11:04 application/octet-stream pfnano/30511/scatter/pfnanooutput1.root
 29M 2026-05-05 20:11:12 application/octet-stream pfnano/30511/scatter/pfnanooutput2.root
 28M 2026-05-05 20:11:13 application/octet-stream pfnano/30511/scatter/pfnanooutput3.root
 30M 2026-05-05 20:11:14 application/octet-stream pfnano/30511/scatter/pfnanooutput4.root
 30M 2026-05-05 20:11:19 application/octet-stream pfnano/30511/scatter/pfnanooutput5.root
 31M 2026-05-05 20:11:20 application/octet-stream pfnano/30511/scatter/pfnanooutput6.root
 27M 2026-05-05 20:11:11 application/octet-stream pfnano/30511/scatter/pfnanooutput7.root
 31M 2026-05-05 20:11:20 application/octet-stream pfnano/30511/scatter/pfnanooutput8.root
  35 2026-05-05 18:40:14 text/plain;charset=utf-8 simple.txt
235M

If you wish, you can download the logs and output files locally with

swift download mystorage --prefix pfnano/30511/logs/ --output-dir logs/testjob
swift download mystorage --prefix pfnano/30511/scatter/ --output-dir pfnano/testjob

In this example, only 8000 events were processed. For any real-scale processing, keep in mind that the output files can be big so avoid unnecessary downloads.

Delete the cluster

Once finished, delete the cluster to avoid unnecessary costs. You can do it from the Kubernetes tab of the Cloud Computing dashboard in the Infomaniak Web GUI:

This brings to the cluster page where you find the delete option under Manage:

Both the control plane and node pool will be deleted and you are asked to confirm the choice.

Costs

The cost of these quick exercises is less than 0.5 €, depending on the uptime of the cluster (approximately 0.1 €/hour for a small 2-node cluster).

Episode

Scaling up

Input data

The optimal cluster configuration depends on the input dataset. Datasets consist of files, and the number of files can vary. Files consist of events, and the number of events can vary.

In practical terms, the input to the parallel processing steps is a list of files to be processed. We do not consider directing events from one file to different processing steps. It could be done, but would require a filtering list as an input to the processing.

In an ideal case, the parallel steps should take the same amount of time to complete. That would guarantee that the resources are used efficiently so that no idle nodes - for which we still pay for - remain while the longer jobs still continue. However, this is usually not the case because

• input files are not equal in size
• processing time per events can vary.

The best approach remains to have the same amount of files in each parallel step. Eventually, the files could be sorted according to their size and their share to the nodes could be optimized.

Cluster and job configuration

vCPUs, memory and disk

In earlier investigations on Google Cloud Platform, kubectl top node and kubectl top pods were used to evaluate the resources needs for the processing - the most demanding step in the workflow chain. These earlier estimates were used.

    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=10250
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls

kubectl apply -f components.yaml

Without constraints (i.e. other parallel tasks running), the processing jobs take 80% of the available CPU, and require up to 1.8 GB of memory. However, choosing nodes with 4 vCPUs 8 GB memory resulted jobs failing because the node was low on resources. Therefore, nodes with larger memory (16 GB) were chosen.

The output is written to the local disk on the node. For the type of processing in this example - adding information to the usual NanoAOD format - the output size for processing a single MiniAOD file is around 2 GB. The node disk needs to be large enough to accommodate the outputs of all jobs on the node before they are moved to the object storage at the end of the job. It also needs to fit the CMSSW container image which is around 30 GB (uncompressed). Therefore, of the available options, the disk size of 80 GB was chosen. A 50 GB disk could have been large enough, if each job processes a single output file. However, the difference in cost is small, around 10%.

Number of nodes and job configuration

In the example case, we have used the MuonEG MiniAOD dataset with 353 input files. For that number of files, we deployed a cluster with 45 nodes 4-vCPU nodes, providing a total of 180 vCPUs.

We first make sure that the large CMSSW image is present on each node. This is done in a prepull step with an artificial resource request requiring more than half of the available vCPUs, making sure that only one pull happens per node. In this cluster configuration, we require 3 CPUs as the nodes have 4 vCPUs

      resources:
        requests:
          cpu:  "3"

Note that this value needs to be changed is the node has more (or less) than 4 vCPUs.

For the processing, we define a workflow with 353 jobs, and set the resource request so that one job runs on each vCPU at a time. The relevant part of the workflow definition is the resource request of the final processing step:

      resources:
        requests:
          cpu: "800m"

For the processing of the full dataset, this results to running a first set of 180 jobs parallel, and when they complete, the remaining 173 will start. We will have a close to full occupation of the cluster.

Autoscaling

Autoscaling is a feature that automatically removes idle nodes from the cluster. It can reduce cost, as the total cost depends on the uptime of nodes in the cluster. We are not using it, because it would delete the nodes that remain idle while waiting for the prepull step to finish on all nodes. Once the processing step starts, the cluster would automatically scale up to the number of defined nodes. However, in our case, we do not want that happen as the prepulled image on each node is used in the processing step.

Automating

The optimal cluster configuration depends on the number of files in the dataset. One could envisage a tool that

• inspects the dataset(s) to be processed
• creates the cluster with optimal numbers of nodes
• inserts the relevant parameters - the number nodes for the prepull and the number of jobs for the processing - to the workflow definition.

This has not been implemented in this tutorial example to keep it simple.

Costs

The processing over the full example dataset took 8 hours, including setup. The resulting cost of the cluster was 14 CHF. The graph below shows comparison with the earlier work using different cluster configurations and node types on Google Cloud Platform (GCP).

Episode

Discussion

What worked?

Setting up resources was straight-forward. The documentation is adequate, very good for starting a project and for storage resources, but a bit less elaborated for Kubernetes services. However, this was not found to be an issue as the setup is standard Kubernetes without provider-specific add-ons.

Various Kubernetes roles and bindings to handle access between computing and storage resources were not needed, which simplifies the cluster setup. Access to the storage is handled with standard OpenStack credentials and they can be smoothly used in the Argo workflow definitions.

The initial quota for different resources during the free trial period was large enough for full-scale processing on CPU nodes. When enquired, the support team promptly increased them, but finally none of these extra resources were needed in this type of processing.

The Infomaniak support team replied to questions within expected delays whenever help was needed.

What difficulties were observed?

Two useful functionalities are not directly available.