Creating a K8s cluster

Overview

Teaching: 5 min
Exercises: 20 min

Questions

How do I manually create a K8s cluster on the GCP

Objectives

Learn how to create a K8s cluster from scratch on GKE

Introduction

In this demonstration we will show you the very basic way in which you can create a computer cluster (a Kubernetes cluster to be exact) in the cloud so you can do some data processing and analysis using those resources. In the process we will make sure you learn about the jargon. During the hands-on session of the workshop (cloud computing), a cluster similar to this one will be provided to you for the exercises.

Creating your own cluster on GKE

For the hands-on part of this lesson you will not have to create the cluster for yourself, it will be already done for you. For pedagogical reasons, however, we will show an example of how to do it by hand. The settings below should be good and cheap enough for CMSSW-related workflows.

Here you will find some instructions on how to use preemptible machines in GKE. Preemptible VMs offer similar functionality to Spot VMs, but only last up to 24 hours after creation, this might be taken into account if reducing costs is of your concern.

Get to the Console
Create a new project or select one (if you already have one). Note that you will not be able to create clusters in the GCP project that we used for the workshop.
Click on the Kubernetes engine/clusters section on the left side menu
Click Create
Select Standard (not autopilot)
Give it a name
Change the zone to e.g. europe-west1-b (for data streaming speed, choose a zone close to CERN)
Many ways to configure the cluster, but let’s try an efficient one with autoscaling
Go to “Default pool”
Choose size: 1 node
Autoscaling 0 to 4
Go to “Nodes”
Choose a machine e2-standard-4
Check “Enable spot VMs” down in the menu
Leave the rest as it is
Hit create
Creation will take a while While we wait, let us inspect the Cloud shell…
Cloud shell

GCP provides an access machine so you can interact with their different services, including our newly created K8s cluster. This machine (and the terminal) is not really part of the cluster. As was said, it is an entry point. From here you could connect to your cluster.

The gcloud command

The gcloud command-line interface is the primary CLI tool to create and manage Google Cloud resources. You can use this tool to perform many common platform tasks either from the command line or in scripts and other automations.

Connect to your cluster

Once the cluster is ready (green check-mark should appear)
Click on the connect button of your cluster:
Execute that command in the cloud shell:
You’ve connected to your shell, now press enter to link to your GKE cluster:
Authorize Shell