Virtual Machines on Microsoft Azure

Introduction

Overview

Teaching: 10 min
Exercises: 5 min
Questions

  • How to create a VM over Microsoft Azure?

  • How to install docker with the CMS open data software in a single virtual machine?

Objectives

  • Create a single virtual machine on Microsoft Azure using the Ubuntu Server operating system.

  • Learn the process of installing Docker on the Ubuntu virtual machine.

  • Install the CMSSW, ROOT, and Python Docker images on the Ubuntu virtual machine.

  • Verify the successful installation of CMSSW, ROOT, and Python Docker images on the virtual machine.

Overview

In this hands-on exercise, you will learn how to use Microsoft Azure’s cloud platform to create a virtual machine (VM) and harness the capabilities of containerization.

You will install three essential container images on the VM:

  1. CMS software (CMSSW)
  2. ROOT
  3. Python tools

The VM environment offers the added convenience of a graphical user interface accessible through Virtual Network Computing (VNC). With VNC, you can seamlessly access the graphical interface of the containerized applications directly in a browser window, making it effortless to interact with and visualize your data.

By the end of this exercise, you will have a fully functioning virtual machine on Microsoft Azure, equipped with the CMSSW, ROOT, and Python container images, all accessible through an intuitive browser-based graphical interface.

Let’s dive in and explore the process of creating the VM, installing the container images, and utilizing the VNC interface to empower your data analysis and exploration.

Creating a VM on Microsoft Azure

To begin, let’s take advantage of Microsoft Azure’s Azure for Students offering, which provides free access to the cloud. You can access the Azure for Students page by visiting Azure Portal and signing in with your academic account email.

Upon accessing the link, click on the Start Free button. If this is your first time accessing Azure, you may be required to provide some personal information, but rest assured that credit card details should not be requested. Upon successful registration, you will be granted 100 USD credit to get started.

If you wish to use Azure with a non-academic account, you will receive a 200 USD credit to use within the first 30 days after signing up. However, it is important to note that you may be required to provide credit card details during the sign-up process. For more detailed information, you can access the documentation or resources available from Azure free account FAQ

Now, let’s proceed with creating your VM.

  1. Click on Deploy a virtual machine and follow Create a Linux virtual machine.
  2. Locate the Ubuntu Server 20.04 LTS image from the Azure Marketplace service. Select the Create option to proceed.
  3. In the Basic Data tab, fill in the following mandatory details:
    • Subscription: Depends on your account Azure for Students or Azure subscription
    • Resource group: (New) Resource Group
    • Virtual machine name: Choose any name you prefer
    • Region: Choose your preferred location
    • Image: Ubuntu Server 20.04 LTS - x64 Gen2
    • Size: Standard_B2ms - 2 vCPUs, 8 GiB memory
    • Type of authentication: Public key SSH
    • Username: azureuser
    • Key pairs name: Ubuntu_key
    • Public inbound ports: Allow selected ports
    • Select inbound ports: SSH (22)

You can leave the remaining fields with their default values, as well as the other tabs.

  1. Once all the necessary information is provided, click on Review + create at the bottom of the page. You will be redirected to a summary of the selected options and the associated price (which should be within your student credits, approximately 0.0832 USD/hr).
  2. After reviewing the details, click on Create to initiate the deployment process.
  3. A pop-up message will appear, prompting you to generate a new key pair. Select the Download private key and create resource option. This action will initiate the download of a file named <VMName>_key.pem.

    Please note that the deployment process may take some time.

  4. Once the deployment is complete, Click on Go to resource. You will find information about your VM, including the public IP address. Make sure that your VM is in the Running status, and the Start icon (triangle) at the top left side of the page should be disabled.
Congratulations! You have successfully created your VM.

Key Points

  • Azure

Accessing the VM with SSH

Overview

Teaching: 10 min
Exercises: 20 min
Questions

  • How do I connect to my Azure VM?

Objectives

  • Establish an SSH connection to the Azure VM using PuTTY.

  • Configure PuTTY to establish tunnels for VNC, and Jupyter connections

Secure Shell (SSH)

You can use any SSH client to connect to your VM. Here, we provide the steps using PuTTY for Windows and Linux operating systems.

For Linux

  1. Open a terminal on your Linux machine and navigate to the path of the .pem file you downloaded from Azure before.

  2. Set the correct permissions for the pem file by running the following command:

     chmod 600 <VMName>_key.pem

  3. Execute the following command:

     ssh -i <VMName>_key.pem -L 5901:localhost:5901 -L 6080:localhost:6080 -L 8888:localhost:8888 azureuser@<AZURE_VM_IP>

For Windows

Using Windows Subsystem for Linux (WSL)

Using WSL, users can seamlessly operate within their Ubuntu shell and effortlessly follow the Linux instructions.

Using Command Prompt

  1. Open Command Prompt.

  2. Run the following command:

      ssh -i C:\Users\<WindowsUserName>\Downloads\<VMName>_key.pem -L 5901:localhost:5901 -L 6080:localhost:6080 -L 8888:localhost:8888 azureuser@<VM_IP>

Using the PuTTY application

  1. Download the latest version of PuTTY from the official website: PuTTY

  2. Open PuTTYgen and select the Conversions option.

  3. Then, choose Import key to locate and select the .pem file you downloaded previously from Azure.

Note: Do not select Generate.

  1. Go to the File menu, select Save private key, and confirm by clicking Yes in the pop-up window.

  2. Save the new file as Any_name_you_want.ppk to your computer.

  3. Close the PuTTYgen window.

Configuration of PuTTY

  1. Open PuTTY and navigate to the left part of the PuTTY window. Select Category > Session and fill in the following values:

    • Host Name (or IP address): Azure_VM_IP
    • Port: 22

  2. In the left part of the PuTTY window, select Category > Connection > SSH > Auth. Under Private key file for authentication, click on Browse and load the .ppk file created earlier.

  3. For VNC connections, navigate to Category > Connection > SSH > Tunnel.

    • To add the VNC tunnel, set Source Port to 5901, Destination to localhost:5901, and click the Add button.
    • To add the WebVNC tunnel, set Source Port to 6080, Destination to localhost:6080, and click the Add button.
    • To add the Jupyter tunnel, set Source Port to 8888, Destination to localhost:8888, and click the Add button.

    Note: Tunnels will be required later. They will allow you to easily connect to VNC or Jupyter when the corresponding container is launched.

  4. After defining the above tunnels, save the SSH connection settings and establish the connection by clicking the Open button. In the pop-up window, select Accept and enter the VM credentials (by default, azureuser).

Congratulations! You now have remote access to your VM created in Azure.

Key Points

  • Once connected, you will have command-line access to the VM, allowing you to execute commands and perform administrative tasks remotely.

Containers

Overview

Teaching: 10 min
Exercises: 10 min
Questions

  • What is the purpose of installing Docker in the Azure VM?

  • How can you access the graphical user interface of the CMSSW container?

  • What is the recommended method for validating the successful installation of CMSSW, ROOT, and Python containers?

Objectives

  • Installation process of Docker on your Ubuntu VM created on Azure

  • Perform a validation test to ensure that the installation of CMSSW, ROOT, and Python containers is successful.

Installation of Containers

In this section, we will guide you through the installation process for Docker and the CMSSW, ROOT, and Python containers.

Docker is a container platform that enables packaging applications with their dependencies to facilitate quick and consistent deployment and execution across different environments.

Docker Installation

To install Docker on your Ubuntu system, you can follow the steps outlined below. You can also refer to the Official Docker Documentation for a detailed guide.

  1. Update the apt package index and install packages to allow apt to use a repository over HTTPS:

     sudo apt-get update
 sudo apt-get install ca-certificates curl gnupg -y

  2. Add Docker’s official GPG key:

     sudo install -m 0755 -d /etc/apt/keyrings
 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
 sudo chmod a+r /etc/apt/keyrings/docker.gpg

  3. Use the following command to set up the repository:

     echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] \
 https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list

  4. Update the apt package index:

     sudo apt-get update

  5. Install Docker Engine, containerd, and Docker Compose. To install the latest version, run:

     sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

  6. Execute the next commands for adding the user to the “docker” group and starting a new shell session with the “docker” group as the primary group, the user will gain the necessary permissions to interact with Docker without requiring root privileges.

     sudo groupadd docker
 sudo usermod -aG docker $USER
 newgrp docker

  7. Verify that the Docker Engine installation is successful by running the hello-world image:

     docker run hello-world

    You should be able to view these lines:

    Hello from Docker!

    This message shows that your installation appears to be working correctly.

Start Containers from CMS Open Data Container Images

To get access to CMS software (CMSSW) and ROOT Python tools for open data analysis, you can follow the steps outlined below.

CMSSW

For detailed information about CMSSW, you can refer to the documentation CMS Open Data Guide.

  1. Download the docker image for CMS open data and start a container:

     cd
 mkdir cms_open_data_work
 docker run -it --name my_od -P -p 5901:5901 -p 6080:6080 -v \
 ${HOME}/cms_open_data_work:/code cmsopendata/cmssw_7_6_7-slc6_amd64_gcc493 \
 /bin/bash

  2. Once you are inside the container, you can start the VNC service to access the graphical user interface. Run the following command to start VNC:

     start_vnc

    After starting VNC, you can access the VNC service in two ways:

    1. VNC Client: Use a VNC client and connect to localhost:1 to access the graphical interface.
    2. Web Browser: Access the VNC service in your web browser by navigating to http://localhost:6080/vnc.html. Password: cms.cern

  3. To stop the VNC service and exit the container, run :

     stop_vnc
 exit

With the CMSSW container running, you can now proceed with analyzing the open data from CMS using the CMSSW software.

ROOT

To start a container with ROOT installed, you can follow the steps outlined below.

  1. Download the docker images for ROOT and start the container:

     cd
 mkdir cms_open_data_root
 docker run -it --name my_root -P -p 5901:5901 -p 6080:6080 -v ${HOME}/cms_open_data_root:/code gitlab-registry.cern.ch/cms-cloud/root-vnc:latest

  2. To start VNC inside the container:

     start_vnc

    After starting VNC, you can access the VNC service in two ways:

    1. VNC Client: Use a VNC client and connect to localhost:1 to access the graphical interface.
    2. Web Browser: Access the VNC service in your web browser by navigating to http://localhost:6080/vnc.html. Password: cms.cern

  3. To stop VNC and exit:

     stop_vnc
 exit

Python Tools

To start a container with Python tools installed, you can follow the steps outlined below.

  1. Download and install Python tools container: 
     cd
 mkdir cms_open_data_python
 docker run -it --name my_python -P -p 5901:5901 -p 6080:6080 -p 8888:8888 -v ${HOME}/cms_open_data_python:/code gitlab-registry.cern.ch/cms-cloud/python-vnc:latest

    This will download and install the necessary container for Python tools.

  2. To start Jupyter (notebook server) inside the container, run the following command:

     jupyter-lab --ip=0.0.0.0 --no-browser

    When you start a notebook server with token authentication enabled (default), a token is generated for authentication. The token information is logged to the terminal, allowing you to copy and paste the URL into your web browser. Here is an example of the logged token: “The Jupyter Notebook is running at: http://localhost:8888/?token=c8de56fa4deed24899803e93c227592aef6538f93025fe01” Jupyter can be accessed in your web browser by navigating to that link.

  3. To stop the Jupyter server, since it takes over the shell, press CTRL + C and type y when prompted with the message Shutdown this Jupyter server (y/[n])?.

  4. To exit the container, run the following command:

     exit

    Please note that the commands provided for CMSSW, ROOT, and Python Tools containers are a summary of the process. For more detailed instructions and additional configurations, refer to the Docker pre-exercise.

Returning to the Same Container

If you need to return to a previously used container, you can do so using the docker start command:

docker start -i my_od
docker start -i my_root
docker start -i my_python

By running these commands, you can resume your work in the desired container. The -i flag ensures that you can interact with the container’s terminal. Remember to use the appropriate container name (my_od, my_root, or my_python) based on the container you want to start. With these commands, you can easily return to the same container and continue your work without any loss of progress or data.

Note: Make sure that the containers are not already running before using the docker start command.

Key Points

  • Installing Docker and the necessary containers (CMSSW, ROOT, Python) enables you to easily set up and utilize a containerized environment for efficient data analysis and exploration in the Azure VM.

Cleaning up

Overview

Teaching: 5 min
Exercises: 5 min
Questions

  • How do I delete containers and associated data from my Azure VM?

Objectives

  • Delete containers created on Azure

  • Perform a validation test to ensure that the installation of CMSSW, ROOT, and Python containers is successful.

Deleting Docker Containers

To delete the Docker containers and associated data, you can use the following commands:

CMSSW

docker rm my_od
rm -rf cms_open_data_work

ROOT

docker rm my_root
rm -rf cms_open_data_root

PYTHON

docker rm my_python
rm -rf cms_open_data_python

Running these commands will remove the respective docker containers from your system. Additionally, it will delete the corresponding directories (cms_open_data_work, cms_open_data_root, cms_open_data_python) containing the data associated with each container.

Please note that deleting the containers will permanently remove the data stored inside them. Make sure to back up any important data before executing these commands.

Validation Test

To validate the installation and ensure that everything is working fine, you can perform a validation test. Follow the steps of this test page.

Note: This page provides instructions on how to test the functionality of CMSSW, ROOT, and Python within the docker containers.

Stopping a Virtual Machine on Microsoft Azure

If you intend to reuse the created VM in the future, it is recommended to stop the VM. Stopping a VM in Azure may be advisable to prevent the consumption of your free credit or to avoid incurring additional costs.

To stop a virtual machine on Microsoft Azure, follow these steps:

  1. Shut down the VM from the Ubuntu shell.
  2. Go to the Home page in the Azure portal.
  3. Click on the Virtual machines icon.
  4. The list of created VMs will be displayed.
  5. Select the VM you wish to stop and choose the Stop option at the top.
  6. A confirmation message will appear: “Do you want to stop all the selected virtual machines?” Select the Yes option.
  7. The stopping process may take a few seconds.
  8. Once completed, you will find a notification at the top indicating: “Executed stop command on 1 selected items. Succeeded: 1, Failed: 0, Canceled: 0.”

Deleting a Virtual Machine on Microsoft Azure

If you do not intend to reuse the created VM in the future, it is recommended to delete the VM. Deleting a VM in Azure may be advisable to prevent the consumption of your free credit or to avoid incurring additional costs.

To delete a virtual machine and its associated resources on Microsoft Azure, follow these steps:

  1. Go to the Home page in the Azure portal.
  2. Click on the Virtual machines icon.
  3. The list of created VMs will be displayed.
  4. Check the VM you want to delete and select the Delete option at the top.
  5. You will receive a confirmation message: “Do you want to delete all the selected virtual machines?” Type “yes” in the Confirm delete box.
  6. The deletion process may take a few seconds.
  7. Once completed, the VM should no longer be visible in the virtual machines list.

Key Points

  • Containers will permanently remove the data stored inside them. Therefore, it is recommended to back up any important data before executing these commands.