Overview

This section of the documentation covers the infrastructure and deployment methods used by the MUSES project to provide the web-based data and computational services. This cyberinfrastructure (CI) is in fact one of the proposed deliverables of the project, alongside the core calculation engine and its integrated EoS and output packages. In other words, we not only want to provide the core computational packages, we also want to provide a flexible, reproducible, and ultimately scalable deployment infrastructure for the scientific community.

While the MUSES project will maintain a flagship instance of the web-based services, individual organizations and institutions are encouraged to run the full MUSES CI in the form best suited to their needs. Their experiences can provide valuable feedback to improve the flexibility and capability of the CI to the benefit of all.

The diagram below captures some of the relationships between the various components involved in our infrastructure. The cluster is bootstrapped using a Terraform script, which

  • connects to the NCSA Radiant OpenStack API to provision VMs and networking resources and

  • connects to the Rancher API to install Kubernetes on the VMs and establish a Kubernetes cluster.

image

Radiant

The NCSA Radiant elastic computing service provides the hardware required for service development and deployment.

Rancher

NCSA maintains a Rancher instance that we use to provision and manage the k8s cluster nodes. This provides a robust set of tools for managing the cluster, including authentication and authorization for team members, cluster state snapshots, rolling upgrades for Kubernetes components (e.g. kube-apiserver, kubelet, etc.)

Terraform

We use Terraform to create and synchronize our infrastructure, allowing us to bootstrap a Kubernetes cluster automatically and reproducibly.

GitOps and ArgoCD

ArgoCD is a continuous deployment system that supports the GitOps paradigm. GitOps in a nutshell:

GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools.

The core idea of GitOps is having a Git repository that always contains declarative descriptions of the infrastructure currently desired in the production environment and an automated process to make the production environment match the described state in the repository. If you want to deploy a new application or update an existing one, you only need to update the repository - the automated process handles everything else. It’s like having cruise control for managing your applications in production.

The goal is to capture the complete state of the deployed services in a reproducible and revertable way by following the GitOps methodology. The combination of ArgoCD and a deployment git repo allows in most cases for apps to be developed and deployed without requiring kubectl access to the Kubernetes cluster directly.