Skip to main content

Infrastructure

The clusters​

The backend runs on three EKS clusters in eu-central-1, one per AWS account:

ClusterAccountEnvironments
dev-emerald194455872934dev4, dev5, devtw
staging-silver056007870958staging
prod-pita388786193518prod, prod-it

The clusters themselves are defined in eks-cluster, owned by Platform. What runs on them is split across three repositories:

WhatWhere
workloads — Deployments, Services, Ingresses, HPAsk8s-gitops-twpro, synced by ArgoCD
secret valuesAWS Secrets Manager, declared in twpro-infra — see Deployment secrets flow
imagesECR, built by the GitHub Actions workflows in CD - Continuous Deployment

Nothing is applied by hand: a change is a commit to k8s-gitops-twpro that ArgoCD reconciles.

Basic concepts​

A short list of the Kubernetes concepts the backend system uses.

Namespace​

!ref Namespaces It contains a set of isolated objects, and for the backend project they keep each application in its own group. Namespaces are named be-<app>-env-<environment> — be-admin-env-prod, be-main-env-prod, be-admin-env-staging, be-main-env-dev4, and so on. That naming is load-bearing, not cosmetic: it is what the secrets access model keys on.

!!!info Isolation Objects have visibility over the whole namespace by default, meaning that if there are 2 services service1 and service2 in namespace1 namespace, from service1 it's possible to reach service2 directly. Instead, from namespace2 namespace, a service can access service2 by forcing the namespace in the URL: service2.namespace1. NOTE: that is a shortcut for service2.namespace1.svc.cluster.local !!!

Workloads​

!ref Workloads These are the most used objects in the project, because they include application runners as well.

There can be multiple types of workload and they all define image name, command, environment variables, etc; these are the most used in our system, here there are some examples:

  • CronJobs: used for database backup cron
  • Deployments: used for web server, background worker, etc
  • Jobs: executions of CronJobs
  • StatefulSets: used for Redis cluster, Kafka cluster, etc
Access a pod console shell and logs​

With kubectl — see Usage below for the commands, and ArgoCD for a read-only view of what is deployed.

Services and ingresses​

!ref Services Services are used to achieve discoverability between objects, in particular to create a network between workloads and let them communicate. Ingresses on the other hand define routes that the K8S cluster's ingress controller will then make use of.

For example, a web server Deployment expose a Service (with a properly configured selector) that allows other workload, such as an haproxy load balancer Deployment, to communicate with it. Then, the haproxy load balancer Deployment expose another Service that will be used by an Ingress and allow the external internet communication.

Kubectl​

!ref Cheatsheet

  1. Install instructions: install-kubectl-macos
  2. Log in to the account with AWS SSO and write the context yourself — there is no kubeconfig to download:
aws sso login --profile treatwell-x-uala-development.view-eks
aws eks update-kubeconfig \
  --name dev-emerald \
  --region eu-central-1 \
  --profile treatwell-x-uala-development.view-eks \
  --alias dev-emerald

Without --alias the context is named after the full cluster ARN, which is unpleasant to type. The profile prefix differs per account: treatwell-x-uala-development for dev-emerald, treatwell-saas-staging for staging-silver, treatwell-saas-production for prod-pita.

Extra: shell autocompletion

What your profile lets you do​

Profile suffixKubernetes role
.view-eksview — read everything except Secrets
.deploy-eksview cluster-wide, plus edit in the TwPro namespaces on staging-silver
.devops, .adminsystem:masters

!!!warning view-eks opens a shell on dev and prod, but not on staging dev-emerald and prod-pita aggregate pods/exec into the view role, so .view-eks is enough to open a shell there. staging-silver does not, and its tw-pod-exec ClusterRole is bound to nobody — on staging a shell needs .deploy-eks, whose namespace-scoped edit carries exec.

Port-forward is unaffected: all three clusters grant it to view-eks, staging through a separate tw-pod-portforward binding. !!!

Usage​

Assuming you have multiple contexts (eg. dev-emerald and prod-pita) in your ~/.kube/config file, you can switch between them using the --context flag (or even better you can use kubectx to manage context and namespaces):

!!! In all the following commands you can use --context <context name> to select a specific cluster within your ~/.kube/config file. !!!

# List the pods of a specific environment (ex. _be-main-env-dev4_):
kubectl -n be-main-env-dev4 get pods

# Open a shell in a specific deployment (ex. _toolbox_):
kubectl exec -ti -n be-main-env-dev4 deploy/toolbox -- bash

# Open a Rails console directly:
kubectl exec -ti -n be-main-env-dev4 deploy/toolbox -- bin/rails c

# Show last 10 log lines of a pod:
kubectl -n be-main-env-dev4 logs web-b2b-5698f9587d-p8r47 --tail 10

# Tail the logs from all pods with the same hash (useful for example to monitor a set of running jobs):
kubectl -n be-main-env-dev4 logs -l pod-template-hash="cf9985dd8" -f

# Show status & extended info about a pod:
kubectl -n be-main-env-dev4 describe pods web-b2b-cf9985dd8-dz2fh

# Find an ingress in all namespaces:
kubectl get ingresses -A -o=custom-columns='NAMESPACE:metadata.namespace, DOMAIN:spec.rules[*].host' | grep 'api.uala.it'

# Copy a local file to a pod:
kubectl -n be-main-env-dev5 cp /path/to/local/file database-69dfd8f557-tkhn6:/tmp/filename.ext

Connect local ruby code to dev databases​

Once you have configured your kubectl locally and you have access to cluster you can proxy a local port to remote database to test and debug your local code against dev databases. Pay attention on what you're doing ☠️ ☠️

Add in config/database.yml the following under development.main key url: <%= ENV["DATABASE_URL"] %>, so that result is something like

development:
  main:
    <<: *default_postgres
    url:  <%= ENV["DATABASE_URL"] %>
    migrations_paths: db/main

Retrieve cluster database url from any pod running backend, lens or k9s are the best tools for this task, just fire a shell and type

echo $DATABASE_URL
DATABASE_URL=postgres://something:somepass@database-lb-rw-service:6432/uala_development

Once you got that you need to proxy any of your free local port to database service in the ENV above with a kubectl command like this one:

kubectl port-forward service/database-lb-rw-service -n be-main-env-devtw 26432:6432

Where 26432 is the chosen port. In order to use that tunnel just use this command locally and you're ready to go

DATABASE_URL='postgresql://<user>:<pass>@127.0.0.1:26432/<database_name>' rails console

Essentially you need to replace service name with 127.0.0.1 (not localhost!!!) and port with local port you choose earliear when you launched port-forward command.

Once you finished debugging or whatever you need to:

  • Close tunnel command with Ctrl+C
  • Revert edits made in database.yml

Some useful script​

kubectl_bash (ex. kubectl_bash backend-env-4)

#!/bin/bash

target=$1

if [ ! -z "$target" ]; then
  pod=$(kubectl -n $target get pods | grep -Eo 'web-[^-]*-[^- ]* ')
  echo -e "> kubectl bash on ${target} \\ ${pod}\n"
  kubectl exec -n ${target} --stdin --tty ${pod} -- /bin/bash
fi

Autoscaling​

!!!info Replicas updating Keep in mind that the autoscaler changes the replicas value, so a workload changing pod count during the day is normal.

The overlays in k8s-gitops-twpro deliberately leave replicas unset on every Deployment an HPA targets — no Deployment is both HPA-scaled and replicas-pinned. That is not an oversight: the TwPro ApplicationSet syncs with selfHeal: true, so a replicas value committed for an HPA-scaled Deployment would have ArgoCD reverting the autoscaler every time it scaled. !!!

!ref HPA !ref Automatically scale your Rails application with HPA

We leverage on K8S HPA feature to scale the system based on different rules, that mainly consider the current system load.

For Puma web servers, the "average RPM by workload metric is considered": when the RPM value is too high, a new replica is spinned.

Similarly, for Sidekiq background workers, both the "Avg jobs/min per worker By Queue (HPA)" and "Sidekiq Latency By Queue" metrics are used to solve different cases.