Skip to content

In case of emergency

This page provides a quick reference for the most common actions needed when responding to an incident.

Connecting to Cloud environments

Access to AWS is managed through the AWS Web Console and the AWS CLI. Access to Scaleway is managed through the Scaleway Web Console and the Scaleway CLI. Both require valid credentials for the relevant AWS account.

For AWS, navigate to the AWS access portal and sign in with SSO. Here you can select the target AWS account and role to assume, which will open a new browser tab with the AWS Web Console.

For Scaleway, navigate to the Scaleway access portal and sign in with SSO. Here you can select the target Scaleway project, and manage its resources.

Connecting to AWS instances

All EC2 instances are enrolled into AWS Systems Manager (SSM). This allows shell access without requiring a direct SSH connection, open inbound ports, or a bastion host.

Via the cloud-auth helper script

Ensure GETRECT is installed and configured. If your system is set up with the cloud-auth helper script, you can connect directly from your local terminal:

cd ~/projects/gitlabhost-prod/
cloud-auth gitlab-rails-1

Via the AWS Web Console

  1. Open the EC2 console and navigate to Instances.
  2. Select the instance you want to connect to.
  3. Click the Connect button and choose the Session Manager tab.
  4. Click Connect to open a root shell in your browser.

Connecting to Scaleway instances

Scaleway can be accessed over SSH through the managed Bastion host. This is the Public Gateway of each project.

Via the cloud-auth helper script

Ensure GETRECT is installed and configured. If your system is set up with the cloud-auth helper script, you can connect directly from your local terminal:

cd ~/projects/gitlabhost-prod/
cloud-auth gitlab-rails-1

Via the SSH client

Connect through the Bastion host using the following command. Replace <prefix> with the project prefix, <username> with your username, and <resource-name> with the name of the instance, or its private IP address.

You need to be connected to the VPN for this to work.

ssh -J bastion@bastion.<prefix>.glhc.nl:50022 <username>@<resource-name>

Viewing cluster monitoring data

A centralised Grafana instance is available for browsing metrics and logs across all clusters: https://grafana.int.glhc.nl

The following data sources are pre-configured:

Data source Description
Prometheus / Thanos Infrastructure and application metrics (CPU, RAM, request rates, error rates, etc.)
Loki Centralised log aggregation from all nodes — including /var/log, nginx, systemd journal, and GitLab application logs such as production_json.log
  1. Open the Grafana URL above and log in.
  2. Use the Dashboards menu to browse pre-configured dashboards for each cluster.
    • The most useful dashboards for troubleshooting are located under the GitLab Environment Toolkit folder:
      • Server Performance: Overview of server metrics.
      • GitLab Runner Overview: Overview of GitLab Runner metrics.
  3. Use Explore to view and search through logfiles across each cluster.