A Basis Life: Docker: Setting up Prometheus, Grafana and Postgres with TLS via Step-Ca

Docker: Setting up Prometheus, Grafana and Postgres with TLS via Step-Ca

Following on from my post about setting up Gitea and the adventures into docker here https://www.rjruss.info/2024/11/docker-setting-up-gitea-with-postgres.html

I decided to add Grafana and Prometheus into my homelab setup. Using TLS via step-ca as well, as using encryption is an increasingly important activity in my day job.

It got complicated via some cyber security reasons around storing passwords in clear text. Some security frameworks state that this issue is an instant fail/breach. Grafana stores the database password in clear text. I did mask the Grafana password by using Postgres pure “cert” authentication and not use the actual password but chose to use a “-” character in the Grafana ini file.

The Postgres setup was the main driver to split out my docker compose and Dockerfiles into single targeted service related files. I wanted to use my Postgres config in my Gitea project but it was tangled up inside the global Docker and compose files of my original Gitea project. So I chose to split it all out into individual compose and docker files (that are dependent) and combine them into a controlling compose file using the “include” notation. It does not use any provided docker hub or other setup, I chose to install the software during the build on a prepared base image. The version control is in the .env file. And also the entry command scripts are based on one general control script “template_start_script.sh” to try and keep consistency in the approach.

An important step for enabling consistent certificate generation is time syncing across servers. I set up the following on a new Ubuntu server connecting to the step-ca running on an Alma linux host. I use Google and Cloudflare time sync for my linux servers, time sync is not covered here but an example to check time sync source follows. Creating certificates will fail if the time between servers is out of the permitted range.

Ubuntu checks with timedatectl

# timedatectl show-timesync

SystemNTPServers=time.google.com time.cloudflare.com

FallbackNTPServers=ntp.ubuntu.com

Alma checks with chronyc

# chronyc sources

MS Name/IP address Stratum Poll Reach LastRx Last sample

===============================================================================

^+ time2.google.com 1 6 77 17 -1734us[ +245us] +/- 11ms

^* time.cloudflare.com 3 6 77 16 +1937us[+3929us] +/- 15ms

The following is based on downloading the git repo and setting up the pre-reqs, it's been tested on Alma and Ubuntu linux systems.

*All steps need to be executed by the root user unless otherwise stated

Step 1 setup local user and directory

Creating a dedicated docker user in this example dockeruser1 - if this is changed the .env file needs to be adapted in step 2

The directory where the downloaded project is stored can be adapted as well.

The example uses /srv/docker-config

Example using the name of the downloaded zip as prometheus-grafana-postgres-docker

sudo su -

useradd -m -s /bin/bash -u 1245 dockeruser1

mkdir /srv/docker-config

cd /srv/docker-config

unzip {downloaded zip file} -d .

cd /srv/docker-config/prometheus-grafana-postgres-docker

./initialGitHubScript.sh

Step 2 Adapt the .env file

Change bold entries if required and ignore any other settings in the .env file

Local docker user and group needs to match the user created in step 1

# check Env values

#LOCAL_SETUP - used by first_setup.sh scripts

LOCAL_DOCKER_USER=dockeruser1

LOCAL_DOCKER_GROUP=dockeruser1

LOCAL_DOCKER_VOLUME_DIR=/docker-vol1

DOMAIN=rjruss.org

MX_SHARED_ENV_GROUP=5501

BASE_HOST=ubhost-

STEP_HOST=${BASE_HOST}step

POSTDB_HOST=${BASE_HOST}post1

# STEP value for default app is the container name - indicated by *-run host

TX_STEP_HOST=https://newhost-step.rjruss.org:9011

GRAFANA_HOST=${BASE_HOST}grafana

TX_GRAFANA_PORT=5009

PROMETHEUS_HOST=${BASE_HOST}prom

TX_PROMETHEUS_PORT=9091

TX_ALERTMANAGER_PORT=9094

NODE_EXPORTER_HOST=${BASE_HOST}nodeexp

TX_NODE_EXPORTER_PORT=9021

MY_HOST_FQDN=hawubtest.rjruss.org

** it is important that the TX_STEP_HOST variable points to a working step-ca service check with the following command (using this as an example). This should display the root certificate of the step-ca service.

curl -k https://newhost-step.rjruss.org:9011/roots.pem

The above would be allow the services to be accessed

Via the docker user

Adapt the dockeruser1 - group dockeruser1 is used by default

Via the shared group

Adapt the group id of 5501 a new group will be created with that gid

Via a new directory/volume (it will be created if it does not exist)

Adapt the /docker-vol1

Via the following domain, hostname and ports

rjruss.org = this needs to be adapted to the required domain

ubhost-step = for step-ca hostname

ubhost-post1 = postgres hostname

ubhost-grafana = grafana hostname

ubhost-prom = prometheus hostname

ubhost-nodeexp = node exporter hostname

Grafana port = 5009

Prometheus port = 9091

Alert Manager port = 9094

Node Exporter port = 9021

MY_HOST_FQDN = FQDN of current host server

(postgres is not by default open to connections other than on the docker network and uses the standard postgres port 5432 in the docker network)

Step 3 adapt the age related password files

The age (see step 4) command is used to store passwords in an encrypted file on a docker volume. These can be shared between containers via the bash scripts in the base docker container build.

These .*.info files should be deleted ***after completing the step 4 process***

#Grafana DB connection password (not used but can be used if postgres config adapted)

Step 4 Create Volumes, Networks and encrypted passwords

#Edit file .GRAFANA.info

DB_PW="testing!1234!"

#Edit file .POST.info

POSTGRES_PASSWORD="testing!123"

#Edit file .STEP.info

PW="testing!234"

Following script installs operating system pre-req packages, volumes, networks and encrypts the passwords.

./first_setup.sh

After running the script the following volumes and network will be available.

[root@zhawalma monitor]# docker volume ls |grep tx

local tx_monitor_grafana_vol1

local tx_monitor_info_vol1

local tx_monitor_keys_vol1

local tx_monitor_node_exporter_vol1

local tx_monitor_pgdata_vol1

local tx_monitor_prometheus_vol1

[root@zhawalma monitor]# docker network ls |grep tx

5c122c720cc6 tx-monitor-net1 bridge local

[root@zhawalma monitor]#

**** delete these password files ****ENSURE YOU KNOW THE PASSWORDS before deleting**** the current password used in all files is testing!123

ls -a .*.info

rm -rf .GRAFANA.info .POST.info .STEP.info .TEST.info

Step 5 build and run it

The script “build-it-and-run-it.sh” checks the step-ca hostname via TX_STEP_HOST in the .env file. As all services rely on TLS and step-ca then the script will fail if it cannot resolve the defined step host.

su - dockeruser1

cd /srv/docker-config/prometheus-grafana-postgres-docker

./build-it-and-run-it.sh

Step 6 Setup host windows/client computer

Install step-ca root certificate on host windows

Download the root CA certificate and ignore the certificate with the “-k” option as it will be subsequently installed. This is based on the step-ca running in this project.

#curl -k -o stepCA.crt https://10.10.50.216:9011/roots.pem

#use hostname and port set in .env for the step-ca service (step 2 above)

curl -k -o stepCA.crt https://newhost-step.rjruss.org:9011/roots.pem

Using Powershell add the downloaded certificate as trusted.

Import-Certificate -FilePath ".\stepCA.crt" -CertStoreLocation cert:\CurrentUser\Root

Answer YES at the prompt