Docker: Setting up Prometheus, Grafana and Postgres with TLS via Step-Ca

 

Docker: Setting up Prometheus, Grafana and Postgres with TLS via Step-Ca

Project Github link

Following on from my post about setting up Gitea and the adventures into docker here https://www.rjruss.info/2024/11/docker-setting-up-gitea-with-postgres.html

I decided to add Grafana and Prometheus into my homelab setup. Using TLS via step-ca as well, as using encryption is an increasingly important activity in my day job.

It got complicated via some cyber security reasons around storing passwords in clear text. Some security frameworks state that this issue is an instant fail/breach. Grafana stores the database password in clear text. I did mask the Grafana password by using Postgres pure “cert” authentication and not use the actual password but chose to use a “-” character in the Grafana ini file. 

The Postgres setup was the main driver to split out my docker compose and Dockerfiles into single targeted service related files. I wanted to use my Postgres config in my Gitea project but it was tangled up inside the global Docker and compose files of my original Gitea project. So I chose to split it all out into individual compose and docker files (that are dependent) and combine them into a controlling compose file using the “include” notation. It does not use any provided docker hub or other setup, I chose to install the software during the build on a prepared base image. The version control is in the .env file. And also the entry command scripts are based on one general control script “template_start_script.sh” to try and keep consistency in the approach. 

An important step for enabling consistent certificate generation is time syncing across servers. I set up the following on a new Ubuntu server connecting to the step-ca running on an Alma linux host. I use Google and Cloudflare time sync for my linux servers, time sync is not covered here but an example to check time sync source follows. Creating certificates will fail if the time between servers is out of the permitted range.

Ubuntu checks with timedatectl 

# timedatectl show-timesync

SystemNTPServers=time.google.com time.cloudflare.com

FallbackNTPServers=ntp.ubuntu.com

Alma checks with chronyc

# chronyc sources

MS Name/IP address         Stratum Poll Reach LastRx Last sample

===============================================================================

^+ time2.google.com              1   6    77    17  -1734us[ +245us] +/-   11ms

^* time.cloudflare.com           3   6    77    16  +1937us[+3929us] +/-   15ms

The following is based on downloading the git repo and setting up the pre-reqs, it's been tested on Alma and Ubuntu linux systems.

*All steps need to be executed by the root user unless otherwise stated

Step 1 setup local user and directory

Creating a dedicated docker user in this example dockeruser1 - if this is changed the .env file needs to be adapted in step 2

The directory where the downloaded project is stored can be adapted as well.

The example uses  /srv/docker-config

Example using the name of the downloaded zip as prometheus-grafana-postgres-docker

sudo su -  useradd -m -s /bin/bash -u 1245 dockeruser1 mkdir /srv/docker-config cd /srv/docker-config unzip {downloaded zip file} -d . cd /srv/docker-config/prometheus-grafana-postgres-docker ./initialGitHubScript.sh

Step 2  Adapt the .env file

Change bold entries if required and ignore any other settings in the .env file

Local docker user and group needs to match the user created in step 1

# check Env values #LOCAL_SETUP - used by first_setup.sh scripts LOCAL_DOCKER_USER=dockeruser1 LOCAL_DOCKER_GROUP=dockeruser1 LOCAL_DOCKER_VOLUME_DIR=/docker-vol1 DOMAIN=rjruss.org MX_SHARED_ENV_GROUP=5501 BASE_HOST=ubhost- STEP_HOST=${BASE_HOST}step POSTDB_HOST=${BASE_HOST}post1 #  STEP value for default app is the container name - indicated by *-run host TX_STEP_HOST=https://newhost-step.rjruss.org:9011 GRAFANA_HOST=${BASE_HOST}grafana TX_GRAFANA_PORT=5009 PROMETHEUS_HOST=${BASE_HOST}prom TX_PROMETHEUS_PORT=9091 TX_ALERTMANAGER_PORT=9094 NODE_EXPORTER_HOST=${BASE_HOST}nodeexp TX_NODE_EXPORTER_PORT=9021 MY_HOST_FQDN=hawubtest.rjruss.org

** it is important that the TX_STEP_HOST variable points to a working step-ca service check with the following command (using this as an example). This should display the root certificate of the step-ca service.

curl -k https://newhost-step.rjruss.org:9011/roots.pem

The above would be allow the services to be accessed 

1. Via the docker user

Adapt the dockeruser1 - group dockeruser1 is used by default

2. Via the shared group 

Adapt the group id of 5501 a new group will be created with that gid

3. Via a new directory/volume (it will be created if it does not exist)

Adapt the /docker-vol1

4. Via the following domain, hostname and ports

rjruss.org = this needs to be adapted to the required domain

ubhost-step = for step-ca hostname

ubhost-post1 = postgres hostname

ubhost-grafana = grafana hostname

ubhost-prom = prometheus hostname

ubhost-nodeexp = node exporter hostname

Grafana port = 5009 

Prometheus port = 9091

Alert Manager port = 9094

Node Exporter port = 9021

MY_HOST_FQDN = FQDN of current host server

(postgres is not by default open to connections other than on the docker network and uses the standard postgres port 5432 in the docker network)

Step 3 adapt the age related password files

The age (see step 4) command is used to store passwords in an encrypted file on a docker volume. These can be shared between containers via the bash scripts in the base docker container build.

These .*.info files should be deleted ***after completing the step 4 process***

#Grafana DB connection password (not used but can be used if postgres config adapted)

Step 4 Create Volumes, Networks and encrypted passwords

#Edit file .GRAFANA.info DB_PW="testing!1234!" #Edit file .POST.info POSTGRES_PASSWORD="testing!123" #Edit file .STEP.info  PW="testing!234"

Following script installs operating system pre-req packages, volumes, networks and encrypts the passwords.

./first_setup.sh

After running the script the following volumes and network will be available.

[root@zhawalma monitor]# docker volume ls |grep tx

local     tx_monitor_grafana_vol1

local     tx_monitor_info_vol1

local     tx_monitor_keys_vol1

local     tx_monitor_node_exporter_vol1

local     tx_monitor_pgdata_vol1

local     tx_monitor_prometheus_vol1

[root@zhawalma monitor]# docker network ls |grep tx

5c122c720cc6   tx-monitor-net1   bridge    local

[root@zhawalma monitor]#

**** delete these password files  ****ENSURE YOU KNOW THE PASSWORDS before deleting**** the current password used in all files is testing!123

ls -a .*.info

rm -rf .GRAFANA.info  .POST.info  .STEP.info  .TEST.info

Step 5 build and run it

The script “build-it-and-run-it.sh” checks the step-ca hostname via TX_STEP_HOST in the .env file. As all services rely on TLS and step-ca then the script will fail if it cannot resolve the defined step host.

su - dockeruser1 cd /srv/docker-config/prometheus-grafana-postgres-docker ./build-it-and-run-it.sh

Step 6 Setup host windows/client computer

Install step-ca root certificate on host windows 

Download the root CA certificate and ignore the certificate with the “-k” option as it will be subsequently installed. This is based on the step-ca running in this project.

#curl -k -o stepCA.crt https://10.10.50.216:9011/roots.pem #use hostname and port set in .env for the step-ca service (step 2 above) curl -k -o stepCA.crt https://newhost-step.rjruss.org:9011/roots.pem

Using Powershell add the downloaded certificate as trusted.

Import-Certificate -FilePath ".\stepCA.crt" -CertStoreLocation cert:\CurrentUser\Root

Answer YES at the prompt

I have a homelab Hyper-V landscape with a windows DNS host, and it is important to set up dns resolution for Grafana and Prometheus.

For my local windows based P.C I updated the host file with the Grafana and Prometheus hostnames so they resolve.

By default the hostname is ubhost-grafana.rjruss.org and ubhost-prom.rjruss.org

Step 7 Access Grafana, Prometheus and Alert Manager

Grafana - user is “admin” with password set in .GRAF.info

https://ubhost-grafana.rjruss.org:5009/login

Prometheus

https://ubhost-prom.rjruss.org:9091/

Check the targets setup for node-exporter and prometheus - both should be “up”

Alert Manager 

https://ubhost-prom.rjruss.org:9094/

Miscellaneous actions

Remove demo certs from certstore, from the example code only - if the root CA name is changed then the *basestep* match won't match any…..

Get-ChildItem -Path Cert:\CurrentUser\Root | Where-Object { $_.Subject -like '*basestep*' } | ForEach-Object {     certutil -user -delstore Root $_.Thumbprint }

Docker: Setting up Gitea with Postgres using TLS via Step-Ca

 

Docker: Setting up Gitea with Postgres using TLS via Step-Ca

Project Github Link

I started with a simple objective to learn git as it was becoming more essential in my work in infrastructure. I was coming across more CI/CD pipelines with git repos and also thought it would be good to manage my own scripting in git. I also wanted to combine this with docker to improve my knowledge of docker and start using it for my setup on my homelab. 

It got complicated via some cyber security reasons around storing passwords in clear text. Some security frameworks state that this issue is an instant fail/breach. And my chosen Gitea implementation of git stores the database password in clear text. In the end it also stores tokens in clear text so I abandoned thoughts of bringing Gitea into my work setting but still using it on my homelab setup. I did mask the Gitea password by using Postgres pure “cert” authentication and can use a “-” character in the Gitea ini file. I moved the tokens out into dedicated files separate from the ini file and inside the container (not the docker volume). 

Also complicated by initially using one combined docker compose implementation and then wanting to use postgres in another use case. So I chose to split it all out into individual compose and docker files (that are dependent) and combine them into a controlling compose file using the “include” notation. It does not use any provided docker hub or other setup, I chose to install the software during the build on a prepared base image. The version control is in the .env file. And also the entry command scripts are based on one general control script “template_start_script.sh” to try and keep consistency in the approach. 

The following is based on downloading the git repo and setting up the pre-reqs, it's been tested on Alma and Ubuntu linux systems.

*All steps need to be executed by the root user unless otherwise stated

Step 1 setup local user and directory

Creating a dedicated docker user in this example dockeruser1 - if this is changed the .env file needs to be adapted in step 2

The directory where the downloaded project is stored can be adapted as well.

The example uses  /srv/docker-config

Example using the name of the downloaded zip as gitea-postgres-stepca-docker

# su to root  e.g.  sudo su -  useradd -m -s /bin/bash -u 1245 dockeruser1 mkdir /srv/docker-config       cd /srv/docker-config # Install zip if required - (“apt install -y zip” on ubuntu) or (“dnf install -y zip” alma/rocky) unzip {downloaded zip file} -d .

Run the initial script to copy the TEMPLATE env and info files into the actual files used in the setup

cd /srv/docker-config/gitea-postgres-stepca-docker ./initialGitHubScript.sh

Edit the .env file to adapt the entries in bold to the requirements on the host

The example would be the following,

Step 2  Adapt the .env file

Change bold entries if required and ignore any other settings in the .env file

Local docker user and group needs to match the user created in step 1

# check Env values #LOCAL_SETUP - used by first_setup.sh scripts LOCAL_DOCKER_USER=dockeruser1 LOCAL_DOCKER_GROUP=dockeruser1 LOCAL_DOCKER_VOLUME_DIR=/docker-vol1 DOMAIN=rjruss.org MX_SHARED_ENV_GROUP=5501 BASE_HOST=newhost- STEP_HOST=${BASE_HOST}step POSTDB_HOST=${BASE_HOST}post GITEA_HOST=${BASE_HOST}gitea #  STEP value for default app is the container name - indicated by *-run host MX_APP_STEP_PORT=9011 MX_APP_GITEA_PORT=3011 MX_APP_GITEA_SSH_PORT=2222

The above would be allow the services to be accessed 

1. Via the docker user

Adapt the dockeruser1 - group dockeruser1 is used by default

2. Via the shared group 

Adapt the group id of 5501 a new group will be created with that gid

3. Via a new directory/volume (it will be created if it does not exist)

Adapt the /docker-vol1

4. Via the following domain, hostname and ports

rjruss.org = this needs to be adapted to the required domain

newhost-step = for step-ca hostname

newhost-post = postgres hostname

newhost-gitea = gitea hostname

Step-ca port = 9011

Gitea port = 3011

Gitea ssh port = 2222

(postgres is not by default open to connections other than on the docker network and uses the standard postgres port 5432 in the docker network)

Step 3 adapt the age related password files

The age (see step 4) command is used to store passwords in an encrypted file on a docker volume. These can be shared between containers via the bash scripts in the base docker container build.

These .*.info files should be deleted ***after completing the step 4 process***

#Gitea DB connection password (not used but can be used if postgres config adapted)

#Edit file .GITEA.info DB_PW="testing!1234!" #Edit file .POST.info POSTGRES_PASSWORD="testing!123" #Edit file .STEP.info  PW="testing1234"

Step 4 first setup script

The first_setup.sh script performs the following and the setup will fail if these are not installed.

• Install standard operating system packages for docker, wget, curl, jq, ca-certificates, git and openssl

Also downloads and installs the following

• age “age is a simple, modern and secure file encryption tool, format, and Go library.” 

https://github.com/FiloSottile/age

• yq “a lightweight and portable command-line YAML, JSON and XML processor”

https://github.com/mikefarah/yq

The script also runs the scripts to create docker volumes based on the directory defined in variable LOCAL_DOCKER_VOLUME_DIR above. Also it creates a dedicated docker network for the containers in this project. It also uses the age command to encrypt passwords/secrets in the related docker volumes.

sudo su -  cd /srv/docker-config/gitea-postgres-stepca-docker ./first_setup.sh

After running the script the following volumes and network will be available.

docker volume ls

DRIVER    VOLUME NAME

local     mx_base_gitea_vol1

local     mx_base_info_vol1

local     mx_base_keys_vol1

local     mx_base_pgdata_vol1

local     mx_base_step_vol1

docker network ls

NETWORK ID     NAME           DRIVER    SCOPE

f9137c47b08f   bridge         bridge    local

eb0a2e835235   host           host      local

e9d5110fe553   mx-base-net1   bridge    local

854f22748de4   none           null      local

**** delete these password files  ****ENSURE YOU KNOW THE PASSWORDS before deleting**** the current password used in all files is testing!123

ls -a .*.info

rm -rf .GITEA.info  .POST.info  .STEP.info  .TEST.info

Step 5 build and run it

su - dockeruser1 cd /srv/docker-config/gitea-postgres-stepca-docker ./build-it-and-run-it.sh

Step 6 Setup host windows/client computer

Install step-ca root certificate on host windows 

Download the root CA certificate and ignore the certificate with the “-k” option as its will be subsequently installed.

#use hostname and port set in .env for the step-ca service (step 2 above) curl -k -o stepCA.crt https://newhost-step.rjruss.org:9011/roots.pem

Using Powershell add the downloaded certificate as trusted.

Import-Certificate -FilePath ".\stepCA.crt" -CertStoreLocation cert:\CurrentUser\Root

Answer YES at the prompt

Setup host/dns resolution for Gitea

Update the host file with the Gitea hostname so it resolves.

By default the hostname is newhost-gitea.rjruss.org

Step 7 Install Gitea and register User

Access the Gitea URL newhost-gitea.rjruss.org:3011

Do not change any settings on the Initial Configuration screen but scroll down to the bottom of the page

Hit the “Install Gitea” and wait for the green tea cup to change to the login/register user page

No users have been created for Gitea so register the required user.

Local git setup

  git config --global user.email "Email@used.above"

  git config --global user.name "Username-used-above"

Step 8 Restart the compose project 

By default at initial install Gitea insists on creating the LFS_JWT_SECRET token in the app.ini. By restarting the script will move this out to a dedicated file. This will only occur at the next restart of Gitea. Therefore restart the compose project.

docker compose -f c_docker_compose.yml stop

Output should be as follows with the default configuration

[+] Stopping 3/3

 ✔ Container base-gitea-run   Stopped                                                                                                                                                                   10.3s

 ✔ Container base-postdb-run  Stopped                                                                                                                                                                   10.2s

 ✔ Container base-step-run    Stopped 

The compose file c_docker_compose.yml uses the include notation to add all the individual compose files into one manageable file.

Start again

docker compose -f c_docker_compose.yml start

Output should be as follows with the default configuration

[+] Running 3/3

 ✔ Container base-step-run    Started                                                                                                                                                                    0.3s

 ✔ Container base-postdb-run  Healthy                                                                                                                                                                    5.7s

 ✔ Container base-gitea-run   Started 

Further posts to follow on adapting this Docker repo for other use cases.

Miscellaneous actions

Remove demo certs from certstore, from the example code only - if the root CA name is changed then the *basestep* match won't match any…..

Get-ChildItem -Path Cert:\CurrentUser\Root | Where-Object { $_.Subject -like '*basestep*' } | ForEach-Object {     certutil -user -delstore Root $_.Thumbprint }