I decided to operate the monitoring system on a separate VPS, so I set one up using my basic hosting repository where everything is prepared for a use-case like this (for details, see this article). My plan was to use Stefan Prodan's package as a foundation, which will reside behind the proxy, so once the basic setup was ready, I issued the following command:
How to use Grafana to monitor your servers
|
Read time:
3
minute(s)
In the past I had a lot of issues with one of my VPS: it slowed down from time to time to such a level that I couldn't even log in via SSH, so I had a hard time to find out what caused the slowdowns. So I set up a complete monitoring system where I can see on graphs which containers of which project uses how many resources, which ultimately helped fix the root cause.
bash
git clone [email protected]:stefanprodan/dockprom.gitBefore setting up the monitoring service by following the README, I have made some changes to the configuration.
Connect the monitoring stack to the reverse proxy
We have to add the hosting-with-docker_nginx_reverse_proxy network to the dockprom/docker-compose.yml file and to the services networks, so our monitoring service will be available on a public URL. Insert the following snippet into the dockprom/docker-compose.yml file (at around line 96), under the environment key of the grafana service. This tells the reverse proxy to route the requests coming to monitoring.domain.tld to the grafana service's port 3000, where the web UI is listening.
- LETSENCRYPT_HOST=monitoring.domain.tld
- VIRTUAL_HOST=monitoring.domain.tld
- VIRTUAL_PORT=3000We should also add
- hosting-with-docker_nginx_reverse_proxyto the networks key, so the section describing the grafana service will look something like this:
grafana:
image: grafana/grafana:7.1.5
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
environment:
- GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
- GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
- GF_USERS_ALLOW_SIGN_UP=false
- LETSENCRYPT_HOST=monitoring.domain.tld
- VIRTUAL_HOST=monitoring.domain.tld
- VIRTUAL_PORT=3000
restart: unless-stopped
expose:
- 3000
networks:
- monitor-net
- hosting-with-docker_nginx_reverse_proxy
labels:
org.label-schema.group: "monitoring"Add some sources!
In order to access the usage metrics of our VPS, we have to add them as sources. For this, open dockprom/prometheus/prometheus.yml, and add the following snippet under scrape_configs key:
- job_name: 'Trolo - nodeexporter'
static_configs:
- targets: ['1.2.3.4:9100']
- job_name: 'Trolo - cadvisor'
static_configs:
- targets: ['1.2.3.4:8080']This points Grafana and its underlying service, Prometheus, to our VPS named Trolo with IP address 1.2.3.4, at ports 9100 and 8080.
Set up our source VPS
The usage metrics referenced above are not exposed by default, something has to export them - this is what the dockprom/docker-compose.exporters.yaml file is for. Log in to your source VPS and clone the dockprom repository there, but instead of issueing the usual docker-compose up -d command, run this instead:
bash
docker-compose -f docker-compose.exporters.yml up -dThis will start a different set of services - once you are ready with this, you can go back to the monitoring VPS and start the services with docker-composer up -d. Visit monitoring.domain.tld where you should already see the Grafana web UI.
This is my Grafana dashboard.
Setting up a firewall
During this tutorial you might have been wondering, if we expose sensitive usage metrics about our VPS without any authentication, anyone could access them. You are right, so we should set up some firewall rules where we block requests for ports 8080 and 9100. If you use DigitalOcean, you can easily add a firewall ruleset, where you allow traffic for ports 80 and 443, while blocking all traffics for ports 8080 and 9100, except if it is coming from our monitoring VPS (just select the monitoring droplet from the dropdown).
Summary
Congratulations, you have successfully set up your monitoring system - you are at the start of the road, though. You can set up useful and awesome things with Grafana (alerts, uptime monitoring, etc.), but this takes time and tinkering.
