How to Install Prometheus and Grafana on Rocky Linux 10 (SELinux-Ready, with Node Exporter)
Prometheus collects and stores time-series metrics, Node Exporter exposes the health of a Linux host, and Grafana turns it all into dashboards. Together they are the most common open-source monitoring stack for servers, containers, and Kubernetes.
This guide installs the three components natively with systemd on Rocky Linux 10 (AlmaLinux 10 and RHEL 10 work the same), keeps SELinux enforcing, opens only the ports you need in firewalld, connects Grafana to Prometheus, imports a ready-made dashboard, and finishes with security and troubleshooting. A Docker-based alternative is covered in our existing Docker article; native installs are easier to patch with your normal OS process and to debug with journalctl.
Table of Contents
- Architecture and ports
- Prerequisites
- Step 1: Install Prometheus
- Step 2: Install Node Exporter
- Step 3: Configure scraping
- Step 4: Install Grafana
- Step 5: Add the data source and a dashboard
- SELinux notes
- Firewall and exposure
- Securing the stack
- Storage, retention and sizing
- Useful PromQL queries
- Next steps: alerting and logs
- Troubleshooting
- FAQ
1. Architecture and ports
+--------------------+ +----------------------+ +-----------------------+
| Monitored hosts | | Monitoring server | | Your browser |
| | | | | |
| node_exporter | <----- | Prometheus :9090 | <----- | Grafana :3000 |
| :9100 (metrics) | scrape | - scrapes targets | query | dashboards, alerts |
| | (pull) | - stores TSDB data | | data source = |
| other exporters | | - evaluates rules | | http://prometheus |
+--------------------+ | - sends alerts --> | +-----------------------+
| Alertmanager |
| :9093 |
+----------------------+
| Component | Port | Role |
|---|---|---|
| Prometheus | 9090 | Scrapes, stores, and queries metrics |
| Node Exporter | 9100 | Exposes CPU, memory, disk, network, and systemd metrics |
| Grafana | 3000 | Dashboards and visualization |
| Alertmanager | 9093 | Alert routing (see the Alertmanager guide) |
Prometheus is pull-based: the server fetches /metrics from each target. That means monitored hosts only need an open port toward the Prometheus server, not outbound access.
2. Prerequisites
| Requirement | Detail |
|---|---|
| OS | Rocky Linux 10 (AlmaLinux 10 and RHEL 10 work the same) |
| Resources | 2 vCPU and 2 GB RAM for a small setup; plan more for many targets |
| Disk | 20 GB or more for metrics (see section 11) |
| Access | sudo user; outbound HTTPS to GitHub and rpm.grafana.com |
| SELinux | Enforcing (we keep it) |
sudo dnf -y update
sudo dnf install -y tar curl
sudo timedatectl set-ntp true # accurate time matters for metrics
3. Step 1: Install Prometheus
Prometheus is not in the default Rocky repositories, so install the official binary. Choose a version you have checked on the release page, and verify the checksum.
# Set to the version you verified (the 3.13 LTS line or the latest 3.x release)
PROM_VERSION=3.13.0
cd /tmp
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/sha256sums.txt"
grep "prometheus-${PROM_VERSION}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -
tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
cd "prometheus-${PROM_VERSION}.linux-amd64"
Create the user, directories, and install the binaries:
sudo useradd --system --no-create-home --shell /sbin/nologin prometheus
sudo mkdir -p /etc/prometheus/rules /var/lib/prometheus
sudo install -m 0755 prometheus promtool /usr/local/bin/
sudo cp prometheus.yml /etc/prometheus/prometheus.yml
sudo cp -r consoles console_libraries /etc/prometheus/ 2>/dev/null || true # not present in every release
sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo chown -R root:prometheus /etc/prometheus
sudo chmod -R 750 /etc/prometheus
prometheus --version
Create the systemd service /etc/systemd/system/prometheus.service:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--web.listen-address=127.0.0.1:9090 \
--web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
[Install]
WantedBy=multi-user.target
Notes on the flags:
| Flag | Meaning |
|---|---|
--storage.tsdb.retention.time=30d | Keep 30 days of data (default is 15 days) |
--web.listen-address=127.0.0.1:9090 | Local only; Grafana on the same host can still reach it. Change deliberately when you need remote access |
--web.enable-lifecycle | Allows POST /-/reload; it is unauthenticated, so keep Prometheus on localhost or behind authentication |
Start and check:
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
systemctl status prometheus --no-pager
curl -s http://127.0.0.1:9090/-/healthy
4. Step 2: Install Node Exporter
Node Exporter should run on every host you want to monitor, including the monitoring server itself.
NODE_EXPORTER_VERSION=1.10.2 # verify the latest release on the project page
cd /tmp
curl -fsSLO "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz"
curl -fsSLO "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/sha256sums.txt"
grep "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -
tar xzf "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz"
sudo useradd --system --no-create-home --shell /sbin/nologin node_exporter
sudo install -m 0755 "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64/node_exporter" /usr/local/bin/
Create /etc/systemd/system/node_exporter.service:
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
--collector.systemd \
--web.listen-address=:9100
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | head -5
--collector.systemd adds service state metrics, which are useful for alerts such as “unit failed”. On remote hosts, restrict who can reach port 9100 (section 9).
5. Step 3: Configure scraping
Edit /etc/prometheus/prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ["127.0.0.1:9093"] # remove until you install Alertmanager
rule_files:
- /etc/prometheus/rules/*.yml
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["127.0.0.1:9090"]
- job_name: node
static_configs:
- targets: ["127.0.0.1:9100"]
labels: { role: monitoring }
- targets: ["192.168.10.21:9100", "192.168.10.22:9100"]
labels: { role: web }
Validate and reload:
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
Open http://<server>:9090/targets (or use an SSH tunnel) and confirm every target shows UP. For many hosts, replace static lists with file-based service discovery (file_sd_configs) so you can add targets without editing the main config.
6. Step 4: Install Grafana
Grafana provides an official RPM repository. The OSS edition is installed from it with signature checking.
sudo tee /etc/yum.repos.d/grafana.repo > /dev/null <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF
sudo dnf install -y grafana
grafana-server --version
Check which line you received with dnf info grafana. The 12.4.x branch continues to get patch releases, and the 13.x line is the newest; for a production system pick one major line and plan upgrades deliberately. To pin a version, use dnf install grafana-<version> and add exclude=grafana* to the repo file when you do not want automatic upgrades.
Start it:
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
systemctl status grafana-server --no-pager
Grafana listens on port 3000. Log in at http://<server>:3000 with the default admin / admin and change the password immediately when prompted.
Harden the configuration in /etc/grafana/grafana.ini:
[server]
http_addr = 127.0.0.1
http_port = 3000
domain = grafana.example.com
root_url = https://grafana.example.com/
[security]
disable_gravatar = true
cookie_secure = true
[users]
allow_sign_up = false
[analytics]
reporting_enabled = false
check_for_updates = false
sudo systemctl restart grafana-server
With http_addr = 127.0.0.1, access goes through a reverse proxy with TLS; see Nginx with Let’s Encrypt on Rocky Linux 10. While testing, leave the default listener or use an SSH tunnel: ssh -L 3000:127.0.0.1:3000 user@server.
7. Step 5: Add the data source and a dashboard
Add Prometheus as a data source:
- In Grafana, open Connections > Data sources > Add data source > Prometheus.
- URL:
http://127.0.0.1:9090. - Click Save & test. You should see a success message.
Alternatively, provision it from a file so it survives rebuilds. Create /etc/grafana/provisioning/datasources/prometheus.yml:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://127.0.0.1:9090
isDefault: true
sudo systemctl restart grafana-server
Import a dashboard:
- Dashboards > New > Import.
- Enter dashboard ID 1860 (the community “Node Exporter Full” dashboard) and click Load.
- Select your Prometheus data source and import.
You now have CPU, memory, disk, network, and system graphs per host, with a drop-down to switch between instances. Treat community dashboards as a starting point: check which metrics they use and remove panels you do not need.
8. SELinux notes
Keep SELinux enforcing. In practice:
- Binaries in
/usr/local/binthat are started by systemd typically run in the genericunconfined_service_tdomain, so Prometheus and Node Exporter work without extra policy. Verify withps -eZ | grep -E 'prometheus|node_exporter'. - The Grafana RPM ships with its own file contexts and works out of the box.
- If you use a custom data directory for Prometheus on another disk, make sure it is accessible: set ownership with
chown prometheus:prometheusand check for denials withsudo ausearch -m AVC -ts recent. - If Nginx proxies to Grafana and returns 502, allow the connection with
setsebool -P httpd_can_network_connect 1or label the port; details are in the SELinux troubleshooting guide.
9. Firewall and exposure
Open only what each host needs. Use rich rules to limit Prometheus access to Node Exporter:
# On every monitored host: allow port 9100 only from the Prometheus server
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.10.5" port port="9100" protocol="tcp" accept'
sudo firewall-cmd --reload
# On the monitoring server: Grafana only for your admin network (or keep it on localhost behind Nginx)
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/24" port port="3000" protocol="tcp" accept'
sudo firewall-cmd --reload
Do not open 9090 or 9100 to the internet. Node Exporter reveals host details that help attackers, and Prometheus has no built-in authentication. For more patterns see the firewalld guide.
10. Securing the stack
| Risk | Mitigation |
|---|---|
| Unauthenticated Prometheus | Listen on localhost, or put it behind a reverse proxy with basic auth or TLS; Prometheus also supports --web.config.file for TLS and basic auth |
| Exposed exporters | Firewall to the Prometheus address; use TLS and auth via --web.config.file on exporters when crossing untrusted networks |
| Default Grafana password | Change on first login; disable sign-up; use SSO (LDAP or OAuth) for teams |
| Secrets in dashboards or data sources | Use provisioning with environment variables, and restrict who can edit data sources |
| Unpatched components | Subscribe to release notes; patch Grafana through DNF, and update Prometheus and exporters on a schedule |
| Lifecycle endpoint abuse | Keep --web.enable-lifecycle only if the API is not reachable by untrusted users |
| Admin API | Leave --web.enable-admin-api off unless needed |
11. Storage, retention and sizing
Prometheus stores samples in a local time-series database (TSDB). A rough sizing formula:
disk = retention seconds x ingested samples per second x bytes per sample (about 1 to 2)
# Current samples per second ingested (in the Prometheus UI or via API)
rate(prometheus_tsdb_head_samples_appended_total[5m])
# Disk used by the TSDB
du -sh /var/lib/prometheus
| Environment | Rough scale | Guidance |
|---|---|---|
| Homelab, 5 to 20 hosts | Thousands of series per host | 2 GB RAM, 20 GB disk, 15 to 30 days retention |
| Small business, 50 to 200 hosts | Hundreds of thousands of series | 8 GB RAM, SSD, tune scrape interval |
| Larger or long-term needs | Millions of series | Use remote storage (Thanos, Mimir, VictoriaMetrics) and federation |
Control cost with a sensible scrape_interval (15 to 60 seconds), by dropping unneeded metrics with metric_relabel_configs, and by setting --storage.tsdb.retention.size as a safety cap, for example --storage.tsdb.retention.size=50GB.
Back up the TSDB with snapshots if you enable the admin API (/api/v1/admin/tsdb/snapshot), or accept that metrics are replaceable and back up only configuration and dashboards. Store /etc/prometheus, /etc/grafana, and dashboard JSON in Git.
12. Useful PromQL queries
| Purpose | Query |
|---|---|
| Is the target up? | up |
| CPU usage % per host | 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) |
| Memory used % | (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 |
| Root disk used % | 100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) |
| Network receive rate | rate(node_network_receive_bytes_total{device!="lo"}[5m]) |
| Disk I/O time | rate(node_disk_io_time_seconds_total[5m]) |
| System load per core | node_load1 / count by (instance) (node_cpu_seconds_total{mode="idle"}) |
| Failed systemd units | node_systemd_unit_state{state="failed"} == 1 |
Use rate() on counters and compare gauges directly. Always test queries in the Prometheus Graph tab before putting them into dashboards or alert rules.
13. Next steps: alerting and logs
- Alerts to Slack or email: Prometheus Alertmanager: Alert Rules, Routing & Slack Notifications
- Logs next to metrics: Loki + Grafana Log Aggregation Setup
- Kubernetes and application traces: OpenTelemetry on Kubernetes
- Choosing between tools: Zabbix vs Nagios vs HertzBeat
- Certificate expiry and uptime probes: add the blackbox exporter and alert before certificates run out, which matters more as certificate lifetimes shrink.
14. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Prometheus will not start | YAML error, permissions, or port in use | promtool check config ...; journalctl -u prometheus -n 50; ss -ltnp | grep 9090 |
permission denied on /var/lib/prometheus | Wrong ownership or SELinux | chown -R prometheus:prometheus /var/lib/prometheus; check ausearch -m AVC |
Target shows DOWN connection refused | Exporter not running or firewall | systemctl status node_exporter; test with curl http://host:9100/metrics |
Target DOWN context deadline exceeded | Network blocked or exporter too slow | Check firewalld and routing; raise scrape_timeout (must be below scrape_interval) |
| Config reload has no effect | Reload not accepted | Check journalctl; use systemctl restart prometheus once to confirm |
| Grafana “Bad Gateway” or data source error | Wrong URL, or Prometheus bound elsewhere | Use http://127.0.0.1:9090; if Grafana is remote, bind Prometheus appropriately and firewall it |
| Dashboard shows “No data” | Wrong data source, job label, or time range | Select the data source; check that metric names exist in Prometheus |
| Dashboard variables empty | Labels differ from the dashboard’s assumptions | Edit variable queries to match your labels (instance, job) |
dnf cannot reach the Grafana repo | DNS or proxy issue | curl -I https://rpm.grafana.com; configure the proxy for dnf |
| Disk fills up quickly | High cardinality or long retention | Reduce series, drop metrics, set retention.size, add disk |
| High memory use | Too many series | Check prometheus_tsdb_head_series; drop high-cardinality labels |
| Time drift errors, out-of-order samples | Clock skew | Enable chrony; check NTP on all hosts |
| Nginx returns 502 for Grafana | SELinux blocks proxy connection | setsebool -P httpd_can_network_connect 1 or label port 3000 |
| Grafana redirect loop behind proxy | root_url or headers wrong | Set root_url to the external URL; pass Host and X-Forwarded-Proto |
15. FAQ
Is Prometheus available in the Rocky Linux repositories?
Not in the default ones. You install the official binary, as shown here, or use a container or Kubernetes operator.
Which Prometheus version should I choose?
Use an LTS line for stability, or the latest release if you want newer features. Check the release page, verify the checksum, and plan regular updates.
Do I need Grafana?
Prometheus has a basic UI, but Grafana provides far better dashboards, sharing, and alert management. Most teams use both.
Can I monitor Windows servers and network devices?
Yes. Use the Windows exporter for Windows hosts and the SNMP exporter for switches and routers.
How long should I keep metrics?
Fifteen to thirty days is typical for troubleshooting. For long-term trends, use downsampling or remote storage.
Docker or native install?
Both work. Native installs use your existing OS patching and systemd tooling, and Docker makes it easy to move or replicate the stack. Pick the one your team can maintain.
How do I upgrade Prometheus?
Download and verify the new release, stop the service, replace the binaries, and start it again. Read release notes for flag changes, and test the configuration with promtool first.







