How to Install Prometheus and Grafana on Rocky Linux 10 (SELinux-Ready, with Node Exporter)

Prometheus collects and stores time-series metrics, Node Exporter exposes the health of a Linux host, and Grafana turns it all into dashboards. Together they are the most common open-source monitoring stack for servers, containers, and Kubernetes.

This guide installs the three components natively with systemd on Rocky Linux 10 (AlmaLinux 10 and RHEL 10 work the same), keeps SELinux enforcing, opens only the ports you need in firewalld, connects Grafana to Prometheus, imports a ready-made dashboard, and finishes with security and troubleshooting. A Docker-based alternative is covered in our existing Docker article; native installs are easier to patch with your normal OS process and to debug with journalctl.

Table of Contents

  1. Architecture and ports
  2. Prerequisites
  3. Step 1: Install Prometheus
  4. Step 2: Install Node Exporter
  5. Step 3: Configure scraping
  6. Step 4: Install Grafana
  7. Step 5: Add the data source and a dashboard
  8. SELinux notes
  9. Firewall and exposure
  10. Securing the stack
  11. Storage, retention and sizing
  12. Useful PromQL queries
  13. Next steps: alerting and logs
  14. Troubleshooting
  15. FAQ

1. Architecture and ports

 +--------------------+        +----------------------+        +-----------------------+
 |  Monitored hosts   |        |  Monitoring server   |        |  Your browser         |
 |                    |        |                      |        |                       |
 |  node_exporter     | <----- |  Prometheus  :9090   | <----- |  Grafana  :3000       |
 |  :9100  (metrics)  | scrape |   - scrapes targets  | query  |   dashboards, alerts  |
 |                    | (pull) |   - stores TSDB data |        |   data source =       |
 |  other exporters   |        |   - evaluates rules  |        |   http://prometheus   |
 +--------------------+        |   - sends alerts --> |        +-----------------------+
                               |     Alertmanager     |
                               |     :9093            |
                               +----------------------+
ComponentPortRole
Prometheus9090Scrapes, stores, and queries metrics
Node Exporter9100Exposes CPU, memory, disk, network, and systemd metrics
Grafana3000Dashboards and visualization
Alertmanager9093Alert routing (see the Alertmanager guide)

Prometheus is pull-based: the server fetches /metrics from each target. That means monitored hosts only need an open port toward the Prometheus server, not outbound access.

2. Prerequisites

RequirementDetail
OSRocky Linux 10 (AlmaLinux 10 and RHEL 10 work the same)
Resources2 vCPU and 2 GB RAM for a small setup; plan more for many targets
Disk20 GB or more for metrics (see section 11)
Accesssudo user; outbound HTTPS to GitHub and rpm.grafana.com
SELinuxEnforcing (we keep it)
sudo dnf -y update
sudo dnf install -y tar curl
sudo timedatectl set-ntp true    # accurate time matters for metrics

3. Step 1: Install Prometheus

Prometheus is not in the default Rocky repositories, so install the official binary. Choose a version you have checked on the release page, and verify the checksum.

# Set to the version you verified (the 3.13 LTS line or the latest 3.x release)
PROM_VERSION=3.13.0

cd /tmp
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/sha256sums.txt"
grep "prometheus-${PROM_VERSION}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -

tar xzf "prometheus-${PROM_VERSION}.linux-amd64.tar.gz"
cd "prometheus-${PROM_VERSION}.linux-amd64"

Create the user, directories, and install the binaries:

sudo useradd --system --no-create-home --shell /sbin/nologin prometheus
sudo mkdir -p /etc/prometheus/rules /var/lib/prometheus

sudo install -m 0755 prometheus promtool /usr/local/bin/
sudo cp prometheus.yml /etc/prometheus/prometheus.yml
sudo cp -r consoles console_libraries /etc/prometheus/ 2>/dev/null || true   # not present in every release

sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo chown -R root:prometheus /etc/prometheus
sudo chmod -R 750 /etc/prometheus

prometheus --version

Create the systemd service /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.listen-address=127.0.0.1:9090 \
  --web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target

Notes on the flags:

FlagMeaning
--storage.tsdb.retention.time=30dKeep 30 days of data (default is 15 days)
--web.listen-address=127.0.0.1:9090Local only; Grafana on the same host can still reach it. Change deliberately when you need remote access
--web.enable-lifecycleAllows POST /-/reload; it is unauthenticated, so keep Prometheus on localhost or behind authentication

Start and check:

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
systemctl status prometheus --no-pager
curl -s http://127.0.0.1:9090/-/healthy

4. Step 2: Install Node Exporter

Node Exporter should run on every host you want to monitor, including the monitoring server itself.

NODE_EXPORTER_VERSION=1.10.2     # verify the latest release on the project page

cd /tmp
curl -fsSLO "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz"
curl -fsSLO "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/sha256sums.txt"
grep "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -

tar xzf "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz"
sudo useradd --system --no-create-home --shell /sbin/nologin node_exporter
sudo install -m 0755 "node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64/node_exporter" /usr/local/bin/

Create /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --collector.systemd \
  --web.listen-address=:9100
Restart=on-failure
RestartSec=5s
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | head -5

--collector.systemd adds service state metrics, which are useful for alerts such as “unit failed”. On remote hosts, restrict who can reach port 9100 (section 9).

5. Step 3: Configure scraping

Edit /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["127.0.0.1:9093"]     # remove until you install Alertmanager

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["127.0.0.1:9090"]

  - job_name: node
    static_configs:
      - targets: ["127.0.0.1:9100"]
        labels: { role: monitoring }
      - targets: ["192.168.10.21:9100", "192.168.10.22:9100"]
        labels: { role: web }

Validate and reload:

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

Open http://<server>:9090/targets (or use an SSH tunnel) and confirm every target shows UP. For many hosts, replace static lists with file-based service discovery (file_sd_configs) so you can add targets without editing the main config.

6. Step 4: Install Grafana

Grafana provides an official RPM repository. The OSS edition is installed from it with signature checking.

sudo tee /etc/yum.repos.d/grafana.repo > /dev/null <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

sudo dnf install -y grafana
grafana-server --version

Check which line you received with dnf info grafana. The 12.4.x branch continues to get patch releases, and the 13.x line is the newest; for a production system pick one major line and plan upgrades deliberately. To pin a version, use dnf install grafana-<version> and add exclude=grafana* to the repo file when you do not want automatic upgrades.

Start it:

sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
systemctl status grafana-server --no-pager

Grafana listens on port 3000. Log in at http://<server>:3000 with the default admin / admin and change the password immediately when prompted.

Harden the configuration in /etc/grafana/grafana.ini:

[server]
http_addr = 127.0.0.1
http_port = 3000
domain = grafana.example.com
root_url = https://grafana.example.com/

[security]
disable_gravatar = true
cookie_secure = true

[users]
allow_sign_up = false

[analytics]
reporting_enabled = false
check_for_updates = false
sudo systemctl restart grafana-server

With http_addr = 127.0.0.1, access goes through a reverse proxy with TLS; see Nginx with Let’s Encrypt on Rocky Linux 10. While testing, leave the default listener or use an SSH tunnel: ssh -L 3000:127.0.0.1:3000 user@server.

7. Step 5: Add the data source and a dashboard

Add Prometheus as a data source:

  1. In Grafana, open Connections > Data sources > Add data source > Prometheus.
  2. URL: http://127.0.0.1:9090.
  3. Click Save & test. You should see a success message.

Alternatively, provision it from a file so it survives rebuilds. Create /etc/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true
sudo systemctl restart grafana-server

Import a dashboard:

  1. Dashboards > New > Import.
  2. Enter dashboard ID 1860 (the community “Node Exporter Full” dashboard) and click Load.
  3. Select your Prometheus data source and import.

You now have CPU, memory, disk, network, and system graphs per host, with a drop-down to switch between instances. Treat community dashboards as a starting point: check which metrics they use and remove panels you do not need.

8. SELinux notes

Keep SELinux enforcing. In practice:

  • Binaries in /usr/local/bin that are started by systemd typically run in the generic unconfined_service_t domain, so Prometheus and Node Exporter work without extra policy. Verify with ps -eZ | grep -E 'prometheus|node_exporter'.
  • The Grafana RPM ships with its own file contexts and works out of the box.
  • If you use a custom data directory for Prometheus on another disk, make sure it is accessible: set ownership with chown prometheus:prometheus and check for denials with sudo ausearch -m AVC -ts recent.
  • If Nginx proxies to Grafana and returns 502, allow the connection with setsebool -P httpd_can_network_connect 1 or label the port; details are in the SELinux troubleshooting guide.

9. Firewall and exposure

Open only what each host needs. Use rich rules to limit Prometheus access to Node Exporter:

# On every monitored host: allow port 9100 only from the Prometheus server
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.10.5" port port="9100" protocol="tcp" accept'
sudo firewall-cmd --reload

# On the monitoring server: Grafana only for your admin network (or keep it on localhost behind Nginx)
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/24" port port="3000" protocol="tcp" accept'
sudo firewall-cmd --reload

Do not open 9090 or 9100 to the internet. Node Exporter reveals host details that help attackers, and Prometheus has no built-in authentication. For more patterns see the firewalld guide.

10. Securing the stack

RiskMitigation
Unauthenticated PrometheusListen on localhost, or put it behind a reverse proxy with basic auth or TLS; Prometheus also supports --web.config.file for TLS and basic auth
Exposed exportersFirewall to the Prometheus address; use TLS and auth via --web.config.file on exporters when crossing untrusted networks
Default Grafana passwordChange on first login; disable sign-up; use SSO (LDAP or OAuth) for teams
Secrets in dashboards or data sourcesUse provisioning with environment variables, and restrict who can edit data sources
Unpatched componentsSubscribe to release notes; patch Grafana through DNF, and update Prometheus and exporters on a schedule
Lifecycle endpoint abuseKeep --web.enable-lifecycle only if the API is not reachable by untrusted users
Admin APILeave --web.enable-admin-api off unless needed

11. Storage, retention and sizing

Prometheus stores samples in a local time-series database (TSDB). A rough sizing formula:

disk = retention seconds x ingested samples per second x bytes per sample (about 1 to 2)

# Current samples per second ingested (in the Prometheus UI or via API)
rate(prometheus_tsdb_head_samples_appended_total[5m])

# Disk used by the TSDB
du -sh /var/lib/prometheus
EnvironmentRough scaleGuidance
Homelab, 5 to 20 hostsThousands of series per host2 GB RAM, 20 GB disk, 15 to 30 days retention
Small business, 50 to 200 hostsHundreds of thousands of series8 GB RAM, SSD, tune scrape interval
Larger or long-term needsMillions of seriesUse remote storage (Thanos, Mimir, VictoriaMetrics) and federation

Control cost with a sensible scrape_interval (15 to 60 seconds), by dropping unneeded metrics with metric_relabel_configs, and by setting --storage.tsdb.retention.size as a safety cap, for example --storage.tsdb.retention.size=50GB.

Back up the TSDB with snapshots if you enable the admin API (/api/v1/admin/tsdb/snapshot), or accept that metrics are replaceable and back up only configuration and dashboards. Store /etc/prometheus, /etc/grafana, and dashboard JSON in Git.

12. Useful PromQL queries

PurposeQuery
Is the target up?up
CPU usage % per host100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
Memory used %(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
Root disk used %100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)
Network receive raterate(node_network_receive_bytes_total{device!="lo"}[5m])
Disk I/O timerate(node_disk_io_time_seconds_total[5m])
System load per corenode_load1 / count by (instance) (node_cpu_seconds_total{mode="idle"})
Failed systemd unitsnode_systemd_unit_state{state="failed"} == 1

Use rate() on counters and compare gauges directly. Always test queries in the Prometheus Graph tab before putting them into dashboards or alert rules.

13. Next steps: alerting and logs

14. Troubleshooting

SymptomLikely causeFix
Prometheus will not startYAML error, permissions, or port in usepromtool check config ...; journalctl -u prometheus -n 50; ss -ltnp | grep 9090
permission denied on /var/lib/prometheusWrong ownership or SELinuxchown -R prometheus:prometheus /var/lib/prometheus; check ausearch -m AVC
Target shows DOWN connection refusedExporter not running or firewallsystemctl status node_exporter; test with curl http://host:9100/metrics
Target DOWN context deadline exceededNetwork blocked or exporter too slowCheck firewalld and routing; raise scrape_timeout (must be below scrape_interval)
Config reload has no effectReload not acceptedCheck journalctl; use systemctl restart prometheus once to confirm
Grafana “Bad Gateway” or data source errorWrong URL, or Prometheus bound elsewhereUse http://127.0.0.1:9090; if Grafana is remote, bind Prometheus appropriately and firewall it
Dashboard shows “No data”Wrong data source, job label, or time rangeSelect the data source; check that metric names exist in Prometheus
Dashboard variables emptyLabels differ from the dashboard’s assumptionsEdit variable queries to match your labels (instance, job)
dnf cannot reach the Grafana repoDNS or proxy issuecurl -I https://rpm.grafana.com; configure the proxy for dnf
Disk fills up quicklyHigh cardinality or long retentionReduce series, drop metrics, set retention.size, add disk
High memory useToo many seriesCheck prometheus_tsdb_head_series; drop high-cardinality labels
Time drift errors, out-of-order samplesClock skewEnable chrony; check NTP on all hosts
Nginx returns 502 for GrafanaSELinux blocks proxy connectionsetsebool -P httpd_can_network_connect 1 or label port 3000
Grafana redirect loop behind proxyroot_url or headers wrongSet root_url to the external URL; pass Host and X-Forwarded-Proto

15. FAQ

Is Prometheus available in the Rocky Linux repositories?
Not in the default ones. You install the official binary, as shown here, or use a container or Kubernetes operator.

Which Prometheus version should I choose?
Use an LTS line for stability, or the latest release if you want newer features. Check the release page, verify the checksum, and plan regular updates.

Do I need Grafana?
Prometheus has a basic UI, but Grafana provides far better dashboards, sharing, and alert management. Most teams use both.

Can I monitor Windows servers and network devices?
Yes. Use the Windows exporter for Windows hosts and the SNMP exporter for switches and routers.

How long should I keep metrics?
Fifteen to thirty days is typical for troubleshooting. For long-term trends, use downsampling or remote storage.

Docker or native install?
Both work. Native installs use your existing OS patching and systemd tooling, and Docker makes it easy to move or replicate the stack. Pick the one your team can maintain.

How do I upgrade Prometheus?
Download and verify the new release, stop the service, replace the binaries, and start it again. Read release notes for flag changes, and test the configuration with promtool first.

(Visited 11 times, 2 visits today)

You may also like