Monitoring with Prometheus and Grafana

You can monitor your Uyuni environment using Prometheus and Grafana. Uyuni Server and Proxy are able to provide self-health metrics. You can also install and manage a number of Prometheus exporters on Salt clients.

Prometheus and Grafana packages are included in the Uyuni Client Tools for:

  • SUSE Linux Enterprise 12

  • SUSE Linux Enterprise 15

  • CentOS  6

  • CentOS  7

  • CentOS  8

  • and openSUSE 15.x

You need to install Prometheus and Grafana on a machine separate from the Uyuni Server. We recommend you use a managed Salt client as your monitoring server.

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to monitored clients. Clients must have corresponding open ports and be reachable over the network. Alternatively, you can use reverse proxies to establish a connection.

Prometheus and Grafana

Prometheus

Prometheus is an open-source monitoring tool that is used to record real-time metrics in a time-series database. Metrics are pulled via HTTP, enabling high performance and scalability.

Prometheus metrics are time series data, or timestamped values belonging to the same group or dimension. A metric is uniquely identified by its name and set of labels.

   metric name               labels                 timestamp     value
┌────────┴───────┐  ┌───────────┴───────────┐    ┌──────┴──────┐  ┌─┴─┐
http_requests_total{status="200", method="GET"}  @1557331801.111  42236

Each application or system being monitored must expose metrics in the format above, either through code instrumentation or Prometheus exporters.

Prometheus Exporters

Exporters are libraries that help with exporting metrics from third-party systems as Prometheus metrics. Exporters are useful whenever it is not feasible to instrument a given application or system with Prometheus metrics directly. Multiple exporters can run on a monitored host to export local metrics.

The Prometheus community provides a list of official exporters, and more can be found as community contributions. For more information and an extensive list of exporters, see https://prometheus.io/docs/instrumenting/exporters/.

Grafana

Grafana is a tool for data visualization, monitoring, and analysis. It is used to create dashboards with panels representing specific metrics over a set period of time. Grafana is commonly used together with Prometheus, but also supports other data sources such as ElasticSearch, MySQL, PostgreSQL, and Influx DB. For more information about Grafana, see https://grafana.com/docs/.

Set up the Monitoring Server

To set up your monitoring server, you need to install Prometheus and Grafana, and configure them.

Install Prometheus

If your monitoring server is a Salt client, you can install the Prometheus package using the Uyuni Web UI. Otherwise you can download and install the package on your monitoring server manually.

Procedure: Installing Prometheus Using the Web UI
  1. In the Uyuni Web UI, open the details page of the system where Prometheus is to be installed, and navigate to the Formulas tab.

  2. Check the Prometheus checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Prometheus tab in the top menu.

  4. In the Uyuni Server section, enter valid Uyuni API credentials. Make sure that the credentials you have entered allow access to the set of systems you want to monitor.

  5. Customize any other configuration options according to your needs.

  6. Click Save Formula.

  7. Apply the highstate and confirm that it completes successfully.

  8. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

For more information about the monitoring formulas, see salt:formula-monitoring.adoc.

Procedure: Manually Installing and Configuring Prometheus
  1. On the monitoring server, install the golang-github-prometheus-prometheus package:

    zypper in golang-github-prometheus-prometheus
  2. Enable the Prometheus service:

    systemctl enable --now prometheus
  3. Check that the Prometheus interface loads correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

  4. Open the configuration file at /etc/prometheus/prometheus.yml and add this configuration information. Replace server.url with your Uyuni server URL and adjust username and password fields to match your Uyuni credentials.

    # {productname} self-health metrics
    scrape_configs:
    - job_name: 'mgr-server'
      static_configs:
        - targets:
          - 'server.url:9100'  # Node exporter
          - 'server.url:9187'  # PostgreSQL exporter
          - 'server.url:5556'  # JMX exporter (Tomcat)
          - 'server.url:5557'  # JMX exporter (Taskomatic)
          - 'server.url:9800'  # Taskomatic
        - targets:
          - 'server.url:80'    # Message queue
          labels:
            __metrics_path__: /rhn/metrics
    
    # Managed systems metrics:
    - job_name: 'mgr-clients'
      uyuni_sd_configs:
       - host: "http://server.url"
         username: "admin"
         password: "admin"
  5. Save the configuration file.

  6. Restart the Prometheus service:

    systemctl restart prometheus

For more information about the Prometheus configuration options, see the official Prometheus documentation at https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Install Grafana

If your monitoring server is a Salt client, you can install the Grafana package using the Uyuni Web UI. Otherwise you can download and install the package on your monitoring server manually.

Procedure: Installing Grafana Using the Web UI
  1. In the Uyuni Web UI, open the details page of the system where Grafana is to be installed, and navigate to the Formulas tab.

  2. Check the Grafana checkbox to enable monitoring formulas, and click Save.

  3. Navigate to the Grafana tab in the top menu.

  4. In the Enable and configure Grafana section, enter the admin credentials you want to use to log in Grafana.

  5. On the Datasources section, make sure that the Prometheus URL field points to the system where Prometheus is running.

  6. Customize any other configuration options according to your needs.

  7. Click Save Formula.

  8. Apply the highstate and confirm that it completes successfully.

  9. Check that the Grafana interface is loading correctly. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

Uyuni provides pre-built dashboards for server self-health, basic client monitoring, and more. You can choose which dashboards to provision in the formula configuration page.

For more information about the monitoring formulas, see salt:formula-monitoring.adoc.

Procedure: Manually Installing Grafana
  1. Install the grafana package:

    zypper in grafana
  2. Enable the Grafana service:

    systemctl enable --now grafana-server
  3. Check that the Grafana interface is loading correctly. In your browser, navigate to the URL of the server where Grafana is installed, on port 3000 (for example, http://example.com:3000).

monitoring grafana example

For more information on how to manually install and configure Grafana, see https://grafana.com/docs.

For more information about the monitoring formulas with forms, see salt:formula-monitoring.adoc.

Configure Uyuni Monitoring

With Uyuni 4 and higher, you can enable the server to expose Prometheus self-health metrics, and also install and configure exporters on client systems.

Server Self Monitoring

The Server self-health metrics cover hardware, operating system and Uyuni internals. These metrics are made available by instrumentation of the Java application, combined with Prometheus exporters.

These exporter packages are shipped with Uyuni Server:

These exporter packages are shipped with Uyuni Proxy:

The exporter packages are pre-installed in Uyuni Server and Proxy, but their respective systemd daemons are disabled by default.

Procedure: Enabling Self Monitoring
  1. In the Uyuni Web UI, navigate to Admin  Manager Configuration  Monitoring.

  2. Click Enable services.

  3. Restart Tomcat and Taskomatic.

  4. Navigate to the URL of your Prometheus server, on port 9090 (for example, http://example.com:9090)

  5. In the Prometheus UI, navigate to menu:[Status > Targets] and confirm that all the endpoints on the mgr-server group are up.

  6. If you have also installed Grafana with the Web UI, the server insights are visible on the Uyuni Server dashboard.

monitoring enable services

Only server self-health monitoring can be enabled using the Web UI. Metrics for a proxy are not automatically collected by Prometheus. To enable self-health monitoring on a proxy, you need to manually install exporters and enable them.

Monitoring Managed Systems

Prometheus metrics exporters can be installed and configured on Salt clients using formulas. The packages are available from the Uyuni client tools channels, and can be enabled and configured directly in the Uyuni Web UI.

These exporters can be installed on managed systems:

When you have the exporters installed and configured, you can start using Prometheus to collect metrics from monitored systems. If you have configured your monitoring server with the Web UI, metrics collection happens automatically.

Procedure: Configuring Prometheus Exporters on a Client
  1. In the Uyuni Web UI, open the details page of the client to be monitored, and navigate to the menu:Formulas tab.

  2. Check the Enabled checkbox on the Prometheus Exporters formula.

  3. Click Save.

  4. Navigate to the Formulas  Prometheus Exporters tab.

  5. Select the exporters you want to enable and customize arguments according to your needs. The Address field accepts either a port number preceded by a colon (:9100), or a fully resolvable address (example:9100).

  6. Click Save Formula.

  7. Apply the highstate.

monitoring configure formula

Monitoring formulas can also be configured for System Groups, by applying the same configuration used for individual systems inside the corresponding group.

For more information about the monitoring formulas, see salt:formula-monitoring.adoc.

Network Boundaries

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to monitored clients. By default, Prometheus uses these ports:

  • Node exporter: 9100

  • PostgreSQL exporter: 9187

  • Apache exporter: 9117

Additionally, if you are running the alert manager on a different host than where you run Prometheus, you also need to open port 9093.

For clients installed on cloud instances, you can add the required ports to a security group that has access to the monitoring server.

Alternatively, you can deploy a Prometheus instance in the exporters' local network, and configure federation. This allows the main monitoring server to scrape the time series from the local Prometheus instance. If you use this method, you only need to open the Prometheus API port, which is 9090.

For more information on Prometheus federation, see https://prometheus.io/docs/prometheus/latest/federation/.

You can also proxy requests through the network boundary. Tools like PushProx deploy a proxy and a client on both sides of the network barrier and allow Prometheus to work across network topologies such as NAT.

For more information on PushProx, see https://github.com/RobustPerception/PushProx.

Reverse Proxy Setup

Prometheus fetches metrics using a pull mechanism, so the server must be able to establish TCP connections to each exporter on the monitored clients. To simplify your firewall configuration, you can use reverse proxy for your exporters to expose all metrics on a single port.

Procedure: Installing Prometheus Exporters with Reverse Proxy
  1. In the Uyuni Web UI, open the details page of the system to be monitored, and navigate to the Formulas tab.

  2. Check the Prometheus Exporters checkbox to enable the exporters formula, and click Save.

  3. Navigate to the Prometheus Exporters tab in the top menu.

  4. Check the Enable reverse proxy option, and enter a valid reverse proxy port number. For example, 9999.

  5. Customize the other exporters according to your needs.

  6. Click Save Formula.

  7. Apply the highstate and confirm that it completes successfully.

For more information about the monitoring formulas, see salt:formula-monitoring.adoc.

Security

Prometheus server and Prometheus node exporter offer a built-in mechanism to secure their endpoints with TLS encryption and authentication. Uyuni Web UI simplifies the configuration of all involved components. The TLS certificates have to be provided and deployed by the user. Uyuni offers enabling the following security model:

  • Node exporter: TLS encryption and client certificate based authentication

  • Prometheus: TLS encryption and basic authentication

For more information about configuring all available options, see salt:formula-monitoring.adoc.

Generating TLS certificates

By default, Uyuni does not provide any certificates for securing monitoring configuration. For providing security, you can generate or import custom certificates, self-signed or signed by a third party certificate authority (CA).

This section demonstrates how to generate client/server certificates for Prometheus and Node exporter minions self-signed with SUSE Manager CA.

Procedure: Creating server/client TLS certificate
  1. On the Uyuni Server, at the command prompt, run following command:

    rhn-ssl-tool --gen-server --dir="/root/ssl-build" --set-country="COUNTRY" \
    --set-state="STATE" --set-city="CITY" --set-org="ORGANIZATION" \
    --set-org-unit="ORGANIZATION UNIT" --set-email="name@example.com" \
    --set-hostname="minion.example.com" --set-cname="minion.example.com" --no-rpm

    Ensure that the set-cname parameter is the fully qualified domain name (FQDN) of your Salt client. You can use the the set-cname parameter multiple times if you require multiple aliases.

  2. Copy server.crt and server.key files to the Salt minion and provide read access for prometheus user.