3. Monitoring and logging on OpenShift Operations Training

3.1 Rules and dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.1.1: Provided dashboards

OpenShift provides a set of default dashboards for cluster adminstrators and OpenShift users.

Cluster dashboards

To get a sense of where these dashboards can be found, let’s take a look at the performance of the etcd cluster.

Navigate to Observe -> Dashboards on the web console and select etcd in the dashboard drop-down list.

User dashboards

Switch to the Developer console and select the uptime-app-prod project. Then navigate to the Observing tab. Select the uptime-app in the Workload drop-down list. This will display a wide selection of different metrics relevant to your app like CPU usage, memory usage and networking performance.

3.2 Monitoring stack configuration

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.2.1: Enable alerting

Let’s take a look at an example of how you can send alerts to a receiver of your choice. Configuring a new receiver can either be done using the web console or by directly configuring the appropriate secret. Before going into detail on where this configuration lies, let’s have a look at an example:

---
apiVersion: v1
kind: Secret
metadata:
 name: alertmanager-main
 namespace: openshift-monitoring
stringData:
 alertmanager.yaml: |
 ---
 global:
 resolve_timeout: 5m
 smtp_from: monitoring@ops-training.openshift.ch
 smtp_smarthost: smtp.ops-training.openshift.ch:587
 smtp_hello: ops-training.openshift.ch
 smtp_auth_username: test
 smtp_auth_password: test
 smtp_require_tls: true
 inhibit_rules:
 - equal:
 - namespace
 - alertname
 source_match:
 severity: critical
 target_match_re:
 severity: warning|info
 - equal:
 - namespace
 - alertname
 source_match:
 severity: warning
 target_match_re:
 severity: info
 receivers:
 - name: Default
 - name: Watchdog
 - name: Critical
 - name: mail
 email_configs:
 - to: ops@ops-training.openshift.ch
 route:
 group_by:
 - namespace
 group_interval: 5m
 group_wait: 30s
 receiver: Default
 repeat_interval: 12h
 routes:
 - match:
 alertname: Watchdog
 receiver: Watchdog
 - match:
 severity: critical
 receiver: mail 
type: Opaque

The main components that can be configured when applying a custom Alertmanager configuration comprise:

3.3 User-defined monitoring rules

Mon, 01 Jan 0001 00:00:00 +0000

Task 3.3.1: Enable the user-workload monitoring stack

As mentioned in a lab before, it is not supported to alter the OpenShift-provided monitoring stack. If you want to add additional monitoring targets or add custom Prometheus rules, you need to enable the user-workload monitoring stack.

And this is exactly what we are going to do. First, edit the cluster-monitoring-config configmap in the openshift-monitoring namespace.

Hints

oc -n openshift-monitoring edit cm cluster-monitoring-config

Add the line enableUserWorkload: true under data/config.yaml (leave the rest as it is).

3.4 Logging

Mon, 01 Jan 0001 00:00:00 +0000

In this lab we will install and configure the logging stack.

Task 3.4.1 Install cluster logging

You can install OpenShift Logging to aggregate all the logs from your OpenShift cluster, such as node logs, application logs and infrastructure logs.

To deploy the cluster logging stack from the CLI, we need to create the following objects. Have a look at them first:

OpenShift Elasticsearch Operator Namespace

apiVersion: v1
kind: Namespace
metadata:
 name: openshift-operators-redhat 
 annotations:
 openshift.io/node-selector: ""
 labels:
 openshift.io/cluster-monitoring: "true"

Red Hat OpenShift Logging Namespace

apiVersion: v1
kind: Namespace
metadata:
 name: openshift-logging
 annotations:
 openshift.io/node-selector: ""
 labels:
 openshift.io/cluster-monitoring: "true"

OpenShift Elasticsearch Operator OperatorGroup

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
 name: openshift-operators-redhat
 namespace: openshift-operators-redhat 
spec: {}

Red Hat OpenShift Logging OperatorGroup

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
 name: cluster-logging
 namespace: openshift-logging 
spec:
 targetNamespaces:
 - openshift-logging

OpenShift Elasticsearch Operator Subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: "elasticsearch-operator"
 namespace: "openshift-operators-redhat" 
spec:
 channel: "stable" 
 installPlanApproval: "Automatic"
 source: "redhat-operators" 
 sourceNamespace: "openshift-marketplace"
 name: "elasticsearch-operator"

Red Hat OpenShift Logging Subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
 name: cluster-logging
 namespace: openshift-logging 
spec:
 channel: "stable"
 name: cluster-logging
 source: redhat-operators 
 sourceNamespace: openshift-marketplace

Now either copy and paste above resource definitions and then apply them to your cluster or directly use our provided files: