Skip to main content

Prometheus

Cloudders Health Check operator exposes by default Prometheus Metrics.

Our helm chart creates Prometheus Operator ServiceMonitor resource by default in the monitoring namespace.

If the prerequsites are met and the ServiceMonitor is enabled in the helm chart here.

Prerequisites​

You need to have kube-prometheus-stack installed on your cluster.

The installation is quite simple if you follow the guideline in their installation section of the helm chart readme.

Exposed metrics​

The operator exposes the following metrics:

  • healthcheck_response_codes: counter - This is counter type of metrics for the HTTP reponse codes received by the HealthCheck. In case a different response code is received a new time series counter will be created for this response code

  • healthcheck_response_time: gauge - This gauge is for the HealthCheck response time. Its measured in milliseconds.

  • healthcheck_status: gauge - The healthcheck_status gauge represent 1 if the target of the HealthCheck is UP and 0 if the target is DOWN.

Understanding the labels​

The following labels are exporter for the metrics:

  • name: this is the name of the HealthCheck CRD (available for all metrics)
  • code: response code of the target (only available for the healthcheck_response_codes metrics)

Exploring the metrics​

You can query the Prometheus service endpoint with cURL to explore the metrics exposed by the Operator, or via port-forwarding, if you haven't exposed the Prometheus UI outside your cluster.

cURL method​

To get the Prometheus metrics you can execute the following commands, assuming that your Prometheus operator is installed in the monitoring namespace.

kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n monitoring

and in a new shell you can cURL the endpoint with the following command:

curl -X POST  http://localhost:9090/api/v1/query\?query\="healthcheck_status"

Prometheus UI​

The Prometheus UI provides user friendly way to query metrics, explore targets, see current alerts from the operator.

To access the UI if you haven't exposed your Prometheus you can simply type this command:

kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n monitoring

Navigate your browser to the following URL: http://localhost:9090 - this assumes your Prometheus does not run over SSL.

You can now query the metrics from the Graph section of Prometheus like on the following screenshot:

Prometheus UI

Troubleshooting​

Here you can find some troubleshooting guides, if the metrics are not exposed via the ServiceMonitor target.

ServiceMonitor​

Make sure your have setup the serviceMonitor configuration in the helm chart.

There are two important parameters:

  • namespace: the namespace where the Prometheus is installed, if you are not scanning all namespaces for ServiceMonitor resources.
  • prometheusReleaseName: this is the name of the helm release of your kube-prometheus-stack.