In this tutorial we want to show you, how to collect Hazelcast metrics via Prometheus. The main reason for this how-to is, that the manual provided by Hazelcast does not give any hints on how to obtain the metrics in detail. They refer to their full-blown zip-file / Docker images and start it with one parameter. However, if you are running in a K8s (sidecar) container or with your own microservice, this will not help much. You can also get the metrics through the Management Center if it is available, but this is limited to Hazelcast Enterprise and a corresponding license.

Prerequisites

You have a customised Hazelcast Microservice and are not using the full-blown zip-file / Docker images.

Introduction

Hazelcast (more specifically Hazelcast Platform since version 5.0) is an in-memory database/cache solution like Redis. In Hazelcast you can store different data types in various data structures such as Maps, Sets, Lists, Queues, …. Furthermore, you can create backups and partitions. As Hazelcast is cloud agnostic you can deploy it on a any hyperscaler or your own on-prem cloud. Hazelcast also supports persistence to disk, which is an enterprise feature. If you combine backups with a zone_aware partitioning, K8s TopologySpread and PDB, your cache can withstand whole datacentre outages in one zone without any data loss.

Step-by-Step Guide

To collect metrics from Hazelcast, we need to extend an existing Hazelcast microservice with the following steps:

1. Add the Prometheus dependency to the pom.xml

   <dependency>
       <groupId>io.micrometer</groupId>
       <artifactId>micrometer-registry-prometheus</artifactId>
   </dependency>

2. Update template.yaml

   1. Add the Prometheus label(s) from your configured Prometheus to the service, so Prometheus can see it, e.g.,

      monitored: Prometheus

      or

      prometheus.io/scrape: "true"
      prometheus.io/path: "/actuator/prometheus"
      prometheus.io/port: "8081"

   2. Add an additional port to the service:

      - name: http-prometheus
        protocol: TCP
        port: 8081
        targetPort: 8081

3. Add the lines below to your Hazelcast microservice configuration, either with Java code or in the yaml file to activate the metrics. They should be already activated by default and we explicitly disable the Management Center as we do not use it:

   c.getMetricsConfig().setEnabled(true);
   c.getMetricsConfig().getJmxConfig().setEnabled(true);
   c.getMetricsConfig().getManagementCenterConfig().setEnabled(false);

4. Create or update - if it already exists - the application.yml file to pass the metrics through the web server:

   management:
     endpoints:
       web:
         exposure:
           include: 
             - info
             - health
             - prometheus
       metrics:
         enabled: true
       prometheus:
         enabled: true
     metrics:
       export:
         prometheus:
           enabled: true
     info:
       env:
         enabled: true

5. Add the JMX Prometheus Agent

   1. Download the latest version and add it to the folder libs inside your microservice. (link)
   2. Download the configuration (link) and add it also into the folder libs.

6. Update the Dockerfile to inject the agent.

   1. Copy the files from step 5 in the root directory:

      COPY libs/jmx_prometheus_javaagent-0.17.2.jar /jmx_prometheus_javaagent.jar
      COPY libs/jmx_agent_config.yaml /jmx_agent_config.yaml

   2. Inject the agent via JAVA_OPTS:

      -javaagent:/jmx_prometheus_javaagent.jar=8081:/jmx_agent_config.yaml

7. Update your ServiceMonitor under “endpoints”, so that Prometheus can find and poll for Hazelcast metrics:

   - path: /actuator/prometheus
     port: http-prometheus

8. The metrics in Prometheus are now available and are prefixed with

   hazelcast_

Hints

• If you are using a port other than 8081, you need to update it in the Dockerfile as well.
• If the libs folder is named differently, you need to update it in the Dockerfile as well.
• If you have downloaded a newer version of the agent, you need to update it in the Dockerfile as well.
• In the ServiceMonitor, the port name must match the name in the service.

Sources

• Monitoring (hazelcast.com) (link)
• GitHub - prometheus/jmx_exporter: A process for exposing JMX Beans via HTTP for Prometheus consumption (link)
• hazelcast/distribution/src/bin-filemode-755/hz at master · hazelcast/hazelcast · GitHub (link)
• Jmx exporter is not showing hazelcast related information · Issue #119 · hazelcast/hazelcast-docker · GitHub (link)
• Spring boot - Prometheus endpoint error when using micrometer - Stack Overflow (link)

Credits

Title image by Travel mania on Adobe Stock