Welcome to Kafka Overwatch documentation ¶

Docker Build Status BUILD

Docs Build status DOCS_BUILD

TL;DR ¶

This microservice monitors Kafka topics in intervals, examining offsets, consumer groups, and lag to identify active topics with new messages. It exposes metrics for validation and generates a user-friendly report, helping users discern actively utilized topics from dormant ones.

It can be monitor multiple Kafka clusters in-parallel, using different configurations and settings.

Features ¶

Generates a usage report on topics
Exposes metrics via prometheus
- Topics count
- Partitions count
- Number of new messages (measured with topic offsets)
AWS Secret integration for client config values

Extras ¶

Here are some extra features that serve additional needs

Generates commands script to re-create all the topics in case of DR (store local or to S3)
Governance: evaluate your resources configuration

Input configuration ¶

Before running it, you need to define an input configuration file, with the different Kafka clusters you wish to scan. Head to Input Configuration to check the input specifications.

For example below, as-used in the docker-compose.override.yaml

           ---
# Local testing Kafka configuration

clusters:
  test-simple: # defaults filters out internal topics
    cluster_config:
      kafka:
        bootstrap.servers: broker:29092
        client.id: kafka-overwatch.test-simple
    topic_exclude_regexes:
      - ^_(.*)$
    x-test: whatever
    reporting_config:
      local: ./generated_reports
      evaluation_period_in_seconds: 120
      exports:
        S3:
          bucket_name: usera-sftp
          prefix_key: kafka-overwatch/reports

  test-exclude: # testing to exclude topics
    cluster_config:
      kafka:
        bootstrap.servers: broker:29092
        client.id: kafka-overwatch.test-exclude
    topic_exclude_regexes:
      - ^test(.*)$
    reporting_config:
      exports:
        local: ./generated_reports
      evaluation_period_in_seconds: 60


  test-include: # show me only include topics, using default exclude rules
    cluster_config:
      kafka:
        bootstrap.servers: broker:29092
        client.id: kafka-overwatch.test-include
    topic_include_regexes:
      - '^_(.*)$'
      - '^test(.*)'
    reporting_config:
      exports:
        local: ./generated_reports
      evaluation_period_in_seconds: 300

          

Install | Deploy ¶

With docker ¶

            docker run --rf -v ${pwd}/config.yaml:/tmp/config/config.yaml public.ecr.aws/johnpreston/kafka-overwatch -c /tmp/config/config.yaml

           

With python ¶

            # Inside a python virtual environment
python3 -m venv venv
source venv/bin/activate
pip install pip -U
pip install kafka-overwatch

# For your user only, without virtualenv
python3 -m pip install kafka-overwatch --user

           