kafka_overwatch.processing package

Submodules

kafka_overwatch.processing.clusters module

kafka_overwatch.processing.clusters. generate_cluster_report ( kafka_cluster , topics_df , groups_df )

Evaluates whether time to produce the report has passed. If so, generates and updates next monitoring time.

Return type :

None

kafka_overwatch.processing.clusters. measure_consumer_group_lags ( kafka_cluster , consumer_group_lag_gauge )

Evaluates, if consumer groups were retrieved, the consumer groups lags and export metrics to Prometheus.

kafka_overwatch.processing.clusters. process_cluster ( kafka_cluster , overwatch_config , stop_flag )

Initialize the Kafka cluster monitoring/evaluation loop. Creates the cluster, which creates the Kafka clients.

kafka_overwatch.processing.clusters. process_cluster_resources ( kafka_cluster , stop_flag )

Makes sure that no signal was received in between each instruction

kafka_overwatch.processing.schema_registries module

kafka_overwatch.processing.schema_registries. backup_schema_registry_subjects ( schema_registry )

Attempts to back up all the subjects and their associated schemas to S3.

Return type :

None

kafka_overwatch.processing.schema_registries. init_schema_registry_prometheus_reporting ( schema_registry , overwatch_config )

kafka_overwatch.processing.schema_registries. process_schema_registry ( schema_registry , runtime_key , overwatch_config , stop_flag )

Process function for the schema registry. Responsible for :rtype: None

• retrieving all the schemas & subjects

• backup the subjects & schemas if configured.

kafka_overwatch.processing.schema_registries. process_schemas ( schema_registry , sr_client , stop_flag )

Process all schemas in the schema registry. If getting schemas from /schemas fails, fall back to /subjects

Return type :

None

kafka_overwatch.processing.schema_registries. retrieve_from_schemas ( schema_registry , sr_client )

Fastest way, used if /schemas worked

Return type :

None

kafka_overwatch.processing.schema_registries. retrieve_from_subjects ( schema_registry , sr_client , stop_flag )

Much longer way to retrieve all the schemas & subjects, using the subjects endpoints. Using threading to speed up the processing, but still slower by an order of magnitude than retrieving the schemas directly.

Return type :

None

kafka_overwatch.processing.schema_registries. set_prometheus_metrics ( subjects_count , schemas_count , schema_registry )

kafka_overwatch.processing.schema_registries. write_schema_registry_mmap ( schema_registry )

Writes the schema registry to a mmap file.

Return type :

None

kafka_overwatch.processing.signals module

kafka_overwatch.processing.signals. handle_signals ( pid , frame )

Module contents

kafka_overwatch.processing. ensure_prometheus_multiproc ( prometheus_dir_path )

Just in case the env_var had not propagated among processes, setting in child env var.

kafka_overwatch.processing. wait_between_intervals ( stop_flag , time_to_wait , too_short_desc = None )

Return type :

None