kafka_overwatch.overwatch_resources package

Subpackages

Submodules

kafka_overwatch.overwatch_resources.clusters module

class kafka_overwatch.overwatch_resources.clusters. KafkaCluster ( name , config , overwatch_config ) [source]

Bases: object

assign_sns_channels ( overwatch_config , config ) [source]

Maps the SNS channels defined at the root of the configuration to the ones defined for the cluster. This will make it easy to send notifications for each of these channels

check_replace_kafka_clients ( ) [source]
property config : ClusterConfiguration
consumer_client ( ) [source]
Return type :

cimpl.Consumer

get_admin_client ( ) [source]
Return type :

confluent_kafka.admin.AdminClient

get_schema_registry ( ) [source]

If the SR has produced a binary file for it, retrieves and loads the class. This works around the fact that the SR data is processed in a different process therefore the data it has is not updated in the cluster thread. If not found or not set, ignore and consider Kafka Cluster does not have SR data available yet. Which can occur if the SR processing has not yet yielded results.

Return type :

typing.Optional [ kafka_overwatch.overwatch_resources.schema_registry.SchemaRegistry ]

init_cluster_prometheus_reporting ( overwatch_config ) [source]
property prometheus_collectors
render_report ( topics_df , groups_df ) [source]
Return type :

None

render_restore_files ( ) [source]

Generates scripts/config files to re-create the Topics of a cluster

Return type :

None

set_cluster_connections ( ) [source]
Return type :

None

set_cluster_properties ( ) [source]
Return type :

None

set_reporting_exporters ( ) [source]
kafka_overwatch.overwatch_resources.clusters. generate_cluster_consumer_groups_pd_dataframe ( kafka_cluster ) [source]
Return type :

pandas.core.frame.DataFrame

kafka_overwatch.overwatch_resources.clusters. generate_cluster_topics_pd_dataframe ( kafka_cluster ) [source]
Return type :

pandas.core.frame.DataFrame

kafka_overwatch.overwatch_resources.groups module

class kafka_overwatch.overwatch_resources.groups. ConsumerGroup ( group_id , init_members , init_state ) [source]

Bases: object

fetch_set_lag ( topic_name = None ) [source]

Returns the lag for a topic and its partitions If topic_name is set, returns the lag for that topic alone.

Return type :

dict [ str , dict ]

property group_id
property init_members : tuple [ list [ MemberDescription ] , datetime ]
property is_active : bool
property members : list [ MemberDescription ]
property partitions_offsets
property pd_frame_data : dict
property state : ConsumerGroupState

kafka_overwatch.overwatch_resources.topics module

class kafka_overwatch.overwatch_resources.topics. Partition ( topic , partition_id , init_start_offset , init_end_offset , init_time ) [source]

Bases: object

Class to represent a partition of a topic, with methods to make it easy to track them

cleanup ( ) [source]
property end_offset : tuple [ int , datetime ]
property first_offset : tuple [ int , datetime ]

Until the first offset of the partition changes, we use the init start value.

get_end_offset_diff ( ) [source]

Returns the numerical difference in offset and the elapsed time between the two measures

Return type :

tuple

has_new_messages ( ) [source]

Returns True if the topic has new messages

Return type :

bool

property init_start_offset : tuple [ int , datetime ]

Represents the offsets as the kafka-overwatch started, allowing to do the long term evaluations

property partition_id : int
property total_messages_count : int
class kafka_overwatch.overwatch_resources.topics. Topic ( name , cluster , properties = None ) [source]

Bases: object

property cluster
property config : dict [ str , ConfigEntry ]

Returns the ConfigEntry

generate_kafka_create_topic_command ( ) [source]
has_active_groups ( ) [source]

Returns whether any of the consumer groups are active, or not.

Return type :

bool

has_new_messages ( ) [source]

Returns True if the topic has new messages by checking if any partition had new messages

Return type :

bool

property name : str
new_messages_count ( ) [source]
Return type :

tuple [ int , int ]

property pd_frame_data : dict

Module contents