kafka_overwatch.kafka_resources package

Submodules

kafka_overwatch.kafka_resources.groups module

kafka_overwatch.kafka_resources.groups. describe_update_consumer_group_offsets ( consumer_group , kafka_cluster )

Retrieve a given Consumer group details, given that the list_consumer_group_offsets cannot take multiple groups as argument. Updates the kafka_cluster.groups.

Return type :

None

kafka_overwatch.kafka_resources.groups. get_consumer_groups_desc ( kafka_cluster , groups_list , groups_desc = None )

Return type :

dict [ str , confluent_kafka.admin._group.ConsumerGroupDescription ]

kafka_overwatch.kafka_resources.groups. retry_kafka_describe_update_consumer_group_offsets ( future , futures_to_data , executor )

kafka_overwatch.kafka_resources.groups. set_update_cluster_consumer_groups ( kafka_cluster , stop_flag )

Lists all Consumer Groups Checks if all the listed CGs were present in the existing cluster CGs. If not, remove. Describe all Consumer Groups List all Consumer Group Offsets (serial, no support for list of CGs) If CG not in the cluster groups, add to it

Return type :

None

kafka_overwatch.kafka_resources.groups. set_update_filter_cluster_consumer_groups ( kafka_cluster )

Lists all the consumer groups from the Kafka cluster Filters out of the Kafka cluster groups attribute all the consumer groups that are not in the list Retrieves the consumer group description/details from the cluster

Return type :

None

kafka_overwatch.kafka_resources.groups. tidy_consumer_groups ( kafka_cluster , query_groups_list )

Return type :

None

kafka_overwatch.kafka_resources.groups. update_set_consumer_group_topics_partitions_offsets ( kafka_cluster , consumer_group )

Groups topic partitions offsets per topic Assigns partitions offsets to the consumer group Maps consumer group to topic

Return type :

None

kafka_overwatch.kafka_resources.topics module

kafka_overwatch.kafka_resources.topics. define_topic_jobs ( kafka_cluster , desc_topics )

Goes over all the topic configurations, and if authorization allows for describe configs, plans for the list of configurations to return. Returns the dict of topic description jobs to perform, and the topic descriptions list.

Return type :

tuple [ dict , list ]

kafka_overwatch.kafka_resources.topics. describe_update_all_topics ( kafka_cluster , stop_flag )

Lists all topics

Return type :

None

kafka_overwatch.kafka_resources.topics. describe_update_topics ( kafka_cluster , desc_topics , stop_flag )

Leverages threads to retrieve the topic offset watermarks which cannot be sent in a single call to Kafka.

Return type :

None

kafka_overwatch.kafka_resources.topics. get_topic_descriptions ( topic_names , admin_client )

Return type :

dict [ str , confluent_kafka.admin._topic.TopicDescription ]

kafka_overwatch.kafka_resources.topics. get_topic_partition_watermarks ( consumer_client , topic_name , partition_id )

kafka_overwatch.kafka_resources.topics. get_topics_list ( kafka_cluster )

Return type :

list [ str ]

kafka_overwatch.kafka_resources.topics. init_set_partitions ( topic_obj , consumer_client , topic , now )

kafka_overwatch.kafka_resources.topics. update_set_topic_config ( kafka_cluster , topics_configs_resources )

Return type :

None

Module contents

kafka_overwatch.kafka_resources. set_admin_client ( settings )

Creates a new librdkafka Admin client

Return type :

confluent_kafka.admin.AdminClient

kafka_overwatch.kafka_resources. set_consumer_client ( settings )

Creates a new librdkafka Consumer client

Return type :

cimpl.Consumer

kafka_overwatch.kafka_resources. wait_for_result ( result_container )

Return type :

dict