Slow initial rebalance on remote consumer

[Arisawa-v]

I’m using confluent-kafka-go to run a consumer on a remote server and a producer locally. Every time I start the consumer with a new group.id (I generate it using the PID), the first partition assignment takes 1–2 minutes. Subsequent restarts finish in under 5 seconds, and local-to-local runs are always fast (< 5 s).

My consumer config is:

groupID := fmt.Sprintf("kafka-go-client-%d", os.Getpid())
c, err := kafka.NewConsumer(&kafka.ConfigMap{
    "bootstrap.servers":       os.Getenv("KAFKA_BOOTSTRAP_SERVERS"),
    "group.id":                groupID,
    "auto.offset.reset":       "latest",
    "enable.auto.commit":      "true",
    "auto.commit.interval.ms": "5000",
    "security.protocol":       "SASL_PLAINTEXT",
    "sasl.mechanisms":         "PLAIN",
    "sasl.username":           os.Getenv("KAFKA_CLIENT_USERNAME"),
    "sasl.password":           os.Getenv("KAFKA_CLIENT_PASSWORD"),
})

Questions:

1. Why does the first rebalance on the remote server take so long compared to later restarts?
2. Which client or broker settings should I tune to speed up that initial assignment?
3. Would using static membership (group.instance.id) remove this long delay, and how do I configure it in confluent-kafka-go?

[milindl]

Is it possible to add "debug": "all" to your configuration for the first consumer? That will help us determine exactly which step might be taking time, and we can see how we can speed it up. It is not expected that it should take so much extra time for the first rebalance.

Additionally, it might be helpful to know the broker details as well, ie. version of Apache Kafka being used, or if you're using something like confluent cloud.