For a topic with replication factor N, we will tolerate up to N-1 server failures without losing any records committed to the log.
for Eg for a given topic there are 11 brokers/servers and for each topic the replication factor is 6. That means the topic will start loosing data if more than 5 brokers go down.
Consumers label themselves with a consumer group name
maintain separate consumer group per tenant basis. Helps to scale out when we have more load per tenant.
the only metadata retained on a per-consumer basis is the offset or position of that consumer in the log. This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads records, but, in fact, since the position is controlled by the consumer it can consume records in any order it likes.
partition offset maintained by kafka. Offset number is maintained so that if the consumer goes down nothing breaks.
the retention policy is set to two days, then for the two days after a record is published,
Might have to tweek this based on the persistence level we want to keep.
replication factor N, we will tolerate up to N-1 server failures without losing any records
Replication Factor means number of nodes/brokers which could go down before we start losing data.
So if you have a replication factor of 6 for a 11 node cluster, then you will be fault tolerant till 5 nodes go down. After that point you are going to loose data for a particular partition.