SubscribePattern allows you to use a regex to specify topics of interest
This can remove the need to reload the kafka writers in order to take consume messages.
regex - "topic-ua-*"
2 Matching Annotations
- Nov 2017
-
spark.apache.org spark.apache.org
-
-
The cache for consumers has a default maximum size of 64. If you expect to be handling more than (64 * number of executors) Kafka partitions, you can change this setting via spark.streaming.kafka.consumer.cache.maxCapacity.
You might need this for keeping track of all partitions consumed.
-