2 Matching Annotations
  1. Nov 2017
    1. SubscribePattern allows you to use a regex to specify topics of interest

      This can remove the need to reload the kafka writers in order to take consume messages.

      regex - "topic-ua-*"

    2. The cache for consumers has a default maximum size of 64. If you expect to be handling more than (64 * number of executors) Kafka partitions, you can change this setting via spark.streaming.kafka.consumer.cache.maxCapacity.

      You might need this for keeping track of all partitions consumed.