When we insert data into the comments table, we need it to route to be inserted into a partition instead of the actual table. For that, we can create a trigger: add_partition_trigger :comments, :comments_by_year, [ { if: 'NEW.year = 2016', insert: :comments_2016 }, { elsif: 'NEW.year = 2017', insert: :comments_2017 }, { else: "RAISE EXECEPTION 'comments_by_year recieived an unexpected value: %', NEW.year;" } ]

Example1.49.Recall from Example 1.36 that given a setXwe defineEXto be theset of partitions onX, and that a partition may be defined using a surjective functions:XPfor some setP.Any surjective functionf:X!Yinduces a monotone mapf:EY! EX, going“backwards”. It is defined by sending a partitions:YPto the compositef:s:XP

You measure the throughout that you can achieve on a single partition for production (call it p) and consumption (call it c). Let’s say your target throughput is t.

Designing a High Level Consumer

no data loss will occur as long as producers and consumers handle this possibility and retry appropriately.

The first limitation is that each partition is physically represented as a directory of one or more segment files. So you will have at least one directory and several files per partition. Depending on your operating system and filesystem this will eventually become painful. However this is a per-node limit and is easily avoided by just adding more total nodes in the cluster.

the number of partitions -- there's no real "formula" other than this: you can have no more parallelism than you have partitions.