95 Matching Annotations
  1. Jun 2019
    1. The value of organic imports during Jan.-Aug. was up 25 percent compared to the same period in 2016, the trade data showed, while the value of organic exports during the first eight months was up 14 percent. Last year, the U.S. organic products trade deficit hit nearly $1.2 billion, its highest level ever, with U.S. organic imports reaching $1.7 billion, while U.S. organic exports came in at $547.6 million. Check out the Top 10 U.S. organic imported and exported commodities for 2016.
    1. , demand for organic food is growing so fast that consumer demand is outstripping some domestic supplies. Once a net exporter of organic products, the United States now spends more than $1 billion a year to import organic food, according to the USDA, and the ratio of imported to exported products is now about 8-to-1.
  2. Feb 2019
    1. As with neoliberalism more generally, New Public Management is invisible, part of a new “common sense” that has somehow become hegemonic, whereby the “entrepreneurial spirit” has infused the public sector, leading to “businesslike government”. As with the claims of neoliberalism more generally as to its positive outputs in terms of prosperity, NPM has never been shown to have been successful even in its own terms. NPM “introduced punishments and rewards to produce better services with lesser staff. Instead of having freed energies and creativity of employees formerly shackled by their bureaucratic turfs, NPM reforms have bound energies into theatrical audit performances at the cost of work and killed creativity in centralizing resources and hollowing out professional autonomy... Fundamental deprivation of the legitimacy of public employees . . .has traumatized many most-committed employees and driven others toward a Soviet-type double standard.” (Juha Siltala, New Public Management : The evidence-based worst practice?, Administration; Vol. 45, No. 4.; 2013 pp. 468-493) Sekera quotes Christopher Pollitt et al., who “after compiling a database of 518 studies of NPM in Europe, determined that “more than 90% of what are seen by experts as the most significant and relevant studies contain no data at all on outcomes” and that of the 10% that had outcomes information, only 44% of those, or 4% of the total, found any improvements in terms of outcomes.” But in the end, the point of NPM is less that of measureable outcomes, and more that of the ideological victory of turning the public and its good into customers exercising their “choices” (see tax revolt example in Duggan), along of course with the radical disempowering of public administration workers and their unions, instituting “cost savings” by cutting their real income and putting more and more of the public sector’s production directly into the profit-making market.
  3. Nov 2017
    1. SubscribePattern allows you to use a regex to specify topics of interest

      This can remove the need to reload the kafka writers in order to take consume messages.

      regex - "topic-ua-*"

    2. The cache for consumers has a default maximum size of 64. If you expect to be handling more than (64 * number of executors) Kafka partitions, you can change this setting via spark.streaming.kafka.consumer.cache.maxCapacity.

      You might need this for keeping track of all partitions consumed.

  4. Jun 2017
    1. You measure the throughout that you can achieve on a single partition for production (call it p) and consumption (call it c). Let’s say your target throughput is t.

      t = throughput (QPS) p = single partition for production c = consumption

    1. In merced, we used the low-level simple consumer and wrote our own work dispatcher to get precise control.

      difference between merced and secor

    1. A better alternative is at least once message delivery. For at least once delivery, the consumer reads data from a partition, processes the message, and then commits the offset of the message it has processed. In this case, the consumer could crash between processing the message and committing the offset and when the consumer restarts it will process the message again. This leads to duplicate messages in downstream systems but no data loss.

      This is what SECOR does.

    2. By electing a new leader as soon as possible messages may be dropped but we will minimized downtime as any new machine can be leader.

      two scenarios to get the leader back: 1.) Wait to bring the master back online. 2.) Or elect the first node that comes back up. But in this scenario if that replica partition was a bit behind the master then the time from when this replica went down to when the master went down. All that data is Lost.

      SO there is a trade off between availability and consistency. (Durability)

    3. keep in mind that these guarantees hold as long as you are producing to one partition and consuming from one partition.

      This is very important a 1-to-1 mapping between writer and reader with partition. If you have more producers per partition or more consumers per partition your consistency is going to go haywire

    1. On every received heartbeat, the coordinator starts (or resets) a timer. If no heartbeat is received when the timer expires, the coordinator marks the member dead and signals the rest of the group that they should rejoin so that partitions can be reassigned. The duration of the timer is known as the session timeout and is configured on the client with the setting session.timeout.ms. 

      Time to live for the consumers. If the heartbeat doesn't reach the co-ordindator in this duration then the co-ordinator redistributes the partitions to the remaining consumers in the consumer group.

    2. The high watermark is the offset of the last message that was successfully copied to all of the log’s replicas.

      High Watermark: messages copied over to log replicas

    3. Kafka new Client which uses a different protocol for consumption in a distributed environment.

    4. Kafka scales topic consumption by distributing partitions among a consumer group, which is a set of consumers sharing a common group identifier.

      Topic consumption is distributed among a list of consumer group.

    1. Kafka consumer offset management protocol to keep track of what’s been uploaded to S3

      consumers keep track of what's written and where it left off by looking at kafka consumer offsets rather than checking S3 since S3 is an eventually consistent system.

    2. Data lost or corrupted at this stage isn’t recoverable so the greatest design objective for Secor is data integrity.

      data loss in S3 is being mitigated.

    1. incidents are an unavoidable reality of working with distributed systems, no matter how reliable. A prompt alerting solution should be an integral part of the design,

      see how it can hook into the current logging mechanism

    2. Consumers in this group are designed to be dead-simple, performant, and highly resilient. Since the data copied verbatim, no code upgrades are required to support new message types.

      exactly what we want

  5. May 2017
    1. The Kafka cluster retains all published records—whether or not they have been consumed—using a configurable retention period. For example, if the retention policy is set to two days, then for the two days after a record is published, it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so storing data for a long time is not a problem.

      irrespective of the fact that the consumer has consumed the message that message is kept in kafka for the entire retention policy duration.

      You can have two or more consumer groups: 1 -> real time 2 -> back up consumer group

    1. Consumer Federation of America

      This may be a front group. Investigate, find additional sources, and leave research notes in the comments.

    2. Consumer Federation of America

      This may be a front group. Investigate, find additional sources, and leave research notes in the comments.

    3. Consumer Federation of America

      This may be a front group. Investigate, find additional sources, and leave research notes in the comments.

    1. For a topic with replication factor N, we will tolerate up to N-1 server failures without losing any records committed to the log.

      for Eg for a given topic there are 11 brokers/servers and for each topic the replication factor is 6. That means the topic will start loosing data if more than 5 brokers go down.

    2. The way consumption is implemented in Kafka is by dividing up the partitions in the log over the consumer instances so that each instance is the exclusive consumer of a "fair share" of partitions at any point in time. This process of maintaining membership in the group is handled by the Kafka protocol dynamically. If new instances join the group they will take over some partitions from other members of the group; if an instance dies, its partitions will be distributed to the remaining instances.

      The coolest feature: this way all you need to do is add new consumers in a consumer group to auto scale per topic

    3. Consumers label themselves with a consumer group name

      maintain separate consumer group per tenant basis. Helps to scale out when we have more load per tenant.

  6. Sep 2016
    1. A recent Hewlett-Packard printer software update changed the printers so they would not work with third-party ink cartridges. Worse, the change was made as part of a security update.

      https://act.eff.org/action/tell-hp-say-no-to-drm Petition HP to fix this wrongdoing, and promise not to repeat it. They are also being asked to promise not to invoke the DMCA against security researchers who find vulnerabilities in their products.

  7. Feb 2016
  8. Nov 2015
    1. If this were true for modern society, it has multiplied in ourage of social media, in which control and value are indissolubly linked to the machine ensemblesthat comprise contemporary digital infrastructures.

      I have studied in my International Marketing course here how social media is a cultural institution in society and has an extremely powerful influence on societal structures regarding preferences, levels of acceptance of products/technology, and how consumers are influenced to use them.

  9. Aug 2015