79 Matching Annotations
  1. Dec 2017
    1. The three processes needed to be separated into three small, disposable programs.

      Pipelines FTW

    2. The right way is to build the smallest program possible that's hard-coded for one vendor's API. The number of times you'll need to switch between vendors is so seldom that the investment to abstract it won't pay off, and you won't have to cram another vendor's paradigm into the abstraction you've just invented.

      Isn't this over-specialization and over-automation?

    3. Your software system will need the equivalent of hacksaws, Dremels, and putty.

      Good point. Any system that is monolithic and magical is doomed to fail.

    4. and system needs to leave enough edge-cases un-automated so that the users are continuously practiced and know how to use the tools well.

      Again, I don't agree. The system should be reflective enough to allow people to reacquaint themselves with the system as necessary instead of paying the constant tax of manual labor.

    5. Imagine a machine that receives wood in one end and outputs furniture. It's a completely sealed unit that's automated for safety and efficiency, but when a splinter gets stuck somewhere the machine's way of dealing with the problem is to dump the entire pile of unfinished parts into a heap. As the machine only has one input, and that input only takes raw wood, there's no way to fix the cause of the fault and resume the process where it left off.

      This seems like not having the proper debugging capabilities instead of being "over-automated". The designers assumed perfect operation and so did not add the proper entry points for debugging and visibility.

    6. When a peculiarity rears its head and gets in the way of a necessary change, there'll be less to demolish and rebuild.

      Modularity is good but it adds overhead in other places. Maybe the overhead is justified but it requires discipline to keep under control because it too can balloon out of proportion relative to useful work being done by the system.

    7. There is no such thing as the ultimate generalized system (unless you count the Turing Machine itself) and you will often not realize what a peculiarity is until it's suddenly getting in the way.

      Are we doomed to the Turing tarrpit if we want to make a generic system?

    8. These are slow deaths because the cost to work around them on a daily basis eventually overwhelms the payoff from making the change. If the organization depending on this system doesn't die, then it'll be forced to replace the entire system at great risk.

      Usually the organization will leverage cheap labor to work around the issues.

    9. a peculiarity of your business got baked into the way the system works

      The system is over-specialized for the present instead of the future.

    10. their parts can't be swapped out

      Incidental complexity from the way the problem was coded.

    11. But most systems and computer programs are written to resist change. At the system level it's a problem of ignorance, while at the program or code level it might also be a consequence of reality, since code has to draw the line somewhere.

      Many things are dynamic but code is static. This echoes Gerald Sussman's talk about us not knowing how to code.

    1. The workflow has to begin with the EDI document and use it as the bible all the way to the end. You pull the minimum details into your database for tracking and indexing, and build an abstraction layer to query the original document for everything that isn't performance sensitive (such as getting the spec for the shipping label). Now when the document's format changes, you only have to change the abstraction layer

      Sure sounds like a monad. Build up the computation tree and then run it by passing in the required parameters.

    2. You need to design your system so that validation takes place after a transaction has been received and recorded, and that the transaction is considered "on hold" until validation is performed

      Sometimes regulatory practices can prevent this. Even though the user has a shitty experience there is nothing that can be done from the software side because the regulation spells out exactly how and what needs to happen in what sequence.

    3. Your customers can probably tolerate a delay in fulfillment, but they'll go elsewhere if they can't even submit an order at all

      This layer of indirection can be helpful but has added another point, the queue, that can fail. As long as the queue is less error prone than the DB then this is fine but usually DBs are way more robust than queues.

    4. Address correction services, for example, can identify addresses with missing apartment numbers, incorrect zip codes, etc. They can help you cut back on reshipping costs and penalty fees charged by UPS for false or inaccurate data. If this service fails you'll want your software to transparently time-out and pass the address through as-is, and the business will simply cope with higher shipping costs until the service is restored

      Pass through modes tend to go unnoticed. I think it is better to fail loudly rather than simply pass things through.

    5. All that you've done is create more possible failure modes.

      Because each insurance policy is another component with its own failure modes.

    6. A failure mode is a degradation of quality, but it is not yet a catastrophe. In your personal life and in your work, you should always think about what kind of quality you'll be limping along with if some component or assumption were to fail. If you find that quality is unpalatable, then it's time to go back to the drawing board and try again.

      I've never seen this exercise performed.

    7. At 4:00:36 a.m. on March 28, 1979 the pumps feeding the secondary cooling loop of reactor number 2 at the Three Mile Island nuclear plant in western Pennsylvania shut down. An alarm sounds in the control room, which is ignored becuase the backup pumps have automatically started. The backup pumps are on the loop of a 'D'-shaped section in the pipework. At the corners of this 'D' are the bypass valves, which are normally open, but were shut a day earlier so that technicians could perform routine maintenance on the Number 7 Polisher. Even though they completed the maintenance, they forgot to reopen the valves, meaning that the backup pumps are pumping vacuum instead of cooling water.As pressure in the primary cooling system rose--from being unable to shift heat into the secondary loop--a Pressure Relief Valve (PORV) on top of the reactor vessel opens automatically and begins to vent steam and water into a tank in the floor of the containment building.Nine seconds have elapsed since the pumps failed and now control rods made of boron and silver are automatically lowered into the reactor core to slow down the reaction. In the control room the indicator light for the PORV turns off, but the PORV is still open: its failure mode is "fail open", like how a fire-escape door is always installed on the outer rim of the door-jam.Water as well as steam begin to vent from the PORV, a condition known as a Loss Of Coolant Accident. At two minutes into the accident, Emergency Injection Water (EIW) is automatically activated to replace the lost coolant. The human operators see the EIW has turned on, but believe that the PRV is shut and that pressure is decreasing, so they switch off the EIW.At the eight minute mark, an operator notices that the bypass valves of the secondary cooling loop are closed, and so he opens them. Gagues in the control room falsely report that the water level is high, when in fact it has been dropping. At an hour and 20 minutes into the accident, the pumps on the primary cooling loop begin to shake from steam being forced through them. An operator mistakes this to mean the pumps are malfunctioning, so he shuts off half of them. These are the last two that were still operation, so now there is no circulation of cooling water in the core at all, and the water level drops to expose the top of the core.Superheated steam reacts with the zirconium alloy in the control rods, producing hydrogen gas that escapes through the PORV.At two hours and 45 minutes, a radiation alarm sounds and a site emergency is declared and all non-essential staff are evacuated. Half of the core is now exposed, but the operators don't know it, and think that the temperature readings are erroneous.Seven and a half hours into the accident, the operators decide to pump water into the primary loop and open a backup PORV valve to lower pressure.Nine hours, and the hydrogen in the containment vessel explodes. This is heard as a dull thump, and the operators think it was a ventilator damper.Fifteen hours in, and the primary loop pumps are turned back on. Half the core has melted, but now that water is circulating the core temperature is finally brought back under control.But even if the operators had done nothing at all, Three Mile Island had an inherently high-quality failure mode: it was a negative void coefficient reactor. This meant that as steam increased (voids), the nuclear reaction decreased (negative coefficient).Compare this to a reactor with a positive void coefficient, and a much lower quality failure mode.

      This is a cool story. Seems like everything went wrong.

    1. A treadmill, forever. But we can make the treadmill nicer.

      Red queen thinks this is a slow kind of country.

    2. Progress must come from the evolution of computer languages, and not from the artifacts built with them. Each generation folds another layer of experience and insight into new syntax, not new modules. Next year's batch of kids will still make a shitty web site, and they'll graduate to make shitty payroll and inventory management systems for the companies they work for. Everything they make will have to be thrown away. But each generation's produce will get better and better in quality because the tools they use will get better. Since the dawn of computers in business, each re-imagination of software for managing tea shops gets better and takes less time to develop.

      The lowest reusable component in programming is the programming language and that is why things mostly suck. There is no such thing as software reuse.

    3. In a way, state and community colleges have already figured this out and have their students rebuild the school's web site and class signup software every few years as part of their CS curriculum. It gets thrown away a few years later by another batch of kids, and they always do a shitty job of it each time, but this is what it's like in the enterprise, too.

      Many enterprises already buy into this model and outsource all the work to places like Palantir, IBM, Accenture, HP, etc.

    4. This is ultimately loss aversion

      Good observation. Sunk cost and loss aversion consistently mutate good things into horrible monsters.

    5. Data doesn't like to live in multiple places, so the programs closest to the data accrue functions to be everything to everyone. Now you get software that's both a floor wax and a salad dressing, and your training costs for both users and maintenance developers increases--for doing nothing more than explaining to them why things are they way they are.

      More entropy winning.

    6. institutionalized cluelessness

      Interesting phrase. I'd heard it as institutional brain damage.

    7. But the universe itself is the most malicious partner; it works hard to contrive problems that--by nature--will get worse if you try to solve them with the ponies and unicorns of your time. It sets you up for a fall before you've even started writing code.

      Entropy always wins.

    8. Ten years before it was XML, Java, Extreme Programming and Design Patterns. Ten years in the future it will be Functional languages, Solid-State hard drives and schema-less databases.

      Wise words. Time is a flat circle.

    9. And there is such a strong economic incentive to solve a new problem with an old trick that good problems go misdiagnosed.

      Always comes back to incentives and the failures of human heuristics.

    1. Machine instructions are also encoded such that code size is compact while decoding them, which is ultimately done in hardware by the machine, is still fast. This is, of course, extremely important since every single instruction a processor executes needs to be decoded into opcode and operands before execution.

      What are the constants involved in this slowdown?

    2. There are applications that are correct even when such wrap-arounds occur.

      What are these applications?

    3. int* foo;

      This is assuming a 32-bit machine I think. I had to replace all ints with uint64_ts.

    4. Similarly, all characters written to files and the screen are written from left to right using just one line of code in selfie.c.

      Is this related to endianess?

    5. could increment 01010101 according to elementary arithmetic making 01010101 represent 85

      So what would be the incremented binary sequence? 01010110

    6. similar way than

      "as", not "than"

    7. We conclude this chapter by pointing out that there was a time when there was no compiler for C and in fact no compiler for any programming language. The first compilers were therefore written in machine code or in some programming language and then compiled into machine code by hand. Thus even selfie’s origins go back to that time!

      This is where my head spins. The process of bootstrapping systems and carrying on the binary encoding of those bootstrapped compilers from one generation to the next. It all seems like a house of cards waiting to topple at any instant.

    8. This also means that bits can sometimes be data and sometimes be code. Selfie is a non-trivial example of that phenomenon.

      Code/data quality in action.

    9. However, delivering just a single bit by truck may take days. Thus the truck clearly provides terrible latency not suitable to host, say, a skype call. Got it?

      Don't get it. What about time? I can skype over fiber but can't skype over truck.

    10. kibi, mebi, gibi, tebi

      Good to know.

    11. ./selfie -c selfie.c -o selfie5.m -m 3 -l selfie5.m -y 2 -l selfie5.m -y 2 -l \ selfie5.m -y 2 -l selfie5.m -y 2 -c selfie.c -o selfie6.m

      I believe it. Passthrough is better than full emulation.

    12. ./selfie -c selfie.c -o selfie3.m -m 2 -l selfie3.m -y 2 -c selfie.c -o selfie\ 4.m

      Clearly not as slow as the emulator on top of the emulator.

    13. Hypster is a hypervisor that mimics mipster without executing any machine code itself but instead asks the mipster on which it runs to execute the machine code on its behalf. For the machine code there is no difference except that it runs much faster.

      So is this a level of pass-through?

    14. A way to avoid the slowdown of running emulators is virtualization.

      Not a bummer.

    15. This means three mipsters take days, four take months, and five take years!

      Exponential slowdown. Bummer.

    16. ./selfie -c selfie.c -o selfie.m -m 1 -l selfie.m -m 1

      That was super slow.

    17. Let us try that using the -d option which invokes mipster in debugging mode.

      This is pretty nifty. More tools should have such features.

    18. ./selfie -c selfie.c -s selfie.s

      It's only ~30k lines

      ...
          3 0x1C920(~7439): 0xDFBF0000: ld $ra,0($sp)
          2 0x1C924(~7439): 0x67BD0018: daddiu $sp,$sp,24
          1 0x1C928(~7439): 0x03E00008: jr $ra
      29260 0x1C92C(~7439): 0x00000000: nop
      
    19. ./selfie -c selfie.c -o selfie1.m -m 2 -c selfie.c -o selfie2.m

      Yup, checks out

      $ diff selfie1.m selfie2.m
      $ echo $?
      0
      
    20. However, it does work and this is what counts here:

      It actually worked but took a really long time

      $ ./selfie -c selfie.c -m 6 -c selfie.c
      ./selfie: this is selfie compiling selfie.c with starc
      ./selfie: 199244 characters read in 7438 lines and 988 comments
      ./selfie: with 116630(58.82%) characters in 30827 actual symbols
      ./selfie: 268 global variables, 317 procedures, 447 string literals
      ./selfie: 2095 calls, 800 assignments, 73 while, 585 if, 277 return
      ./selfie: 125984 bytes generated with 29260 instructions and 8944 bytes of data
      ./selfie: this is selfie executing selfie.c with 6MB of physical memory on mipster
      selfie.c: this is selfie compiling selfie.c with starc
      selfie.c: 199244 characters read in 7438 lines and 988 comments
      selfie.c: with 116630(58.82%) characters in 30827 actual symbols
      selfie.c: 268 global variables, 317 procedures, 447 string literals
      selfie.c: 2095 calls, 800 assignments, 73 while, 585 if, 277 return
      selfie.c: 125984 bytes generated with 29260 instructions and 8944 bytes of data
      ./selfie: selfie.c exiting with exit code 0 and 1.73MB of mallocated memory
      ./selfie: this is selfie terminating selfie.c with exit code 0 and 1.46MB of mapped memory
      ./selfie: profile: total,max(ratio%)@addr(line#),2max(ratio%)@addr(line#),3max(ratio%)@addr(line#)
      ./selfie: calls: 55549216,18593526(33.55%)@0x3010(~1368),9459267(17.03%)@0x3070(~1373),9130155(16.44%)@0x30EC(~1378)
      ./selfie: loops: 3125924,2595870(83.33%)@0x6CA0(~2348),325459(10.41%)@0x387C(~1466),91571(2.92%)@0x5E98(~2083)
      ./selfie: loads: 354840870,18593526(5.24%)@0x3024(~1368),9459267(2.66%)@0x3084(~1373),9130155(2.57%)@0x3100(~1378)
      ./selfie: stores: 234841894,18593526(7.91%)@0x3014(~1368),9459267(4.02%)@0x3074(~1373),9130155(3.88%)@0x30F0(~1378)
      
    21. ./selfie -c selfie.c -m 2 -c selfie.c

      What happens if I increase the memory? Does it go faster?

    22. ./selfie -c selfie.c -m 1

      The code is still compiled to mipster but just remains in memory. I'm going to guess since that's what -c did before I'm assuming it continues to do that even if -m option is passed.

    23. After loading selfie.m the -m 1 option directs mipster to emulate a computer with 1 megabyte of memory (abbreviated 1MB, explained below) for executing selfie.m. Since selfie.m is invoked without any options, which could appear after the -m 1 option, it responds, just like selfie without options before, with its usage pattern and then terminates. After that mipster terminates and outputs a summary of its builtin performance profiler.

      I ran

      ./selfie -l selfie.m -m 1 -c selfie.c
      

      and it actually worked. It started compiling itself while emulating itself.

    24. it also includes mipster, which is an emulator of the computer that can execute selfie.m and any other machine code generated by starc. That computer is so simple that everyone can understand it in a reasonable amount of time even though it corresponds to parts of a real machine.

      Looks like that virtual machine is mipster. Is it a MIPS machine?

    25. selfie.m is that selfie is executable on our computer whereas selfie.m is executable on a computer that we do not have. This is not the end of the story though.

      Sounds like there is a virtual machine somewhere. So I'm going to guess selfie has a virtual machine specification as well.

    26. ./selfie -c selfie.c

      Looks like selfie can't deal with macros. I added

      #define uint64_t unsigned long long
      

      to get rid of syntax highlighting errors in VSCode and selfie crapped out

      ./selfie: this is selfie compiling selfie.c with starc
      ./selfie: error in selfie.c in line 85: found unknown character '#'
      
    27. cc -w -m32 -D'main(a,b)=main(a,char**argv)' selfie.c -o selfie

      My output was

      cc -w -O3 -m64 -D'main(a,b)=main(int argc, char** argv)' -Duint64_t='unsigned long long' selfie.c -o selfie
      

      I'm guessing because I'm working on 64-bit system.

    28. It tells us what numbers are, not just how to add, subtract, multiply, and divide them!

      Context is important. Data on its own is meaningless.

    29. When it comes to a machine the meaning of bits and thus characters or any kind of symbol has to be created mechanically. The key insight is that the meaning of bits is in the process of changing bits, not in the bits themselves.

      Dynamics and not statics is what gives semantics. How does one reconcile this with the static type advocates?

    30. 1,411,264 bits

      1.5M bits for a self-reflective system. Again, not too shabby.

    31. selfie.c

      Looked at the file an it looks like about ~8k lines of code and comments. Not so large for a self-reflective system in one file.

    32. The second reason is that whatever you can say using any number of values per digit or character greater than two you can also say using two values with only an insignificant difference in the number of digits or characters you need to say the same thing.

      Sounds like a computable isomorphism with a constant factor slow down.

    33. By now your mind might be spinning but you at least understand why selfie is called selfie.

      Sounds like the system is fully self-reflective.

    34. Selfie shows how to do that. This may sound difficult but can actually be done in a systematic fashion with a strong focus on basic principles.

      Turtles all the way down.

    35. The reason is that computers and software are so expressive that it is unlikely that any other computational machine in the future will be more expressive. Anything that can be computed can be computed now, if you have enough time, space, and energy.

      Turing completeness is a blessing and a curse because just because all Turing machines are equivalent that doesn't mean all languages are equally expressive for all tasks.

    1. So far this flu season more than 6,000 people in the U.S. have tested positive for influenza and 856 have been hospitalized for laboratory-confirmed flu-associated reasons, according to the CDC. The most effective way to prevent the flu and its many potentially lethal complications is to get vaccinated.

      I thought vaccination didn't work on the flu since it always mutates too quickly for any vaccine to work. Maybe I need to revisit the science.

    2. In a typical season most flu-related deaths occur among children and the elderly, both of whom are uniquely vulnerable. The immune system is an adaptive network of organs that learns how best to recognize and respond to threats over time. Because the immune systems of children are relatively naive, they may not respond optimally. In contrast the immune systems of the elderly are often weakened by a combination of age and underlying illness. Both the very young and very old may also be less able to tolerate and recover from the immune system's self-attack. Apart from children between six and 59 months and individuals older than 65 years, those at the greatest risk of developing potentially fatal complications are pregnant women, health care workers and people with certain chronic medical conditions, such as HIV/AIDS, asthma, and heart or lung diseases, according to the World Health Organization.

      The system is always on the precipice of decline. It's a wonder anything in biology works at all. The cycles are all unstable or barely stable.

    3. Sometimes Guillain–Barré leads to a period of partial or near-total paralysis, which in turn requires mechanical ventilation to keep a sufferer breathing. These complications are less common, but can be fatal.

      Partial paralysis from flu. Nature is fucking scary.

    4. Based on autopsy studies, Kathleen Sullivan, chief of the Division of Allergy and Immunology at The Children's Hospital of Philadelphia, estimates about one third of people who die from flu-related causes expire because the virus overwhelms the immune system; another third die from the immune response to secondary bacterial infections, usually in the lungs; and the remaining third perish due to the failure of one or more other organs.

      These all seem like shit ways to die.

    5. In most healthy adults this process works, and they recover within days or weeks. But sometimes the immune system's reaction is too strong, destroying so much tissue in the lungs that they can no longer deliver enough oxygen to the blood, resulting in hypoxia and death.

      Evolution is a fucking joke. There is no intelligent design in any of this. Humans are an agglomeration of disparate pieces that barely just work.

    1. If the entire tables are constant over a long time, you could generate them as modules. Nowadays, compile-time constants (even complex ones) are placed in a constant pool associated with the module. So, you could generate something like this: -module(autogen_table). -export([find/1]). find(0) -> {some_struct, "hello", ...}; ... find(99) -> {some_other_struct, <<"hi!">>} find(X) -> throw({not_found, X}). As far as I know, these constants will not be copied to the private heaps of the processes. The generated code will also give you the fastest possible lookup.

      This is a pretty cool trick. So looks like a way to bypass the copying overhead is to make a function that only has constant outputs and doesn't actually compute anything.

    1. To summarize: without queuing mechanism: same Erlang node: 5.3 million messages/min; different Erlang nodes: 700 K messages/min. with queuing mechanism: same Erlang node: 5.3 million messages/min; different Erlang nodes: 2.1 million messages/min. The complete code to run this on your machine is available here. This whole ‘queuing idea’ is still an experiment, and I’d be more than delighted to hear your feedback, to see whether you are getting the same results, you know how to improve the concept or the code, or you have any considerations at all you would like to share.

      I got here from the discord blog on how they optimized their performance and it looks like the trick is to batch messages when sending to remote nodes. Seems kinda obvious though that batching messages would improve performance.

      A trick to keep in the back pocket.

    1. :ets.update_counter/4, which performs atomic conditional increment operations on a number inside an ETS key. Since we needed high concurrency, we could also run ETS in write_concurrency mode but still read the value out, since :ets.update_counter/4 returns the result. This gave us the fundamental piece to create our Semaphore library. It is extremely easy to use and performs really well at high throughput:

      ETS has atomic counters! Is there anything Erlang doesn't have?

    2. In most other languages, we could use an atomic counter to track outstanding requests and bail early if the number was too high, effectively implementing a semaphore.

      Slightly disappointed with the actual answer. Why doesn't the same trick work in Elixir?

    3. We knew how we would solve this in other languages, but how would we solve it in Elixir?

      Umm... how? How would this be solved in another language?

    4. After doing some research, we found mochiglobal, a module that exploits a feature of the VM: if Erlang sees a function that always returns the same constant data, it puts that data into a read-only shared heap that processes can access without copying the data. mochiglobal takes advantage of this by creating an Erlang module with one function at runtime and compiling it.

      This is a cool trick and it sounds like partial evaluation and just-in-time compilation.

    5. If the session server were to crash and restart, it would take about 30 seconds just for the cost of lookups on the ring. That does not even account for Erlang de-scheduling the single process involved in the ring for other processes’ work. Could we remove this cost completely?

      What is this ring structure? Why is it so expensive to reload it?

    6. An Erlang VM responsible for sessions can have up to 500,000 live sessions on it.

      This is pretty impressive: half a million processes.

    7. Finally, those workers send the messages to the actual processes. This ensures the partitioner does not get overloaded and still provides the linearizability guaranteed by send/2. This solution was effectively a drop-in replacement for send/2:

      Opening says they weren't going to shard but how is this not sharding?

    8. We knew we had to somehow distribute the work of sending messages. Since spawning processes in Erlang is cheap, our first guess was to just spawn another process to handle each publish. However, each publish could be scheduled at a different time, and Discord clients depend on linearizability of events. That solution also wouldn’t scale well because the guild service was also responsible for an ever-growing amount of work.

      So seems like the plan was to do the send in another spawn call but that would be a hell of a lot of processes and as they mention lose the "linear" aspect of publishing messages.

    9. session process (a GenServer), which then communicates with remote Erlang nodes that contain guild (internal for a “Discord Server”) processes (also GenServers)

      So looks like the coordination happens across nodes with these "guild" processes

    10. How Discord Scaled Elixir to 5,000,000 Concurrent Users

      Is this across the entire set of clusters or is this per single node or set of nodes for a given "guild"?