It is only through broad public conversations and beginning to see the consequences of some of the approaches I was taking that I have come to fully appreciate the severe limits of technocracy. In that case, as in all those above, there is a severe danger of great technical minds being wasted on an arrogant pursuit of remaking the world in their image, rather than contributing to a broader conversation.

those who don’t usually end up in jail

A primary goal of AI design should be not just alignment, but legibility, to ensure that the humans interacting with the AI know its goals and failure modes, allowing critique, reuse, constraint etc.

Yet the lack of technical formalizability does not imply there is nothing systematic about achieving legibility.

Designers must explicitly recognize and design for the fact that there is critical information necessary to make their designs succeed that a) lies in the minds of citizens outside the technocratic/designer class, b) will not be translated into the language of this class soon enough to avoid disastrous outcomes and c) does not fit into the thin formalism that designers allow for societal input.

Keynesian planning and neoliberal privatization drives are superficially quite opposite tendencies. Yet deeper down they share the view that a thin formalism, based on aggregate statistics like inflation, GDP growth, output, interest rates, etc. as defined in the theory, are enough to process the wide range of social feedback necessary for sensible political and economic decision-making.

Brasilia stands as a monument in glass, steel and stone to the arrogance of technocracy

supported by a community of scientists and engineers that police the boundaries of what is considered valid and valued work within such a knowledge system

Let us call this goal “fidelity”, as it tries to make the formal system as true to the world as possible and contrasts with “optimality”. Yet, as the same time, they must recognize that whatever they design, it will fail to capture critical elements of the world. In order to allow these failures to be corrected, it will be necessary for the designed system to be comprehensible by those outside the formal community, so they can incorporate the unformalized information through critique, reuse, recombination and broader conversation in informal language. Let us call this goal “legibility”.

Constraints on this process based on democratic legitimacy or explicability, “common sense” restrictions on what should or shouldn’t be optimized, unstructured or verbal input into the process by those lacking formal training, etc. are all viewed as harmful noise at best and as destructive meddling by ill-informed politics at worst.

“teaching engineer”

Despite all the talk about “leveling the playing field” and disrupting old, powerful institutions, the Web replicates many pre-existing inequalities; it exacerbates others; it creates new ones. I think we have to work much harder to make the Web live up to the rhetoric of freedom and equality. That’s a political effort, not simply a technological one.

But the readable, writable, programmable Web is an important development in education technology. Perhaps one of the most important. As such, we can’t just let that go. We can’t just surrender the Web to the technology industry, just as we shouldn't surrender ed-tech to programmed instruction.

Designing the machine. Designing the material.

But my experience with the correspondence course did help me understand, I think for the first time in my life as a student I think, how our models and our theories and our practices in education shape and are shaped by the technologies we use.

“Wait, what are we doing? Why?”

Me, I rewound and replayed those statistics videos a lot. It didn’t help.