I'm not an expert in biology or anything, but perhaps the moral there is we should rethink this 'cell' idea? IIRC astronomers continue to talk about 'planets' even though the longer you examine the concept the more incoherent it becomes. (For an extended example, see the infamous Discourse about whether "Pluto is a planet", which led to hilarious goal post stretching where people kept trying to find a definition of 'planet' that exactly fit the traditional celestial objects we classify as planets without having to include any new ones or exclude existing ones)

There is obviously no rule that says the categorizations we come up with for stuff when a field is young should be expected to have infinite inferential reach as that field of knowledge expands.

Has anyone ever actually proposed something like this?

Actually, let me rephrase, has anyone we have any good reason to take philosophically seriously ever actually proposed something like this, in a context I would care about?

Something like, an important philosopher that people-who-do-stuff take seriously (it's not my job to police whatever degenerate things academia gets up to separated from course correcting incentives) making a serious unironic proposal along these lines that nontrivial resources were invested into?

This argument strikes me as malformed and perverse.

Obviously there's a conflation here between two different questions, but I think a deeper error is implied by the fact of the conflation.

You seem to suppose there is some objective 'literal' sense of the statement "Is there water in the fridge?". Considering you wrote earlier that beliefs are not strings of words (reasonable), it's weird to then use that as a premise to argue against Bayesian Epistemology. Words are a way to locate things in concept space, and epistemology doesn't exist separate from neurology. There are no real agents that make this mistake, because it's more or less based on a type error. A Bayesian Epistemology is de-facto not enough to replicate human cognition, if it was then artificial intelligence would be solved. That doesn't stop it from being pretty powerful in conjunction with the world modeling capabilities of a human brain.

More to the point: At the moment of posing the question, any sensibly designed agent capable of being in this situation wielding a Bayesian Epistemology knows exactly which of these two questions it is posing when it thinks "Is there water in the fridge?" in its internal monologue. The conflation only exists when an outsider observer tries to impute some literal meaning to a thing that is contextually in a system of meaning extending beyond the words in the sentence.

I tend to find it's useful to handle this case with the notion of category membership having a spatial organization. Leading to the whole idea of things being central, noncentral, midway, etc members of categories. 'Eggplants are noncentrally berries' communicates precisely what we mean, they are berries, but they're also not what is typically imagined as a berry.

Unfortunately the full description of 'reasonableness' is not currently present, but it's given in short as "everyday informal thinking and acting". Which, to wit: So what is that made of anyway? It's not like informal reasoning just exists an a magic phenomena. Clearly, there is some kind of machine (us) existing in the world that implements this informal reasoning algorithm. Obviously we can't know exactly how it works, but I think of a lot of the 'rationality' critiqued here as increasingly powerful models of what that thing is made of; that is the entire point arguably.

That these models suck is fair game, but it's important not to conflate the people who geek out about formal systems with the people trying to derive the code to human cognition with the people trying to get a decent working model of human cognition so they can augment it with heuristics et al.

It's common in speech coding (e.g. a vocoder) to rely on a thing that reconstructs the 'meaning' of a signal by predicting its 'full' representation.

This act of predicting then is also a form of compressing, by predicting the full representation from its lossy analogue you require less bandwidth to transmit messages just like if you'd used a non-stochastic compression technique.

Not quite. You use esoteric equipment and methods to get the opportunity to make an observation of some tiny bit of reality that would help you narrow down hypothesis space.

I kind of feel like I'm being Motte-Bailey'd here. This book feels like it's responding to Eliezer Yudkowsky, yet this is literally the kind of scenario you have the notion of a 'prior' for. My prior probability that the moon is made of green cheese is very low. So low in fact that not even a crazy statistical analysis is going to convince me otherwise, the lie contagion factor is off the charts. That is, for the moon to be made of green cheese lots of other stuff I'm pretty confident is true has to be false.