Reviewer #1 (Public review):
The authors describe a learning algorithm that is meant to deal with the problem of inference in large, complex environments. They argue that simpler but ensemble learning systems with heterogeneous learning rates often perform better than more complex approaches. They also relate the ideas of their model to a circuit implementation in the mushroom body, an associative learning area in insects.
I found the argument about the value of sublearners remaining simple to be thoughtful and intriguing. I also appreciated the connection to biology and, although there remain details to be worked out, it seems that the proposed approach has what I'd call a "biologically plausible" instantiation.
Comments on the major claims:
The paper claims that algorithms that learn stochasticity and volatility "necessarily result in worse short-term performance than Hetlearn." The results with Autocovariance Least Squares and the model of Piray and Daw are consistent with this statement, but the authors seem to be making a more generic claim. Is there a rigorous justification of this? In particular, I found the section "Competing sublearners should stay simple" to straddle a line between opinion and fact, and I wasn't sure where it intended to be regarding that line.
Relatedly, regarding the statement just before that section: "The exponential convergence of Hetlearn weightings stands in contrast to the slow convergence of stochasticity/volatility estimates (Figures 2A and C), which result in poor initial performance of the Kalman learner." Is the claim here that the Kalman learner doesn't converge exponentially, or that it converges more slowly?
The paper would benefit from a comparison to other models of multiple learning systems with different timescales. The classic one would be the complementary learning systems ideas about hippocampus and neocortex (McClelland, McNaughten, O'Reilly 1995), and the many subsequent follow-ups. Two other relevant ones to consider are Roxin & Fusi 2013 and Lindsey & Litwin-Kumar 2024, which also consider multiple memory systems with different plasticity rates.
Comments on clarity:
(1) Page 2: There could be some more unpacking of the relationship between x and z in equations 1a, 1b. If z_t are time-varying, shouldn't theta, if it represents what is learned about "large-world factors," also depend on time?
The description of second-order conditioning on page 6 is confusing. Second-order conditioning tests whether S2 acquires a positive predicted value after pairing with S1, which had previously been paired with reward. The text makes it seem like the goal of second-order conditioning is to learn S2->S1->no reward.
(2) Page 6: "Representations decay on a slower timescale than stimulus presentation. Hence, the representation x2 after experiencing S2 → S1 holds traces of the S2 representation." I couldn't understand this. Maybe an equation would help?
(3) Page 7: The phrase "unconditioned stimulus" is introduced without being described. Is this a reward or punishment? What does it have to do with valence lower-case v, or value upper-case V? The authors seem to mix notations from different subfields, leading to a rather confusing description of the model.
Comments on relationship to biology:
(1) Page 6: It is assumed that Kenyon cells interact recurrently through plastic weights. This is an unconventional assumption for mushroom body models, and it is made without much discussion. The authors cite papers that argue for changes in the calyx during learning (although these effects are generally much weaker than the well-documented plasticity onto MBONs). However, KC-to-KC interactions are largely in the lobes, not the calyx. The authors should provide more discussion of this assumption. What happens if the recurrence is weak or absent? A relevant reference to consider is Manoim et al. Current Biology 2022, which shows that KC-to-KC interactions are inhibitory.
The model seems to imply that each MBON compartment encodes only one memory, or did I misunderstand? Is this necessary? Typically, it is thought that because of the sparse, decorrelated representations in the KCs, an individual MBON could store multiple associations without too much interference.

