response relevant

response

interaction

greater during the Refresh

greater during Unexpected versus Expected

Refresh-Expected to Repeat-Expected trials

n ACC

Repeat-Unexpected to Repeat-Expected

prepared response

need for control beyond

correct feedback stimuli, com-pared to error feedback stimuli, engaged a distributednetwork of brain areas consisting of the bilateral caudatenuclei, the right putamen, the right rostral and posteriorcingulate cortices, bilateral middle and inferior prefron-tal gyri, bilateral superior temporal gyri, and the rightlateral and medial occipital cortex

a strong prediction of the reinforcement learning the-ory is increased activation following error feedbackcompared to correct feedback, in the absence of con-flict related to action planning or expectancy violation

a conflictview of ACC functioning would not necessarily predictits activation following error feedback, unless this feed-back elicits conflict between conceptual representations,requires a task set reconfiguration, or violates a strongexpectancy

A neural area within this ROI (area 32:x = 4, y = 18, z = 44) was indeed more activated by error feedback thanby correct feedback on trials with random mappings

a comparison between error feedback and correct feedback on trialswith random mappings did not reveal any neural areas in which feed-back-related error activity was greater than feedback-related correctactivity

According to the RL-ERN theory, the ERN isgenerated when a phasic decrease in mesencephalicdopaminergic activity (indicating that ongoing events areworse than expected) disinhibits the apical dendrites ofneurons in anterior cingulate motor cortex;

inconsistent with theresponse inhibition hypothesis: larger inhibitory effortshould result in more effective suppression of the erro-neous response, thus in a shorter RT.

only in the RI condition would oneinhibit an incorrectly primed response

(b) a system, involving the ACC, that is responsible only for last-minute conflict resolution.

(a) a system coming into play in anticipation of demanding activities and sustained through the course of such activities

van Veen, Cohen, Botvinick, Stenger, & Carter, 2000

Specifically, it shares extensive connections with prefrontal cortex

testable predictions. One of these is that slowdowns in responding should occur not only after errors, but also after correct trials involving a high degree of conflict

focus more effectively

control conditions, the presented letter series contained no Xs.

pressing a button with each presentation but omitting this response if the presented letter was an X.

error, surprise,posterior,andposterior variance in the ex-post 264period

To satisfy these 115conditions at the population level, the constituent single neurons must betuned to combinations 116of several cognitive variablesat once(“non-linear mixed selectivity”)(Rigotti et al., 2013)

normative neuralnetwork model that yields event-predictive encodings

In the real world, we often receive signals from com-plementary cognitive systems that foreshadow when the cur-rent event is about to end (e.g. from the visual system asthe perceived distance between one’s hand and a glass ofwater approaches zero).

This module re-ceives event boundary information in the form of an increasedvalue when a transition is about to happen

Pavlovian attraction parameterthat had proved important in the behavioral study (Huys et al., 2012).This captured the attraction of states based on their average future con-sequences, regardless of whether sufficient choices remained on a trial toexploit those consequences.

To the extent that participants do not perceive any action–outcome contingencies, f ERN amplitude appears to be less sensitive to reward probability

pseudorewards are distinct from primary rewards

option terminates when a particularsubgoal is attained, which generates an option-specificprediction error, referredtoasapseudorewardpre-diction error (PPE)

Many neurons were alsosensitive to multiple task events

Formalizing this account of choice may require us to reformulate the RL problem as being one of minimizing distance to goals rather than maximizing discounted future reward

prioritizing locations that will cause a substantial change in the future behavior of the agent

When visual range is reduced, such as in nocturnal vision, plan-based control may only exist for stable environments over a previously established cognitive map

bility to detect the structure of a complex, cluttered environment with high temporal and spatial resolution

dorsomedialPFC

Fine et al., 2020

This may even be of greater importance than for traditional reinforcement learning research

task set representations will be inhibited after an unexpectedorsurprisingevent

Nunez Castellar et al

performed a task set

N-2 Repetition Cost

(Mayr & Keele, 2000)

Interestingly,if the density is above zero then there is a slight gain by adding an ISI of 2s but no suchbenefit exists when the density is 0

factoring out the negative lin-ear effect of rCV in rule-free trials

r

‘archive’ of the different states

only found forwords in the backward condition

no main effects of Community

. The PRO model thus accurately accounts for dACC activity over the course of the entire trial in our speeded decision-making task

The EVC posits that activity in dACC reflects ‘expected value of control’—a trade-off between cost and benefits resulting in the selection of an optimal control signal.

contextofclassicalconditioning

a number of candidate learning signals measured through fMRI do not reflectlearning rate when considering a broader set of statistical contexts

In humans, con-flict signals are apparent in the firing rates of single dACC neu-rons

only a main effect of phasic versus no-phasic LC response

“fake-beep” trials

ACC rsc tended to be larger after versus before the beep stimulus, but only for the subset of trials in which it also elicited the characteristic phasic LC response

quenching

slower responses to the first than the second orthird targets

frontoparietal network early in the preparation interval thatwas activated when the cues validly indicated that the task wouldchang

the benefit of updating task-set on switch trials(switch-tovs.switch-away)

Frontoparietal theta (FPθ) activity has also been reported.

strategic mPFC dynamics whereongoing activity is modulated in response to changes in context

non-informativecues

separate regressors were not constructed for each positionas in the onsets model above

deciding which task toperform in the future

exploration of alternative strategies doesnot seem to be driven by unreliability, conflict, or errors

Do neurally defined eventboundaries in a continuous movie, evoked by subtler transitionsbetween related scenes, generate the same kind of hippocampalsignature?

or when our goals change

action domain

OFC neuronsencoded evidence associated with each shape in the shape se-quence, but only transiently

Functional MRI data, however, suggeststhe opposite pattern for the mPFC/ACC, which are more active inresponse to gains compared to losses

two possible states of the transition probabilities: top/left/bot-tom/right and top/right/bottom/left

three possiblestates of the reward probabilities for the left/right ports

interaction between reward and task sequence did notreach significance

We were especially interested in the effect of reward on proactivecontrol mechanisms, wherein goal-relevant information is encoded inpreparation for upcoming task demands.

approach behavior

needed to assess whether this result truly reflects an aspect of [DA] signaling that is inherently slow (tonic) or could instead be explained by rapidly changing [DA] levels, that signal a rapidly changing decision variable

reward rate is a key decision variable

ACC-to-VTA 4-Hz communication increased in the vertex of the mazewhen animals made pre-reversal decisions to the LCLR arm

re-emergence of prolonged dopamine elevations by the larger than expected reward could be due to the presence of positive pre-diction errors, but could also result from a unidirectional attention, alerting, or motivating function

both prior to and at asymptotic performance the amplitude of dopamine elevation prior to sequence initiation significantly negatively correlated with time to complete the immediately following sequence

self-paced (i.e., free-operant) action-se-quence task devoid of experimenter-provided initiation cues

persistent motivation is required to drive behavior from initial actions through a series of events to earn a distal reward

it could be argued that good performance in theinstructed goal task requires intrinsic motivation

prediction error responses indopamine neurons are influenced by task representations, which are putatively maintained inthe prefrontal cortex

interaction between executive and lower-level instrumentallearning circuits

In the present study, strategic ad-justments in behavior were promoted by both small rewards andby surprising ones

consistent with the idea that monkeys treat am-biguous options as providing less information about outcomes

The activity of 55% of neurons (n51/92, 35in monkey E, 16 in monkey O) signaled the size of the rewarddelivered following risky choices

nexpected small rewards promoted largerwillingness to choose the redder option

non-specific RPEsignals did not differ

feature of the chosen stimulus, i.e., they encoded feature-specificRPEs

can lead to increases in phasic DA activity

persistently chose one of thegoal arms (only LL reward in 14 behavioral recording sessionsand only SS reward in 3 sessions) even though they sampledboth reward sizes in the preceding forced-choice trials.

the lOFC may identify in changes in rein-forcement contingencies and signal the mOFC to update apprais-als concerning actions that may be more profitable.

well trained

lOFC hasbeen reported to attenuate phasic dopamine

welltrained rats

inactivation of some ofthe main OFC and medial PFC inputs to the accumbens

lesions of the dorsolateral PFC (dlPFC) in primates or medialPFC in rats impairs shifts between different strategies or atten-tional sets

It is currently unclear as to why signed predictionerrors were observed in the LFPs, but not at the level of theensembles.

The present data were collected using the same task in which alocal field potential (LFP) was recorded (Warren et al., 2015)thatexhibited properties consistent with the FN observed in the EEGof humans

at the ensemblelevel, responses did not uniformly increase or decrease for unex-pectedly ‘‘good’’ versus unexpectedly ‘‘bad’’ outcomes but onlyconveyed information that the trial was R or NR, regardless ofwhether or not the particular outcome was unexpected

while prior historycaused eFSRs to progressively resemble FSRs, the precisetiming and the direction of change varied across the population

eFSR tracked past outcomes in a port-specific manner, afinding that is consistent with the single-unit analysis whereonly 1% of the total cells had expectancy-related changes infiring for both reversal ports.

actual outcome feedback scents will be referred to asthe FSR

‘‘early feedback-like scent response’’ (eFSR)

only the responseto the NP that changed, which appeared to reflect the mostcommonly encountered outcome at a given port

internalized representation of a future event,it is difficult to show clearly that a neuron encodes this informa-tion

set of allshortest-path problems within the grap

explicitdecisionsthatpertaineddirectlytotheprobabilisticfeaturesoftheongoingtaskmodeling volatile environments. The latter characteristics possibly ex-plain why the present results contrast with earlier findings

Participants were the fasteston random high-probability triplets

P3 amplitude modulations in both di-rections could indicate the implicit sensitivity to or the implicitacquisition of predictable relations in a stimulus stream.

minimize the RMS prediction error with respect to each subject’sobserved reaction times, RMSE¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1TPtrtðÞ^rtðÞðÞ2q, whereTis the number oftrials

‘RT~log(Trial)*Stage+Target+Recency+(1+log(Trial)*Stage+Recency|ID)’

modular-lattice effect

onsider the possibleconfound of stimulus recency: the tendency for people to respondmore quickly to stimuli that have appeared more recently

modular graph with three communities offive densely connectednodes

15 dif-ferent stimuli represented a node in an underlying transitionnetwork

no decr ease in performanc e occurr ed during these fir st sess ion blocks; in Exper iment 1 we even obs erved a modes t performance incr ease

random-high tr ial

patte rn trial

Third,andperhapsmostimportant,in theBaldwinand Kutas study,partic-ipantswererequiredtorespondonly onthosetrials onwhichthetargethadperformedacertainpredefinedmovement,whereasin thepresentexperiment,a responsehad to be given on each trial.

larger P3b effect in the second experi-mentalhalfthan in the first half

if ACC is responsible for computing changes in belief

the relationship between pupil dilation andchange in representation strength seems to hold in cases where uncertainty is increasing, but notwhen it is decreasing

weightings of representations of specific option

which option was the current high-reward option

irst process is evidence-drivenupdating of beliefs

n-adjacentdependencies

tbothprocessesinvolvethelateralprefrontal cortical regions that subserve several cognitive functions

one of themost potent antecedent conditions for the P300 component

In our study both perceptual and motor deviants di€ered perceptually fromthe next most likely, regular stimulus. Thus, it makes sense that both bear an e€ecton the N200.

200 was a€ected by both types of deviants and the P300 bymotor deviants only,

t all three types of stimuli were processed bythe system in the very same manne

hierarchical structure

when a more effort-ful task was used (semantic categorization), robust learning wasobserved

The task consisted of a number(1, 2, or 3) appearing on a computer screen,and the participant was instructed to pressthe corresponding key on the number padwith the right hand as quickly and as accu-rately as possible

novelty represents a deviation from the ex-pected likelihood of an event on the basis ofboth previous information and internal es-timates of conditional probabilities

irrespective of whether the instruction cues indicate exactlyhow to respond (i.e. to switch to the opposite response) or to guessthe response, so long as the instructions pertain to the present trialrather than to the following trial.

infrequently occurring stimulus

(despite the fact that both regions are less active overall at thistime),

fundamental differencein how the hidden variable context is represented in HPCcompared to PFC brain areas

SROmappings changed simultaneously for all 4 conditions

PEsignalsaregeneratedbythedACCandthenconvergetowardthebrainstemnuclei

notusefulinclassicalconditioning

ACC neural activity has also been shownto covary with pupil modulations under passiveviewing conditions, over both short (singlespike) and longer (several seconds) timescales [38].

facilitates integrating representations

bot-tom-up excitation might be conceived of as determining a probability distribution overhigher-level event schemata

instead of updating our beliefs about representations that we have already learned (infer-ence), we must update our beliefs about theparametersof the generative model itself

it would be wasteful to invest resources in pre-activatingupcoming information that is irrelevant to our current comprehension goals

Pro-gressively higher levels of the generative model would have increasingly larger (longer)temporal “receptive fields.”

a large prediction error that leads us to infer that the statistical struc-ture of the environment has fundamentally changed is known asunexpected surprise

can still beexplainedby this overall goal

At any given time, this event model, in turn, provides new information that is passedup to the third level of the hierarchy,

schema-relevant information is latent within long-term memory, but linked to goalrepresentations so that it can be proactively retrieved

Goal end states are usuallyconceptualized as the desired future state of affairs that is associated with a goal’s fulfill-ment

The change that isinduced by each incoming event can be conceptualized as the change in the probabilitydistribution induced by this new event (the Kullback–Leibler divergence), and it can bethought of as animplicit prediction error.

two-level generative model is often insufficient to explainthe complex and multidimensional structure of our environmental observations

probabilistic dependencies

mostuncertain

If we were able to observe all the events together as a batch, detecting the boundariesbetween event models would be less challenging

encoded probabilistically such that the events and event sequences that aremost likely to occur within a given schemaclustertogether in representational space

end state of even a single event functions as aprecondition

Surprise?and Novelty?contribute each separately to the ERP components at around 300ms

we hypothesize that participants estimate the probabilities89of transitions from a given state to another state when performing a given action

surprise independent of novelty

find the shortest path to the goal image

10 states with 4 possible actions per state plus one goal state

Surprise is triggered by the violation of expectations and manifests itself in pupil dilation [21]33and EEG signals

in studies that separately assess encoding of outcome identity versus outcome value, activity in brain areas that are often considered to be emblematic of economic value (in particular, the OFC) turns out to correlate with outcome identity instead

Indeed, the idea that value exists on a single scale, also called a “common currency,” has been extended to encompass not only goods, but also effort costs and time delays

neural network associated with sur-prise is largely distinct from that of valence

aMCC, dMCC, the pre-SMA

h vary categorically along positive-negative axes,

, in additio

These later valenceand surprise signals appeared in spatially distinct but temporally over-lapping neural signatures

human electroencephalogra-phy (EEG) studies, attempting to offer a temporal account of the corti-cal dynamics associated with RPE processing, did not find a systematicmonotonic response profile consistent with a single RPE representationbut instead offered evidence suggestive of separate representationsfor valence and surprise at the macroscopic level of responses recordedon the scalp.

suppresses N200 amplitude while concomitantly reducingevoked theta power (together with total theta power, to whichit contributes), but less so for induced theta power, with whichthe dopamine signal is relatively inconsistent in phase acrosstrials

N200amplitude relative to total theta power provides a more sensitiveindex of a reinforcement learning signal

the effect of probabil-ity on N200 amplitude can be predicted by total theta powerwhereas the effect of valence on total theta power can be predictedby N200 amplitude

when the effect of N200 ampli-tude is statistically controlled, the evidence for sensitivity of totaltheta power to outcome probability compared to outcome valenceincreases from “weak” to “strong.”

sensi-tivity of N200 amplitude to outcome probability is mostly drivenby its correlation with ongoing theta power, as statistically control-ling for this correlation reduced the effect of probability

very strongly support Prob-ability over Valence,pBICª.99

induced theta powerwas greater for the infrequent oddball condition compared to thefrequent oddball condition

Valence overProbability,pBICª.9

calculated the correlationbetween N200 amplitude and total, evoked, and induced thetapower.

Evoked theta power was determined directly from theaveraged ERP

induced portion of frontalmidline theta would be relatively more sensitive to outcomeprobability

ERP technique assumessuch information to be noise

oscillatory activity that is consistent in phase to an event acrosstrials contributes to the generation of an ERP to that event

sensitivity to important cognitive events in general ratherthan to errors in particular

sensitive to thevalenceof feedback

Intertrial phase coherence