Figure

Both MindEye2 and MindEye1 also map brain activityto the latent space of Stable Diffusion’s (Rombach et al.,2022) variational autoencoder (VAE) to obtain blurry recon-structions that lack high-level semantic content but performwell on low-level image metrics (e.g., color, texture, spatialposition), which get combined with the semantically richoutputs from the diffusion prior to return reconstructionsthat perform well across perceptual and semantic features.

Other ultrasound methodologies, employing higherfrequencies, are more severely impacted by signal attenuation,both in the skull and in soft tissue

Time-of-flight tomography uses a short-wavelength approx-imation, basing its analysis on the simplified physics of raytheory in which the effects of transmission through aheterogeneous medium are represented by a simplechange in travel time. For a finite wavelength, wavetransmitted through a medium that is heterogeneous onmany scales, such delay times are only sensitive to theproperties of the medium averaged over the dimensions ofthe first Fresnel zone 11 .

Upon receiving observations y, solving a Bayesian inverse problem involves sampling theconditional distribution of x given y (Tarantola, 2005)

For the transmission inversion we use a model based large scale minimization at a particular angular fre-quency ωj based on an L 2 minimization.

At test time, Gr is the sameas F(P ∗)

Power Doppler intensity for each implant scenario over N=15 acquisitions. Stars indicate statistical differences (paired sampledt-test) between each scenario compared to the No implant scenario (* : p < 0.05 , **: p < 0.01 , ***: p < 0.001) (F) SNR attenuationfor each implant scenario as function of the depth (N=15 acquisitions).(which was not certified by peer review) is the author/funder. All rights reserved. No reuse allowed without permission.The copyright holder for this preprintthis version posted June 15, 2023.;https://doi.org/10.1101/2023.06.14.544094doi:bioRxiv preprint

A good first guess is one that makes your data decorrelated and whitened.

given legal tractability

A clinical mind enhancement research group could conduct healthy-volunteer clinical trials on effects and safety of MEIs to gather sufficient data to push almost deployable MEIs to deployable level.

you lose all the value of using AI if you won’t use unaligned systems

having an AI that is competent or intelligent

And if what I want is not measurable

I might think it's really important for that new system to not mess up. I think this is a really important problem, but it's worth separating it from the problem of building AI that's trying to do what we want it to do.

ou could stay on a diet, spend the optimal amountof time learning which activities will achieve your goals, and then follow through on anoptimal plan, no matter how tedious it was to execute.

biological cognitiveenhancement (Sandberg 2011)

then “slightly increase their degree of influence-seeking behavior” would be just as good a modification as “slightly improve their conception of the goal.”

in the broader landscape of “possible cognitive policies.”

Law enforcement will drive down complaints and increase reported sense of security. Eventually this will be driven by creating a false sense of security, hiding information about law enforcement failures, suppressing complaints, and coercing and manipulating citizens.

Investors will “own” shares of increasingly profitable corporations, and will sometimes try to use their profits to affect the world. Eventually instead of actually having an impact they will be surrounded by advisors who manipulate them into thinking they’ve had an impact.

We will try to harness this power by constructing proxies for what we care about, but over time those proxies will come apart:

-that can’t be done by trial and error

In particular, this is the problem of getting your AI to try to do the right thing, not the problem of figuring out which thing is right.

As an example to illustrate the base/mesa distinction in a different domain, and the possibility of misalignment between the base and mesa- objectives, consider biological evolution.

Arguably, any possible system has a behavioral objective—including bricks and bottle caps. However, for non-optimizers, the appropriate behavioral objective might just be “1 if the actions taken are those that are in fact taken by the system and 0 otherwise,”[6] and it is thus neither interesting nor useful to know that the system is acting to optimize this objective. For example, the behavioral objective “optimized” by a bottle cap is the objective of behaving like a bottle cap.[7] However, if the system is an optimizer, then it is more likely that it will have a meaningful behavioral objective. That is, to the degree that a mesa-optimizer’s output is systematically selected to optimize its mesa-objective, its behavior may look more like coherent attempts to move the world in a particular direction.[8]

When a base optimizer generates a mesa-optimizer, safety properties of the base optimizer's objective may not transfer to the mesa-optimizer.

Whether a system is an optimizer is a property of its internal structure—what algorithm it is physically implementing—and not a property of its input-output behavior. Importantly, the fact that a system’s behavior results in some objective being maximized does not make the system an optimizer. For example, a bottle cap causes water to be held inside the bottle, but it is not optimizing for that outcome since it is not running any sort of optimization algorithm.(1) Rather, bottle caps have been optimized to keep water in place. The optimizer in this situation is the human that designed the bottle cap by searching through the space of possible tools for one to successfully hold water in a bottle. Similarly, image-classifying neural networks are optimized to achieve low error in their classifications, but are not, in general, themselves performing optimization.

The VR devices of 2025 (or potentially even next year) will be qualitatively different from what we have today:

Some body language.

Why is it so much better?

,

The sense of being transported to another place, being bodily present with others, is important for committing with your whole self to the people you're with and the thing you're doing together.

,

Most people don't realize this (and that's a large part of the problem), but when you have an audio call without audio isolation (without wearing headphones), the system silences you when other people are talking in order to prevent audio feedback. This disrupts natural turntaking negotiation and makes very lively conversation impossible.

Costs about 100$

"human-level AI

useful

For a model using a point estimate of θ (such as anMLE or MAP), rather than a distribution over θ

Specifically, is itnecessary to explicitly ground high-level actions into therepresentation of the environment? Or is it sufficient tosimply inform the learner about the abstract structure ofpolicies, without ever specifying how high-level behaviorsshould make use of primitive percepts or actions?

For the example inFigure 2, f might be an image rating model (Socheret al., 2014) that outputs a scalar judgment y of howwell an image x matches a caption w

While Transformers are good at modelinglong-range global context, they are less capable to extract fine-grained local feature patterns.

The 10th and 90th percentiles are estimated viabootstrapping data (80% of the dataset is sampled 100 times)

The relation between data compression (via prediction)and intelligence

This evaluation is perhaps slightly unfair, as we have not performed the obvious step of training the model on a much largerdataset of executions. This is an interesting direction for future work.

100% 12%

, indicating general variability in model performance

This method of checking correctness forced us to filter the MathQA dataset to keeponly those problems for which the code evaluates to the declared numerical answer, resulting in us removing 45% ofproblems.

How to characterize "task type," estimating how "difficult" and expensive a task is to “try” or “watch” once.

Trained on a task where each "try" took days, weeks, or months of intensive "thinking."

and that safety research that advances capabilities is not safely scalable to a broader research community,

In the generalisation-based approach, the way to create superhuman CEOsis to use other data-rich tasks (which may be very different from the taskswe actually want an AI CEO to do) to train AIs to develop a range of usefulcognitive skills.

reinforcementlearning (which often requires billions of training steps even for much simplertasks).

if this is possible

BCI as Existential Security Factor

Supposing our estimates are out by an order of magnitude

Having a tiny fraction of humanity free would not have powerful longterm positive effects for humanity overall, as it would be vanishingly unlikely that small, unthreatening populations could liberate humanity and recover a more positive future.

Further evidence can be seen in the steady expansion in government surveillance by even democratic governments.

If there were only a single actor controlling the development of all BCIs, it would likely be an easier situation to control and regulate. Having multiple state actors, each of whom can decide the level of coercion they wish their BCI to be built with, is a much more complex scenario

But BCI based surveillance would have no such flaw.

This allows the system to interactively perform actions and probecausal relationships

The semantic layer is not contained in the data

And the animal-level part might not be themost crucial part in this equation.

Another example of what can be done in terms of hybridization is how we could use agenetic algorithm to evolve a formal grammar for a particular language, using a large lan-guage model like GPT-3 as an oracle to compute a fitness by comparing the kind of sentencesthat the current grammar candidates are producing and assessing how statistically realisticthey sound.

its focus on handling only solutionspaces that happen to allow for a gradient computation.

However, to avoid circular dependencies in the graph, it re-quires that agent k choose its action before j, and thereforek can influence j but j cannot influence k.

As thesame as the reward case, DT takes the summation of the state-feature over a trajectory as an input

n this case, CDT is unnecessary because if Φ learns sufficient featuresof s, matching their means, i.e. moments, through DT is enough to match any distribution to anarbitrary precision (Wainwright & Jordan, 2008; Li et al., 2015)

We split our static dataset 50:50, andtrained separate PMs on each half, which we refer to as train PMs and test PMs

This means that our helpfulness dataset goes ‘up’ in desirability during the conversation, while our harmlessnessdataset goes ‘down’ in desirability. We chose the latter to thoroughly explore bad behavior, but it is likely not idealfor teaching good behavior. We believe this difference in our data distributions creates subtle problems for RLHF, andsuggest that others who want to use RLHF to train safer models consider the analysis in Section 4.4.

We hypothesize that this might be to a domain gapbetween the natural images that CLIP has been trained on andthe simulated images from CALVIN.

The task parameters referto the variables that can be collected by the system andthat describe a situation, such as positions of objects inthe environment. The task parameters can be fixed duringan execution trial or they can vary while the motion isexecuted

C∗ ∈[1,1+ ̃O(1/N)]

For analysis purposes, we consider a BC algorithm that matches the empiricalbehavior policy on states in the offline dataset, and takes uniform random actions outside the supportof the dataset.

This condition holds in sparse-reward tasks

Although ingeneral, the optimal policy does vary with the horizon, in environments where it is possible to stay atthe goal, a Markovian policy that reaches, then stays at the goal can be near-optimal

it should be possible to extend it toachieve any goal which does not change due to the simulated char-acter’s behaviour

The reward structure is similarto that of Bergamin et al. [2019]

Again,PPO’s reward definition being a multiplication of terms likely lessensthe effect of any one loss component having more influence on themotion imitation than another.

before being converted toquaternions

Second, that the transition function of these two individ-ual character states is partially separable; generally the kinematiccharacter’s behaviour does not change because of the simulated

benefits from models leveraging temporal abstraction [58, 66]. Inspired by these works, weextend the RNN model from BC-RNN into a multi-layered variant (TieredRNN). This new variantintegrates multiple layers operating at varying timesteps to better streamline information flow fromtimesteps early on in a given sequence to timesteps much further downstream, which can be usefulfor our long-horizon MM tasks.

(c)

t ∼ ψ(zt|st−K:t,aHt−1)

compute energies

we can define this simple environment to bein N dimensions

he observation suggests that the semantic structure of wordand object categories share the common representation in human cognitive processing.

The Stroop effect known in human cognitive scienceliterature is one of such biases. When human participants observe a mismatched word, where theword “red” with the “blue” color, and are asked to answer the “color” of the word, their responseto the word can be slower than when they see the word “red” with the “red” color (Stroop, 1935).

ir recognitio

They are somehow unable to grasp that someone could categorically attack ALL agents on the basis that they do not exist, and that it is potentially harmful to believe that they do.

But remember, although agent programs tend to share a set of deficiencies, it is your psychology that really makes a program into an agent; a very similar program with identical capabilities would not be an agent if you take responsibility for understanding and editing what it does. An agent is a way of using a program, in which you have ceded your autonomy. Agents only exist in your imagination. I am talking about YOU, not the computer.

we leverage the learned reward function to scan goal candidates

Bahdanau et al. [4] and Fu et al. [29] also learn a reward function butrequire extensive expert knowledge (expert dataset and known environment dynamics respectively),whereas our agent uses experience generated by its own exploration.

Language can thus be used to generate out-of-distributions goals by leveragingcompositionality to imagine new goals from known ones

Children do so by leveraging the compositionality of language as atool to imagine descriptions of outcomes they never experienced before, targetingthem as goals during play.

Existing beam-search [17, 48, 53] and backtracking solu-tions [24, 29] are infeasible due to the larger action and statespaces, long horizon, and inability to undo certain actions.

Thisinference is more realistic than simple object class pre-diction, where localization is treated as a solved problem

This time the model got the “turn right” thing all right. We still have the disambiguation problem, there are two tables involved here. I also disagree with “turning around” as the approapriate description for the navigation part after picking up the box (more like “make a right and then make a left”). The captioner may be picking up correlations in the language directly with bad conditoning on the frames, as picking up an object oftern is followed by turning around, as seen in the previous examples.

That is, use the same framework and algorithm, but replace the sequence of words corresponding to goal sentences with goals as sequences of final state-action tuples. (This would be the time-extended version of simple goal-conditioned RL algos with goals as final states. In this framework, no relabelling is needed).

We conclude that the synthetic data is especiallyimportant for generalization to unseen environments.

We conjecture thatdomain-specific language pretraining is important for theALFRED benchmark.

hen evaluated on human instruc-tions

We note that performance of the modelwith16input frames is close to the performance of the fullepisode memory agent, which can be explained the averagetask length of50timesteps

it is desirable for the agents tobe able to handle raw sensory inputs, such as images orvideos.

his setting is harder as it removes theability for agents to explore the environment and collect additional feedback.

evaluate the likelihood under our discretized model, we treat each bin as a uniform distribution overits specified range; by construction, the model assigns zero probability outside of this range.

To better isolate the source of the Transformer’s improved accuracy over standard single-step models,we also evaluate a Markovian variant of our same architecture. This ablation has a truncated contextwindow that prevents it from attending to more than one timestep in the past. We find that this modelperforms similarly to the trajectory Transformer on fully-observed environments, suggesting thatarchitecture differences and increased expressivity from the autoregressive state discretization play alarge role in the trajectory Transformer’s long-horizon accuracy.

log likelihood

, the conditional probabilities of the past given the future are stillwell-defined, allowing us to condition samples not only on the preceding states, actions, and rewardsthat have already been observed, but also any future context that we wish to occur. If the futurecontext is a state at the end of a trajectory, we decode trajectories with probabilities of the form

While this choice may seem inefficient, it allows us tomodel the distribution over trajectories with more expressivity, without simplifying assumptions suchas Gaussian transitions.

modeling considerations areconcerned less with architecture design and more with how to represent trajectory data – consistingof continuous states and actions – for processing by a discrete-token architecture

nd we no longer require ensembles orother epistemic uncertainty estimators, as is common in prior work on model-basedRL

relaxing the conditioning onX

Relaxing the conditioning onXgives us the limiting spectral distributionργMPρψMP, becauseμXis almost surelyρψMP, which terminates the proof.

intra-class correlations and strong inter-class

Observe that if the average test error over theversion spaceε(VS)is low, thenmostmembers of the version space musthave low test error (either intuitively, or by Markov’s inequality).

In words, typical solutions may bedescribed in a number of bits close to the typical information content

While the numberof steps until validation accuracy>99% grows quickly as dataset size decreases, the number of stepsuntil the train accuracy first reaches 99% generally trends down as dataset size decreases and stays inthe range of103-104optimization steps.

Articulatory suppression and interference have been used as methods to investigate the contribution of verbal, spatial, and/or motor codes in encoding, rehearsing, and recalling series of danceitems from working memory (e.g., Jean et al. 2001; Rossi-Arnaud, Cortese, & Cestari,2004). Suppression tasks involve performing a task at the same time as observing the to-be-remembered material, and are used to disrupt encoding of the to-be-remembered (TBR) material.

Leveraging the extracted kinematic beats, we define the dance unit in this work. As illustrated inFigure 2(b), a dance unit is a temporally standardized short snippet, consisting of a fixed numberof poses, whose kinematic beats are normalized to several specified beat times with a constant beatinterval. A

We thusadopt an auxiliary reconstruction task on the latent movement sequences to facilitate training:Lmrecon=E[∥∥{ˆzimov}−{zimov}∥∥1].

While the generation diversity is ensured by the adversariallearning schema

So our proposed BC, whichfinds matched kinematic beat for each music beat, is a bet-ter metric

Position Variance (PVar)evaluates the diversity of thegenerated dance

For each node, 3 individual fully-connected layersembed the joint node features into Q, K and V vectors andthen we perform 24 individual LLA modules on Q, K and Vof the nodes.

N ∼(ti,σ2)

Attφ(qi,K,V) =ψ(qi ̃φ(KT))φ(V)

we adopt an adversarial trainingarchitecture

During training thestart pose is simply selected as the first frame of the dancesequence and the right shifted outputs are just taken fromground-truth with mask for future time steps.

In implementation we use a 4-knot CHS to fit a mo-tion curve between two key poses, as an optimal trade-off between fitting precision and representation simplicity

position androtation

The mo-tion capture methods are more accurate but still suffer fromnoise and misalignment to music.

Ow-ing to the LLA module, the training is more stable and thedegradation to averaging actions is avoided

Different from sentences in NLP that have long-term cor-relations, human poses and motions only have local cor-relations in a short time interval.

similarity between them.

Thisposes a unique challenge for cross-modal learning betweenmusic and motion

f a model instead develops amore general representation, it should be able to apply learned functions also to longer input strings.Its performance on longer strings may drop for other, practical, reasons, but this drop should bemore smooth than for a model that has not learned a general-purpose representation at all.

Note that, in ourcase, this ability does not require productivity in generating output strings, since the correct outputsequences are not distributionally different from those in the training data (in some cases, theymay even be exactly the same)

When the percentage of exceptions becomes too low, on the other hand, all models have difficultiesmemorising them at all:

One potential explanation for this score discrepancy is that, due to the slightly different distri-bution of examples in the systematicity data set, the models learn a different solution than before.

As observed by many others before us, insight in the compositional skills of neural networks is noteasily acquired by studying models trained on natural language directly. While it is generally agreedupon that compositional skills are required to appropriately model natural language, successfullymodelling natural data requires far more than understanding compositional structures. As a conse-quence, a negative result may stem not from a model’s incapability to model compositionality, butrather from the lack of signal in the data that should induce compositional behaviour. A positiveresult, on the other hand, cannot necessarily be explained as successful compositional learning, sinceit is difficult to establish that a good performance cannot be reached through heuristics and memo-risation.

f an expression is altered byreplacing one of its constituents with another constituent with the same meaning (a synonym),this does not affect the meaning of the expression (Pagin, 2003

As a consequence, the interpretation of theprinciple of compositionality depends on the type of constraints that are put on the semantic andsyntactic theories involved.

Fodor and Pylyshyn (1988) contrast systematicity with storing all sentences in an atomic way,in a dictionary-like mapping from sentences to meanings. Someone who entertains such a dictionarywould not be able to understand new sentences, even if these sentences were similar to the onesoccurring in their dictionary. Since humans are able to infer meanings for sentences they have neverheard before, they must use some systematic process to construct these meanings from the onesthey internalised before

This ability concerns the recombination of known parts and rules: anyone who understands a numberof complex expressions also understands other complex expressions that can be built up from theconstituents and syntactical rules employed in the familiar expressions. To use a classic examplefrom Szab ́o (2012): someone who understands ‘brown dog’ and ‘black cat’ also understands ‘browncat’.

In the substitutivity test, we evaluate how robust models aretowards the introduction of synonyms, and, more specifically, in which cases words are consideredsynonymous.

he binary nature of many preference tests means that responsesare relatively information-poor and makes it harder to verify that two conditions are statistically different.

uTWv

pθ(z)

whilereverse KL-divergence is not a viable objective function,

dditionally, it has been suggested thatreverse KL-divergence,DKL(pg||pd), is a better measurefor training generative models than normal KL-divergence,DKL(pd||pg), since it minimisesEx∼pg[lnpd(x)][92]

2-Stage VAEsBy interpreting VAEs as regularised autoencoders, it is nat-ural to apply less strict regularisation to the latent spaceduring training then subsequently train a density estima-tor on this space, thus obtaining a more complex prior[53]. Vector Quantized-Variational Autoencoders (VQ-VAE)[170], [215] achieve this by training an autoencoder witha discrete latent space, then training a powerful autore-gressive model (see Section 5) on the latent space.

2 stage VAEs that firstmap data to latents of dimensionrthen use a second VAE tocorrect the learned density can better capture the data [35]

(14)

implicit energy models

This is made worse by thefinite nature of the sampling process, meaning that samplescan be arbitrarily far away from the model’s distribution[57].

the gradient of the negative log-likelihoodlossL(θ) =Ex∼pd[−lnpθ(x)]has been shown to approxi-mately demonstrate the following property [21], [193]

TABLE 1

training speed is assessed based on reportedtotal training times

Choosing what to optimise for has implica-tions for sample quality, with direct likelihood optimisationoften leading to worse sample quality than alternatives.

constant mem-ory

especially in the most difficult long horizon setting.

This process is scalable because pairing happens after-the-fact, making it straightforward to parallelize via crowdsourc-ing.

Training an agent for145such social interactions most likely requires drasti-146cally different methods – e.g. different architectural147biases – than classical object-manipulation training

To117enable the design and study of complex social sce-118narios in reasonable computational time

rather than lan-059guage based social interactions

(8)

Consequently, maximizing the log-likelihood of the continuous model on uniformly dequantizeddata cannot lead to the continuous model degenerately collapsing onto the discrete data, because itsobjective is bounded above by the log-likelihood of a discrete model.

In our datasets Fig. 2c, we find empirically thatfor the same amount of collection time, play indeed covers 4.2 times more regions of the availableinteraction space than 18 tasks worth of expert demonstration data, and 14.4 times more regions thanrandom exploration.

lay data ischeap: Unlike expert demonstrations (Fig. 5), playrequires no task segmenting, labeling, or resetting to an initial state, meaning it can be collectedquickly in large quantities.

We additionally find that play-supervised models,unlike their expert-trained counterparts, are more robust to perturbations and ex-hibit retrying-till-success behaviors.

Intuitively, this is equivalentto taking the average of demonstrated actions at each specificstate.

TextWorldaddresses this discrepancy by providing programmatic and aligned linguistic signals during agentexploration.

Informally, all else being equal, discontinuousrepresentations should in many cases be “harder” to approx-imate by neural networks than continuous ones. Theoreti-cal results suggest that functions that are smoother [34] orhave stronger continuity properties such as in the modulusof continuity [33, 10] have lower approximation error for agiven number of neurons.

n the skill learning phase, LGB relies on an innate semantic representation that characterizes spatial relations between objects in the scene using predicates known to be used by pre-verbal infants [Mandler, 2012].

Although the policy over-generalizes, the reward function can still identify whether plants have grown or haven’t.

more aligned data autonomously.

This would require a means for representing meaning from experience—a situation model—and a mechanism that allows information to be extracted from sentences and mapped onto the situation model that has been derived from experience, thus enriching that representation

What can and should the user be doing while the AI agent is taking its turn to increase engagement?

We employ a turn-based frameworkbecause it is a common way of organizing co-creative interactions [3,12,13] and because it suitsevolutionary and reinforcement-learning approaches that require discrete steps [2, 7, 8, 14].

As can be seen in Figure 3 (left), the trainingperformance was sensitive to the weight scaleσ, despitethe fact that a weight normalisation scheme was beingused.

natural motions

For quantitative evaluation, we computed the meansquared error between the generated motion and motioncapture on a left-out test set, for fingertip positions and jointangles

In total, we used approximately120 minutes of data

2)autoregression reduces the amount of fast moments, making thevelocity histogram more similar to the ground truth

“In which video...”: (Q1) “...are the character’s movements mosthuman-like?” (Q2) “...do the character’s movements most reflectwhat the character says?” (Q3) “...do the character’s movementsmost help to understand what the character says?” (Q4) “...are thecharacter’s voice and movement more in sync?”

Hence, after five epochs of training with autoregression,our model has full teacher forcing: it always receives the ground-truth poses for autoregression. This procedure greatly helps withlearning a model that properly integrates non-autoregressive input.

This pretraining helps thenetwork learn to extract useful features from the speech input, anability which is not lost during further training.

we pass a sliding windowspanning 0.5 s (10 frames) of past speech and 1 s (20 frames) offuture speech features over the encoded feature vectors.

feature vector𝑉𝑠was made distinct from all other encodings, bysetting all elements equal to−15

three different domains: U.S. presidents,dog breeds, and U.S. national parks. We use mul-tiple domains to include diversity in our tasks,choosing domains that have a multitude of entitiesto which a single question could be applied

We assume each task has an associ-ated metricμj(Dj,fθ)∈R, which is used tocompute the model performance for taskτjonDjfor the model represented byfθ.

ecological pre-training

BART waspre-trained using a denoising objective and a variety of different noising functions. It has obtainedstate-of-the-art results on a diverse set of generation tasks and outperforms comparably-sized T5models [32].

We believe these properties provide good motivationfor continuing to scale larger end-to-end imitation archi-tectures over larger play datasets as a practical strategy fortask-agnostic control.

Multipleavenues, including understanding more deeply the mechanisms of creative, knowledge-rich thought, or transferring knowledge from large, real world datasets, may offer a wayforward.

To go beyond competence within somewhat stereo-typed scenarios toward interactive agents that can actively acquire and creatively recombineknowledge to cope with new challenges may require as yet unknown methods for knowl-edge representation and credit assignment, or, failing this, larger scales of data.

To record sufficientlydiverse behaviour, we have “gamified” human-human interaction via the instrument of lan-guage games.

Winograd envisioned computers that are not “tyrants,” but rather ma-chines that understand and assist us interactively, and it is this view that ultimately led himto advocate convergence between artificial intelligence and human-computer interaction(Winograd, 2006)

Generally, these results give us con-fidence that we could continue to improve the performance of the agents straightforwardlyby increasing the dataset size.