Figure 1: Mathematics can illuminate the ways that ReLU-based neural networks shatter input space into countless polygonal regions, in each of which the model behaves like a linear map [2, 3, 4]. These decompositions create beautiful patterns. (Figure made with SplineCam [5]).

Likemany existing option discovery methods, we too make theassumption that all options are available everywhere, i.e.,∀s ∈ S, ∀ω ∈ Ω : s ∈ Iω . However, we show that ourapproach ends up relaxing this assumption, in effect, andprovides an elegant way to learn distinct initiation sets foroptions

The introduction of Transformers, such as GPT-4, took the field of natural language processing (NLP) and established benchmarks for several natural language tasks. Longer sequences have long been a thorn in the side of transformers as they significantly hamper their efficiency. This deficiency is where Mamba excels. Namely, mamba can process lengthy sequences more quickly than transformers and does so more simply due to its unique architecture.

We simplify prior deep sequence model architectures by combining the design of prior SSM architectures(Dao, Fu, Saab, et al. 2023) with the MLP block of Transformers into a single block, leading to a simple and homogenousarchitecture design (Mamba) incorporating selective state spaces

Every Model Learned by Gradient Descent Is Approximatelya Kernel Machine

the meaning of observational concepts is influenced by theoretical assumptions and presuppositions. For example, the concepts “mass” and “length” have different meanings in Newtonian and relativistic mechanics; so does the concept “temperature” in thermodynamics and statistical mechanics

This position has been adopted by Karl R. Popper, Rudolf Carnap and other leading figures in (broadly) empiricist philosophy of science. Many philosophers have argued that the relation between observation and theory is way more complex and that influences can actually run both ways (e.g., Duhem 1906 [1954]; Wittgenstein 1953 [2001]). The most lasting criticism, however, was delivered by Thomas S. Kuhn (1962 [1970]) in his book “The Structure of Scientific Revolutions”.

the rewards are divided through by the standard deviation of a rolling dis-counted sum of the reward

we find that they dramatically affect the performanceof PPO. To demonstrate this, we start by performing a full ablation study on the four optimizationsmentioned above

In brief, DCD takes a step from CD and attempts to provide a rationale bridge towards EMD for abetter sense of point distribution rather than being blinded by its nearest neighbour. Compared withEMD, it is not only more efficient but also stricter with local structures. A balanced distributionand good preservation of detailed structures are both important factors for the visual quality of thecompletion result.

Chamfer Distance between two point sets S1 and S2 is defined as

Using input transformation givesa 0.8% performance boost.

Q(λ) is a variant of Q-learning where eligibility traces are used to calculate the TD error. Asmentioned previously, the backwards view of traces is traditionally used

to provide a forecast Probability of Fire (PoF) on a given day within a 9 by9 km grid cell

To effectively fuse the multi-view information, we propose a geometrically-guided projective attentionmechanism. Instead of applying full attention to densely aggregate features across spaces and views,it projects the estimated 3D joint into 2D anchor points for different views, and then selectivelyfuses the multi-view local features near to these anchors to precisely refine the 3D joint location. wepropose to encode the camera rays into the multi-view feature representations via a novel RayConvoperation to integrate multi-view positional information into the projective attention. In this way, thestrong multi-view geometrical priors can be exploited by projective attention to obtain more accurate3D pose estimation.

MvP designs a novel geometricallyguided attention mechanism, called projective attention, to more precisely fuse thecross-view information for each joint.

Since the number of queries is larger than the actualnumber of people, we train an MLP-based classifier fβ (.) topredict a score for each query based on the appearance termto remove the “empty” ones.

We adopt a hierarchical query embed-ding scheme proposed in [36] to reduce the number of learn-able parameters.

Most closely related to our work, MvP [36]extends DETR for multi-view 3D human pose estimation.

We propose a variant of autoencoderswhich can work with two views of the data, while being explicitly trained to achieve all

his suggests that in scenarios with relativelylow amounts of data, Decision Transformer can outperform %BC by using all trajectories in thedataset to improve generalization, even if those trajectories are dissimilar from the return conditioningtarget. Our results indicate that Decision Transformer can be more effective than simply performingimitation learning on a subset of the dataset. On the tasks we considered, Decision Transformer eitheroutperforms or is competitive to %BC, without the confound of having to select the optimal subset

Formally, the learned reward function can be defined as ΦR : (G, V ) → R that maps a language goalG and a video snippet V to a scalar reward. An ideal ΦR should return a high reward if the behaviordepicted in the video faithfully follows the language description, and a low reward otherwise

Agents developed in popular RL benchmarks [ 119 , 146 ] often rely on meticulously crafted dense andtask-specific reward functions to guide random explorations. However, these rewards are hard or eveninfeasible to define for our diverse and open-ended tasks in MINEDOJO. To address this challenge, ourkey insight is to learn a dense, language-conditioned reward function from in-the-wild YouTubevideos and their transcripts. Therefore, we introduce MINECLIP, a contrastive video-languagemodel that learns to correlate video snippets and natural language descriptions (Fig. 4). MINECLIPis multi-task by design, as it is trained on open-vocabulary and diverse English transcripts

simpler tree search that relies upon thissingle neural network to evaluate positions and sample moves, without performing any Monte-Carlo rollouts.

new reinforcement learning algorithm thatincorporates lookahead search inside the training loop, resulting in rapid improvement and preciseand stable learning

Unlike earlier versions of AlphaGo, Zero only perceived the board's stones, rather than having some rare human-programmed edge cases to help recognize unusual Go board positions.

We use a reward function r(s) that is zero for allnon-terminal time steps t < T. The outcome zt = ± r(sT) is the termi-nal reward at the end of the game from the perspective of the currentplayer at time step t: +1 for winning and −1 for losing.

Using no searchat all, the RL policy network won 85% of games against Pachi

When played head-to-head, the RL policynetwork won more than 80% of games against the SL policy network.

The net-work predicted expert moves on a held out test set with an accuracy of57.0% using all input features, and 55.7% using only raw board posi-tion and move history as inputs, compared to the state-of-the-art fromother research groups of 44.4% at date of submission

Monte Carlo tree search in AlphaGo.

The policy network

the search attempts to prune sequences which are less relevant. In some cases, a play can lead to a very specific line of play which is significant, but which is overlooked when the tree is pruned, and this outcome is therefore "off the search radar"

it achieves better results than classical algorithms in games with a high branching factor

pure Monte Carlo tree search does not need an explicit evaluation function

Most contemporary implementations of Monte Carlo tree search are based on some variant of UCT

The main difficulty in selecting child nodes is maintaining some balance between the exploitation of deep variants after moves with high average win rate and the exploration of moves with few simulations.

An illustration of alpha–beta pruning. The grayed-out subtrees don't need to be explored (when moves are evaluated from left to right), since it is known that the group of subtrees as a whole yields the value of an equivalent subtree or worse, and as such cannot influence the final result. The max and min levels represent the turn of the player and the adversary, respectively.

For example, the chess computer Deep Blue (the first one to beat a reigning world champion, Garry Kasparov at that time) looked ahead at least 12 plies, then applied a heuristic evaluation function.[6]

Comparing Monte Carlo tree search searches, AlphaZero searches just 80,000 positions per second in chess and 40,000 in shogi, compared to 70 million for Stockfish and 35 million for Elmo. AlphaZero compensates for the lower number of evaluations by using its deep neural network to focus much more selectively on the most promising variation.[1]

AZ has hard-coded rules for setting search hyperparameters.

Overall, our results highlight the necessity of designing deep RL methods in a modular manner.

It turns out that the clipping mechanism is not necessary to achieve high performance—we findthat PPO-NOCLIP performs uniformly better than PPO-M, despite the latter employing the corePPO clipping mechanism.

We find that varying the use of code-level optimizations impactsperformance significantly more than varying whether the PPO or TRPO step is used.

The theory justifying TRPO actually suggests using a penalty instead of a constraint, i.e.,solving the unconstrained optimization problemmaximizeθˆEt[ πθ(at | st)πθold (at | st) ˆAt − β KL[πθold (· | st), πθ(· | st)]](5)for some coefficient β

PPO has been around for a relatively long time

At this point in the RLHF system, we have an initial language model that can be used to generate text and a preference model that takes in any text and assigns it a score of how well humans perceive it.

By comparing model outputs in head-to-head matchups, an Elo system can be used to generate a ranking of the models and outputs relative to each-other. These different methods of ranking are normalized into a scalar reward signal for training.

rankings are used to compare the outputs of multiple models and create a much better regularized dataset.

Human annotators are used to rank the generated text outputs from the LM.

The underlying goal is to get a model or system that takes in a sequence of text, and returns a scalar reward which should numerically represent the human preference.

OpenAI fine-tuned on human-generated text that was “preferable” and Anthropic generated their initial LM for RLHF by distilling an original LM on context clues for their “helpful, honest, and harmless” criteria

Through experimental methods focusing on PG methodsfor continuous control, we investigate problems with repro-ducibility in deep RL. We find that both intrinsic (e.g. randomseeds, environment properties) and extrinsic sources (e.g. hy-perparameters, codebases) of non-determinism can contributeto difficulties in reproducing baseline algorithms. Moreover,we find that highly varied results due to intrinsic sourcesbolster the need for using proper significance analysis. Wepropose several such methods and show their value on asubset of our experiments.

Can model-free reinforcement learning explain deontological moraljudgments?Alisabeth AyarsUniversity of Arizona, Dept. of Psychology, Tucson, AZ, USA

Briefly, these gridded datasets were built using an observed, satellite-derived measure of fire severity (Parks et al. 2014) and statistical models in which the probability of stand-replacing fire was modeled as a function of fuel, topography, climate, and weather. For a subset of ecoregions in our study area (Colorado Plateau, AZ–NM Mountains, and Apache Highlands), Parks et al. (2018b) also produced gridded datasets representing the probability of stand-replacing fire under extreme fire weather conditions.

GPT-4 System CardOpenAIMarch 23, 2023

denote dimensions0 through i − 1 of the state

τ<t to denote a trajectory from timesteps 0 through t − 1

lower-diagonal attention mask

Transformer architectures feature a “causal” attentionmask to ensure that predictions only depend on previous tokens in a sequence

We can use this directly as a goal-reaching method by conditioning on a desired final state sT .

If we set the predictedsequence length to be the action dimension, our approach corresponds exactly to the simplest form ofbehavior cloning with an autoregressive policy

Pθ (· | x)

Uniform discretization has the advantage that it retains information about Euclidean distance inthe original continuous space, which may be more reflective of the structure of a problem than thetraining data distribution.

modeling considerations are concerned lesswith architecture design and more with how to represent trajectory data – potentially consisting ofcontinuous states and actions – for processing by a discrete-token architecture

Concurrently with our work, Chen et al. (2021) also proposed an RL approach centered aroundsequence prediction, focusing on reward conditioning as opposed to the beam-search-based planningused by the Trajectory Transformer.

Modeling the states and actions jointly already provides a biastoward generating in-distribution actions, which avoids the need for explicit pessimism

model-based RL

stimateconditional distributions over actions

While such works demonstrate the importance of such models for representingmemory (Oh et al., 2016), they still rely on standard RL algorithmic advances to improve performance

The Trajectory Transformeris a substantially more reliable long-horizon predictor than conventional dynamics models

When decoded with a modified beam search procedure that biases trajectory samples according totheir cumulative reward,

K = 50 for Pong, K = 30 for others

loss = mean (( a_preds - a )**2)

We feed the last K timesteps into Decision Transformer, for a total of 3K tokens (onefor each modality: return-to-go, state, or action)

Does Decision Transformer perform behavior cloning on a subset of the data?

we use the GPT architecture [ 9 ], which modifies the transformer architecture with a causal self-attention mask to enable autoregressive generation, replacing the summation/softmax over the ntokens with only the previous tokens in the sequence (j ∈ [1, i]).

this allows the layer to assign “credit” by implicitly forming state-returnassociations via similarity of the query and key vectors (maximizing the dot product)

We then use a similar QA summarization framework as Wu et al. (2023) which produces QA dialogueon game mechanics

LATEX source code

all prior works require expert or human generated example trajectories

Wu et al. (2023) proposes a summary (Read) and reasoning (Reward) through a QA promptingframework with an open-source QA LLM Tafjord and Clark (2021). The framework demonstratesthe possibility of an using real-world human-written manuals to improve RL performance on populargames, despite limiting the interaction types to only “hit”. Our framework handles all 17 kinds ofinteractions available in the game. Moreover, our framework makes use of information on tech-treedependencies, and suggestions on desired policies extracted from the academic paper

Indicate their priority out of 5

The visual descriptor takes the last two gameplay screens as input, andoutputs their descriptions in language (dt, dt−1)

Answer to the final question qa is mapped to environment action usingsub-string matching.

Experimentally, we find that prompting the LLM with only the direct parents of a question greatlyreduces the context length, and helps LLM to focus on the most relevant contextual information

model-based methods like DreamerV2 Hafner et al. (2020);DreamerV3 Hafner et al. (2023)

We add the prompt “DO NOT answer inLaTeX.” to all of Qgame to prevent the LLM from outputting the list in LATEX format

in an environmentwhere control tasks are less required

zero-shot LLM-based (GPT-4) policy

,we promote and regulate in-context chain-of-thought reasoning in LLMs to solvecomplex games. The reasoning module is a directed acyclic graph (DAG), with questions as nodesand dependencies as edges. For example, the question “For each action, are the requirements met?"depends on the question “What are the top 5 actions?", creating an edge from the latter to the former.For each environment step, we traverse the DAG computing LLM answers for each node in thetopological order of the graph. The final node of the DAG is a question about the best action to takeand the LLM answer for the question is directly translated to environment action

decidingthe paragraphs that are relevant for playing the game

the environment is OOD to them.

In a nutshell, the CHT seems to disprove the scaling hypothesis.Or does it? In this work, we argue that foundation models might be exploiting a “loop hole” in the CHT4.Namely, what happens if the causal assumptions (which are required, by the CHT, for causal inference) arerepresented in observational data itself?

Plato. Republic: Allegory of the cave, 375 BC

Same Implication, Different Representations

we expect P (YX←1 = 1) = P (Y = 1) since intervening on X will notchange Y

the probability of a high number of Nobel laureates if the given chocolate consumption were to behigh.

We call these concepts ‘meta’ since they are one level above ‘regular’, simple SCM in thesense that they encode information about answering causal questions in another SCM.

More intriguingly, it does not matter where that L2 fact comes from since the formulation is independent ofwhether the model learns the fact and simply requires that the model knows about the fact. We state oursecond key insight as

Example 1 serves to show how the rather abstract definition of an SCM can be made tangible to communicatewhat we believe about our observed data and more so the underlying data generating process.

The Pearl’s Causal Hierarchy

It is clear how the observed correlation in this case corresponds to a directcausation according to

These models are castles in theair. They have no foundations whatsoever.” discrediting the models for lacking any identifiable notion tocausality.

Our explanation for this is that they are not only ‘stochastic parrots’ as already suggested by Benderet al. (2021) but sometimes also ‘causal parrots’ since they will also encounter correlations over causal factsduring training in their vast oceans of textual data.

parameterizedvariants of SCMs such as the neural ones presented in (Xia et al., 2021

y meta SCM

However, this conclusion is arguably nothing new, as most people wouldagree, and this is partly so because such obtained knowledge has been embedded as textual articles into en-cyclopedias such as Wikipedia, which are freely accessibl

IPEEE denotes the exogenousdistribution

to our real worldintuition since there is a bidirected edge X ↔ Y ∈ G(M2) with E3 being the underlying confounder

The following block paragraph serves as a summary

we take the former perspectivepro causal AI/ML. We argue that the questions around causality can fuel research also on questions of recentdebates such as how much ‘real’ progress towards AGI has been made since the advent of large scale models

counteringopinions start to speak out against causal AI/ML (Bishop, 2021)

Performing optimization in the latent space can more flexibly model underlying data distributions than mechanistic approaches in the original hypothesis space. However, extrapolative prediction in sparsely explored regions of the hypothesis space can be poor. In many scientific disciplines, hypothesis spaces can be vastly larger than what can be examined through experimentation. For instance, it is estimated that there are approximately 1060 molecules, whereas even the largest chemical libraries contain fewer than 1010 molecules12,159. Therefore, there is a pressing need for methods to efficiently search through and identify high-quality candidate solutions in these largely unexplored regions.

Petersen, B. K. et al. Deep symbolic regression: recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations (2020).

Reinforcement learning uses neural networks to generate a mathematical expression sequentially by adding mathematical symbols from a predefined vocabulary and using the learned policy to decide which notation symbol to be added next140. The mathematical formula is represented as a parse tree. The learned policy takes the parse tree as input to determine what leaf node to expand and what notation (from the vocabulary) to add

In chemistry, models such as simplified molecular-input line-entry system (SMILES)-VAE155 can transform SMILES strings, which are molecular notations of chemical structures in the form of a discrete series of symbols that computers can easily understand, into a differentiable latent space that can be optimized using Bayesian optimization techniques (Fig. 3c).

Neural operators are guaranteed to be discretization invariant, meaning that they can work on any discretization of inputs and converge to a limit upon mesh refinement. Once neural operators are trained, they can be evaluated at any resolution without the need for re-training. In contrast, the performance of standard neural networks can degrade when data resolution during deployment changes from model training.

Standard neural network models can be inadequate for scientific applications as they assume a fixed data discretization. This approach is unsuitable for many scientific datasets collected at varying resolutions and grids.

generating hypotheses

Applications of symbolic regression in physics use grammar VAEs150. These models represent discrete symbolic expressions as parse trees using context-free grammar and map the trees into a differentiable latent space. Bayesian optimization is then employed to optimize the latent space for symbolic laws while ensuring that the expressions are syntactically valid. In a related study, Brunton and colleagues151 introduced a method for differentiating symbolic rules by assigning trainable weights to predefined basis functions. Sparse regression was used to select a linear combination of the basis functions that accurately represented the dynamic system while maintaining compactness. Unlike equivariant neural networks, which use a predefined inductive bias to enforce symmetry, symmetry can be discovered as the characteristic behaviour of a domain. For instance, Liu and Tegmark152 described asymmetry as a smooth loss function and minimized the loss function to extract previously unknown symmetries. This approach was applied to uncover hidden symmetries in black-hole waveform datasets, revealing unexpected space–time structures that were historically challenging to find.

to address the difficulties that scientists care about, the development and evaluation of AI methods must be done in real-world scenarios, such as plausibly realizable synthesis paths in drug design217,218, and include well calibrated uncertainty estimators to assess the model’s reliability before transitioning it to real-world implementation

However, current transfer-learning schemes can be ad hoc, lack theoretical guidance213 and are vulnerable to shifts in underlying distributions214. Although preliminary attempts have addressed this challenge215,216, more exploration is needed to systematically measure transferability across domains and prevent negative transfer.

Another approach for using neural networks to solve mathematical problems is transforming a mathematical formula into a binary sequence of symbols. A neural network policy can then probabilistically and sequentially grow the sequence one binary character at a time6. By designing a reward that measures the ability to refute the conjecture, this approach can find a refutation to a mathematical conjecture without prior knowledge about the mathematical problem.

foresighted

AI methods have become invaluable when hypotheses involve complex objects such as molecules. For instance, in protein folding, AlphaFold210 can predict the 3D atom coordinates of proteins from amino acid sequences with atomic accuracy, even for proteins whose structure is unlike any of the proteins in the training dataset.

Transformer architectures

Such pretrained models96,97,98 with a broad understanding of a scientific domain are general-purpose predictors that can be adapted for various tasks, thereby improving label efficiency and surpassing purely supervised methods8.

In the analysis of scientific images, objects do not change when translated in the image, meaning that image segmentation masks are translationally equivariant as they change equivalently when input pixels are translated.

Symmetry is a widely studied concept in geometry69. It can be described in terms of invariance and equivariance (Box 1) to represent the behaviour of a mathematical function, such as a neural feature encoder, under a group of transformations, such as the SE(3) group in rigid body dynamics.

Another strategy for data labelling leverages surrogate models trained on manually labelled data to annotate unlabelled samples and uses these predicted pseudo-labels to supervise downstream predictive models.

To identify rare events for future scientific enquiry, deep-learning methods18 replace pre-programmed hardware event triggers with algorithms that search for outlying signals to detect unforeseen or rare phenomena

Recent findings demonstrate the potential for unsupervised language AI models to capture complex scientific concepts15, such as the periodic table, and predict applications of functional materials years before their discovery, suggesting that latent knowledge regarding future discoveries may be embedded in past publications.

inductive biases (Box 1), which are assumptions representing structure, symmetry, constraints and prior knowledge as compact mathematical statements. However, applying these laws can lead to equations that are too complex for humans to solve, even with traditional numerical methods9. An emerging approach is incorporating scientific knowledge into AI models by including information about fundamental equations, such as the laws of physics or principles of molecular structure and binding in protein folding. Such inductive biases can enhance AI models by reducing the number of training examples needed to achieve the same level of accuracy10 and scaling analyses to a vast space of unexplored scientific hypotheses11.

Box 1 Glossary

and coupled with new algorithms

geometric deep learning (Box 1) has proved to be helpful in integrating scientific knowledge, presented as compact mathematical statements of physical relationships, prior distributions, constraints and other complex descriptors, such as the geometry of atoms in molecules

Causal Deep Learning Authors:Jeroen Berrevoets, Krzysztof Kacprzyk, Zhaozhi Qian, Mihaela van der Schaar

Bias-variance trade-off

To avoid such bias, a fundamental aspect in the research design of studies of causalinference is the identification strategy: a clear definition of the sources of variation in the datathat can be used to estimate the causal effect of interest.

Matching: This approach seeks to replicate a balanced experimental design usingobservational data by finding close matches between pairs or groups of units andseparating out the ones that received a specified treatment from those that did not, thusdefining the control groups.

Quantitatively, SPRING with GPT-4 outperforms all state-of-the-art RLbaselines, trained for 1M steps, without any training.

It should not be used as a primary decision-making tool, but instead as a complement to other methods of determining the source of a piece of text.

Training language models to follow instructionswith human feedback

LaMDA: Language Models for Dialog Application

“Rung 1.5” Pearl’s ladder of causation [1, 10] ranks structures in a similar way as we do, i.e., increasing amodel’s causal knowledge will yield a higher place upon his ladder. Like Pearl, we have three different levelsin our scale. However, they do not correspond one-to-one.

We find empirically that for best-of-n (BoN) sampling

d

RL

LaMDA pre-training as a language model.

Safety does not seem to benefit much from model scaling without fine-tuning.

How LaMDA handles groundedness through interactions with an external information retrieval system