method of moments allows modelling a RV via it moments - mean, variance etc. In reality we can say very little when the inputs are high dimensional as we will see no samples for almost all combinations.

the chance that some word will be emitted is proportional to the exponential of the inner product of its latent vector with the the current context. The proportional indicates that we are omitting the normalization term, which is a sum over all such exponents for all possible words.

but what does this model generate ?

this is a great insight.

PMI was considered as an information theoretic foundation to explain the success of TD-IDF with 1-hot encoded word representations. While how the latter works has been called mysterious, PMI is has clear probabilistic underpinnings.

Distributional Representation or Distributed Semantics would a better definitions, for word embeddings and it due to Co-occurrence. Because vector representations include 1-Hot encodings and TD-IDF methods

Both consider semantics but PMI is localised at the word level while Topic models aggregate semantics over collections of words.

Global vectors use the full co-occurrence matrix

Without reweighting and dimensionality reduction PMI doesn't scale to higher dimensions. (I.e. for large vocabulary).

consider a brief review

MLE are not in the bayesian tradition

what is this mixing time ?

This suggest this method should be good for low dimensional embeddings.

In higher dimensions, PMI faces the following challenges 1. Curse of Dimensionality: As the dimensionality increases, the data points become more sparse, i.e. data points are generally far apart, making co-occurrence statistics less reliable. PMI relies heavily on meaningful co-occurrence counts, which become sparse and noisy in high-dimensional spaces, leading to instability in the calculated PMI values.

clustered or isotropic

This section sheds light on the model - it is like the missing motivation. It helps by considering the model in the context of an experiment.

Not clear where this analysis comes from. Kernel suggest a gaussian process

The Bayesian tradition includes writing the equations of the model one on top of the other and explaining the parameters latent variables etc.

means non trivial

isotropy is motivated by need in integration.

holds empirically (

the Isotropy assumptions simplifies integration but seems to fit with experiment

This need further consideration.

There are different hypothesis for the existence of the "power law" and some of them my not fit with this isotropy.

since there is an inner product of v_twith c_t in R^d the word vectors must all have length d so they have the same length ?

the model is a random walk with t being the word and the RV being a random vector called a discourse vector of dimension d. This vector is a distributed representation of the semantics at word t.

what is the discourse - is it one hot encoded, is it orthogonal, is it sparse, is it disentangled, is it compositional ? What is a small change in a single dimension???

What is a slow/quick random walk ? How many steps are allowed and in how many dimensions?

A bigger point is that the meaning of words dont seem to follow this model ? Adjacent words in a sentence can have dramatically different semantics. I.E. despite Firth - only a few words are strongly related. The majority are not.

we need a second equation for the change in c_t - what is small here ?

Important point

but how often can we have big jumps without effecting the word bigram probabilities? Much of english is words like The and it which don't have semantic content.

also is the correlation short term of long term

err I recall that that model of options doen't have 2000 dimensions - typicaly just one latent variable - the value of the underlaying stock at some future date and the value is real number. But the semantics are spread over a high dim space and far from real valued. So AFAIKS this is stretching the analogy too far.

Doesn't that paper uses a modified model and not the same model? If this paper a prior or a posterior of that paper :-)

What assumptions are the prior encoding here?

how are the different dynamics expected to create better dynamics (which of gaussian or log linear aggregation more realistic?)

Topics are statistical aggregates over a window or some collective and so it makes sense to model them as slowly changing. However Topics are likely also subject to significant change at say pragraph and sentence boundries and the small changes are likely an artifact of the sampling rather then of the generating process. Also text can belong to multiple topics. Words on the other hand should not be modeled as having multiple meanings.

This comment highlights the gap in the theoretical understanding of why the PMI matrix exhibits an approximately low rank. It raises the question of whether this low-rank property could be formally proven or if it is purely an empirical observation.

this is the basic building block of the model

this section discusses how we can inject communication errors into the game. The form used creates a tradeoff between more signals (expressivity) and accuracy (expected no of correct messages)

if stems have a hierarchy via phonemic similarity and there is a rich morphology, learning words may be greatly simplified by a learning a few systematic rules. Idealy we need to learn a lexicon with just one base form and apply a group action to get all the forms. In this case we only need to memorize a small lexicon.

Here we may be considering a form of compositionality, where semantics are established by preferential selection of languages with related patterns of phonemes. What is missing is that these patterns are not grounded in the structural features if the state being expressed..

decoding the state is mandated on decoding the each signal in the sequence

this q is a re-normalised transposed of p<br /> this is what receivers learn in lewis games.

having this transpose solution means that for each agent there is randomly chosen a lewis equilibrium in place at the start of equilibrium and selection is acting the fitness function which is highest for a signaling system. As n increases the number of partial pooling equilibria grows much faster then the number of separating equilibria

the payoffs per interaction are symmetric but the aggregations are not!!!

this is a combination of a cooperative step and a competitive step

This is basically a probabilistic belief of the signaling system by the speaker and listener in a lewis signaling game.

When agents get the same rewards they are in a cooperative game. This is not expressed clearly.

And again in line with the Lewis Signaling Game

This notion of errors driving language evolution is central to this paper. And is also one that has gone under appreciated in later research.

When agent A can transmit the full state in one symbol the agents have coordinated on a completely separating equilibrium and Q has perfect information. Also RL algs will not have a reward signal to find a more refined equilibrium unless some reward shaping is done to incentivize agents to coordinate on agent's Q's state as well.

some problems with neural dialog modles

Task and Talk is Clearly based in the Lewis signaling game. However this setup seems to lead to entangled solutions at least for the minimal number of required signals.<br /> A needs to send one of three signals - the one he does not need. Then B needs to send one of four symbols for the state of one then the other (order is selected by symmetry breaking and learned thoroug reinfocement.) In this minimal version the signals are reused. If agent A has 8 signals he can respond fully without needed a second round.

so 64 states and an state space that both symmetrical and has three normal subgroups - so agent could learn to choose composable representations if one provides that right incentives to pick these over other equilibria.

so AFAIK this is what we expect in RL - a quick and efficent solution not the Oxford English dictionary. Or that solving a maze should be done with style.

so they just mean compsitional language

this suggests a dynamic system with issues of different phases like chaotic dynamics, periodicity etc unless the authors can demonstrate this always converges.

This is best understood through two english examples 1. The irregualr verb "to be" is used in many other constructions e.g. the future tense, so instability would have significant impact on learning. 1. there are many loan words in english which preserved thier forien morphology. This makes learning them as a group easier (they are irregualr but follow a template)in

I think that one on one language games fosters strong co-adaptation and eliminates the frictions that pressure the development of more efficent representations. Using multiple students can probably get faster evolution.

This is the main point of the paper.