GLUT DOUBLE: double buffered window GLUT RGB: RGB colors GLUT_DEPTH: Cache z-values

Params passed in here

https://stackoverflow.com/questions/19719055/what-are-the-differences-between-glu-glew-glut-qt-sdl-opengl-and-webgl

For full list of comparisons

tmain is a windows only thing, when I tried to understand it from a SO answer my head hurt

Compiler directive

Again, this might relate back to the idea that there is some connection, but not a cause and effect relationship.

I am glad to hear this, but again have trouble accepting the viewpoint from Bozell, who is a staunch Trump supporter and writes in a way that stray from academic objectivity.

This article demonstrates the type of scholarship produced by Brent Bozell. He degrades his opponents by calling them "survivors of failed abortions" and then proceeds to question the validity of the "liberal news media." His statements and input and expertise may need to be taken with a grain of salt.

It is great that they relied on the studies on this subject! I wonder what the effect will be?

This meeting was very similar, but included scientific research from leading professors, and lacked sensationalized video game anecdote. In this meeting, Biden was looking for facts and evidence, and did not result to trying to persuade the audience, which may have been the goal with the current White House and their Youtube publication.

This video was a little hard to watch, but only shows anecdotal evidence. This process might be flawed, since it is based on very violent scenes, not on the scientific data that gets to the root of the question.

From my research, it appears as though many of the studies show that violent video games may be a factor in aggression, but they are not the sole cause. This statement seems pretty fair, since it is claiming a correlation, not a causation.

I think that this is interesting, since the science points to the alternative. I wonder how he came to this conclusion.

After this comment, the NRA then had an emergency meeting with Trump, with which they announced that he might be retracting these type of statements. The New York Times covered this development.

It is great that the video gamer publishers went right to the source. I am wondering if this is the first time in the ongoing debate when a publisher met with a president. I am glad to see that they are communicating, instead of having law makers create legislation based on one view point.

Here we go again, This is an issue that has been constantly brought up and constantly refuted with objective evidence.

I do agree with this as there has been no good way of teaching us in a way that we retain it, the one way that I, personally think works for me is when teachers have Kahoots or similar games made up for us to test our knowledge prior to the test. I find that playing a game makes me for some reason remember the information.

the game is safer than it ever has been

https://via.hypothes.is/https://www.bestadvisor.com/handheld-game-consoles

It is interesting to see how he came up from the D-League and became an NBA all-star. He had to start as a nobody and turn himself into a somebody. His rookie year he didn't get to play or score a lot but then he turned himself into a scoring machine. It was like he couldn't be guarded and it was all because of how hard he pushed himself.

I really like this because it shows the stats of the teams and how to break down who has the best chance to win each game. It gives you the best chance to get a perfect bracket or very close.

very true!! observers know more, as they are watching or observing every side of the story.

This is a profoundly important statement. How do we teach students to interact and therefore learn on a deeper level within the classroom?

<font size="+3">FOUR PINNOCHIOS</font> The states committed to funding 50 percent of Gateway costs in 2015. Read the full fact check.

This statement has been analyzed by a member of the [International Fact Checking Network](https://www.poynter.org/channels/fact-checking). This annotation is provided by Hypothesis as a public service.

The states committed to funding 50 percent of Gateway costs in 2015. Read the full fact check.

This statement has been analyzed by a member of the [International Fact Checking Network](https://www.poynter.org/channels/fact-checking). This annotation is provided by Hypothesis as a public service.

So, th ere were lots of the difference between regular comics with the underground comix. And we can observe that underground comix have no limits on their contents and their freedom of expression but the regular comics might appeal to mass audience.

This is definitely true for video games I think, and it starts early -- the best example I think would be Princess Peach and Mario. While not the typical example, for decades Peach has always been in need of saving throughout each Mario game.

46's mom comes to her game for the first time, the girls react as if shes weird (except 00) (123)

00 talks for the first time ever before a game quoting 46's song she says "... it's catchy" before she runs off to vomit, shows 46 is creating change within one of the girls at least. (65)

Unlike gun violence statistics, which largely don't exist thanks to NRA lobbying and dismal legislating, there's plenty of statistical information on the impact of video games on the human psyche. I just don't think they're going to find the answers they're hoping to find. Then again, this administration is fact-resistant at best.

uh, you're the parent. you LET him.

Exactly, so why waste time and resources on this?

Does is sound like the ESA is trying to skirt the issue though?

That would be nice.

As an educator, I have to say that video games should be low on the list of priorities for school safety. What about actually making school buildings more secure and eliminating access to assault weapons? Those seem like more pressing concerns.

Daphne Bavelier, neuroscientist who studies video game's effects, should be in attendance as an expert in this field. See some of her work here: "Your Brain on Video Games" Ted Talk and Brain Plasticity Through the Life Span: Learning to Learn and Action Video Games

Why is this an immediate focus of the responses? Did the shooter have a history of video game use? A Google search does not find any solid connections, just opinions. And further research does not directly link video game play to violent actions.

I wonder what research will be looked at, or will public opinion be the only data analyzed?

Macheath's company is planning a robbery to take place after a high-stakes game of cards. They talk of what and whom to rob

Essentially questioning the whole research process. Rather than being teleological, the interview is reflexive. Immersive interviewing. Instead of looking at research from one angle: results; this research intends to inspect itself and be conscious of the biases. Whereas most interviews lose their value, because the sample size is not large enough, this research does not lose its value. It's a mini world to be inspected. The interview determines its value. I determine its value. The subjects determine its value.

This may be an interesting context for the ESP game, where two users play until they come up with a same label for a given picture. http://ael.gatech.edu/cs6452f13/files/2013/08/labeling-images.pdf

This is true. Women are set to high standards physically and mentally and they are degraded for things they're equally good or better at like driving for no reason.

He is the mommy called in to soften the stance. Modi is wily enough to know how to play the game.

If Allen had been a computer/video game nerd, she might have made a joke here, by referring to these "most selective universities" as "l337 institutions"

Most people sleep late all the time because they play video games for hours. And they will have trouble sleeping because they forgot to do homework or anything else important to do.

I myself know this since I always work in a team. If you work alone even in video games you're social skills won't improve. Since you're in some type of group, you'll be talkative.

To prevent from being obese or overweight from situations like this, get a game console that moves your body around, not just your hands. Good ones are the Wii, Xbox Kinect, and maybe a VR.

It's telling you to play more strategy games to increase your knowledge. Or you can get a game that has a lot of dialogue, but you should be able to enjoy the same time while reading it.

This is showing you the details between a person who plays an huge amount of high violence video games compared to a low violence one and shows violent games desensitized aggression and violence.

A game to play as you begin this annotation or one to end with after you have done with it: an empathy map.

Here is another set of directions.

this idea would be pretty damn revolutionary and change the tech game

This would be absolutely game-changing. Athletes find it very difficult to find time to be prepared for all their classes when they have to devote so much time to their athletic teams. I would personally be less stressed if I had 8 years to finish my college degree instead of 4.

This would really change the recruiting game for numerous universities. Previously offering players money outside of the scholarship is against NCAA rules, and coaches and teams recently have been getting in trouble for doing this. If teams were just allowed to pay athletes all of this sketchy rule breaking and going behind the NCAA's back would disappear. And it really would make the athlete's decision easier. Whatever school would offer more money, would usually get the top player.

La integración de los participantes como desarrolladores y usuarios a quienes se dirige este proyecto (Connected Community)

ery interesting the game developed

for us these is a great page. We dont know about the impact of these topics in LGBT people. These videos share not just information, also hummanity #team 15

This shows his humble beginning and shows the kids that anyone can do it.

I see this as someone who is sick is more able to recognize others that are sick.

Acting like it is a game because he is a kid

These "friends" of Spendall, while manipulative, are also, at least according to Nan Tickleman's speech, themselves in economically precarious situations; they may be squeezing Spendall for what they can get, but they are also playing a tricky game of avoiding bankruptcy and debt.

Technology allows gaming industry to grow

This is so sad. Individuals would be targeted for something such as their strength and would be brought in for a "crime" such as hitching a ride on the road. This system is extremely messed up an sad. The people they would target were determined by the number of workers they needed. This is sad that people were targeted in this way.

What do you make of this recurrent twist? Is Julius just a hustler, or is there a game behind the game? How might readers have responded to these twists around 1900?

it seems they are both playing hard to get when they both obviously want each other . A game of cat and mouse if you ask me.

wow , she played him , she knew he would fall for the words .. leading him into her trap with the game on mystery .

This is interesting and a very good idea. Today, young people are normally using computers and other technologies as tools for learning and creating. Fusing a specific subject into a computer game is likely to obtain attention and attraction to the subjects we study today.

I believe that videogames will take over the medium of visual and audio story telling. Games become more and more immersive in how they are able to morph around player choices, so may replace more mediums like movies if they grow larger.

There is talk actually of implementing Esports into the next summer game. Esports streams have taken over the scene to the point of surpassing some sports, which goes to show that there is enough of an audience and clearly enough competition to make it a viable sport. Even watching the "videogame awards" shows we can see each year how larger and larger it grows with the boom in popularity.

Gestural modes are arguably the most important mode of our day to day social interactions. 55% of communication comes from gestures and body language alone, most of the time (https://www.psychologytoday.com/blog/beyond-words/201109/is-nonverbal-communication-numbers-game). Certainly, something to consider when giving a speech or interviewing.

Just this exercise alone demonstrates how moving away from technology in the classroom changes the way the class works. It is apparent when open discussion is encouraged in a classroom, many tend to shrink into their laptop and avoid engagement. There are classes I have taken where open discussion is prevalent, but often times it takes the entire semester to get to that point, and I think phones and laptops can attribute to that.

By using this exercise, Rheingold showed how easy it is for students to get distracted in the classroom because of the easy access to information. In class, the students with the laptop open are able to search on any site or social media app and not fully pay attention in class. This gives them the ability to pay attention while not actively paying attention which means they are actually distracted. Even the students who didn't have their laptop open were distracted by the ease of information access by other students. Students had to wait to use their laptop which makes them distracted while trying to pay attention to the professor. The use of technology in the classroom allows students the ability to not actively pay attention to the lecture but to also search for other things. This is bad for the students attention to the class and is a good example of distracted technology usage.

When I think about the Gilded Age (between the 1860ish and 1890), which was brimming with corruption, this is almost ironic. Ethnic minorities were expected to act as traditional Americans while the these "traditional Americans" were busy rigging the voting booths and building Robber Barron personas

Does this move have some sort of meaning is it symbolic of something? Is it trying to convey an idea?

1. We can identify organized crime by finding groups of people who share a similar disregard for societal norms. If we compare it to disorganized or random crime, we can spot the differences in planning and execution. Disorganized crime is more often opportunistic on a small scale, while organized crime is more likely to be premeditated or a long term operation.
2. No, they are not limited to criminals. More generally, a sharpie could be considered someone who takes advantage of the suckers in normal society to advance, even if not through illegal means, and wise guys would be those who take advantage to gain power but not necesarily social standing.
3. Even from secondhand descriptions, Cersi Lannister from Game of Thrones is a sharpie, more than willing to take advantage of and break societal norms for more power.
4. Nick Carraway is a sucker in The Great Gatsby because of his refusal to abandon societal norms and laws when the opportunity to amass money and power arises. Gatsby is a sharpie, because he uses illegal means to obtain fortune and social standing. Wolfsheim is one example of a wiseguy because while he doesn't follow societal laws, he has used this disregard to accumulate money but not necessarily social standing.
5. From Hamilton I would say that Hamilton is a sharpie from his willingness to do anything to raise his social standing (it was part of his reasoning for marrying Eliza, see the song Satisfied). Burr is more of a sucker because he waits (song: Wait for It) until he can grow legitimately and doesn't take every opportunity available.
6. If the American Dream is suceptable to corruption, then it would make logical sense if the path to the goal was also corrupted, "crooked" as it were. If the only goal is wealth and respect, and the path to get there doesn't matter, then what reason is there to stay on the straight and narrow?

1. Organized crime can be identified despite having been around for so long by using one of the defining characteristics mentioned by Lupsha. Organized crime is that which is done contrary to traditional American values which is also illegal.

2. No, these terms can apply in very legitimate pursuits as well. Though criminals would consider those actually working to all be suckers, the distinctions are found within business too. Someone who rises to the top by taking advantage of suckers would probably be a sharpie. Those who game their way to the top without putting in much work are the wise guys.

3. I am not entirely sure. Those caught in corporate scams who are lower down in the company are probably suckers. CEOs could be considered suckers if they worked legitimately for their money and position, but many have been shrewd and taken some short cuts, making them sharpies. Modern criminals involved in things like Ponzi schemes (Bernie Madoff) are wise guys.

4. Gatsby was a sharpy, working for his wealth but doing it illegally; Tom was more of a wise guy because he did not have to put in any work. Nick is quite apparently a sucker.

5. Hamilton works incredibly hard for his success, making him more of a sharpy or a sucker. He is accused of using illegal means for his advancement, but he makes it clear that he did not. By the end, he is a sucker because Burr, Jefferson, and Madison, acting as sharpies, have figured out a way to make Hamilton essentially self-destruct. Jefferson is also a wise guy. He doesn't have to work for his success because of his influence, and he is just offered positions like Secretary of State. He avoids the war by being the ambassador to France.

6. Because American values can be corrupted, the American dream can also be corrupted. By virtue of being a dream of success, people will do what is necessary to gain that success. Their desire goes beyond the "white picket fence" and a comfortable life, turning into a desire to gain as much wealth as is possible. The methods of achieving this dream are often contrary to what most people consider American because of this corruption.

2017 marked the first edition of the official baseball rules that made specific mention of or provisions for minor league baseball and minor league teams.

2017 marked the first edition of the official baseball rules that made specific mention of or provisions for minor league baseball and minor league teams.

2017 marked the first edition of the official baseball rules that made specific mention of or provisions for minor league baseball and minor league teams.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

This trend appears more directly in the CBA documents, but as baseball's global presence expanded, Major League Baseball opted to capitalize on that popularity, rather than collaborate with foreign professional leagues, an attitude reflected in this section's language.

By numbering the issues with the land, and purposefully creating short, choppy sentences, the author provides emphasis to each point. The emphasis creates a literary punch to appeal to the audiences guilt.

This is also a big info breaker as the brain knows when to and how to impress the people around it as when with a girlfriend the brain knows not to impress them by being risky but by showing them that they can be safe.

This shows how peer pressure affects the thinking patterns of not only teens but adults as well

Liangelo had pulled out of UCLA after his China case. Gelo had not played a game with the Bruins yet in the regular season. At the time, Lavar Ball said "We are seeking other options"

AP

Instagram game/Instagram prowess. Also known as "Flexing"

we are at a crossroads. We can continue to use the old methods or try to use new methods that would change the game.

Meliora students, what is the meaning of this word? Provide an example of another story (book, film, video game, etc.) that includes a simulacrum.

I'd say this has a lot to do with what the company was originally labelled as a game changer in that industry. Maybe a similar example would be Disney which produces and funds much more than movies and theme parks but that's what we know it best for.

Sophie Woodman establishes her stance that this preconceived notion is nothing but false. Artifacts and objects alike can have a multitude of meanings and underlying cultural aspects that exist within that object. Haltman himself though has vouched that the process of recognizing these aspects take repetitious action and a thorough game plan.

was Russia ahead of the game here by using social media to meddle with the US election?

No equivalent pool was created for minor league players to profit from championship or post-season appearances.

Since the CBA only articulates "leagues" governed by this CBA as the National League and American League, the various affiliated minor leagues (collections of minor league clubs) and their all-star game activities are not covered by this sub-article.

No equivalent pool was created for minor league players to profit from championship or post-season appearances.

Since the CBA only articulates "leagues" governed by this CBA as the National League and American League, the various affiliated minor leagues (collections of minor league clubs) and their all-star game activities are not covered by this sub-article.

Since the CBA only articulates "leagues" governed by this CBA as the National League and American League, the various affiliated minor leagues (collections of minor league clubs) and their all-star games are not covered by this sub-article.

Since the CBA only articulates "leagues" governed by this CBA as the National League and American League, the various affiliated minor leagues (collections of minor league clubs) and their all-star games are not covered by this sub-article.

Since the CBA only articulates "leagues" governed by this CBA as the National League and American League, the various affiliated minor leagues (collections of minor league clubs) and their all-star games are not covered by this sub-article.

It seems to me that in most of these readings that anyone who thinks different than the majority of society is labeled as crazy. The world thinks very black and white, but to me these people are just open minded and are able to think outside of the box. Obviously today we know that there aren't just two ways of looking at things. Do you think these people would still be labeled as crazy in today's world or just open minded?

"Merits and demerits" of this angelic parallelism is provided in 1.1 Related Work section later.

curr 3

cur 4

curr 2

curr 1

That's too bad!

Again, as seen previously in the CBA, players are given limited ability to utilize their baseball skills for financial gain outside the playing season established by major league baseball in the CBA.

Since the CBA only articulates "leagues" governed by this CBA as the National League and American League, the various affiliated minor leagues (collections of minor league clubs) and their all-star games are not covered by this sub-article.

Since the CBA only articulates "leagues" governed by this CBA as the National League and American League, the various affiliated minor leagues (collections of minor league clubs) and their all-star games are not covered by this sub-article.

Again, as seen previously in the CBA, players are given limited ability to utilize their baseball skills for financial gain outside the playing season established by major league baseball in the CBA.

Again, as seen previously in the CBA, players are given limited ability to utilize their baseball skills for financial gain outside the playing season established by major league baseball in the CBA.

Again, as seen previously in the CBA, players are given limited ability to utilize their baseball skills for financial gain outside the playing season established by major league baseball in the CBA.

Good start to show the counter to my argument.

Owl-eyes has MAD game. Jordan is a beautiful, bronze skinned athlete who is single and belonging to nobody. Owl-eyes really is going to take a good long look, up and down on Jordan while she is standing directly next to Nick. This situation poses a most important question. Is owl-eyes just that bold? Or is Jordan just that gorgeous? This meme is what I think about even though Owl-eyes doesn't have a girl.

http://time.com/4919007/love-triangle-meme-interview/

foo

Here, she's surprised that she can save the game in her own name, which makes me think about the stories from Pandora's Breeches where the women were not given credit for their work or they were being overshadowed by the males in their lives.

This goes over with every idea I have on this matter in one sentence.

I think social media influence people more that video games. Because people of all age don't play video game but everyone is connected to social media.

Reading this was quite interesting and oddly disturbing. However, it did allow me to remember the frequent use of ashes as jewelry such as turning them into diamonds in the game, "Metal Gear Solid V". Ultimately, the existence of Mr. Wolfheim presents readers like me hints about Gatsby's origins and his true connections untold so far.

The fact that Gatsby traveled the capitals of Europe means he should be well educated in other cultures. I remember for my Bat Mitzvah, I chose the option of traveling abroad in all the capitals that he visited instead of a massive party like normal tweens had. I'm so grateful I chose traveling over a party because now I feel more cultured like Gatsby.

The description of the area Tom and Nick have arrived to reminds me of a location called the Boreal Valley in the game Dark Souls 3. The setting: dark, gloomy, and filled with dust, similar to the atmosphere Fitzgerald has set for the first half of the second chapter as well as the idea behind Myrtle's origin.

Reading this statement caused me to delightfully remember the first of every time I've played a Pokemon game where each professor asks the player to determine their gender. Furthermore, like the games, the scene also showcases the unimportance of gender towards attaining happiness (regarding that the owner who buys that dog will be happy regardless of its gender).

This description of Mr. McKee reminds me of a character named Jhin in the game League of Legends. As a feminine male character who adheres to chivalry, Jhin's personality and play style really symbolizes my idealization of Mr. McKee. Consequently, this causes me to wonder on Fitzgerald's creation of characters as Mr. McKee definitely fits an older western personality.

Douglass, yet again inserts quiet irony to subvert our ideas of power. Covey's behavior and rhetoric is almost childish or playful. "Ha, ha!" and "dash on!" sound more like cries of a kids game than of a figure of authority. Furthermore, they are and far less articulate than Douglass' words.

what were been affected are not just their language, but also their thoughts. They started to be changed without even notice. The author throws a slide of hope, plants a thought of the "evil" will eventually turns into "good". but instead, they kick "Bartleby" out off the game, because what right makes them feel uncomfortable. Another human nature.

Question from a non-native speaker: is this suppose to be "Support"? or sport has other means more than "game"?

This brings up some anecdotal memories. I used to attempt to be as good at Sid Meier's Pirates! on the Commodore 64 as my mom when I was a kid. And my dad and I used to have 4th and Inches tournaments with our friends. Each experience brought us closer together as a family and helped me socialize with adults before I was a teenager.

That seems complicated. What is a "quality" game? According to Metacritic, Grand Theft Auto V is the highest rated video game in Playstation 4 history. Is that quality, or is there a different definition according to this article? http://www.metacritic.com/browse/games/score/metascore/all/ps4/filtered

Again we come across the term "too much" with no clear definition of what that means. There is no statement that "studies show" this either. Is this just the writer's opinion?

This poem reminds me of all the unspoken social rules that we follow as human beings. It makes me think of how we're constantly walking by other human beings, many of whom intentionally do not make eye contact, although they too clearly recognize the human passing by them. Yet we continue with these unspoken social conventions, because going against them might make others uncomfortable. The notion of the natural borders that separate Frost from his neighbor serve to show how ridiculous and ambiguous our social "rules" are, although we still devoutly follow them.

On 2016 May 22, Lydia Maniatis commented:

(Third comment) Arbitrary methodological choices, data presentation

The title of the paper is “Seeing sets,” and question of interest is said to be “to determine what observers know about the members of a set and what they know about the statistical properties of the set (mean and distribution)” and that “sets of objects could be represented in a qualitatively different way than single items.”

These are quite broad objectives, and don't imply any particular methodology.

The author uses a very specific methodology that places demands on subjects that go beyond straightforward perception, and imposes conditions that compromise straightforward perception.

He uses brief presentation times, 500ms for the first figure and 500s for the comparison figure, with no blank space in between.

This means that there is time for about two saccades and fixations (barely), and involves potential masking effects. How did he choose these parameters? Do they matter? Would we expect the same results with longer presentation times? With a blank interval? Also, the circles increase in size geometrically, not linearly and the mean that we are talking about is the geometric, not the arithmetic mean. Why not a linear progression and an arithmetic mean?

It's been understood for a long time that by adjusting experimental conditions we can get pretty much any result we want, which is why such conditions should come with a theoretical rationale attached. Because the author's choices here seemingly make the task much more difficult than it should be given the goal, they need to be explained, not just flatly asserted.

Another blank check is to be found in a footnote referring to a “small study” which was the basis for selecting/describing one of the parameters in the reported study (“members differed in size from non-members by at least 18%, about three times the size-discrimination threshold for sets of same-size spots [as determined in the “small study”].” In a personal communication, Ariely told me that this study used about ten subjects, and was mostly about studying “within-subject variation.” Such a study could also have given an indication of between-subject variation. Given the link between the two studies, and the oddity of using only two observers (why only two?), some more specific info on results/methods of this "small study" would seems called for.

The presentation of the data is also rather weird. For the mean-discrimination experiment, we aren't given information on the proportions of correct answers the two observers achieved. Maybe this information is hidden in the opaque measure that is provided (and which assumes normality in the data without any justification - isn't that a problem?). This measure is the "mean-discrimination threshold" derived using "a standard profit analysis (Finney, 1971) ...to determine the mean-discrimination threshold (the standard deviation of the best-fitting cumulative normal distribution." Is best-fitting the same thing as "well-fitting"?

Given the method, and assuming that it really is the case that subjects were better at guessing the mean rather than the size of individual members, I could speculate on why that might be. First of all, observers never had to play the "yes-no" game with means (why not?). All they had to do was say "larger-smaller." This seems like an inherently easier task. Further, given the time constraint, there wasn't enough time to inspect each of the four different circle sizes individually. Observers only had time for one or two. So for half of each set, at least, they were out of luck on each trial in the individual member task. For the mean estimation, if they could learn to target the middle two on each trial - not the largest, not the smallest - then they could ballpark the mean. Who knows? The situation is as Runeson (1995) has described for a different type of vision study:

"For perception and cognition research in general, it is noteworthy that the cue-based style of theorising exhibits a certain lack of inherent correctives for ineffectual experiments because of its emphasis on weak and irregular performance. Thus, an experiment that is unsuitable in design or procedure could easily provide supportive-looking data at least as long as cues, salience limits, and trade-off functions are not specified in detail....data that represent suboptimal observer performance seem to have provided spurious support for untenable theoretical commitments,... "

On 2013 Oct 29, Michael Eisen commented:

According to recent Frontline documentary "League of Denial" http://www.pbs.org/wgbh/pages/frontline/league-of-denial/ this paper was the first step in a chain of events that has finally led to the NFL acknowledging that repeated head trauma can potentially cause long term problems for professional football players, and to a series of changes in the game designed to reduce head injuries.

Note that, according to Frontline, the NFL originally disputed the finding and attacked the work and its author - see Casson IR, 2006.

On 2014 May 06, Madhusudana Girija Sanal commented:

Dr. Norman is picturing the great progress in education as a result of the information distribution revolution as if it is an ‘unavoidable evil’! Probably, many of his generation expresses high inertia, still consider, 'physical' universities, libraries, books, lecture halls etc. very much essential! I understand their nostalgia! You know what this means for humanity? Rich countries such as USA hold only a small fraction of the world’s population. Through virtual universities more and more people across the world, irrespective of rich and poor, would be able to attend the best schools and courses. They would be able to take the same exams and get ranked along with the most privileged, rich or intelligent. This would be great! This is 'new' justice! (Although I believe “Justice is ‘man made’ or artificial”). All we want is better tools to evaluate human intellectual qualities, online. I do not think face to face lectures will be better than online recorded, interactive lectures, may be multidimensional (3D-4D-5D) lectures by several professors of the learner’s choice. Lectures will be ranked and paid by based on their quality by student communities and not by bureaucrats or by administrators and politicians. This system is great especially when I consider the advantages! On demand, personalized lectures would be "ready-made" for commonly observed (student) personality traits -say there are 100 personality subtypes and intellectual levels! Custom lectures are available for them all because there are much more people to teach online-lectures need not be real time. You can learn from a "personality" who matches your rare personality, perhaps, one who lived 10 years back. His lectures had to wait for ten years for a student like you! Is not this a very exciting possibility? However, I do agree that face to face lectures can be more individualized and beneficial (for the rich, because they only have money to ‘buy’ good teachers!) Nevertheless, I do not think there is a huge benefit for extreme personalization except for exceptional children who are extremely out-of-the-box in a positive or negative way. It may be, however, noted that overall poor but brilliant students have a better opportunity to come out and stand before the world. This global free learning system will benefit specially those brilliant minds in less privileged countries. Do Dr. Norman has any evidence to support his statements in the editorial? I do not know what Dr. Norman will write if tomorrow we find a new technology which allows direct transmission of knowledge to brains! Then what will happen? Think! Everyone will have equal opportunity to accumulate the same amount of information if he or she wants. Now who will win this game? Those who have money? Probably not! Those fractions of the society who are blessed with right genome, epigenome and the best neural connections!

On 2015 Apr 06, Peter Good commented:

To PubMed Commons: comment to Frye RE, 2013.<br> Frye et al. [2010] previously reported that sapropterin, a synthetic form of tetrahydrobiopterin (BH4), improved behavior in children with autism spectrum disorders (ASD) [Frye RE, 2010]. Their 2013 study [Frye RE, 2013] was intended to test whether sapropterin’s benefit was due to BH4’s role as cofactor for synthesis of the monoamine neurotransmitters dopamine and serotonin (as previous investigators suspected), or BH4’s role as cofactor for nitric oxide synthase (NOS), which produces the critical gaseous molecule nitric oxide (NO). Frye et al. concluded that improvements in communicative language in these children from sapropterin were due to restoration of NOS “coupling” disrupted by lack of BH4, which dysregulated nitric oxide metabolism. In support of their conclusion they cited evidence by Sweeten et al. [Sweeten TL, 2004] and others of high levels of nitric oxide metabolites nitrite and nitrate in blood of ASD children. In their previous study Frye et al. concluded: “[I]t is possible that BH4 in ASD could be depleted by the overactivation of the immune system and inflammatory processes during an excessive production of nitric oxide.” [Frye RE, 2010]

There may, however, be more to this story. A few months after publication of Frye et al. 2013, Stanhewicz et al. reported sapropterin increased reflex vasodilation in aging human skin by increasing release of nitric oxide by endothelial and neuronal nitric oxide synthases [Stanhewicz AE, 2013]. Nitric oxide, the primary dilator of blood vessels in the body, is produced by three different forms of nitric oxide synthase – two constitutive forms present in blood vessel endothelial cells (eNOS) and neurons (nNOS), and a third form (iNOS) induced in brain microglia and other cells of the immune system in response to infections and other agencies. Endothelial nitric oxide maintains the vasodilator tone of blood vessels. Neuronal nitric oxide may be largely responsible for neurovascular coupling – dilation of nearby blood vessels when brain neurons fire. Faraci & Brian: “. . . NO appears to mediate cerebral vasodilatation in response to local neuronal activation.” [Faraci FM, 1994]. Koehler et al.: “. . . NO is required as a mediator of neurovascular coupling in the cerebellum, whereas NO acts as a modulator in the cerebral cortex.” [Koehler RC, 2009]. Inducible nitric oxide is released in large quantities to flush infective agents and toxins, and kill damaged cells.

If nitric oxide is too high in autistic disorders, inducible nitric oxide is the form likely responsible, Frye et al. concluded. Sweeten et al. concluded likewise: “[I]t is reasonable to hypothesize that iNOS is involved in the elevated NO production in autism.” [Sweeten TL, 2004]. Yet inducible nitric oxide is often released to compensate deficiencies of constitutive nitric oxide [Hecker M, 1999;Kubes P, 2000]. One indication neuronal nitric oxide is deficient in children with autistic disorders is their failure of neurovascular coupling – their brains are often hyperexcitable, yet brain blood flow is consistently low [e.g. Ohnishi T, 2000]. Nitrite and nitrate also serve as reservoir forms to deliver nitric oxide elsewhere [Dejam A, 2005]. Lundberg & Weitzberg: “[N]itrate and nitrite should probably be viewed as storage pools for NO rather than inert waste products.” [Lundberg JO, 2005].

Did sapropterin increase endothelial and neuronal nitric oxide in the brains of ASD children in Frye et al. 2013? Why would endothelial and neuronal nitric oxide be deficient in these children? One explanation is deficiency of BH4. Another is deficiency of the amino acid arginine – only substrate for nitric oxide [Wiesinger H, 2001]. Frye et al. found higher baseline levels of blood arginine in these children, and higher ratios of arginine to citrulline, were associated with greater improvements in language from sapropterin. They noted blood arginine and the arginine/citrulline ratio did not change significantly during sapropterin treatment – but also stated improvements in language were greater in children with “an attenuated increase in arginine.” [Frye RE, 2013]

Considerable evidence argues that arginine is deficient in ASD children: (a) high levels of inducible nitric oxide; (b) consistently low brain creatine (arginine + glycine) [Friedman SD, 2003]; (c) frequent high blood ammonia [Filipek PA, 2004] which requires arginine to detoxify to urea; and (d) high levels of arginine vasopressin in autistic boys [Carter CS, 2007; Momeni N, 2005]. Furthermore, NOS produces harmful oxidants superoxide and peroxynitrite when NOS “uncouples” from lack of BH4 – or when arginine is deficient [Xia Y, 1996]. Because most supplemental arginine is taken up by the liver (thus unavailable to other tissues), citrulline (arginine’s precursor) or glutamine (citrulline’s precursor) may be better sources of arginine for NOS [Cynober L, 2010]. The evidence speaks for itself.

Peter Good Autism Studies La Pine, OR www.autismstudies.net autismstudies1@gmail.com

Carter CS. Sex differences in oxytocin and vasopressin: implications for autism spectrum disorders? Behav Brain Res 2007;176:170–186.

Cynober L, Moinard C, De Bandt J. The 2009 ESPEN Sir David Cuthbertson. Citrulline: A new major signaling molecule or just another player in the pharmaconutrition game? Clinical Nutrition 2010;29:545–551.

Dejam A, Hunter CJ, Pelletier MM, et al. Erythrocytes are the major intravascular storage sites of nitrite in human blood. Blood 2005;106:734–739.

Faraci FM, Brian Jr. JE. Nitric oxide and the cerebral circulation. Stroke 1994;25:692–703.

Filipek PA, Juranek J, Nguyen MT, et al. Relative carnitine deficiency in autism. J Autism Dev Disord 2004;34:615–623.

Friedman SD, Shaw DW, Artru AA, et al. Regional brain chemical alterations in young children with autism spectrum disorder. Neurology 2003;60:100–107.

Frye RE, Huffman LC, Elliott GR. Tetrahydrobiopterin as a novel therapeutic intervention for autism. Neurotherapeutics. 2010;7(3):241–249.

Hecker M, Cattaruzza M, Wagner AH. Regulation of inducible nitric oxide synthase gene expression in vascular smooth muscle cells. Gen Pharmacol 1999;32:9–16.

Koehler RC, Roman RJ, Harder DR. Astrocytes and the regulation of cerebral blood flow. TINS 2009;32(3):160–169.

Kubes P. Inducible nitric oxide synthase – a little bit of good in all of us. Glia 2000;47:6–9.

Lundberg JO, Weitzberg E. NO generation from nitrite and its role in vascular control. Arterioscler Thromb Vasc Biol 2005;25:915–922.

Momeni N, Nordström BM, Horstmann V, et al. Alterations of prolyl endopeptidase activity in the plasma of children with autistic spectrum disorders. BMC Psychiatry 2005;5:27–32.

Ohnishi T, Matsuda H, Hashimoto T, et al. Abnormal regional cerebral blood flow in childhood autism. Brain 2000;123(Pt. 9):1838–1844.

Stanhewicz AE, Alexander LM, Kenney WL. Oral sapropterin acutely augments reflex vasodilation in aged human skin through nitric oxide-dependent mechanisms. J Appl Physiol 2013:115:972–978.

Sweeten TL, Posey DJ, Shankar S, McDougle CJ. High nitric oxide production in autistic disorder: a possible role for interferon-gamma. Biol Psychiatry 2004;55(4):434–437.

Wiesinger H. Arginine metabolism and the synthesis of nitric oxide in the nervous system. Prog Neurobiol 2001;64(4):365–391.

Xia Y, Dawson VL, Dawson TM, et al. Nitric oxide synthase generates superoxide and nitric oxide in arginine-depleted cells leading to peroxynitrite-mediated cellular injury. Proc Natl Acad Sci USA 1996;93:6770–6774.

On 2015 May 04, Peter Good commented:

I asked Dr. Cynober whether oral arginine, citrulline, or glutamine would be the best source of arginine for brain nitric oxide and creatine in ASD children. He replied it was controversial whether oral arginine or glutamine produces more citrulline in the intestines. Citrulline enters the brain, he said, but whether it generates creatine there, and the balance between production of nitric oxide (NO) and creatine from citrulline-derived arginine, is unclear.[personal communication 2015]

Their 2010 paper [Cynober L, 2010] presented much evidence critical to autistic disorders (ASD). Citrulline (CIT) is not normally present in protein; its usual sources are arginine (ARG) and glutamine in dietary proteins, which produce CIT in the intestines: “CIT is almost absent from natural foods, watermelon being a notable exception.” CIT bypasses the liver and forms ARG in the kidneys, which limits wasting of nitrogen as urea, and provides ARG to many other tissues, including the brain.

“[D]irect supplementation of CIT should be more useful than ARG supplementation, leaving the kidney to convert CIT into ARG, so avoiding heavy first-pass splanchnic extraction of the ARG and the possible harmful effects of an excessive ARG [therefore NO] supply. . . . CIT is able to sustain NO production through eNOS but not iNOS. . . . CIT could also be a safe way to deliver ARG to endothelial and immune cells, and can certainly prevent excessive uncontrolled nitric oxide production. . . . [A]ntioxidant properties, together with the ability to generate NO, make CIT an excellent candidate for the treatment of pathological situations characterized by oxidative stress and decreased arginine availability . . . .”

Because CIT stimulates protein synthesis when dietary proteins (i.e. ARG and glutamine) are low, it should be given in the postabsorptive state (3–5 hrs after meals) or fasted state (before breakfast). 10–15g/day of oral citrulline in healthy adults showed high bioavailability and no adverse effects. Because CIT is synthesized almost exclusively in the intestines, it may also be a useful biomarker of functional gut tissue.[Cynober L, 2010]

In their 2005 paper [Curis E, 2005] Cynober and colleagues discussed other aspects of citrulline metabolism: “Citrulline presents the common reactivity of the α-amino acid family. In particular, it can form peptide bonds; hence it can therefore be present in proteins. However, since there is no known codon in the genetic table for this amino acid, its presence in a protein must always result from a post-translational modification of the protein. . . .

“The main reason for this citrulline metabolism split between two organs [gut and kidney] is related to the efficacy of the capture of arginine by the liver. In fact, without metabolic adaptation, almost all the arginine coming from food supply would be withdrawn from the portal blood by the liver, leaving only very low amounts of available arginine for other organs. . . .

“[M]any cell types which are able to metabolize arginine into NO are able to uptake circulating citrulline, which explains why citrulline induces certain of the NO effects . . . . The figure seems to be even more complex in the brain, since the recycling of citrulline into arginine is split between various cell types, defining a unique inter-cell-type cycle. Indeed, the brain neurones producing NO are not able to reconvert citrulline into arginine since they do not express the [necessary] enzymes. Hence, citrulline is released from the neurons and taken up by surrounding neural cells where return-conversion to arginine is performed.”[Curis E, 2005]

Romero and colleagues presented other valuable information about citrulline [Romero MJ, 2006]: “With development, intestinal synthesis of L-arginine from glutamine decreases and the small intestine gradually becomes the major site of net L-citrulline production. . . . L-citrulline is largely taken up and metabolized by the kidney, which in turn releases arginine equivalent to ~75% of the L-citrulline taken up. Thus, much of the L-citrulline produced by enterocytes reaches the systemic circulation as L-arginine. This L-arginine/L-citrulline homeostasis allows a proper supply of L-arginine for the whole body. About 60% of dietary L-arginine makes it into the hepatic portal circulation, while the rest is metabolized in the intestine. . . . L-citrulline synthesis in many tissues also occurs as a byproduct of NOS activity. . . . Although NOS is widely distributed throughout the body, its activity does not contribute substantially to whole body L-citrulline flux under normal conditions. . . .

“[A]cute oral administration of L-citrulline appears to be considerably more efficient raising plasma levels of L-arginine than L-arginine itself. Additionally, a recent study in children and young adults showed that five oral doses of L-citrulline every 12 hours (1.9 g/m2/dose) for a total dose of 9.5 g/m2 resulted in 57 and 85% increases in mean plasma levels of L-arginine and L-citrulline, respectively. . . . L-citrulline is generally recognized as safe for oral consumption. In fact, L-citrulline can prevent some of the untoward effects of L-arginine supplementation. . . . L-citrulline is a natural and apparently safe means of providing L-arginine for constitutive NOS production of NO.”[my emphasis] [Romero MJ, 2006]

Deutz also presented important observations about plasma concentrations and interorgan transport of glutamine, citrulline, and arginine [Deutz NE, 2008]: “In daily practice, the plasma concentration is usually viewed as a parameter of production. This is not always correct as the plasma concentration can be high due to an increased production of the substrate and/or a reduced capacity of the body to dispose [of] this substrate. This means that the level of plasma concentration can be misleading and does not always give reliable information whether there is actually an intracellular deficiency of a certain substrate. . . .

“The quantitative main production site [of glutamine] in the body is muscle and the main consumption sites are the gut, liver and the kidney. Liver plays a dual role as it can both produce and consume glutamine, depending on the metabolic state (fasting/fed). . . . [A]bout 80–90% of the citrulline is derived from the gut glutamine to citrulline conversion. Therefore, whole body citrulline production is related to the quantity of gut glutamine conversion to citrulline, and is most likely influenced by the amount of active gut tissue.”

Most important conclusions for ASD: (1) oral citrulline bypasses the liver and becomes arginine in the kidneys, making arginine more available to other tissues, including the brain; (2) oral citrulline supports production of constitutive nitric oxide but not inducible nitric oxide; (3) in light of the benefit of casein-free/gluten-free diets in ASD children, citrulline’s stimulation of protein synthesis when dietary protein is low may be invaluable; (4) citrulline synthesis may be a useful marker of functional gut tissue in these children; (5) oral citrulline is safer than oral arginine. The evidence speaks for itself.

Peter Good Autism Studies www.autismstudies.net autismstudies1@gmail.com

Curis E, Nicolis I, Moinard C, et al. Almost all about citrulline in mammals. Amino Acids 2005;29:177–205.

Cynober L, Moinard C, De Bandt J-P. The 2009 ESPEN Sir David Cuthbertson. Citrulline: A new major signaling molecule or just another player in the pharmaconutrition game? Clin Nutrit 2010;29:545–551.

Deutz NEP. The 2007 ESPEN Sir David Cuthbertson Lecture: Amino acids between and within organs. The glutamate-glutamine-citrulline-arginine pathway. Clin Nutrit 2008;27:321–327.

Romero MJ, Platt DH, Caldwell RB, Caldwell RW. Therapeutic use of citrulline in cardiovascular disease. Cardiovasc Drug Rev 2006;24:275–290.

On 2014 Jan 03, Ian Lyons commented:

In this paper, Park and Brannon showed that training adult subjects on an approximate, nonsymbolic arithmetic task (adding and subtracting estimates of the number of dots in various dot arrays) led to improvement on a symbolic arithmetic task (i.e., using Indo-Arabic numerals). The authors suggest this result points to a causal role for the ‘approximate number system’ (ANS) in more complex, symbolic math processing and may thus inform the development of interventions designed to improve mathematical competence in children and adults. We believe the authors’ work takes an important step forward in terms of understanding the building blocks of mathematical performance, and, using their work as a jumping board, we offer several points of reflection concerning (1) the nature of the ANS, and (2) what it means to train performance on a task versus the process it is meant to measure.

Park and Brannon’s crucial experimental condition trained participants using approximate, nonsymbolic addition. This differs from tasks used more commonly in the literature to measure individual differences in the ANS – adaptation and comparison – which involve simply distinguishing between two approximate quantities (e.g., in comparison tasks, one typically decides which of two arrays contains more dots). The difference in tasks is significant because previous attempts to train participants on just a nonsymbolic comparison task failed to show significant improvement in individuals’ symbolic math performance [Wilson et al., 2006 (http://www.ncbi.nlm.nih.gov/pubmed/16734906); DeWind & Brannon, 2012 (http://www.ncbi.nlm.nih.gov/pubmed/22529786)]. Why, then, does training on nonsymbolic arithmetic lead to improvement in symbolic arithmetic skills, but training on nonsymbolic comparison does not, even though both have been shown to correlate with symbolic arithmetic [e.g., Gilmore et al., 2010 (http://www.ncbi.nlm.nih.gov/pubmed/20347435); Halberda et al., 2012 (http://www.ncbi.nlm.nih.gov/pubmed/22733748)]? One possible conclusion is that it is not enough simply to tap the ANS; instead, accessing the ANS must be structured in a manner that more directly parallels the target skill – symbolic arithmetic. From a broader perspective, such a conclusion suggests that it is time to take a deeper look at what exactly we mean by an ‘approximate number system’, as Park and Brannon’s results may in fact point to an important division between approximate quantity representation and manipulation within the ANS. The view that the ANS is not a unitary construct is also leant support by the fact that performance on nonsymbolic quantity comparison and nonsymbolic arithmetic tasks are uncorrelated [Gilmore et al., 2011 (http://www.ncbi.nlm.nih.gov/pubmed/21846265)].

That nonsymbolic arithmetic (but not nonsymbolic comparison) training leads to improved symbolic arithmetic brings us to a second point: It is crucial to make a distinction between a task meant to measure or be an index of some underlying process, and the process itself. To cure a fever, one does not build a more precise thermometer; and by extension, if one demonstrated that using a more precise thermometer indeed failed to reduce one’s fever, it would be rather rash to conclude ambient bodily temperature is irrelevant to one’s health. A nonsymbolic number comparison task may act like a thermometer, where the underlying process it indexes is ANS acuity. Training on nonsymbolic comparison tasks does not improve math skills (Wilson et al., 2006; DeWind & Brannon, 2012), but this does not mean that the ANS is irrelevant for math. By training on nonsymbolic arithmetic instead of nonsymbolic comparison, Park and Brannon showed that one’s training regimen simply needs to tap the ANS in a way that better parallels the types of cognitive operations used in symbolic arithmetic.

One sees a similar distinction between tasks that index versus train an underlying process elsewhere in the numerical domain: when a person is asked to mark the location of a number on a number line, the linearity of their estimates predicts math achievement [Booth & Siegler, 2006 (http://www.ncbi.nlm.nih.gov/pubmed/16420128), 2008 (http://www.ncbi.nlm.nih.gov/pubmed/18717904)]; but rather than train on this task per se, researchers found success using a board game that trained children to linearize their visuo-spatial representations of symbolic numbers – i.e., the underlying process that was presumably being measured by the number line task. Training on the board game improved performance on both the numberline task as well as math achievement [Siegler & Ramani, 2009 (http://psycnet.apa.org/journals/edu/101/3/545/)].

Further, we believe that Park and Brannon’s own dataset provides yet another example illustrating the distinction between a task meant to measure or be an index of some underlying process, and the process itself. The authors show a correlation between numerical ordering ability and symbolic math ability, replicating our previous work [Lyons & Beilock, 2011 (http://www.ncbi.nlm.nih.gov/pubmed/21855058)]. Nevertheless, training on the ordering task did not lead to improvement in symbolic math beyond what was seen for vocabulary training. If one concludes from this result that understanding ordinality is irrelevant for developing math skills, one is in danger of mistaking a means of measurement for the thing being measured – much as one might have done with the dot comparison or number-line tasks discussed above.

In conclusion, Park and Brannon’s recent paper showing a causal relation between nonsymbolic and symbolic arithmetic, represents a step toward understanding the building blocks of complex arithmetic. Perhaps missed in the excitement, though, is that this work underscores the need for researchers – especially those interested in educational applications – to carefully consider what their tasks and paradigms truly mean with respect to the processes and representations they aim to investigate. Failing to do so risks conflating the means of measurement with what is being measured, and may in turn lead to recommendations for educators to train the wrong thing.

Signed, Ian Lyons and Sian Beilock

On 2013 Oct 05, Daniel J Simons commented:

The paper implies that game training allows adults to perform as well as 20 year olds when multitasking. What it actually shows is that after training on the game, older adults perform as well as younger adults who are playing that same game for the first time. It does not show that training allows the older adults can multitask as well as the younger subjects in any other context. For example, the training did not lead to differential improvements on an a dual-task outcome measure.

The study had a small sample size (about 15/condition), and the analysis did not correct for multiple tests, meaning that it is not clear whether training led to any reliable improvements on the outcome measures.

The paper also did not control for expectations in the training and control group, meaning that any differential improvements upon re-testing could be due to differential placebo effects.

I have posted an extensive post-publication review of the paper here: http://blog.dansimons.com/2013/09/19-questions-about-video-games.html

On 2015 Apr 14, Björn Brembs commented:

“Standing on the shoulders of giants” is what scientists say to acknowledge the work they are building on. It is a statement of humility and mostly accompanied by citations to the primary literature preceding the current work. In today’s competitive scientific enterprise, however, such humility appears completely misplaced. Instead, what many assume to be required in the struggle to survive is to convince everyone that they are the giant, the genius, the prodigy who is deserving of the research funds, the next position, tenure.

The Nature Neuroscience article “Temporal structure of motor variability is dynamically regulated and predicts motor learning ability” by Wu et al. with its accompanying news-type article “Motor variability is not noise, but grist for the learning mill” by Herzfeld and Shadmehr (linked above) can only be described as over-hyping an otherwise very interesting discovery. Both articles claim that the researchers have made the game-changing discovery that something long thought to be a bug in our movement system is actually a spectacular feature. It is argued that this discovery is such a huge surprise, because nobody in their right mind would have ever thought this “unwanted characteristic” to actually serve some purpose.

The problem with this line of argument is that probably most people in the field thought it should be obvious, even to be expected – and not surprising at all. Skinner is largely credited with the analogy of operant conditioning and evolution. This analogy entails that reward and punishment act on behaviors like selection is acting on mutations in evolution: an animal behaves variably and encounters a reward after it initiated a particular action. This reward will make the action now more likely to occur in the future, just as selection will make certain alleles more frequent in a population. Already in 1981, Skinner called this “Selection by Consequences“ (Science Vol. 213 no. 4507 pp. 501-504, DOI: 10.1126/science.7244649). Skinner’s analogy sparked wide interest, e.g. an entire journal issue (Behavioral and Brain Sciences 7(04), 1984), which later appeared in book form (The Selection of Behavior: The Operant Behaviorism of B. F. Skinner: Comments and Consequences. A. Charles Catania, Stevan R. Harnad, Cambridge University Press). Clearly, the idea that reinforcement selects from a variation of different behaviors is not a novel concept at all, but more than three decades old and rather prominent. This analogy cannot have escaped anybody working on any kind of operant/motor learning, except those seriously neglecting the most relevant literature. This interaction of variability and selection is a well-known and not overly complicated concept, based in evolutionary biology and psychology/neuroscience. Consequently, numerous laboratories have been studying various aspects of this interaction for a long time. Skinner’s projection was that increased behavioral variability leads to increased operant learning rates, just like increased mutations rates lead to increased rates of evolutionary change. More than a dozen years ago, Allen Neuringer showed this to be the case in rats (Psychonomic Bulletin & Review 2002, 9 (2), 250-258, doi: 10.3758/BF03196279), but there are studies in humans as well (Shea, J. B., & Morgan, R. B. (1979). Contextual interference effects on the acquisition, retention, and transfer of a motor skill. Journal of Experimental Psychology: Human Learning and Memory, 5, 179–187). That such variability is beneficial, rather than detrimental has been shown even in situations where the variability is so high, that the acquisition rate is reduced, but post-training performance is enhanced (Schmidt RA, Bjork RA (1992): New conceptualizations of practice: Common Principles in Three Paradigms Suggest New Concepts for training. Psychological Science, 3(4): 207-217).

Wu et al. confirm both Skinner’s conjecture as well as previously published reports (some cited above) that indeed the rate of learning in operant conditioning is increased in subjects where the initial variability in the behavior is higher. This is an important and relevant finding. However, instead of citing the wealth of earlier work, Wu et al. claim that their results were surprising: “Surprisingly, we found that higher levels of task-relevant motor variability predicted faster learning”. Herzfeld and Shadmehr were similarly stunned: “These results provide intriguing evidence that some of the motor variability commonly attributed to unwanted noise is in fact exploration in motor command space.”

I regard it as highly unlikely that none of the seven authors in total should have never heard of Skinner or the work over the last four decades by many human movement scientists that have explored the temporal structure of human movement variability and its relationship with motor learning. The work by senior scientists such as Karl Newell, Michael Turvey, Richard Schmidt, and their students published in books and hundreds of journal articles is completely ignored, just as the work by several younger mid-career scientists such as Nick Stergiou, Jeff Hausdorff, Thurmon Lockhart, Didier Dilignieres, and many others. After a thorough review of this literature the authors may realize that some of their results are neither new nor novel. If indeed the authors were unaware of this entire section of literature so relevant to their own research, it would be an indictment in its own right.

It needs to be emphasized explicitly, that the above does not call into question the validity of the research results, nor does it imply that the described results do not merit publication. Clearly, the research described in Wu et al. is relevant, interesting and it was absolutely correct to publish it in its entirety. What ought to have happened, however, is to encourage the authors to include the relevant references in the appropriate sections of their articles.

(Much of this comment was drafted together with Dr. Nick Stergiou)

On 2014 Apr 02, Daniel J Simons commented:

I wrote an extensive post-publication "HI-BAR" review of this paper on my blog (stands for Had I Been A Reviewer). You can access it at http://blog.dansimons.com/2014/04/hi-bar-benefits-of-lumosity-training.html

I posted a list of my concerns about the paper as a comment on the article at PLoS, and I've duplicated that list below. The blog post gives a more detailed discussion and explanation of each point. If the authors respond on PLoS, I'll update my comment and add a link to their response here as well.

In short, I do not think the paper permits the conclusion that game training produced any reliable benefits on the reported outcome measure.

List of questions and concerns:

1) The sample size of 15 in the training group and 12 in the control group is problematically small, especially for correlational analysis, but also for the primary analyses.

2) The "limited-contact" control group does not permit an inference that anything specific to the training led to the transfer effects. See http://pps.sagepub.com/content/8/4/445.full

3) The paper includes no corrections for multiple tests, and the core findings likely would not be significant with correction.

4) The paper does not report the means and variability for the accuracy data, leaving open the possibility of a speed-accuracy tradeoff.

5) The choice of response time cutoffs and exclusions were somewhat arbitrary, so it's not clear how robust these effects would be to other cutoffs.

6) The contrasts used to measure alertness and distraction were not defined. Which conditions were compared?

7) The alertness and distraction tests do not include a test of the difference between the training and control group. The fact that the training group difference was significant (but see below) and the control group difference was not does not mean that the difference between the groups was significant.

8) The training improvements for the alertness and distraction outcome measures were reported to be p=.05 and p=.04. But, they were truncated from p=.0565 and p=.0451. The first was not significant, and truncating the p-values is inappropriate. (Note that neither would be significant after correcting for multiple tests.)

9) The paper reports 20 correlations (each outcome measure with each of the 10 games in the training condition), but does not correct for multiple tests. And, correlations based on N=15 are of questionable reliability anyway. Moreover, correlations between training improvements and improvements on an outcome measure do not provide evidence for the efficacy of training.

10) The conclusion claims support for the idea that training improved "attention filtering," but the study does not test the mechanism that improved (and, the evidence that anything improved is uncertain).

11) The clinicaltrials.gov registration linked from the paper was posted after the paper was first submitted for publication. It is not a pre-registration.

12) The clinicaltrials.gov registration mentions a number of outcome measures that were not reported in the paper and were not mentioned on the PLoS Protocol and Consort Checklist (in the supplementary materials). If these measures were collected, they should be reported in the paper and in the supplemental materials. It is unclear whether these outcome measures just were not significant or were withheld for other reasons. In either case, the presence of unreported outcome measures makes it impossible to interpret the p-values for the one outcome measure reported in the paper.

13) The clinicaltrials.gov registration also lists a 24-week testing session that wasn't mentioned in the paper. Was the reported testing session an interim one?

On 2014 Aug 22, Serge Ahmed commented:

This study is interesting and also quite embarrassing. It is interesting because of the important questions that it asks. It is embarrassing because it shows that when given a choice, most rats prefer cocaine over sweet water – a finding that is strictly the opposite of what we and others have found over the past few years. Of course, contradiction and refutation are the “game of science”. We should not be embarrassed by them and instead welcome them.

My embarrassment comes from the fact that these opposite outcomes were obtained by a former master student of mine – Nathalie Vanhille who is the first author of this study – using a choice protocol initially developed in our lab. When Nathalie was working in our lab using this protocol, she observed that most rats preferred sweet water over cocaine – the opposite of what she now reports in this study despite the use of an identical choice protocol.

But were the choice protocols really identical? Of course not! Like always, the devil lurks into the details and details can sometimes matter a lot! Apparently, this study differs from our previous choice studies in the way rats were given access to sweet water. In our study, access to sweet water was pretty straightforward. Rats had to press a lever to fill a nearby receptacle with sweet water. Then they could obtain additional volumes of sweet water during 20s by continually licking the receptacle. In this study, however, access to sweet water was really contrived for reasons that remain unclear until one reaches the middle of the Discussion. In fact, despite my best efforts and those of other members of the team, we were unable to get a clear final picture of how rats get access to sweet water in this study.

So here is my challenge for the interested readers and researchers: I would really appreciate if someone could help me figure out how exactly rats get access to sweet water in this study.

Thank you!

On 2014 Aug 22, Hilda Bastian commented:

It's great to see such a thorough and rigorous body of work on this subject. This group provides a good overview of the portion downsizing issue, and the limited evidence base on interventions, at Vermeer WM, 2014.

A key part of the intervention in this trial (Poelman MP, 2015) is the interactive web-based PortionSize@warenessTool. Its development and trialing is described at Poelman MP, 2013, with these elements: background reading, an interactive flash game with photos of popular food products in the Netherlands, a flash game where you can upsize/downsize portions on screen, self-test score, information on portions for children and more.

It would be helpful if details about the availability of this intervention could be provided (e.g. where it can be viewed, if the code is open source, and if the license allows translation). The TIDieR checklist (Hoffmann TC, 2014) - the template for intervention description and replication - is a good framework for this. More details on the components of interventions is important for enabling better practice (Glasziou P, 2010).

On 2014 Nov 20, Bernard Carroll commented:

This report breaks some new ground in design – the between-site train-and test exercise and the cross-validation of case assignment to unipolar or bipolar groups. As the authors stated, the data are not sufficiently strong for clinical use. My comments are intended to improve the quality of reporting of this and similar studies.

1. There is selective highlighting of some results and a failure to present all the important findings clearly. In particular, the performance of the classification algorithms in distinguishing patients from normal control subjects was relegated to the Supplementary Material. It can be calculated from eTable 7 that between 27.5% and 38% of controls would be misclassified in the 1-way comparison with unipolar depressed cases. The corresponding Kappa coefficients of concordance would be fair at 0.48 for the SVM method and poor at 0.28 for the GPC method. Results for the bipolar contrast with control subjects were similarly weak. If the method cannot do better than this with normal subjects then clinical use is a very long way away. These sobering data properly belong in the main body of the paper.

2. The cross-site training - testing results for the algorithms were described as “highly significant” (page 1226). Actually, for the train (Munster) and test (Pittsburgh) exercise the Kappa values that can be calculated from Table 3 were weak at 0.28 for SVM and 0.24 for GPC methods. They were only slightly better for the Pittsburgh – Munster exercise (Kappas each 0.38).

3. P-values were given in Table 3 and in eTables 5,6,7 – but there is no statement of what statistical analyses generated these P-values. Were they Goodness of Fit Chi-squared tests? Standard tradecraft requires that such analyses be clearly described.

4. No correction of P-values was made for multiple comparisons. That is another aspect of standard tradecraft.

5. No data were shown for test-retest reliability of the algorithm-derived group assignments.

6. All the analyses were predicated on the untenable assumption that the clinical diagnoses were 100% accurate. As the DSM-5 field trials taught us, that is far from the case in the real world of clinical assessment – the Kappa value for major depressive disorder diagnoses averaged over 4 sites was poor at 0.28 (Regier et al 2013). The authors failed to consider whether this confound degraded the strength of their findings (see a discussion of this issue in Carroll BJ 1989). At the very least, a statement of diagnostic reliability for the cases in this study is needed.

References

Carroll BJ. Diagnostic validity and laboratory studies. Rules of the game.<br> In: The Validity of Psychiatric Diagnosis, eds., L.N. Robins and J.E. Barrett, Raven Press, New York, 1989, pp. 229-245.

Regier DA et al. DSM-5 Field Trials in the United States and Canada, Part II: Test-Retest Reliability of Selected Categorical Diagnoses. Amer J Psychiatry 2013; 170: 59-70.

On 2016 May 17, Lydia Maniatis commented:

(My two PubPeer comments)

Here's the funny thing about vision science these days.

We are told about a prolific area of research - 'visual averaging': “New research tools and imaging techniques coupled with solid psychophysical work have added substantially to the large base of work done in the 20th century.”

Then we learn (caps mine) that: “Some lively debates regarding the scope of the putative sampler/averager and EVEN ITS EXISTENCE have sharpened the questions being asked and reminded researchers to consider scaling issues, experimenter and observer bias, and the multidimensional nature of stimuli and ensembles.”

It is the nature of a dogma and the school that collects around it to act on the basis of assumptions that are never challenged. Research projects are designed so that they cannot challenge the dogma, but only answer questions in its terms. (That something like this is going on here is evidenced by the author's concession that claims of possible non-existence of “averagers” remain viable despite decades of research activity.) It's the nature of science to test its fundamental assumptions before proceeding to elaborate on them. If a claim is falsifiable, in a healthy research environment it will be quickly falsified. (If it cannot be tested even in principle, then it's outside the game of science and cannot constitute a legitimate field of empirical investigation). In the type of environment we have now, it will remain in good standing for the foreseeable future. (This is the case, for example, with the bizarre notion of “spatial frequency filters.”)

On some of the absurdities coming out of “averaging” proponents, see comments here: https://pubpeer.com/publications/90941136CC181AFE4896477BF5BB44 and here:

https://pubpeer.com/publications/26067519

The faith-based tendency to accumulate evidence (supposedly) in favour of the dogma is related to this field's (invalid) adoption of inductive procedures rather than hypothesis-testing. If we just keep measuring things and fitting algorithms to the data, the truth will naturally emerge:

"The psychophysical function and the environmental information on which it is based should be allowed to emerge without predilection, lest the true mental algorithm be obscured by expectation (Anderson, 1968; Levin, 1975).... It is well known that psychological measurement is not impervious to the effects of context, scaling, measurement error, and other issues that can hinder the revelation of the true psychophysical relationship." Expectations, aka hypotheses, should stay well away lest they interfere with revelation.

Despite decades of patient effort and mountains of "evidence," we're still waiting for the true psychophysical function to be revealed. What if it doesn't exist? What if it can never be spotted amongst the confounds, known and unknown?

As we wait, we might ask what it means that "perceptual averages" have never been perceived? In what sense is an unperceived percept perceptible? (If averages were perceptible, there would be no question of their existence; we would be trying to explain perceptual facts rather than waiting for these facts to be revealed.)