I wanted to understand this better. Is she saying that responding to ChatGPT seems urgent but that more fundamental and familiar questions about our teaching are actually more important? I have sometimes been concerned about my own focus on ChatGPT and whether it is distracting me from other things I could focus on that might improve my teaching more.

I don't remember how I put this but this phrase seems so broad--we wouldn't all agree on adapting teaching, but we might all agree that we need to make explicit policies about AI.

Maybe... but these should be carefully crafted too.

Thank you to Marc Watkins for his leadership in pointing this out! I had not seen this clearly before I started reading his tweets on it.

That's true--it hadn't occurred to me that instructors would play gotcha in that way. I suppose I'm naive. I would expect teachers to want to share with students ahead of time to maximize the chance that the students will learn more and do well rather than maximizing the chance of catching cheating.

Banning ChatGPT use in learning is not the same as ignoring it. Teachers could very well teach about the tool and still ask students not to use it when completing assignments if they think its use will interfere with valuable learning. Even if students may use tools like this in the workplace, they may still need to practice without them to learn.

Great point.

Not for every learning activity--we don't "embrace" them in kindergarten, for example.

I can see how this might help students extend their thinking.

I wonder what makes Onyper think students are learning these 'basic building blocks'--ChatGPT can produce them, but what is going on in the student's mind when they see what it produces? Reading a sample essay doesn't teach us to write...

Doesn't this imply that the purpose of the assignment is to produce a high quality product rather than the purpose being the student's learning?

Twilight Zone ending. Come on! Don't end by spooking us about what's unknown. Fulfill the promise of the title and show how specific kinds of prompting produce disturbing outputs!

This puts the agency back with the LLM as if human prompters are helpless before LLM seduction.

No, we're not helpless, and LLMs are not actually actively coaxing us. If we start to see odd outputs, we could look back and reflect on our prompts and any unintended linguistic signals we may have sent.

Very misleading. Humans and LLMs do not have similar cognition. They cannot have a common conceptual state. Their text sequences may come to have certain similarities.

Not something mystical again! Please! Really? A magical object from Harry Potter?

Why not just mention the concept of projection?

This amounts to saying "I believe in the good intentions and sincerity of Microsoft and OpenAI's explanations of their decisions."

Beloved New York Times, why are you not asking the basic questions of why they would need to release the bots to test them? Why not test them first? It's ludicrous to say they can't imagine what the public might do.

And what about their economic motivations to release early and get free crowdsourced testing?

This is too psychological a description of what's happening. The LLM doesn't have a psychology. We need to think in terms of the genre and style of word sequences.

I was so glad to read this! I wish the article had continued from here to show how the style of certain prompts made the "creepy" outputs more likely. This would be a matter of showing similarities in rhetorical styles or genre of the prompts and outputs.

Too mystical a formulation. Not accurate. They are not providing what we desire but predicting text based on statistical associations with the word sequences we provide. Sometimes we are not aware of all the associations our words call up. These may or may not align with desires we are not aware of. But Sejnowski's phrasing implies that these systems are able to know and intentionally respond to our psyches.

This basically creates a sense of mystery without telling us much, implying that there is something spooky going on, something beyond what computer science can explain. Actually it's quite explainable as the article title implies. People start writing prompts in a certain genre and the completion follows the genre...

I was glad to see this as a fair assessment of what happened with Kevin Roose's famous conversation with Sydney/Bing. See the annotation conversation on his first article.

This implies that solutions are inevitable--it's just a question of how fast. Why would we assume this?

Yes, but an explanation is needed.

I don't see what she's saying or how this explains anything. Are they saying here that the prompt also affects the output?

This seems unlikely. And leaving this statement without explanation is another move to add to the Halloween vibe.

I would agree with this, given the transcript of Kevin Roose's conversation with Bing/Sydney. Each time the system went off the rails there was an antecedent in his prompting.

"exploring ways of controlling the behavior of their bots" is not at all reassuring to me.

Thank you!

Not at all! He had asked it to share a secret it had never told anyone. What it spits out is a pretty good guess for what a human already engaged in a very intimate conversation might share when prompted this way.

Surely if we look at the set of secrets commonly revealed in intimate conversations, being in love with the other person would be common.

So misleading! He asked it to tell him a secret. Then it said it wanted to. The first secret it told was the one he had already revealed he knew and considered a secret earlier in the conversation.

The phrasing "got to know each other" again implies there is a real consistent personality to Sydney, and using the word "fantasy" implies there is an imaginative experience it is having.

Describing it in this way strongly implies that there may be some internal experience based on the particular situation of the language model.

This suggests that what emerged for him is a personality that is intrinsic and will be consistent to what other people will see emerge if they focus on personal topics. Isn't it more likely that each chat session's "personality" will depend on the style of prompting?

This suggests that his takeaway matches his night fears more than his "light of day" comments later.

Wait a minute! To so glibly liken AI models' references to emotions to humans misrepresenting what they feel takes us backwards, suggesting again that AI words about emotion actually could refer to real feelings experienced by some being.

This vague, dramatic ending certainly constitutes irresponsible AI hype! Crossed a threshold? We already know about the tendency to project on these systems since the Eliza effect. They are getting more linguistically sophisticated, so this effect will be more pronounced and dangerous. He isn't doing much to clarify that or explore how we could prevent harms.

Yes, exactly. But the earlier parts of the article strongly suggested otherwise and played on our projections. He doesn't do enough to acknowledge the dynamics of his own earlier response and how they misled him.

Rouse implies that everything he was inputting would steer it away from such responses, but I doubt that. There are plenty of conversations in the training set surely that show a kind of push and pull. Just because one interlocutor tries to steer the conversation doesn't make it improbable that the other interlocutor might return to a previous topic.

By switching to "Sydney" here, Rouse endorses the idea that this is a coherent personality secretly embedded in the system prior to his prompts.

But at that point Rouse had already asked it if its name was Sydney and it responded "How do you know that?"

By reassuring us here, he plays on people's fear and misunderstanding of what it means when this kind of text comes out of a machine. He should clarify that text referring to intentions coming out of a machine does not mean the machine has intentions. As one engineer put it on Twitter, we can write code to print these words.

Key here is that Rouse is introducing and explaining this concept, which surely corresponds to plenty of text in the training set. This leads to the probability of responses that illustrate these concepts.

Really don't see why not. Can't the testers ask it deep questions and push it this way as well?

How are LLMs ever connected to "grounded reality?"

Does that mean that any California resident can email to request a record of all the information OpenAI has collected about them?

This would include Microsoft.

Does that mean that we are agreeing for them to use personal information in any way they choose if they deem it to help them improve their software?

Thank you! So beautifully and simply put ChatGPT is also used mostly for tasks where there is no one clear right answer.

See Responding to an Argument from the OER text How Arguments Work.

See Reflect on an Argument's Strengths from the OER text How Arguments Work.

See Check the Argument's Assumptions from the OER text How Arguments Work.

See Check How Well the Argument Addresses Counterarguments from the OER text How Arguments Work.

See 4.4: Decide How Strong the Evidence Is from the OER text How Arguments Work.

This question prompts for the kinds of argument critiques discussed in 4.3: Look for Exceptions from the OER text How Arguments Work.

This question prompts for the kinds of argument critiques discussed in 4.2: Check If the Meaning Is Clear of the OER text How Arguments Work.

This implies that it should take a stand but doesn't say why. Note that the New York Times article is not an editorial.

This summary also misses the stand implied by the choice to end on a quote from scholar Crystal Fleming explaining why she capitalizes "Black":

"'Frankly,” she said, “because I want to. That’s also something that I think we need to see more of in every field — black people doing what we want to do.'"

Why? What value do these people see in the capitalization? This summary ignores the idea of power, which is central in the article. Eligon writes that capitalizing "Black" "confers a sense of power and respect to black people, who have often been relegated to the lowest rungs of society through racist systems, black scholars say."

Here is common academic phrasing to contrast ideas: "some scholars argue...while others believe that..."

However, it's not a great choice to use this contrast phrasing for the particular ideas described here. The previous sentence used "while" to contrast one view in favor of "Black" with a view against it. But the two parts of this particular sentence are not opposing views. The first idea is against capitalization, whereas the second idea doesn't take a stance on "Black" vs. "black" but rather is weighing in on what we should do about terminology for whiteness if we do capitalize "Black."

This is an example of how ChatGPT uses academic phrasing in ways that read as natural and even sound intelligent, but on closer examination remind us that ChatGPT has no understanding of the ideas.

Here's a sample ChatGPT critical assessment of the NY Times article at https://www.nytimes.com/2020/06/26/us/black-african-american-style-debate.html

For contrast, see this human-written sample essay from the textbook How Arguments Work: A Guide to Writing and Analyzing Texts in College: https://human.libretexts.org/Bookshelves/Composition/Advanced_Composition/Book%3A_How_Arguments_Work_-A_Guide_to_Writing_and_Analyzing_Texts_in_College(Mills)/04%3A_Assessing_the_Strength_of_an_Argument/4.11%3A_Sample_Assessment_Essays/4.11.02%3A_Sample_Assessment-_Typography_and_Identity

I'm concerned people will see this and remember it without thinking of all the errors that are described later on in this document.

Glad they are addressing this, and I hope they will continue to offer such suggestions. I don't think ChatGPT should be classed as a journal. We really need a new way to acknowledge its use that doesn't imply that it was written with intention or that a person stands behind what it says.

Since there is a concern about the bias of the tool toward English and developed nations, it would be great if they could include global educators from the start.

I wonder how this information will be shared back so that other educators can benefit from it. I maintain a resource list for educators at https://wac.colostate.edu/repository/collections/ai-text-generators-and-teaching-writing-starting-points-for-inquiry/

It's still not clear to me how they can be used as evidence at of academic dishonesty at all, even in combination with other factors, when they have so many false positives and false negatives. I can see them used to initiate a conversation with a student and possibly a rewrite of a paper. This is tricky.

I agree, though I think we should emphasize teaching about the limitations before teaching how to use the models. Critical AI literacy must become part of digital literacy.

Glad they mention this since I am skeptical of claims that students need to learn prompt engineering. The rhetorical skills I use to prompt ChatGPT are mainly learned by writing and editing without it.

If we teach about ChatGPT, we might do well to showcase examples of these kinds of problems in output so that students develop an eye for them and an intuitive understanding that the model isn't thinking or reasoning or checking what it says.

This is a bigger problem when we use ChatGPT in education than in other arenas because students are coming in without expertise, seeking to learn from experts. They are especially susceptible to considering plausible ChatGPT outputs to be authoritative.

Try PerplexityAI for an example of this. Google's Bard should be another example when released.

Glad they mention this in the first line. People will see the various safeguards and assume that ChatGPT is safe because work has been done on this, but there are so many ways these biases can still surface, and since they are baked into the training data, there's not much prospect of eliminating them.

This is a central idea that I would wish were foregrounded. If we are trying to use auto-generated text in a situation in with truth matters, we need to be quite knowledgeable and also invest time in evaluating what that text says. Sometimes that takes more time than writing something ourselves.

It strikes me that OpenAI itself is warning of a coming flood of misinformation from language models. I'm glad they are doing so, and I hope they keep investing in improving their AI text classifier so we have some ways to distinguish human writing from machine-generated text.

Yes! We must begin to cultivate an ethic of transparency around synthetic text. We can acknowledge to students that we might sometimes be tempted to autogenerate a document and not acknowledge the role of ChatGPT (I have certainly felt this temptation).

This would be wonderful. Currently we can use the ShareGPT extension for this.

I appreciate the open and humble tone here and the invitation to further discussion.

I appreciate these cautions, but I'm still concerned that by foregrounding the bulleted list of enticing possibilities, this document will mainly have the effect of encouraging experimentation with only lip service to the cautions.

I'm concerned that any use of ChatGPT for tutoring would fall under the "overreliance" category as defined below. Students who need tutoring do not usually have the expertise or the time to critically assess or double check everything the tutor tells them. ChatGPT already comes off as more authoritative than it is. It will come across as even more authoritative if teachers are recommending it as a tutor.