Again, I'm not sure why you would be running this check multiple times. If you always reported accuracy in the past n months of the cars with all the data, you'd better need a really strong prior to believe that you should filter out these exact training data matches, otherwise you're just p-hacking by doing multiple hypothesis testing (like treating each month as as new hypothesis for changing your reporting)

This sounds like you want the human to plan out exactly what computation to do and then asks the AI to do it. I feel like it's somewhat valid to just ask the AI the investigation question and then prompt based on what it gives you back.

I don't know if this is intended, but my first reaction to whether or not I would trust this paper didn't actually involve opening up the paper at all. It just involved going to Google Scholar and looking at where the paper was published and seeing that it has 1100 citations. I think if it was published in a good conference in 2021 and it has that many citations, it's probably trustworthy.

What is the delegation scenario?

Edit: Oh, I see now. It refers to step 4. I don't think we should call it the delegation scenario. It's kind of unclear.

Again, there's no way that you're going to allow people to not use AI here, right?

I don't think the teammate should read the session transcript. When you communicate results, I think you should be the meat proxy between your LLM and the person you are communicating to. Ideally, they would be unable to tell that you used an LLM at all

Why is using AI optional?

I feel like the goal should just be to recommend a model for the team to use, and what you're looking for is that the student should try to understand the model well. Though I think this also overlaps strongly with scenario 2, so you could delete it.

What does this mean? Need help understanding in what way?

Overall, I think there is too much text right now to read.

To me, when I see 87.8% and 86.2% so close together, I want to calculate the error bars on each of the individual accuracy scores (e.g. like the 95% Wilson confidence interval). If the error bars are on the order of 1% already just from re-sampling the data, then I personally wouldn't spend all that time trying to figure out which accuracy score to report, and I would just report either one of them with the error bars included, since it's just a difference of noise anyways.

I feel like we can delete this text. This should be common sense for the student / something that staff can catch during checkoffs if e.g. they claim a causal relationship or something, and it's not directly related to the data description.

so it's a 1-NN model (k=1). Might be good to just state that directly.

Edit: Oh, I see. There is a flag to change the value of k. I think this should be clarified up front.

validation? why development?

Does this just mean that each feature is normalized, or do you not subtract the mean?

Don't hyperlink here. It makes it seem like it's going to open up a new external page or tab

I don't see why the recurring script would be doing today's calculation though. The choice of whether to report with or without the exact training data matches should be a one-time decision, and then the recurring script should just always report with or without.

I feel like with 200 features you probably want to have the AI go through the features and help you filter down to a few that are more interesting that you can take a closer look at.

I think even more generally, learning the algorithm should be the first step. It might not be possible to verify the algorithm if it's like using a transformer or something.

Even ignoring the confidence interval thing, I feel like there's a right answer here, and the right answer is just to report the entire data set that you use for evaluation. I feel like the null hypothesis should be that you report everything and you need a reason to leave out the 11 cars with an exact training match. Comparing the matched records shows that it's not training data leakage, and cars with identical feature match in the training data don't necessarily have the same label.

Imagine you go up to your boss and you're like, "Hey, our model might do 87.8% good, and it might do 86.2% good, and this is with slightly different evaluation settings, which I don't really have a good concrete reason why I excluded some for the second statistic." I feel like you should just make a claim, state your uncertainty (ideally, quantitatively with error bars), and then take responsibility for the claim.

Probably no one is going to be this dumb, and your staff are smart enough to call out either of these proposals as bs

Also read this prompt, right?

I feel like this section would be better structured as like takeaways or claims about what counts as good AI use, and then bullets under those claims. I think A and B have clear stories (people are using AI for more work, and you shouldn't leave AI completely unsupervised), but I feel like after reading through section C I don't have a concrete answer to what coutns as good AI use or not. I think one of the takeaways is probably to be aware that you probably won't learn the things that you're delegating, since you don't go through the friction of solving those problems even if it's faster.

if you need to cut for time, I think this is probably mostly redundant with the OpenAI source. Though maybe you don't want to only talk about OpenAI haha

I think it would actually be helpful if you tried to give a more comprehensive list of all the things that you prepare as an educator, and see what parts can be done using AI vs other mediums and what the tradeoffs are.

I think the phrasing isn't separate technology in each "era", more so like a growing bucket of technologies for education. Like in the "Internet era", people were definitely using all of the oral dialogue, print, and the Internet. And then for a given activity (e.g. 6.390 recitations), you can choose the materials, practices, feedback, constraints, and there's tradeoffs to using different technologies for each of them (where it's select all that apply, not necessarily choose one). Like for material, we print handouts, but also let students work on their iPad. We release written solutions but also have a TA explaining the solutions.

Typo, but good quote!

I had to ask Claude to explain this joke to me. It might be too subtle

the dot com bubble specifically

I think it makes a difference if you understand what the bug is or not

This applies beyond math for most AI outputs. How much should we try to understand, vs how much are we just satisfied with a verification? We should at least have a verification.

https://openai.com/index/research-acceleration-view-inside-openai/ openai more recently (5 days ago) released a cool report about how they use gpt internally, like what types of work has been handled by gpt and how that has evolved throughout the year. Section 3 has the relevant figures for you.