 Jan 2021

andymatuschak.org andymatuschak.org

You’ll need to break knowledge down into its discrete components so that you can build those pieces back up as prompts for retrieval practice.
For carefully written manuscripts, authors will have done this in the writing process already. Instructional designers, for instance, routinely do this and it becomes habit.

 Jul 2019

wesharescience.com wesharescience.com

.616*
Correlation between PreClass Engagement and InClass Engagement. In other words, those who were engaged PreClass were also typically engaged InClass. For every 1 point increase in "engaged PreClass" there was a 0.616 increase in "engaged InClass".
Here is a short review on interpreting correlations (sorry about the ads on the page): https://www.dummies.com/education/math/statistics/howtointerpretacorrelationcoefficientr/

M (SD)
Means (M) and Standard Deviations (SD)

p<.01
Reminder on interpreting the pvalue:

70
Though statistically significant, a correlation of 0.27 is fairly low and may not represent any practical significance.

r
r = correlation
Correlations go between 1 and 1, with 0 being no correlation.

Table 2:
Here is a good article describing how to read a regression table: https://www.statology.org/howtoreadandinterpretaregressiontable/

Underlying assumptions of the three statistical analyses were verified with ShapiroWilk’s test, Levene’s test, the VIF and tolerance values, DubinWatson statistics, the scatter plot of the values of the residuals against the values of the outcomes predicted, and the histogram and normal probability, from which the assumptions of multicollinearity, linearity, homoscedasticity, and independent residuals were confirmed
These are just multiple ways that researchers can confirm that their data meets the underlying assumptions required in order to do the regressions. Here is a good overview of the basic assumptions: https://pareonline.net/getvn.asp?n=2&v=8

χ2
Chi Square

goodnessoffit indices
Here is a good video describing Pearson Chi Square, which is one of the most common "goodnessoffit indices" and it thereby explains the basic concept:

maximum likelihood estimation.
Video on MLE:
https://www.youtube.com/watch?v=XepXtl9YKwc
And for those who want to get a little more technical:

To explore the path from influencing factors to FL outcomes
Refer back to Figure 1 to visualize the paths (the arrows between the boxes). This is a fairly simple path model and it is easy to interpret, but they can get more complicated with more arrows going in different directions between variables.

effect sizes
Video on effective sizes (including Cohen's recommendations):

hierarchical multiple regression
Here is a video overview of multiple regression. It builds on the previous video on simple linear regression (start with this video if regression is a new topic for you):
Multiple Regression: https://www.youtube.com/watch?v=dQNpSabq4M
Linear Regression: https://www.youtube.com/watch?v=ZkjP5RJLQF4

analyses of variance (ANOVA
Video on what is ANOVA:

bivariate correlation
A twovariable correlation
https://www.statisticshowto.datasciencecentral.com/bivariateanalysis/

participants’ perceived growth
The extent to which we believe that students can make accurate estimates about their growth from the beginning of the semester to the end of the semester should be considered as we later look at the data on this variable.

and
Again, this would better as two questions.

lectures and materials
Typically researchers will not use "and" in a question since it indicates that you are asking two questions. For example, if a participant here used the preclasses lectures but not the other materials, should they mark yes or no. It would be better asked as two questions (one for materials and one for lectures).

selectively adopted items from a questionnaire developed by Hofer and Pintrich (1997) and translated and validated by Cho (2010)
While it is somewhat common for researchers to select only a subset of questions from a validated instrument, it should be noted that the instrument was only valid in the form that was validated. Variation on the validated instrument may or may not be valid, and a separate study should be done to validate the subset of items to see if they are still valid (and reliable). In this study they do provide the "internal consistency" scores for the items as used, but this is more about reliability (i.e., does it consistently measure what you want to measure) and not as much about validity (i.e., is it accurately measuring the desired construct).

elected and modified item
Note again, they modified an instrument.

Cronbach’s α =
Cronbach's alpha = measure of internal consistency of items

three subconstructs
It is not clear why this is not labeled with the type of coefficient (number) that it is. It is likely the Cronbach's alpha (the most common measure for internal consistency) but later on the page they describe another coefficient as the Cronbach's alpha, thus we are left to wonder if these are or are not similar.

0
This looks to be an editing error since the value should between 0 and 1 (e.g., 0.7).

internal consistency reliabilities
Video on internal consistency:

perceived learning outcomes
Note that this is not their actual learning outcome, but what they thought that they would get.

At the end of the semester
All data was collected at the end of the semester from the students. This is not necessarily a problem, but when reading the results we should keep in mind that this captures where they are at the end of the semester and not at the beginning (for example, questions about their engagement would be attached to the engagement at the end of the semester and not at the beginning or middle of the semester).

0 to 60 minutes to learn.
It appears that this is an estimate based on instructor experience rather than data actually collected from the students. Given that the data on student use would be in Blackboard (the LMS) it would have been good for them to include the actual time students spent preclass engaging with the materials.

two 15week courses
A physics and a chemistry class were used in the study, both used a FL approach. There was no comparison group of students (ideally randomly assigned) taking the same courses but using a FL approach.

research questions
Research questions are an alternative to making hypotheses, though based on the research literature generally researchers should make testable hypotheses. Typically they will leave it at research questions if (a) there is not enough supporting past research to make a good estimate of what is expected, or (b) they haven't done enough background research and therefore can't make a good estimate of what is expected.

Figure 1:
Figures are often quite useful for understanding what the researcher is doing or proposing, so take time to study them.

Even though learner engagement typically comprises cognitive engagement, behavioral engagement, and affective engagement (Archambault, Janosz, Fallu, & Pagani, 2009; Fredricks, Blumenfeld, Friedel, & Paris, 2005; Fredricks & McColskey, 2012), in a context such as FL, engagement should consider the student’s initial commitment to an online preclass learning mode (hereafter, preclass engagement) and subsequent commitment to a F2F inclass learning mode (hereafter, inclass engagement).
This is an important statement since the authors describe how they are going to define "engagement" within the context of this study.

pedagogy
Is FL a pedagogy or tool? You can decide:

epistemological
Here is a video introducing what is "epistemology":

identified diverse outcomes associated with FL course
Again, though technically accurate this may be a little too strong of a statement given the mixed results of FL.

Studies have documented
As mentioned in Molnar (2017), these finding have actually been quite inconsistent. So while it is true that some studies have documented improvements, others have not.

(Molnar, 2017
This is a good example of why it is important to look at the references linked to citations. Molnar's study was not a broad examination of the "numerous reports of improved academic performance and enhanced learning". Rather Molnar actually looked mostly at student perceptions of FL learning. Molnar did include some information on grades, but only for part of the sample and with limited control conditions. So while Molnar's article may generally support that some research has shown positive impacts, it balances that with "Performance measures between traditional and flipped classrooms have also generated inconsistent findings." In the end, after looking at Molnar, I do not think that I agree with the author that Molnar's article is a citation for this statement (not that the statement may not be true, but this conclusion was not the results of Molnar's research).

Practitioner Notes
This section is a standard element for the British Journal of Educational Technology, but it is not common to most research journals. Since the statements do not include citations, you will want to cross check them with statements in the article to determine if you agree (through the citations) that the statements are warranted.


wesharescience.com wesharescience.com

partialη2
n squares (or eta squared) is a statistical tool for examining the effect size of a result. The effect size is an indicator to readers distinguish between results that are statistically significant and those that are practically significant, though in the end the practical significance (or utility) is left to the interpretation of the reader. For example, a difference with a substantial effect size may have practical significance for teachers in urban schools, but at the same time have little practical significance to those in rural schools, based on other characteristics of the study (such a participant profiles, the technologies used, capacity of internet access, etc.).
This video gives a good explanation. https://www.youtube.com/watch?v=RkbmA6WszTo

Chiu
Depending on the topic and discipline, selfcitation (i.e., referencing your own work) can be acceptable. But readers would want to note if the author(s) reference primarily their own work since this could be an indicator of potential bias in the article. Though in some cases, very few researchers are working on a given topic and there are not many others to cite... thus there is no clear "rule" on how much selfcitation is acceptable.

References
Reference sections are very important to science. Not only do they allow readers the opportunity to review (and interpret for themselves) the research being cited by the authors, but they also give readers a quick way to discover what influential articles are guiding the researchers. Many seasoned readers of research will start by skimming the References before going back to read the article  as a preview of coming attractions.

University of Hong Kong
Researchers should also acknowledge who is paying for their research. It is not that research paid for by group's with special interest is "bad" research, but readers should be aware of any potential connections (similar to the conflict of interest statement above).

conflict of interest
In this section the authors should indicate if there are conflicts that might influence how people interpret their research (for example, if research on lung cancer is being done by researchers who are owners of a vape company).

limitations
Researcher will commonly identify limitations to their research and these are important for readers since they may influence how you interpret the findings and implications.

motivate all teachers to integrate mobile devices intotheir teaching practice
This statement gets back to why it would be important for the researchers to monitor and measure the "use" of the mobile technologies of individual teachers in this research.

**
Be sure to note the legend below the table for the meaning of , , and . These are rather routine and typical, but sometimes authors will use different pvalues.

significant
More accurately "...no statistically significant difference..."

p= .01
Reminder video on how to interpret the pvalue

F(1, 28) = 7.15
This video does a good job of describing how ANOVA calculations are made, and how the F test is calculated (including what the 1 and 28 would represent from this example).

before (M= 3.03, SD = 0.74) and after(M= 2.74, SD = 0.59
ANOVA are statistical tests for differences in variation between groups, or in this case preintervention measures and postintervention measures. In other words, rather than just comparing the mean difference (3.03 vs 2.74), the ANOVA uses the variation (from the mean) for individual scores.

0.70
With the posttest, we see that the math and science teachers are more in agreement (i.e., less variation and a lower standard deviation) than they were at the time of the pretest (i.e., dropping from .98 to .70).

0.98
The "perceived ease of use" has a greater standard deviation than most of the other scores, and this illustrates that for math and science teachers there was greater variation in how easy they found the mobile devices to use (i.e., more spread in their individual scores).

SD
SD = Standard Deviation
As a reminder, here is a short introduction to SD.

adoption
As with "use" above, "adoption" here is not defined and no data is provided on actual teacher use. Since there is no control group (i.e., without mobile devices) we don't really know with that data if levels of adoption influenced levels of anxiety, or if some other variables were more influential on levels of anxiety.

used
The "use" of the mobile devices is essential to this study and yet it is not "operationally defined". In other words, you want the researcher to define how they operationalize the key variables in their study. In this case, how will know if teachers actually used the mobile devices? How will they measure this? How will discern differences? For example, one teacher may use the devices on a daily basis and another may use them once a week  are these measured and considered equivalent (i.e., do both cases fall into the definition that the teachers "used" the mobile devices)? This would be important to our interpretation of the research findings.

teachers received training workshops
It would also be useful for the researchers to include what was taught in these workshops, number of hours of training, and participation rates among teachers. Since the study is quasiexperimental (i.e., no control group) then as readers we would benefit from additional details on what interventions the teachers participated in during the study.

got consent.
It is not clear if the research got consent of the principal and the teachers, or just the principal. Typically, all participants have to give consent to participate in a study.
You can watch this for more information on why informed consent is important to ethical research: https://www.youtube.com/watch?v=BXQHDCWStQ

modified
It is important to note if the researchers made any changes to the instruments that they are using in the study. It is common for researchers to make minimal changes in order for the questions to make sense to the participants, but it is critical that these do not go so far as to change the validity or reliability of the instrument. Unfortunately, researchers do often go to far and in this case they used a mix of instruments together. This is not necessarily "wrong", but as the reader we do have to question if this has influenced the validity and reliability of the original instruments  thus compromising the results of the current study. There is typically not clear answer to this question, as in the current paper. But as the reader you should be aware of the potential problems that may stem from altering an instrument from its original state (i.e., the state was tested for validity and reliability).
Here is an overview on validity and reliability of instruments: https://www.youtube.com/watch?v=O4FvBW4Siw

Likerttype questions
Likerttype scales are those that you are commonly used to completing where 1 might indicate that you disagree and a 5 might indicate that you disagree. There is an interesting history to these scales and you can read more about them at on Wikipedia  https://en.wikipedia.org/wiki/Likert_scale

n= 29
"n" refers to the sample size. In this case there were 29 language and humanities teachers among the 62 total teachers in the study.

re and postquestionnaires
Pre  Post designs are common in research, with the Pre being a measure taken before the intervention and Post being the application of the measure after the intervention. For control group studies, the control group would also have the Pre and Post measures but without an intervention inbetween.

62
Sample sizes are typically important since they relate to the "power" of a quantitative research study. Here is a quick overview.

Hong Kong.
Be sure to note demographics of participants in the research. Here, there location (i.e., Hong Kong) might be influential in how you interpret, and potentially apply, the results. Other demographics, such as age, gender, etc. may also be worth noting.

Hypothesis 1
Studies usually present the hypothesis in the "alternative hypothesis" format, rather the "null hypothesis" format, even though the later is the one that is actually tested. Here is a review of the two types.
https://www.youtube.com/watch?v=WtdiMUwWX0k
If you don't recall why scientists use the null hypothesis, here is a review.

information,communications and technology (ICT)
Wiki description of ICTs in Education, with lots of example use cases.
https://en.wikibooks.org/wiki/ICT_in_Education/The_Uses_of_ICTs_in_Education

Evans,2008
It is important to look at the cited references for context. For example, you will note the Evan's research in 2008 looked at podcast use in higher education. Thus, when the researcher here is discussing "using devices in classrooms" the evidence they are applying is related to higher education, and not K12 education  this may, or may not, influence your interpretation.

quasiexperimental desig
Introductory video to quasiexperimental research designs (i.e., studies without control groups).

epeatedmeasure
Short video introduction to Repeated Measures ANOVA

 Jun 2019

wesharescience.com wesharescience.com

ANOVA
ANOVA = Analysis of Variance
ANOVA is a statistical technique used to compare the variations found among two or more groups.
Resources: Wikipedia

ABSTRACT

DOI:
A DOI is an unique Digital Object Identified. They get assigned to articles so that they can be tracked and easily located.

a
ORCID is a nonprofit organization that offers unique identifiers for researchers to automatically link together their contributions.

 May 2019

www.parsingscience.org www.parsingscience.org

citizen scientists named the arc“Steve”

solar wind

 Apr 2019

www.parsingscience.org www.parsingscience.org

β = .035(median 3partial 휂2 = .001, median n = 62,297, median standard error = .004, see Figure 1)
JUST CONCEPT FOR THIS  DO NOT USE
R Jupyter Notebook with code for this analysis: https://rnotebook.io/anon/fa877c0366dc965d/notebooks/DRAFT%20OF%20CONCEPT.ipynb
Or R Jupyter Notebook with Binder: https://github.com/matthewfeickert/RinJupyterwithBinder

NHST
Null Hypothesis Significance Testing (NHST)

screen use
Defining technology use:
https://www.parsingscience.org/wpcontent/uploads/2019/04/ParsingScience047Orben.mp3#t=00:21:47

grounded in SCA

ng SpecificationCurve

garden of forking
Discussion of forking paths:
https://www.parsingscience.org/wpcontent/uploads/2019/04/ParsingScience047Orben.mp3#t=00:11:49
Andrew Gelman unpublished paper on the topic: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf

Andrew K. Przybylski
Story of their collaboration:
https://www.parsingscience.org/wpcontent/uploads/2019/04/ParsingScience047Orben.mp3#t=00:30:50

Supplementary Table 1

Supplementary Table 3
CSV data for Supplemental Table 3:

dolescent wellbeing
Amy discussing measures of adolescent wellbeing:

Specification Curve Analysis(SCA)27
2015 article proposing SCA:

Millennium Cohort Study
Millenium Cohort Study Variable Guide:

Amy Orben


www.parsingscience.org www.parsingscience.org

: https://www.youtube.com/watch?v=tH7bVlGNjWU

methods and best practices applicable to this approach

solar maximum

formal citizen science projectAurorasaurus

conjugate andcoincident flythrough by a lowaltitude spacecraft, such as Swarm, wasneeded to determine the in situ nature of STEVE

significantly equatorward of the auroral ovalduring enhanced activity

aurora

Elizabeth A. MacDonald

itizen science project Aurorasaurus

 Sep 2018

gwu.app.box.com gwu.app.box.com

making poor decisions
Yes, reduce poor decisions  but that isn't the same as guaranteeing good decisions.
We must still realize that we probably can't incorporate all of the perspectives, so some will get left out.

ach discipline’s perspective
Do we have to view through each discipline? Or is it better for us (as interdisciplinary researchers) to question, listen, interpret, and communicate what the experts in each discipline see when they view it through their lens. Me looking through the lens of a biologist isn't of much good, but I can ask questions and listen, and then try to give interpretations.

adequacy, not mastery
This is key  especially as the complexity of the problems is realized.

they provide only partial understandings of the subject under study.
Though, getting a "complete" understanding is next to impossible with many of the complex challenges we want to investigate. Even interdisciplinary will remain partial since it can't possibly go as deep into the all of the specifics for each subdiscipline.

past experiences
Influential professors, courses, readings? Parents?

Internally, everything people experience and learn is “colored” by their epistemic position
Can we really change our position, or do we just become more aware of the positions of others?

phenomenon
What are current examples of research "elephants"? AI? Poverty?

partial understandings
"partial understanding" seems key to this. It is not that any perspective embodies all the necessary dimensions, but that each is missing something. For instance, the sociologist will be missing somethings that the lawyer might pick up.

conflicting insights
Hopefully we can come up with some examples to discuss in class.

gender distinctions
This is an odd statement, esp. since the author gives no context for it.

mere opinion or personal preference
Which may lead us to discount the knowledge generated in other fields. Or even to downgrade the knowledge of our fields.

epistemic
Related to the word Epistemology: https://en.wikipedia.org/wiki/Epistemology

epistemic
The book below on "epistemic cultures" in science actually led to this reading being considered, so I should mention it here: https://www.amazon.com/EpistemicCulturesSciencesMakeKnowledge/dp/0674258940 It is not a great book, but a great concept. We will discuss it in class in a few weeks.

 Nov 2016

blackboard.gwu.edu blackboard.gwu.edu

Benchmark
If you are benchmarking others then you are always playing a game of catchup. Remember, those that you are benchmarking are not sitting still  they are innovating and move forward. So even when you catch up to where they are today, they are still several steps ahead of you. Not that benchmarking is always bad, you just have to balance it with innovations of your own.


blackboard.gwu.edu blackboard.gwu.edu

tacit
Tacit knowledge is that which has not be codified (e.g., written down, archived, shared, etc.).
Here is a helpful link on different types of knowledge: http://www.knowledgemanagementtools.net/differenttypesofknowledge.html

 Oct 2016

blackboard.gwu.edu blackboard.gwu.edu

Oxford English Dictionary
And here is the definition from Webster's Dictionary: http://www.merriamwebster.com/dictionary/knowledge

Drucker, 1994
Here is a link to sample of articles by Peter Drucker: http://www.druckerinstitute.com/peterdruckerslifeandlegacy/adruckersampler/


Local file Local file

eorge Panagiotou suggests an i
It did work. Do you now see this?
