threshold of twenty-five
Annotation explaining how this threshold is determined? Perhaps a source or explanation would be useful here. Can this threshold apply to all cases? What if the researcher is interested in heterogenous effects by subject characteristic (e.g., difference between men and women, minority and majority, party id) would the N need to be 25 per group to generalize?
requests
The focus of this paper is using shadowing to develop empirical evidence. I wonder if it can also be useful for theory developing and how that process may be different than shadowing the the aim of collecting data.
Bias in Observations
I wonder if you want to consider potential researcher bias in addition to bias introduced by the subjects.
As such, a researcher may choose to discount observations from the first day or two of a study, though this is infeasible for shorter studies.
This was also my inclination on how to address this issue. One concern I had, however, is that there may be some sequencing to the week or day or month that researchers may not want to omit. I suppose one way to deal with that is to not the time in the week (day/year) when each observation begins and randomize the start day for subjects.
In the next section,
When you discuss strategies for limiting the Hawthorne effect you may want to reference it directly. Although you note that the next section will address this, you do not explicitly state that you are addressing this issue in the next section.
Unless what you man here is that you address this in the section on Bias. in which case, perhaps you can clarify that here.
Once contact is made,
How do you recommend lining up subjects. Should you choose your sample, and then contact the first person, conclude the shadowing before securing the second subject? Or should you schedule further in advance? These logistics may be useful.
As
What if the subject agrees to most things but only wants to be available for a more limited time or has restrictions on where you can shadow them? Does this mean the subject should be excluded from the study? Or is there a way to include them w/o biasing the results?
An additional benefit of the shadowing approach, is that the shadower can pose questions to the subject about these choices, in order to further inform inferences about their relevance to the study
This was not clear in the earlier parts of the manuscript. Earlier the process was described as "passive observer." It may be useful to have annotation regarding how to think about what the boundaries are when posing questions. If you have teams of enumerators how to you ensure consistency across enumerators. For solo enumerators how to do you ensure consistency across subjects (and is that even necessary--I would think so because it may introduce bias in that some observations may be coded as not happening but this information could have been uncovered with a question). Should scholars devise a systematic approach for knowing when/where to pose questions to the subjects.
In this sense, shadowing may often be less intrusive from the perspective of the politician than it might be for a different type of professional, increasing the chances that a researcher may be able to complete their desired set of studies
Here annotations regarding your success with getting subjects to agree to be shadowed both the N and rate response rate may be useful.
process
This careful approach to shadowing is very important. I had similar concerns about the enumerator. That is, the enumerator may "learn" over time. They may find different things important and wroth noting as they progress through the study. this can also introduce bias. Is that something worth addressing.
In this example, the thematic narrative discussion is buttressed by the presentation of the descriptive statistics on the frequency of described activities among the full set of shadowed presidents.
yes, this is very compelling and helps the reader understand the concepts you are measuring.
A written account may begin with an example from one shadowed individual and then move primarily into a thematic account of the study’s findings.
This seems mostly clear. I can imagine an annotation that uses one of example in all three cases and shows how different data/information is emphasized from a single shadow case in all three examples. That is, in the first case you may have a narrative describing the day of one observation. In the second case, you can illustrate how the exact same shadow case is subject to your coding scheme to calculate the share of shadow cases that fall into each category. In other words, it would be useful to see how you approach the same shadow case differently when aiming to achieve each of these different goals.
Finally, and perhaps most importantly, these qualitative observations offer little to no evidence of partisanship or other conditionalities in the provision of this assistance. Instead, these examples suggest at least a stated adherence to an ethic of public service. In most cases, legislators had only limited information about an individual, and not necessarily even their home village, when making a decision about how to help.
Annotation showing the distribution of the data across different leader's partisanship. Do we have evidence that people are less likely to ask for help if they do not have shared partisanship?
This provides further evidence that these local actors have the potential to resolve these cases but that they had not, for some reason, done so for the individual(s) in question.
annotation regarding the previous evidence?
Second, the provision of assistance to individuals in India is not simply ad hoc: legislators design their meetings with citizens to allow them to respond effectively to a large volume of demands
annotation for coding rule?
Third, the interactions between politicians and their constituents do not typically evoke strong elements of hierarchy, and instead are often characterized by apparent mutual respect between the two parties.
annotation explaining how you code this.
rule-bound decision-making, explaining to individuals the criteria of the program and using these criteria to include or exclude potential applicants.
Clear definition of the concept. Is this the same definition that is supplied to the coders? An annotation here may be useful.
Among the set of shadowed presidents, 86% (12 presidents) intervened in some way to provide assistance to those individuals making requests
Could add annotation to the coding rules for defining providing aid to individuals. E.g., may be important to explain how you define/distinguish between individuals/families/small groups.
Of the fourteen presidents shadowed in-depth for this project, 93% (13 presidents) received requests for assistance with access to government services during the observation period.
Could add annotation to these different examples here. Could add annotation to the coding rule that defines how you code request for government services.
There are three main forms this can take,
maybe preview the 3 forms here.
After discussing other topics
will there be annotation offered about the "other topics discussed?"It is unclear how to know if the enumerator should have offered information on the content of the other topics if we do not yet know what is important during the coding process.
The format for coding memos can be informal and memoing may go through multiple stages
Here (if you do not do so below) it could be useful to show in the annotation examples of a few different stages of the memoing process so readers can see how they evolve.
process of memoing, in which the researcher writes narrative observations of the field materials
Here is the researcher different from the enumerator? Also, is this process done after all of the shadowing is complete?
support a multi-enumerator environment and the potential for quantitative analysis of observations drawn from shadowing.
The choice to develop coding after the shadowing is interesting. I can certainly see the advantages based on the discussion that follows. Are there disadvantages to this? I had expected that the coding would be defined in advance. I wonder if there are advantages to defining the coding scheme in advance? This may merit discussion. Even if the author does not think there are advantages it may be worth noting. Often times in social science we want the enumeration process, the coding process, or other stages of the data collection/coding to be independent of one another to avoid bias. So, for example, when administering lab experiments, often times the researchers administering the experiments do not know what the research question is. Or when coding text data the coders often times do not know what the RQ is. Here the author is suggesting that the coder use the information learned from the shadow notes to inform the coding scheme. There are clearly merits to this approach, but the tradeoffs may merit a bit more discussion.
see here
Link here?
later coding processes
Are the enumerators aware of the coding process when they are shadowing? It seems like this may be important to keep separate so that the enumerators are not biased in what they record and take note of. This may be a more difficult issue to address if there is only one researcher/enumerator.
team of enumerators
Should there be some overlap with enumerators (i.e., multiple enumerators independently observing the same person at the same time) to give us some measure of inner-enumerator reliability? Perhaps this is discussed later.
A
This discussion felt a little under specified. Although I found the example of all city councils vs. Brazilian city councils useful I think it would be useful if the author said a bit more about the aim/goals the researcher should have in mind when she defines subjects before using the example to illustrate the point. This would only require one or two more sentences to flesh out the idea of before turning to the example.
this section is very clear and useful.
reasons she offers for why she d
The enumerator is described as a "passive observer," but does the author intend to suggest here that there is some feedback provided by the individual being shadowed? Or is this intended to suggest that the individual my offer other people (not the enumerator) reasons for her decisions?
medium-N study
I find this idea very appealing. That said, at this point, I am curious how difficult/easy it is to get a medium-N number of legislators (or other elites) to agree to let researchers shadow them. I think a note about feasibility could be helpful in the intro for selling the contribution.
This articl
The data I expect to see: After reading the abstract I suspect that the author will provide field notes from her shadowing experience with several legislators. The contrast with ethnographic work leads me to believe the author will likely develop a coding scheme that she can use to code some key characteristics across different legislators in order to provide descriptive statistics. I anticipate the author will rely on primary data sources--specifically her own observations obtained while shadowing legislators.
Yet, detailed and direct observations of behavior by political elites are rare and, where they occur, often constrained on multiple dimensions. Large-scale observations of behavior, such as roll call votes and audit experiments, offer important insights into single-issue questions, but often have important limitations related to the breadth of inquiry. Small scale, qualitative observation techniques, such as ethnography and interviews, instead offer the opportunity to observe and investigate a wide range of behaviors, but typically on a much smaller sample of individuals, thus limiting the potential generalizability of inferences. These limitations suggest that there may be benefits from adopting tools of observation and data collection that can feasibly leverage the benefits of observing many individuals at a time, while maximizing the amount and quality of data collected on each individual.
After reading this paragraph I wonder if the same researcher needs to do all of the shadowing or if the author believes that the process is sufficiently standardized that a team of scholars/research assistants could shadow different legislators and have confidence in the comparability across observers. This may be worth mentioning here (in the advantages) if the author believes that the same observer does not need to be present for each shadowing experience.
multi-enumerator data
This addresses my previous comment.