And they want more feedback then simply "which journal published it" ?

Make it clear we're a non-profit and we focus on research with potential for global impact.

Publicly Hosted

What do we mean by "decoupled from gatekeeping"? I'm not sure about that one.

Use a colon here, not an em dash

You can publish in a journal too

Add some more vertical slides with the other costs of the existing system/benefits of separating evaluation from "publication", And making the evaluation public

Why is this in bold?

It doesn't really govern dissemination, as noted in the other bullet point... It governs careers and research credibility.

These green slides are not so visually compelling. The text is small, the numbering is not particularly helpful, and there's no image or anything that makes it seem interesting.

That's nice, but I also want to emphasize the value that we're providing in the medium term.

What was meant by "pieces" here?

Recruit an evaluation manager and ...

~2

Use the word 'research' here, not work.

As a team

And evaluators can adjust to this response.

And an evaluation manager summary/synthesis

I don't think this is the right diagram. I think we want the other one illustrating just what the process has been ... People on the team suggest it, give it a rating for prioritization/potential for impact, the whole team votes on it, we finalize it, and liaise with the authors, etc. This is just a diagram about a particular way that we do or do not consider certain things in doing this prioritization.

Drop this. This is obviously already a point being made.

I don't know which building I'm going to be in. Maybe say on this campus or something like that.

"here" Is vague.

Hyperlink the workshop here. Both of them

Research leadership: ...

Obviously, journals never teach this. Also, our research education system doesn't tend to teach this. But it's also not clear that this alone is something that has a lot of career value. It's the methodology, theory, context, etc. that has the career value, as it allows you to do better work.

I don't know that I would emphasize this quite so much. First of all, this only (mainly) holds if you choose to sign your reviews, which not everyone wants to do, particularly early in their career . But also, it does show some value and helps you demonstrate your understanding, but I don't think that the profession rewards refereeing and reviewing quite so much.

A nice thing is you will get feedback on your evaluation from us and potentially from the authors, which will help you improve and learn.

I think this is missing some of the key benefits here. Our public evaluations help you understand what issues other economists care about, as well as, to some extent practitioners, funders and people interested in impact. It's a methodological discussion that will help your own work, as well as help you understand the ways to engage in the peer review process .

This also helps make you part of a conversation involving funders, grantmakers, and people that might be able to help your career and help you have more impact.

This is a bit vague. I'm not sure about this wording here.

I can see something like "sensitive career moments + Work that likely passes the bar" Being a situation where you wouldn't want to have this sort of public criticism or these additional signals. But it depends. If you're at a sensitive career moment but you think you're coming up just below the bar, or you are being systematically undervalued, then it might be helpful to have these additional signals. And if making a commitment to public evaluation itself becomes a positive signal in a screening, sorting equilibrium, you will definitely want to do it in such a situation.

Ask who? What am I supposed to say about this?

I don't see this as clearly argued - The Unjournal isn't going to give these sorts of critiques in a way that I think will be harmful, and I don't want people to keep thinking that public criticism is somehow a deeply harmful thing.

Not sure about mentioning "co-author downside" - rephrase this somehow.

I'm not sure "test" is the right word. It's more like a commitment and a signal.

This seems a bit small, not using the whole page. ?

And Pivotal Questions Advisor

The Metaculus thing is a bit separate now. Go back to the Pivotal questions knowledge base and update on this.

Hyperlink. https://info.unjournal.org/pivotal-questions.html here

Hyperlink the workshops here.

Those aren't the partners on this. The partners on this include Founders Pledge and Animal Charity Evaluators. People from Coefficient Giving and many other organizations have participated in our workshops on these pivotal questions as well.

I think the earlier version of this page was better. It's a little bit confusing because you're talking about evaluators, but you're showing the management team and advisory board here.

I don't like the "plus" here. The substantive reports are equally important, if not more important, than the ratings.

This benchmarking is necessary because we don't quote, accept, or reject, and we don't have a journal tier. So this is the path to Unjournal evaluations being something that has career value as well as value for research users.

This doesn't quite make sense. First of all, we don't publish the research; we publish evaluations. Second of all, where are you getting the 330 figure? Maybe leave this off.

That's not quite getting at the right thing because we actually do favor research that is more well known, as it's likely to be more influential. What we don't favor is research that is just simply seen as deeply intellectually interesting or clever

It shows up as too small on the slide. Can we rearrange it somehow? Maybe make it horizontal?

"lands"

can be

'positive' is more about their work and not about the evaluation ... look for better feedback, including from authors

the "180 evaluators" don't stand behind every evaluation ... this doesn't make sense. We have usually 2 evaluators per evaluation. Also mention the field specialist counts too

incorporate image/text of one of the award-winning evaluations here (or as a bonus vertical slide)

ilustration is OK but it's a bit too 'obvious' -- it should note that a signal could go in either direction ... perhaps should be nested in a graph considering both internal and external signals ... and illstrate cases where the expected value is positive

LaTeX is not really rendered in the diagram below for "Bar C", and also it's not clear what these variables are that need better labeling.

"I don't like 'read it as a working decision aid' so much" -- it's a bit of this, but also a bit of a 'here's how to think about this' discussion document. You can say 'this does not represent The Unjournal's official position or advice' and link our Author FAQ https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/faq-interaction/for-researchers-authors for the latter. But also avoid the 'not this but that' typical AI language. These are 2 separate things

why 'seminar-defensible' -- this could confuse things, people will say "why not present it at seminars"? Also, you mean for such a paper; this could be interpreted as public (UJ) evaluations will make it seminar defensible

Unjournal data

There's some text overlap in the diagram.

what 'costs of negative findings'? This is a bit of a vague sentence.

also state and interpret the equation in terms of 'benefits vs cost' -- easier to interpret I think

remove 'or policy relevance' perhaps -- The Unjournal prioritizes research with global impact potential (although that's not what we mainly rate the research on)

not sure I understand the logic behind the latter part

how would someone know this?

This is a possible set of considerations -- I don't want to state it as definitive

maybe the 'gatekeeper' or 'editor' instead of the 'reader'?

Is it a truly 'public' signal, or only one the evluator sees after reading the paper? Or should we have both?

notation needs improvement, and it should be explained more -- how derived, how to interpret it? Tooltips and expanding sections could help

I suspect another paper has dealt with this question ... 'when noisy signals help the seller' or some such

this needs clarification, I don't quite see why this is the case. Isn't it possible that the author's signal is positive so they submit, but the evaluator reading the paper gets a negative signal?

relevant to whom?

what is 'acts' and what is 'c' -- underexplained

is this 'binary' rhe relevant threshold? Where did it com from? is it sort of generalizable? Consider if it misses some important nuance

who is the 'reader'? I think we mean the later journal evaluator, or career gatekeeper

base

simple

why?

this should be a fold or footnote -- give a quick statistic and footnote yow it was captured

too strong. And takes the 'high dowside' idea for granted too much IMO

too much 'not this but that' AI speak. And 'publicity' is vague here

too much use of bold within paragraphs and prose

this needs further explanation and clarification -- 'usual channels' should already encompass clarification

this is the 'AI language of dichotomy' overused

The language of this is a bit unclear. Try to make it easier to understand.

Don't use bold here.

colore on black text here is very hard to read

Would not attend live but offered to follow up asynchronously.

Fix this. It takes too long to generate.

how many use it now -- put this in context

check and annotate -- what does e.g., 5 wellbys per DALY mean in context, and how does it compare with what people currently do?

This seems like a valid objection, but I think we still phrase the question such that you would give a meaningful answer, or you could give a meaningful answer in this case in terms of the value generated if you were forced to use a single conversion.

The discussion might be more valuable, or I would say is likely to be more valuable than the response, particularly for this question.

let's do better to differentiate this from calibrated well-being. It's not fully clear to someone glancing at this briefly.

this is not a 'subquestion' ... label this page header differently

Add backlink to the workshop and to the elicitation form here.

Make sure you can access the literal question from this interface to know exactly what the respondents are answering. If these things get very long, you can use tooltips.

some visulaizations (bar or donut or mosaic graph) would be helpful here

I don't htink this is a stochastic model? Perhaps an extension of this should give these (correlated?) distributions. ... Squiggle-type modeling

There should also be discplay of the actual equation behind the model, and a folding box or linked page explaining it in more detail

Maybe put this model at the top, after a short preamble?

any prediction markets deal with the fundraising/donation/nonprofit askpets of this?

why? is this just a truism?

this advice seems on the overly generic side?

what is this -- things like this need definitions, in the text with a link and in tooltips

title font too big, taking upd too much screen saace

I'll try to respond / adapt to hypothes.is comments, especially if you flag @daaronr

this is meta -- I don't want meta, or at least put that into an 'opt-in' list

We can swap in here the legal prioritization prototype -- https://uj-prioritization-prototype.netlify.app/legal/ -- please do this -- and note that we're looking for feedback and examples to help improve and train this. Note that we don't envision this prioritization to be mainly driven by AI models -- humans will be making the ultimate decisions -- but these tools can be very helpful in the process.

'the possibility of'

what are these buttons meant to do? Are they supposed to be links?

Or should we focus mainly on the US context because of the greater 'review gap' as well as the greater role for court jurisprudence in the US. On the other hand US legal scholars may be paid more, overcommitted to lucrative and influential work, and thus less willing to do the evaluations.

--> are highly credible to legal audiences, useful to practitioners, and show potential for global impact.

reword this. A 'narrow pilot' is what we will do, that's not a precondition

this is a bit vague, needs clarification

why not?

This itself would be a useful public good, if we curated it well, with feedback from organizations that wanted to use this.

Naturally we will do this in a human-AI collaboration, with AI doing much of the initial search and filtering. (see https://uj-prioritization-prototype.netlify.app/ for a prototype for our main stream)

We did this about 8 months ago, but most of these will probably be stale

Give people the option to suggest/add content.

Show/hide

Proposed and in consideration.

Let us know if you have any questions about this.

this is not Swartz! adjust!

Make this a folding box - and the header should say AI/human collaboration in some way

Another folding box should have the standard call out about how we want feedback, and you can use the hypothesis tool for that.

Add a folding box presenting and explaining the equations behind this!

Give these clear anchors to hyperlink to, and allow people to extract hyperlinks

Let's try to use some referenced values as a starting point, linking them/tooltips. I don't think there's a lot of good research, but still good to start right

Just confirming this is indeed the status

document this better -- is red negative, green positive, and grey close to 0?

maybe we don't want to anchor too much on this --- NB that paper does not involve substitution. "Anchor paper" could be misinterpreted. This is The Unjournal evaluation package that is most strongly connected atm

evan in headers, let's have specific links (perhaps linking section below) for claims

this tracks

Still considering how to frame this workshop, and it depends on interest and participation. One frame is directly targeting what we know about plant-based products, who consumes them, and what it suggests for potential substitution and animal welfare. However, that evidence seems to be rather thin, inconclusive, and premature, perhaps. (See links to EA forum posts, etc.) Furthermore, our evaluation of Bray et al. on experimental versus standard quantitative marketing/I.O. estimates of own price elasticities suggests perhaps deep uncertainty. and lack of ability to be confident in these parameters, not to mention cross-price effects and substitution patterns. This potentially motivates a pivot towards focusing on these methodological questions, as well as framing it in terms of "what can we know and what research is worth pursuing."

Change the date here. The date is still undetermined, probably late summer 2026.

Remove this page for now because it makes it seem like the workshop already happened.

This is not his full evaluation. He gave a very in-depth evaluation, and you've only taken one paragraph here.

1. Firstly, the header does not fully describe the critiques here. It's only one of the critiques.
2. Secondly, even in this scrollytelling depiction, we probably want a bit more about what the evaluators are saying, going into more than one theme very briefly, because this is the core of The Unjournal's value add.

Why "scene"? Are we making a video? Does it even need numbering

these tooltips are not coming up after seconds of hover. -- I only see the "?" -- please fix

We probably want a little bit of a transition here between the issue and the issue of measuring individuals' well-being through self-reports and what the Unjournal is now doing in terms of rating the paper, which is also on certain scales that may have subjective components themselves. Funnily enough. Make the distinction clearer here

This probably needs a little bit more context on why we're trying to measure people's well-being and happiness through self-reports.

the diagram is not fully explained? what does each dot represent? Should we be giving 'names of people' (or IDs, or types of people) to make that clearer?

Make it clear that the number line here is supposed to represent some fixed measure of true well-being.

I know this is meant for a public audience, but it's a little bit oversimplified. Perhaps we can say it in an equally concise and appealing way, but without making the absolute claims like "for one stubborn reason..." there may have been other reasons too. (Note to AI -- try to make this a persistent pattern in your writing. )

Individual "pubs" in the Unjournal PubPub interface

Make it clear that this is what we're calling the "hub" layout.

this is taking up a lot of space -- make it folded by default

--> evaluation package

--> evaluation package on one page

Note -- we'd probably make this additional to PubPub, as the latter goes into standard bibliometrics and information standards.

(Which in turn, brings the risk of divided attention)

let's use actual content and structure from the paper!

This makes me think you're looking at this from a "deontological" standpoint.

In one sense, that's obviously true, as if the risk of causing harm is high enough, the expected value goes negative.

But if you're saying "We should not make even known positive expected value bets if the downside risk is too large"that's a judgment call, and it depends on your moral/ethical worldview.

I'd like to see ambitious research initiatives independent from traditional university/journal processes. If we have the funding, we can build these fields.

I don't fully agree that we need "EA AW community" hands-on involvement in the intermediate and technical steps.

I think it's more a matter of providing funding and incentives and clearly communicating the goals, priorities, and need for rigor to researchers, at enabling coordination.

But I agree that academic incentives on their own are not enough to ensure high-quality, credible work focused on animal welfare implications.

And it would indeed help if the researchers intrinsically cared about animal welfare and thus about producing useful and accurate results. This makes the incentive alignment easier, but I think it can still work even if much of the work is done by people who aren't intrinsically interested in animal welfare or don't think about effectiveness in the same way - as long as they can understand and embody the priorities in their work.

At The Unjournal, we are trying to bring together researchers, practitioners, and funders to do this sort of prioritization and coordination to be able to generate, communicate, powerful, useful evidence, robustly assessing its credibility and improving. Something we're trying to implement through our pivotal questions and workshops, e.g., https://uj-cm-workshop.netlify.app/summary ... and I think we're having some successes.

But I'm not saying we're necessarily the best positioned to do this, and I'd love to work with others or see others move forward on this.

I agree. I strongly believe that some coordination is necessary, and we shouldn't just rely on academic incentives. Large-scale, ambitious evidence and collaborations seem high-value to me.

I would say that this depends on whether you're willing to rely on what might be considered fairly "common sense" priors about the substitution effect in an environment where we have little evidence and it's very hard to collect reliable evidence. Perhaps for any intervention there will be some aspect of the model that we might need to take for granted as just a common sense implication.

I can appreciate that you might not agree that the price and taste equivalent plant-based meat would substantially crowd out the consumption of conventional meat. I find it harder to imagine that the same would hold for cell-based meat, but this is obviously a judgment call.

That said, there's a bit of a chicken-and-egg problem here in that I think it's plausible it's reasonable to assume that until cell-based meat is actually in restaurants and supermarket shelves, we won't have a good sense of whether people will buy it as a replacement for conventional animal meat. But the only way for it to actually get on the shelves is by there being substantial investment. So requiring this high degree of certainty might make it impossible for us to consider potentially high-value but risky investments in this sort of innovation.

The evidence is, in fact, all over the map. -- see e.g., the survey here -- https://forum.effectivealtruism.org/posts/3Eh8MbqLwFBsD7GK2/how-much-do-plant-based-products-substitute-for-animal#Existing_Research

But there are also doubts about whether we can reliably collect evidence in this domain. See https://uj-pba-workshop.netlify.app/context/pbm_fuller_report/ for an (AI-aggregated) synthesis ... Bray et al cast doubt on the reliability of even own-price-elasticity estimates in fairly standard settings -- see our evaluation of this paper here -- https://unjournal.pubpub.org/pub/evalsumbraybray/, and we're working on further discussion about the implications of this for substitution.

(FWIW, intuitively, I have a strong prior that there would be a fairly strong substitution, even if not one that would completely end animal farming. As Bayesians, our prior beliefs should count for a lot in a domain without substantial evidence.)

And they are funding more -- see their most recent post.

Confirmed. There does seem to be deep uncertainty here substantial lack of measurement

I'd like to see more justification of this bolded claim that it's indeed very sensitive to the assumptions. Can you provide a link or a footnote about this? Are there reasonable specifications under which it goes the other way?

The USA and WOR ones would seem to also, although the difference may not attain statistical significance in a conventional sense. But it still contributes to the evidence in that direction.

I would still admit that the evidence, as you presented it, seems to go the other way, though.

This table is somewhat hard to read and somewhat hard to see a synthesis of. I think something like a forest plot could be helpful here. And of course, it would be helpful to just report on the actual meta-analytical results.

What seems important to me is the expected magnitude of the difference in mortality between systems and the implications for the difference in suffering, factoring in the other differences in life quality and health.

In fact, in this case, the aviary mortality is actually lower than conventional cage mortality.

Should correct for multiple comparisons, we'll stop. At the same time, I disagree with drawing conclusions from a lack of statistical significance. To make a conclusion about "it's very unlikely that there's any difference large enough to be meaningful", you need to do something like an equivalence test, or report Bayesian posteriors over the difference.

What is the double arc sign transformation here, and why is it used? AI please remind us

This part certainly suggests that, but we should really present the net with some adding up and weighting for the magnitude of the suffering of each event.

These things and their consequent animal suffering/animal welfare burden could presumably be weighted and aggregated and compared. "More" could be slightly more in one case and far more in another case. What are the aggregate differences for reasonable assumptions here?

AI -- please look this up/clarify

I guess that's my take on what you shared too, but how do authors and experts in the field (other than you) interpret it? A bit of steelmanning+feedback could be useful imo.

AI: look this up, including in comments below, provide sources for other research.

if you want to rewrite or rework a bit, maybe lead with this, so we can understand what the evidence is getting at.

ok this informally suggests important heterogeneity to me, suggesting the need for nontrivial sample sizes.

have you tried to contact the authors? I think that would be high value -- both directly and in terms of field building-- and happy to help facilitate it if I can be of help. They might be particular eager to clarify and happy to hear how their research is valued ... and may also see this as a route to potential future funding.

Is that really how they presented it? Let's double check. That is extremely vague.

a fair comparison to what?

Again, I'm really not sure whether these are good or bad things. Why do we care if they recover their ability to move later if we're normally killing them with this?

OK this is explained a bit further below .... because they may wake up again before being killed

Okay, following the above, I guess the point is that you kind of see similar things happen from ice lorry and electric shock, so it's not clear why one followed by the other is the best. ??

Confused about what this is supposed to mean. Are we considering cutting their head off, or are you saying that if this happens, even though you cut their head off, that means that it probably doesn't indicate anything meaningful, just a sort of knee-jerk response?

are these things good or bad? Not immediately obvious to me.

I found your presentation below on this rather convincing. This also comports with what I've heard from other EAs (although perhaps the same circle of conversation). We need better evidence on which of these AW improving technologies actually reduce animal suffering. I'm in some discussions about possibly building and funding an evaluation service for specific tools and approaches (maybe something between The Unjournal and a fast-review journal, also inspired by Rapid Reviews Infectious Diseases).

Very small sample sizes do not always mean lack of inference. For instance, in very predictable contexts without a lot of noise, like, let's say, Newtonian physics, even a few data points could help us narrow our beliefs substantially. I like Richard McElreath's example about how you can substantially update on the share of a planet that is water by simply, even in the first few random samples, choosing a single point on a planet's sphere.

More intuitive -- if I ask 4 people to taste a drink and they all wince deeply in pain and disgust, I'm going to be highly confident it tastes bad. If all 4 smile and praise it, I'll be fairly confident that it's at least tolerable.

But I don't know that that is the case here. There might in fact be a lot of uncertainty and heterogeneity. What I wonder is whether the sample sizes observing the behavior and bioindicators of these fish are very expensive, or whether it could easily be scaled up with just a small amount of money. as a non-biologist, it seems intuitive that it should be cheap, but I might be missing something

I.e., 6 animals killed with each. See previous comments about sample size. ... what's the within treatment variation etc., and how costly is it per animal? How does this sample size compare with typical measurements in these domains? If the measurements taken are costly, could we get more reliability with cheaper measurements and a larger sample?

I think I would indeed lean in this direction, and I'd suggest that in fact most grants I've seen go the other way. (But I may have some biases here, this goes in line with what we're trying to do at the Unjournal, etc. )

[Consider -- does this post make a clear case for the 'should', demonstrating that the VOI of research here will exceed the expected (?) impact of the best or current 'existing work'?]

v speculative -- this may be highly timely if we think the Anthropic IPO will be driving some money to AW , and donors are evidence driven and 'difference-making risk-averse'

In terms of the 'expected impact'/'uncertainty of impact' tradeoff heard of Animal Welfare as being in between GH&D and X-risk/GCR/AI Risk.

Your concerns may be driven by "Difference-making risk aversion" -- see discussion here, for example https://forum.effectivealtruism.org/s/WdL3LE5LHvTwWmyqj/p/9EENSGhiQiKFaRh4t

Or you might be driven by something more deontological, related to 'do no harm', perhaps?

Maybe to be fair to note that this is along a specific dimension of transition -- conventional caged to regulatory mandated cage-free. Perhaps better evidence for other certified standards of free-range etc?

This is something Unjournal is trying to make happen. Come working with https://www.aw-econ.org/about and others. I think, given the very limited investment in this space, I think there are high marginal returns, although some of the value of information will be learning about what is or is not "epistemically possible".

I think what we have is a substantial lack of evidence in this domain. This is something the Unjournal is trying to remedy. (Unjournal.org, see https://uj-pba-workshop.netlify.app/ for a link to some of our efforts, and https://forum.effectivealtruism.org/posts/3Eh8MbqLwFBsD7GK2/how-much-do-plant-based-products-substitute-for-animal or an earlier take).

Evidence in this domain is very hard to come by, and there are substantial doubts about the extent to which we can even reliably measure these things. (The aforementioned post gets at this. Also see our an evaluation of the Bray et al. paper, which casts substantial doubt on the ability to even measure simpler things like own price elasticity with conventional methods -- https://unjournal.pubpub.org/pub/evalsumbraybray/ -- I'm working to follow up this evaluation package with more detailed evaluation managers' discussion and dissemination, focusing on the applied issues.

Following up on this further -- see https://app.notion.com/p/Validation-Evidence-for-Food-Demand-Elasticities-PBM-Pivotal-Question-376e97e2ad3381a898d3ceb589b265f2 and links within for a preview.

At the same time, there's also just not a lot of economics, social science, and marketing work done that focuses on animal welfare implications (or alternative proteins).

The aforementioned workshop on associated pivotal questions will be focused not only on what we know in this domain (~substitution effects within and between animal and plant-product consumption) , but also on what we can know with given data, what we might be able to reliably learn with more ambitious data collection exercises, experiments, etc., and what (and what is sort of fundamentally unknowable and does not merit further research investment).

I've been hearing this for a while. It's something that organizations like Animal Charity Evaluators are trying to address, but they don't have the dedicated resources and funding that, say, GiveWell has. Furthermore, there's no comparable academic/national resource base for animal-welfare-relevant research. For example, in the economics animal product space, it seems most of the "agricultural economics" is oriented towards supporting the farm industry. (Perhaps with climate change a secondary priority in some countries )

doublecheck -- if it's 50 everywhere just take it out of the data set -- it's just distraction and probably our testing the form

give response dates

relevant yes. But how do we know it's important for these questions?

Fix the tooltip -- it shouldn't be in all caps

give the full question language, and hyperlink the question in context (tooltip if long)

just say 'anonymous 2' .... "H" is confusing

Note -- I did not intent to have CIs over probabilities. This was an artifact of a changed question and vibe coding. Also investigate whether this was the wording of the question when participants answered it

this seems to need more digging into!

this seems worth highlighting, even if it's a rough calculation

state this as percentage

Pre-worskhop discussion !!

there is too much overlap between this and the paragraph beginning "The sceptical concerns"!

We need more of a TOC and sections/navigation on this page. Longer content in folding boxes

shorten this a bit

Skip this one

Research and state this. Also for impossible and. Beyond vs conventionally ground beef