Finally, any up vote based system is just going to push easy to understand, rather than good, papers up to the top.
I don't think this is quite right. An upvote system like reddit's may result in this, but in OpenReview the only upvoting mechanism would be posting reviews. Those reviews could then be reviewed themselves, and if found to be a poor argument, the original review wouldn't affect the parent paper's score as much. We could require that people post an argument that directly addresses the parent post. We could score the post based on how "good" of an argument it is: http://www.paulgraham.com/disagree.html
This could also provide a great way of gathering labeled data for argument mining: people refuting each other, and refuting refutations. It could lead to a virtuous cycle of learning how to mine for logical arguments in text.