Giants that prefer the hyphenated spelling—Merriam-Webster, The Chicago Manual of Style, and The New Yorker, have a good reason for doing so. E-mail is a compound noun, made out of two words—“electronic” and “mail.” The e in e-mail is an abbreviation for “electronic,” and it’s used in a lot of other words as well—e-commerce, e-learning, and e-business, for example. There are also other compound nouns formed from an abbreviation and a noun, like the H-bomb, which is short for hydrogen bomb. The general rule of hyphenation in compound words that combine a single letter (or a number) and a word is to hyphenate them. So, based on tradition, e-mail is the correct way to do it.

English tends to build new compound nouns by simply writing them as separate words with a blank. Once the compound is established (and the original parts somewhat "forgotten"), it's often written as one word or hyphenated. (Examples: shoelaces, aircraft...)

Web site / website seems to be somewhat in a transitional stage, being seen as an "entity" that web page hasn't reached yet. Depending on which dictionary you check you will find web site and website, but only web page, not webpage.

Other languages, German for example, are notorious for very long compunds like this and this, that are made up and written as one word directly. Perhaps the way your native language deals with compounds explains your (or other authors') personal preference and sense of "right"?