The Dunning-Kruger effect is undoubtedly important, but since stupidity has always existed, this doesn't explain why the problem has become worse in recent years.

I think David Riesman hinted at it in his 1959 The Lonely Crowed. Specifically, the transition from a production-oriented economy to a consumption-oriented one has increased the distance between personal experience and expertise that has consequences.

Once there were many workers whose jobs involved listening to and excepting expert guidance. An auto mechanic knew the wrong kind of oil would ruin an engine; a railroad worker knew some steels work better as rails in difference circumstances; a seamstress knew there were important differences between different thread materials. They received expert advice, and saw what happened when it was ignored.

The vast majority of expertise can be denied without any consequence at all to the individual. Even when there are consequences -- such as with the brain-surgeon example from the article -- the denying individual isn't likely to learn any lesson. Honestly, how often can a patient actually see the consequence of that doctor's advice, when alternative narratives are pervasive?

This is a large part of a more general trend towards individualized epistemology, based on each individual's tribal affiliations and social identification.

Education could overcome it, but that requires winning the coordination game that has always crippled education.