This will depend on a very specific and narrow definition of fitness--perhaps one from a very individualistic and libertarian perspective.
There is fitness at the level of the gene, the organ, the individual, and the group, and even possibly larger groupings above that.
What if, by starving out and leaving "uneducated" people like Srinivasa Ramanujan, for example, who surely was marginalized for his time, society is left without them? While on an individual level Ramanujan may have been less fit on some levels as G.H. Hardy and may have otherwise dwindled and disappeared, Hardy adopted him and made both mathematicians better while also making dramatic strides for mankind.
From a statistical mechanics perspective, within some reasonable limits, we should be focusing on improving ourselves as well as the larger group(s) because the end results for humanity and life in general may be dramatically improved. (Though what we mean by improved here may be called into question from a definitional perspective.)
Compare this with [Malcolm Gladwell]]'s argument in My Little Hundred Million.
On a nationalistic level within human politics, Republicans should be less reticent to help out marginalized Americans because it may be from this pool of potential that we may find life saving improvements or even protection from other polities (ie, in our competition or threats from countries like China, Iran, North Korea). Consider how different things may have been had the U.S. not taken in Jewish or other foreign nationals like Albert Einstein, John von Neumann, etc. in the early to mid-1900s.? Now consider, which life changing geniuses we may be preventing reaching their potential by our current immigration policies? our current educational policies?