Research on children supports this, noting that children raised in adverse environments tend to have hindered brain development, increasing their risk of memory issues, learning difficulties, and behavioral problems.
This supports Alter's point in Chapter 8 of Drunk Tank Pink.