This notion, especially after reading the poet's response by asking what he would be walling in or out, seems to speak on the level of trust or distrust between people
Just like what the poet proclaims, walls and fences are borders that serve to keep things in or prevent things from getting out. Both purposes serve as a statement on how much or how little people trust one another, especially their own neighbors.
Whether it be a means of establishing and protecting privacy, the poet's neighbor may be stating that a reasonable amount of private space from one another is what makes them "good neighbors".