What had all this to do with him—this house and Charlotte, the girls and Harold—what did he know about them? They were strangers to him.
This feels like Marriage A La Mode, where William felt similarly about his wife and family. Mr. Neave has spent so much providing for his family that he has not had the opportunity to enjoy it with them. This brings up an interesting conundrum. If Mr. Neave had acted more like Harold, maybe he would have a better relationship with his family. On the other hand, he would not have his business, or likely the other comforts his family can enjoy due to his hard work.