An early life full of neglect, deprivation and adversity leads to people growing up with smaller brains, a study suggests.