"While it might sound like an excuse, the unfortunate reality is that there is a very limited pool of black talent to recruit from," Scharf said in the memo.