Israel's attorney general has ordered an investigation into police surveillance tactics amid reports that law enforcement improperly used a disputed hacking tool.