The new tool announced Thursday is meant to detect organized, malicious efforts that are a threat but fall short of the social media giant's existing rules against hate groups