These two upgrades are used to add mobs into either a blacklist or whitelist, to register a mob, just right-click them.
Animal filter will be usable in animal hunting mode and can only register passive mobs.
Monster filter will be usable in monster hunting mode and can only register hostile mobs.

Starting in 8.1, Both hunting commands have been combined into 1, and by using a new mob filter upgrade you can choose what mobs your rats attack or wont attack