I love this feature, except when placing shulkers. It forces a random shulker into my hotbar that usually makes no sense (not same colour or name), and just makes me have to take it back off of my hotbar. Even named shulkers seem to do this, when they shouldnt be considered a like item any more.

Its frustrating for someone who uses the enderchest as a backpack and carries shulkers like pockets for all my building needs.

From what I've experienced, it doesn't switch if you use different colours. Might want to try that for your main Shulker Boxes!

The order isn't entirely random, upon testing it seems to be ordered by NBT which means it's probably ordered by the default NBT sorting rule

@default_nbt_rule
::enchantments_score
::damage
::display_name
::potion_effect
::nbt_comparator

which I think means it goes by the item name of the first item in the first slot alphabetically so if I have sorted shulkers of mined materials I'd get my andesite first, coal ore second, cobblestone, diorite, dirt, gravel, granite, ironore, stone... since that comes in the NBT data before their item IDs

it'd be nice if there was a way to order/filter this directly within the mod

That said I haven't had the mixing up colours of shulker boxes which means I have the 16 coloured shulkers for important things like tools or specific kits like my redstone box and keep all the general building boxes of stuff all uncoloured after all the sorting works in the same order in inventories as the shulkers get pulled to your hands