My wand currently has 175 spells. However, I can only use around 57 spells because the inventory is full. Is there any way that I can add more hotbars or increase the inventory size.

Click on the wand in your inventory to change pages. It should've told you this in chat when you got more than one page of spells, and there is also an instructions option in the enchant table menu, and finally /mhelp missing spells would've told you this as well.