For items that I have to use in combat or multiple times I have to manually click the icon that appears next to the window multiple times or find the item in my bags and drag that on to my action bars. It'd be a nice small QoL improvement if I could drag it from next to the guide window on to my action bars.

Hmm interesting suggestion, but I think this might be difficult to implement (maybe impossible even I am afraid).

Do you want to drag the button to your bar and then click on it? Would that make things much easier for you that way?
Or would you rather press a key bound to that action bar?
If it is the latter, I kind of have good news for you: You can already bind a key to these buttons. Just open the Key Bindings dialog and click on Guidelime on the left side.