I'm running a server that has HQM installed and want to modify the .dat file so i can just rest or complete 1 persons quests as some are not completing. I know there is the op book, but i don't want to /op player to fix this issue. Im looking for a way to mod the .dat file. any help would be great.

I don't think there is a way of editing the .dat, that is why I moved to json for later releases.
Don't know if you already know this or not, but you can get op books for other players as well. So if it is an issue of opping this given player than that can solve your issue. If you don't want to op at all then this doesn't fix anything.

hmm, what program was used for 1.7.10 as i should be able to be opened and modified from there.

No program was used, it is a binary file format