HTTP Get seems to strip trailing whitespace when getting files
tehbiga opened this issue · 7 comments
Minecraft Version
1.18.x
Version
1.101.0
Details
http.get seems to be stripping trailing whitespace / newlines from response bodies. I was able to compare the responses side by side with a browser.
Story:
Was writing a program that connects to my home computer to transfer files with a generated install script which checks crc8 values and noticed the crc values were always wrong. I realized after a bunch of testing that if I call http.get on my file and dump the contents with res.readAll() the final new line was missing. If I request the same url from a browser and save the file the newline is there.
Computer http.get:
https://i.imgur.com/Kqj18S4.png
File downloaded with browser:
https://i.imgur.com/DXthola.png
If you want an exact match, you probably need to use http.get's binary mode. The default text mode does a whole bunch of extra stuff, including normalising line endings and dealing with unicode.
If you want an exact match, you probably need to use http.get's binary mode. The default text mode does a whole bunch of extra stuff, including normalising line endings and dealing with unicode.
wget uses binary mode and has the same problem.
Hrmr, can you give me a URL to test with and your code? They appear to be preserved for me.
@SquidDev I figured out where all of the confusion is coming from:
Reading the file with mode r is stripping the last newline character.
Reading with mode rb shows the missing newline.
I can switch to using binary reads (and I should have from the beginning), but this also means that the edit program is not reading in binary mode and the editor view is hiding the newline.
This is where all of my confusion was coming from not being able to verify with the built-in editor. When I'm editing files on my real computer I obviously see the newline and in-game I do not, causing the confusion.
Sorry to kind of waste your time, but I didn't realize that not reading in binary would just trim whitespace off the ends. It seems odd to me because file encoding in other languages don't have this issue so I was confused.
Yeah, text mode has all sorts of quirks I'd like to sort out at some point. Maybe even just remove text mode entirely and just treat everything as binary (which is pretty much what PUC Lua does). It's just not clear what the ramifications are of this on user code, and if it's going to break anything!