Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rice append doesn't append compressed zip contents, but plain text? #84

Open
titpetric opened this issue Oct 3, 2016 · 7 comments
Open

rice append doesn't append compressed zip contents, but plain text? #84

titpetric opened this issue Oct 3, 2016 · 7 comments
Labels
enhancement help wanted

Comments

@titpetric
Copy link

Appending two boxes (templates and assets folder), I can inspect the binary files and realize that the appended data is not compressed at all:

--- ping-dashboard-orig.hex     2016-10-03 15:11:35.050537200 +0200
+++ ping-dashboard.hex  2016-10-03 15:08:22.578537200 +0200
@@ -418688,3 +418688,8226 @@
 06637f0: 0000 0000 0000 0000 1031 6600 0000 0000  .........1f.....
 0663800: d000 0000 0000 0000 0000 0000 0000 0000  ................
 0663810: 0100 0000 0000 0000 0000 0000 0000 0000  ................
+0663820: 504b 0304 1400 0000 0000 0000 0000 0000  PK..............
+0663830: 0000 0000 0000 0000 0000 0900 0000 7465  ..............te
+0663840: 6d70 6c61 7465 7350 4b03 0414 0000 0000  mplatesPK.......
+0663850: 0034 5a43 49e5 6bc7 bb73 0e00 0073 0e00  .4ZCI.k..s...s..
+0663860: 0013 0000 0074 656d 706c 6174 6573 2f6d  .....templates/m
+0663870: 6169 6e2e 6874 6d6c 3c21 444f 4354 5950  ain.html<!DOCTYP
+0663880: 4520 6874 6d6c 3e0a 3c68 6561 643e 0a3c  E html>.<head>.<
+0663890: 6d65 7461 2063 6861 7273 6574 3d22 7574  meta charset="ut
+06638a0: 662d 3822 3e0a 3c6c 696e 6b20 7265 6c3d  f-8">.<link rel=
+06638b0: 2273 7479 6c65 7368 6565 7422 2074 7970  "stylesheet" typ
+06638c0: 653d 2274 6578 742f 6373 7322 2068 7265  e="text/css" hre
+06638d0: 663d 222f 2f66 6f6e 7473 2e67 6f6f 676c  f="//fonts.googl
+06638e0: 6561 7069 732e 636f 6d2f 6373 733f 6661  eapis.com/css?fa
+06638f0: 6d69 6c79 3d4f 7065 6e2b 5361 6e73 2220  mily=Open+Sans"
+0663900: 2f3e 0a3c 7374 796c 6520 7479 7065 3d22  />.<style type="
+0663910: 7465 7874 2f63 7373 2220 6d65 6469 613d  text/css" media=
+0663920: 2273 6372 6565 6e22 3e0a 0962 6f64 7920  "screen">..body
+0663930: 7b0a 0909 6261 636b 6772 6f75 6e64 3a20  {...background:
+0663940: 2330 3030 3b0a 0909 636f 6c6f 723a 2023  #000;...color: #
+0663950: 6666 663b 0a09 7d0a 092a 207b 0a09 0966  fff;..}

I'm expecting the zip file would be compressed which doesn't seem to be the case...
@titpetric
Copy link
Author

If I create my own zip and do this, it seems to work, regardless of what rice append produces:

zip -r data.zip assets/ templates/
cat data.zip >> binary-file
zip -A binary-file

So it seems there should be just some step in rice append that would produce a compressed zip file, which can already be read by rice?

@titpetric
Copy link
Author

titpetric commented Oct 3, 2016 •
edited

& there are some issues in regards how folders are added to the archive. Unpacking the binary file doesn't work nicely if the data is appended with the rice command. Boxes/folders are unpacked as 0 byte files.

@jeff-minard-ck
Copy link

I just noticed this as well when my compiled variant was 115mb and the zip appended executable was 137. The files, alone, are 130mb, and when zipped are 11mb (yay xml...).

@jeff-minard-ck
Copy link

Doing some basic research, I think this line (https://github.com/GeertJohan/go.rice/blob/master/rice/append.go#L111) is missing a follow up line of

				zipFileHeader.Method = zip.Deflate

I added that and rebuilt and the appended file now is 19mb (8 for the app pre-append + the 11 I expect) so it looks to, file size, work. However, running the app throws several errors and truncates the embedded files, so clearly the reading back of a compressed archive is also incompatible still.

./application
2016/11/26 10:29:20 error reading data for appended file path/to/file1: EOF
2016/11/26 10:29:20 error reading data for appended file path/to/file2: EOF
2016/11/26 10:29:20 error reading data for appended file path/to/file3: EOF
...etc

@titpetric
Copy link
Author

@jeffminard-ck - I suggest you to try my workaround in comment 2 with zip -A and let me know if you can bypass append and end up with a working app :)

@jeff-minard-ck
Copy link

I did eventually try that and ended up with the same results. The readback code I could find doesn't seem to mention zip.Deflate anywhere so I suspect it doesn't read the archive contents back correctly. I got a lot of half-complete responses and weird behavior.

@wladiz2001
Copy link

Hello

I ran into the same problem. The size of the executable file grows too much because there is no compress option, I have made some changes that allow deflate to be used in the boxes, both as appended and embedded.

I have found that the main problem was due to a problem in the implementation of the algorithm in flate

If you use rc.Read to read the content in a compressed file you will get an EOF error. Then I use ioutil.ReadAll it doesn't generate the error and you don't need make the slice, and to make sure the uncompressed file content is correct it calculate CRC and compare with the store one.

There is a new flag "/c" that allows compress. If the flag is not used the previous process is generated. The boxes are compatible with previous version without changes

I have made a fork on https://github.com/wladiz2001/go.rice/

You can review it and if there is no problem you can merge it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@titpetric @GeertJohan @jeff-minard-ck @wladiz2001