archiver.FileSystem() alternative that takes io.Reader

[tenox7]

What would you like to have changed?

In addition to archiver.FileSystem("filename.ext") there should be a function that takes io.Reader or better io.ReaderAt and size. For example archiver.NewFsReader(r io.ReaderAt, size int64).

Why is this feature a useful, necessary, and/or important addition to this project?

This is necessary to use a file thats already open or for files that do not resize on OS filesystem and cannot be opened with os.Open(). For example afero or any other virtual/in memory file system.

What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

There are no alternatives, currently I can't use archiver.FileSystem with a file on a synthetic FS because archiver assumes the file can be opened with os.Open().

Please link to any relevant issues, pull requests, or other discussions.

Go archive/zip implements NewReader in addition to file Open.

https://pkg.go.dev/archive/zip#NewReader

Similarly SevenZip:
https://pkg.go.dev/github.com/bodgit/sevenzip?utm_source=godoc#NewReader

Thank you!

[mholt]

You want ArchiveFS -- simply specify the Stream field instead of the Path field.

Thanks for using archiver!

(If I've misunderstood your request, feel free to let me know.)

[tenox7]

It worked, thank you! However it was pretty hard to figure out how to use this correctly. It was crashing on me until Format was specified from Identify() with a cast to archiver.Archival. Identify returns io.Reader to be used in further calls but I don't see an easy way to pass it directly to ArchiveFS which requires SectionReader. For this I used io.NewSectionReader from the opened file. But overall its little confusing. It could use an example. Thank you!

[mholt]

Oh... if you don't know the format ahead of time, I guess that can be a little tricky; because Identify() reads some of the bytes, so you have to use the new reader it returns. I didn't realize you had an unknown format.

Maybe we can add an exported API for calling FileSystem on a stream...

[tenox7]

Yeah... this is what I have right now:

	f, _ := os.Open("foo.zip")
	s, _ := f.Stat()
	i, _, _ := archiver.Identify(f.Name(), f)
	a := &archiver.ArchiveFS{
		Stream: io.NewSectionReader(f, 0, s.Size()),
		Format: i.(archiver.Archival),
	}
	a.Sub(".")

So I guess I could reopen f after Indentify()? But still this is little cumbersome. It would be super nice if archiver.FileSystem() could also take stream in addition to filename. Thanks!

[mholt]

M'kay, I'll leave this open and give it some thought. I'd also accept a proposal to review for quicker turnaround!

[aol-nnov]

Well, this is fantastic, thank you! But I have a more sophisticated case with which I'm, ahem, a bit stuck..

1. I have an .iso file, which I turn into fs.FS with (a tweaked version of) go-diskfs (after I've managed to mock Implement standard fs.FS interface and friends  diskfs/go-diskfs#169)
2. ... from which I get an ar file (debian package[1]) which I turn into fs.FS with archiver.ArchiveFS (plus ar implementation of archiver.Archival)
3. ...from which (I'd like to) have a control.tar.gz and data.tar.xz turned into corresponding fs.FS(again, with archiver.ArchiveFS) and pick files from there
4. ...and serve all that with http.FileServer(http.FS(myCrazyStackableFs)) of net/http

So, on each layer of abstraction I have an fs.File, which is, unfortunately, not a io.ReaderAt which is expected as an argument of io.NewSectionReader

As a wild guess, I've made a breaking experiment. I've changed ArchiveFS to the following (which is, I'm quite sure, wrong, but, again, it's a wild guess 😅):

type ArchiveFS struct {
	// set one of these
	Path   string     // path to the archive file on disk, or...
	Stream *io.Reader // ...stream from which to read archive

	Format  Archival        // the archive format
	Prefix  string          // optional subdirectory in which to root the fs
	Context context.Context // optional
}

And surprisingly managed to reach up to point number 3 in my unusual journey, but not further, unfortunately.... Any hints would be appreciated!

The idea behind this is pretty simple: you navigate to a web resource which serves you a list of iso files from a directory on disk. You can dig into iso contents by clicking on corresponding name and navigate deeper and deeper, digging inside all known file types.

[1]: Debian binary package (a .deb file) is actually an ar archive, consisting of three files: control.tar.gz - Debian-specific data, data.tar.xz - actual package contents and debian-binary - a text file. Compression formats may vary.

[mholt]

@aol-nnov Interesting -- thanks for chiming in.

Any reason you can't make the Stream field an io.Reader (no pointer)? Pointers to interfaces are exceptionally unusual.

point number 3 in my unusual journey, but not further, unfortunately....

What happened after that?

[aol-nnov]

@mholt I'm still new to go and some things seem unusual/unexpected, ha!

Any reason you can't make the Stream field an io.Reader (no pointer)?

Indeed, seems, that pointer type was another issue. I'll experiment more with it soon! Thank you for the hint!

What happened after that?

Seems, I've hit another issue, #371 may be. I can list the top-level dir of control.tar.gz but unable to .Open() files from there...

[mholt]

@aol-nnov Thanks for working on this, sorry for the late reply.

Do you want to update your PR with the latest changes? Then maybe we can focus on #371...

[aol-nnov]

Hey, @mholt !

Do you want to update your PR

I do not have any as yet! I think you have me confused with someone else :)

Do you want me to make a PR for Stream io.Reader in ArchiveFS struct?

[mholt]

My bad, you're right -- I guess I'd be curious to see the patch, though -- no guarantees it'll be merged, but curious what it looks like!

[aol-nnov]

@mholt nothing fancy there, but, okay, I'll arrange it soon