Is the zip writer concurrent?

[apprehensions]

Hi, i would like to ask if it is possible to make a archive writer concurrently for performance reasons.

Currently, github.com/klauspost/compress/zip leads me to a zip: write to closed file with the following program:

package main

import (
	"bytes"
	"fmt"
	"io"
	"sync"

	"github.com/google/uuid"
	"github.com/klauspost/compress/zip"
)

func main() {
	buf := new(bytes.Buffer)
	zw := zip.NewWriter(buf)
	wg := new(sync.WaitGroup)
	r := 512

	wg.Add(r)
	for i := 0; i < r; i++ {
		go func() {
			defer wg.Done()
			r, err := zw.Create(uuid.New().String())
			if err != nil {
				panic(err)
			}

			if _, err := io.WriteString(r, uuid.New().String()); err != nil {
				panic(err)
			}
		}()
	}

	wg.Wait()

	if err := zw.Close(); err != nil {
		panic(err)
	}

	fmt.Printf("wrote %d", len(buf.Bytes()))
}

Is it possible to make a archive with a concurrent writer, and if there isn't, are there any alternatives?

[klauspost]

@apprehensions No. There is no concurrent zip writer that I know of.

The only thing I see making sense being concurrent would be the deflate part, since each file needs to be written serially. If you know you are only writing small files I guess you can do the deflate in separate goroutines and use a fan-in to add the compressed data with CreateRaw. But it is not something I have explored too deeply.

[apprehensions]

My usecase is to retrieve zips from a remote source and compile them to one big one, and using CreateRaw seems to deem failure:

	resp, err := http.Get(url)
	if err != nil {
		return err
	}

	fh := &zip.FileHeader{
		Name:   dir + "/",
		Method: zip.Deflate,
	}
	fh.SetModTime(time.Now())

	r, err := zw.CreateRaw(fh)
	if err != nil {
		return fmt.Errorf("create dest %s: %w", pkg.Name, err)
	}

	if _, err := io.Copy(r, resp.Body); err != nil {
		return fmt.Errorf("copy dest %s: %w", pkg.Name, err)
	}

panic: copy dest <zipped file>: zip: write to directory

[klauspost]

Without digging deeper it seems like it is just complaining that you are trying to write data to something with a directory name.

This is not different than the stdlib.

[apprehensions]

Well thanks anyway, i got the answer i needed!

Although that doesn't quite answer my question: is there any sort of alternative available that i can use to concurrently archive to?

[klauspost]

I don't know any. Since most time will be spent with request, you could get away with:

func main() {
	buf := new(bytes.Buffer)
	zw := zip.NewWriter(buf)
	wg := new(sync.WaitGroup)
	type task struct {
		name string
		data []byte
	}
	r := 512
	toComp := make(chan task, r)
	wg.Add(r)
	for i := 0; i < r; i++ {
		go func(i int) {
			defer wg.Done()
			resp, err := http.Get("https://example.com")
			if err != nil {
				panic(err)
			}
			data, err := io.ReadAll(resp.Body)
			if err != nil {
				panic(err)
			}
			toComp <- task{
				name: fmt.Sprintf("file-%d.txt", i),
				data: data,
			}
		}(i)
	}
	var writeWG sync.WaitGroup
	writeWG.Add(1)
	go func() {
		defer writeWG.Done()
		for job := range toComp {
			r, err := zw.Create(job.name)
			if err != nil {
				panic(err)
			}
			_, err = r.Write(job.data)
			if err != nil {
				panic(err)
			}
		}
		if err := zw.Close(); err != nil {
			panic(err)
		}
	}()
	wg.Wait()
	close(toComp)
	writeWG.Wait()

	fmt.Printf("wrote %d", len(buf.Bytes()))
}

There are several alternatives. A) compressing in the request goroutine - adding it as Raw. B) sending the io.Reader - less concurrent, but you don't risk exploding memory. But all with a similar structure, so you can adjust as you need.