Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zipping 1GB+ and splitting to chunks is slow - is there a way to speed it up? #916

Open
alexandis opened this issue Jan 27, 2024 · 0 comments

Comments

@alexandis
Copy link

alexandis commented Jan 27, 2024

I see no possibilities to create a multi-volume archive, so I ended up with creating a bunch of chunks for uploading to the server (and then combine them and unzip there).

Here is the code (I'm uploading the files which names contain subfolders (e.g. '/subfolder/subsubfolder/filename.ext') - I need to recreate the structure of folders on server):

  uploadCases(files: File[], equipmentCode: string, chunkId: string): Observable<ChunkDto> {
    return new Observable<ChunkDto>(observer => {
      this.zip = new JsZip();
  
      files.forEach(file => {
        this.zip.file(file.name.replace(/^\/+/, ''), file, { dir: false, createFolders: true });
      });
  
      const chunks: Blob[] = [];
  
      this.zip.generateAsync({ type: 'arraybuffer', streamFiles: true, compression: 'STORE' }).then((arrayBuffer: ArrayBuffer) => {
        const totalSize = arrayBuffer.byteLength;
        let offset = 0;
        while (offset < totalSize) {
          const chunkSize = Math.min(totalSize - offset, maxChunkBytes);
          const chunk = new Blob([arrayBuffer.slice(offset, offset + chunkSize)], { type: 'application/zip' });
          chunks.push(chunk);
          offset += chunkSize;
        }
  
        const uploadChunk = (chunk: Blob, equipmentCode: string, chunkIndex: number): Observable<ChunkDto> => {
          const formData = new FormData();
          formData.append('chunk', chunk);
          formData.append('equipmentCode', equipmentCode);
          formData.append('chunkIdWithIndex', `${chunkId}_${chunkIndex}`);
          return this.restService.request<FormData, HttpEvent<ChunkDto>>({
            method: 'POST',
            url: '/cases/upload',
            body: formData,
            reportProgress: true
          },
          { apiName: this.apiName, skipHandleError: true, observe: Rest.Observe.Events })
          .pipe(
              map(event => {
                if (event.type === HttpEventType.UploadProgress) {
                  return { uploadedBytes: (event as HttpProgressEvent).loaded, totalBytes: chunk.size, complete: false, id: chunkId } as ChunkDto;
                }
                else if (event.type === HttpEventType.Response) {
                  return { uploadedBytes: chunk.size, totalBytes: chunk.size, complete: true, id: chunkId } as ChunkDto;
                }
                else {
                  return { uploadedBytes: 0, totalBytes: 0, complete: false, id: '' } as ChunkDto;
                }
              })
          );
        }
  
        // Sequentially upload each chunk
        concat(...chunks.map((chunk, index) => uploadChunk(chunk, equipmentCode, index))).subscribe(observer);
      });
    });
  }

However, it is pretty slow on large archives. Probably I am doing something wrong and there's a way to speed it up?

@alexandis alexandis changed the title Is there a way to create a multi-volume archive with a predefined volume size here? Is there a way to create a multi-volume archive with the predefined volume size here? Jan 27, 2024
@alexandis alexandis changed the title Is there a way to create a multi-volume archive with the predefined volume size here? Weird '_' top-level folder is generated in the zip file Jan 28, 2024
@alexandis alexandis changed the title Weird '_' top-level folder is generated in the zip file Zipping 1GB+ and splitting to chunks is slow - is there a way to speed it up? Jan 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant