Slow line list download speeds

[sratcliffe118]

• Do we have a sense for the speeds when a user clicks the download button?
• I just tested on a 70-75Mb per second connection in UK for 137k entries and it is still running +2mins
• This feels too slow for users

@Mougk Are there any download speed benchmarks / datasets we can look to?

[attwad]

Took me 1.4min to start the content download of 130K cases:

That's quite a long time.

After that network seems to be the bottleneck as the servers aren't using any crazy CPU or RAM at all

kubectl top pods
NAME                            CPU(cores)   MEMORY(bytes)
curator-prod-7bd48b496f-tbrtb   3m           53Mi
data-prod-c469b5ccf-rsqtj       18m          44Mi

Download seems to have stopped altogether after 2.2min though and no sign of anything going on other than that spinning widget in the download button.

I'm not sure if we can set some special http headers to start the download immediately in the browser instead of waiting for what seems like the complete file to be sent. @allysonjp715 in case she has any thoughts?

[attwad]

This might be relevant: https://nodejs.org/pt-br/docs/guides/backpressuring-in-streams/

Also, we should probably gzip the CSV before sending it, it will reduce the size of the download and maybe make it faster?
The stream pipeline stuff of node described in this website looks nice, perhaps we can use it in the curator service as well to do this?

[allysonjp715]

Just found this, could be part of the problem: axios/axios#479

[attwad]

Also, I've looked a gzip and nginx already does gzip compression of responses for us (visible in the dev tools nettwork tab), although not for the download it seems, we should be able to configure it so that it does:

https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#gzip-types

[attwad]

Just found this, could be part of the problem: axios/axios#479

I don't think it applies to us though? We're using node on the backend, not in the browser.

[allysonjp715]

That bug is on axios. So IIUC, the response stream can't be handled while it's coming in (if it's going across the browser). So we couldn't start the download until all the data has come in. The slow UI after clicking download makes sense then because it's holding all the data in memory until the full response comes in, only then does the download happen.

Agreed we should try zipping the data.

[attwad]

I've enabled gzipping of csv data in nginx, will send a PR for that.

[attwad]

That bug is on axios. So IIUC, the response stream can't be handled while it's coming in (if it's going across the browser). So we couldn't start the download until all the data has come in. The slow UI after clicking download makes sense then because it's holding all the data in memory until the full response comes in, only then does the download happen.

Agreed we should try zipping the data.

I see, in that that's why the download don't work at all as we can't hold all the data in memory for sure.

[attwad]

Reloaded dev, took me 7s to start the download then browser took it over and it worked nicely, thanks @allysonjp715 for the fix! (I sent you a PR to push to prod as well today).

[attwad]

[allysonjp715]

Unfortunately this isn't working in prod. For 100,000 cases it takes ~30s for the download to start. It shows the URL pending bar in the lower left hand corner, but we should show a loading spinner in the UX for that too.

I've also tried a couple times now and the download keeps hanging at ~46MB. The browser's download object shows continual spinning at that number and doesn't complete or increase after that.

[allysonjp715]

The download is fully working after that last PR 👍 The browser download object shows up immediately and begins the download, and the full download succeeds with ~100,000 cases. The full response is streaming so it should succeed no matter how many cases there are.

[SandraAdele]

Line list download either slow or does not work. I get the error code "502 Bad Gateway" @calremmel

[Mougk]

In addition download speed was very slow (>10minutes for me)

[calremmel]

Switching to a cached download for the entire dataset should help address this. Most filtered subsets should be fine, but we should be mindful for large filtered subsets, such as Brazil, which can have in excess of 100K cases. If many people try to do this, we should think of how we may want to make this more user friendly for them.

[calremmel]

Next steps:

• Write script to export cases as csv in 100K chunks
• Write script to process nested arrays and other formatting per chunk
• Write script to combine chunks into single file and compress
• Schedule pipeline to run nightly at 12am UTC

[calremmel]

Tracked in #1436