Out-of-memory issues with large tsv-file

[Hoeze]

Check list

• I have read through the README
• I have searched through the existing issues

Environment info

• OS
  • Linux
  • Mac OS X
  • Windows
  • Others:

Version

csview 0.3.6

Problem / Steps to reproduce

First of all, thanks for this very nice and useful tool.
I'd like to view a large TSV-type file but it dies because of out-of-memory.

Steps to reproduce:

• wget https://storage.googleapis.com/gcp-public-data--gnomad/release/2.1.1/vcf/exomes/gnomad.exomes.r2.1.1.sites.1.vcf.bgz
• zcat "gnomad.exomes.r2.1.1.sites.1.vcf.bgz" | grep -v "^##" | csview -t | less -S

Would it be possible to limit csview's memory usage and make it play nicely with less?

[wfxr]

Hi @Hoeze, how big is the file ? csview needs to load the entire file into memory to compute alignment. Fow now the easiest way to avoid oom on too large files is to filter the data you focus on then feed them into csview:

preview first 10,000 lines:

zcat "gnomad.exomes.r2.1.1.sites.1.vcf.bgz" | grep -v "^##" | head -n 10000 | csview -t | less -S

preview lines between 100 and 200:

zcat "gnomad.exomes.r2.1.1.sites.1.vcf.bgz" | grep -v "^##" | sed -n '100,200p;201q' | csview -t | less -S

[Hoeze]

I see, thanks for the explanation @wfxr.
Yes, the files can be up to 1TB of compressed size.

One solution could be to calculate alignment on the first 10,000 lines.
Another useful feature would be to split cells which are too large into multiple lines:

+------------------+
| too large column |
+------------------+
| some cell with v |
| ery long text    |
+------------------+

[wfxr]

@Hoeze Unfortunately the crate prettytable-rs which csview depends on does not support streaming rendering. But your advice will be considered when csview implements its own rendering someday.

[wfxr]

Hi @Hoeze, streaming rendering has been implemented by #27. Thanks for your contribution! You can try the latest release. Please let me know if there are any problems.