(csv) Add a CsvParser.Feature.SKIP_EMPTY_LINES to allow skipping empty rows

[cowtowncoder]

Looks like a commonly supported option with CSV (family of formats) is to allow skipping of empty rows; that is, ones that only contain white-space.

[cowtowncoder]

note: Original had 5 thumbs-up, couple of +1s. So highly requested.

[kg29]

Is anyone working on this? I would be happy to pick it up.

[cowtowncoder]

@kg29 No one is workign on this as far as I know, so help would be highly valued!

[kunickiaj]

This is the only thing preventing us from switching from commons-csv.

[cowtowncoder]

@kunickiaj One thing that could help (I may have time to look into this after completing non-blocking json parser) would be a failing unit test that simply shows how you'd like this to work. That is, test that would pass after fix.

[kunickiaj]

Yeah, I may look into this once I get a free moment.

We did some benchmarks against jackson csv + guard clause on our side vs commons-csv and it pretty much wiped out any performance gains from moving to jackson.

Figure fixing it here should be a much better option.

[kunickiaj]

Here's a trivial test which would pass given the desired behavior.

https://github.com/kunickiaj/jackson-dataformats-text/commit/b34aa58b5cb0a6926485d70fc17d2bc296eac925

[cowtowncoder]

Excellent thank you; I'll have a look.

[cowtowncoder]

Quick note: I think this makes more sense as CsvParser.Feature than CsvSchema setting since it's not as much a property of document than handling (although I can see why someone might argue otherwise too). But I think this is more convenient from user perspective.

[cowtowncoder]

Hmmh. Looking at code, this may not be easy to implement. As things are, START_OBJECT (or, without schema, START_ARRAY) is returned without looking at line. Logic would need to check for this setting, then do look-ahead... maybe that works.

[cowtowncoder]

For documentation purposes, I'll note that feature itself was actually added in 2.9, but implementation not. This is unfortunate.

[ehills]

Hi @cowtowncoder how did this go? Has this been implemented? Your last comment was a wee bit confusing.
Cheers,
Ed

[cowtowncoder]

@ehills No, functionality has not been added and does not exist.

[feirorum]

Maybe I misunderstand something here, but from the description in the first comment, I think this should be described as "skip blank lines", i.e. lines with only whitespace. Maybe you'd like to add SKIP_BLANK_LINES instead of changing SKIP_EMPTY_LINES to do this, otherwise code which uses the library could change its behaviour when using the new version.

Also, in the linked unit test I don't find any blank (only whitespace) lines, just empty lines:

Current CSV string:
final String CSV = "1,"xyz"\n\ntrue,\n";

Test for blank line being removed would need something like:
final String CSV = "1,"xyz"\n\ntrue,\n \n";

The rest of the test would be identical.

Cheers, Anders

[jabengozartc]

Not works for me yet

[cowtowncoder]

Good point on "skip blank" vs "skip empty".

[cowtowncoder]

And there was much rejoining for now @vboulaye's patch is merges into 2.10, and the feature shall be supported in patch release 2.10.1!