Parsing a CSV with a non-CSV header

[davetapley]

I'm trying to parse a 'CSV' file with some non-CSV header before it.
Others have asked about CSVs on StackOverflow but the answer is mostly regexes or import csv,
I really do want to do it all in pyparsing (hence asking here on the repo instead!).

That said I've been tearing my hair out and I just can't get it to parse, so I'm hoping someone can help 🙏🏻

I'll post my whole parser, in case there's any other Bad Things.

---

My input file looks like this:

Some fixed title
Foo: bar_pip
More: info
Some number (N): 123.4456
Pip: 369

Header with, spaces and �, but, this is now, a (mostly) regular, csv,
2022-04-24 15:10:00,2022-04-24 20:10:00,329.0,357.0,13.7,26.7,30.4,16,968,0,1050,
2022-04-24 15:00:00,2022-04-24 20:00:00,316.0,328.0,9.1,17.1,29.9,16,968.1,0,1055,
2022-04-24 14:50:00,2022-04-24 19:50:00,349.0,332.0,14,27.5,29.7,16,968.2,0,1057,

And here's what I've come up with (incl. workaround for #392).
This part seems to work okay:

import pyparsing as pp

class BasicMultilingualPlane(unicode_set):
    "Unicode set for Basic Multilingual Plane (BMP)"
    _ranges: UnicodeRangeList = [(0x0000, 0xFFFF)]

start = pp.Suppress(pp.Literal('Some fixed title'))
foo = pp.Keyword('Foo:') + pp.Word(pp.alphas + '_')('foo')

number = pp.common.real.setParseAction(pp.common.convert_to_float)
num = pp.Keyword('Some number (N):') + number('num')

key = pp.Word(BasicMultilingualPlane.printables, exclude_chars=':')
other = pp.Suppress(key + pp.FollowedBy(':') + pp.rest_of_line)

header = start + (foo | num | other)[...]

Now into the 'CSV' part.

My CSV header parser works, but only because the alphas doesn't match the 2022 on the first row, which feels more like a happy accident than actually handling the new line correctly:

csv_header_word =(pp.Word(BasicMultilingualPlane.alphas + '�()/% '))
csv_header = pp.delimitedList(csv_header_word, allow_trailing_delim=True)('csv_header')

Here's where I've spent hours and am getting nowhere:

csv_body_word = pp.Word(pp.nums + ':-.')
csv_row = pp.delimitedList(csv_body_word, allow_trailing_delim=True).setWhitespaceChars('')
csv_body = pp.Group(csv_row[...]('csv_body'))

I feel like setWhitespaceChars('') is what's tripping me up,
but no matter what I try I can't get the output I want which would be csv_body as a list of lists of strings.

[ptmcg]

I think I only changed code from csv_body_word on down.

1. csv_body_word needs to include a ' ' space character, in order to parse timestamps
2. setting whitespace chars to '' means that you have to explicitly add parsing for LineEnds.
3. it looks like you misgrouped the csv_body expression

# csv_body_word = pp.Word(pp.nums + ':-.')
csv_body_word = pp.Word(pp.nums + ':-. ')  # must add space to parse timestamp values
# csv_row = pp.delimitedList(csv_body_word, allow_trailing_delim=True).setWhitespaceChars('')
csv_row = pp.delimitedList(csv_body_word.setWhitespaceChars(''), allow_trailing_delim=True) + pp.LineEnd().suppress()
# csv_body = pp.Group(csv_row[...]('csv_body'))
csv_body = pp.Group(csv_row)[...]('csv_body')

parser = (pp.Group(header)("header")
          + csv_header("csv_header") + pp.LineEnd()
          + csv_body)
result = parser.parse_string(raw_csv, parse_all=True)
print(result.dump())

[davetapley]

Amazing! Thank you! I was so close 🤪