Unable to determine if a value in the CSV is quoted

[jmuehlner]

I am trying to parse some CSV that was generated by a database CSV export, where NULL fields are represented by the unquoted string NULL, whereas the string "NULL" would be represented by a quoted string "NULL".

Currently I have no way to determine the difference between these 2 cases.

It would be really handy if there was some equivalent to the CSV_EMPTY_IS_NULL option, or even better, if the value of csv_parser->quoted was usable in the row and field handlers.

[bobhairgrove]

If your code set up the csv_parser object in the first place, you should still have access to it.

Otherwise, knowing that the first argument (void *) received by your field callback function cb1 is actually the csv_parser->entry_buf pointer, and considering that the definition of the csv_parser struct is known (from the header csv.h:
It should be relatively easy, using pointer arithmetic and some casts, to get either the pointer to the whole struct or just the pointer to the member quoted.

Another idea would be to set the quote character to some byte value you are sure won't occur anywhere in the input -- something in the range of ASCII control codes, for example. Then the csv_parse() function should return any enclosing quotes along with the field data (which you would have to strip off manually, of course, in the callback function). However, this has the major caveat that there are no fields in the input file which have the delimiter character embedded in one of the fields, and that there are no embedded line feed characters in any fields. If either case is true, parsing will not work as expected and will probably exit prematurely with an error.

To me, it looks like the first option would be safer.

[bobhairgrove]

Hmmm ... csv_parser->entry_buf is dynamically-allocated, so this isn't such a good idea after all.

There is a third parameter data, also void *, which can be passed to the csv_parse() function. It will be passed on to the callback function. If it is not aleady being used for something else, this could be set to the address of the csv_parser struct by whatever code calls csv_parse().

[jmuehlner]

Sorry, I should have clarified - I have access to the csv_parser struct, but the value of the quoted field is always 0 when read from within a field or row handler.

And I could do some hackery around quote characters, but then field processing won't work correctly - if a field delimiter is quoted, the library won't properly handle it and I'll potentially get multiple fields or partial fields in a single callback. At that point I'd basically have to implement my own parser.

[bobhairgrove]

OK, I see the problem now. While parsing a field, csv_parse() sets quoted to TRUE, then sets it to FALSE before calling cb1. However, quoted is a local variable, and the csv_parser->quoted member is not touched. There is really no way of knowing whether it was quoted or not based solely on that information.

Is this a general issue, or only regarding "NULL" and NULL?

At the top of the csv_parse() function, there is this section of code:

  /* Store key fields into local variables for performance */
  unsigned char delim = p->delim_char;
  unsigned char quote = p->quote_char;
  int (*is_space)(unsigned char) = p->is_space;
  int (*is_term)(unsigned char) = p->is_term;
  int quoted = p->quoted;
  int pstate = p->pstate;
  size_t spaces = p->spaces;
  size_t entry_pos = p->entry_pos;

Unfortunately, in C++ we could make reference variables easily enough out of these, but we are in C land here...

[jmuehlner]

Yeah, that section of code seems to be the issue. If you got rid of all the local variables there and just use the values directly on the struct it should work fine.

And my issue is specifically with NULL values but somebody else might have the same problem with any other value for which they care about quoting.

[bobhairgrove]

All that I could come up with -- which does involve a change to the source code -- is adding two lines to the macro SUBMIT_FIELD which passes the value of the local variable quoted back into the csv_parser struct, i.e. (untested code follows):

#define SUBMIT_FIELD(p) \
  do { \
   p->quoted = quoted; \    /* <=== added */
   if (!quoted) \
     entry_pos -= spaces; \
   if (p->options & CSV_APPEND_NULL) \
     ((p)->entry_buf[entry_pos]) = '\0'; \
   if (cb1 && (p->options & CSV_EMPTY_IS_NULL) && !quoted && entry_pos == 0) \
     cb1(NULL, entry_pos, data); \
   else if (cb1) \
     cb1(p->entry_buf, entry_pos, data); \
   pstate = FIELD_NOT_BEGUN; \
   entry_pos = quoted = spaces = 0; \
   p->quoted = 0; \    /* <=== added */
 } while (0)

Then you will have access to the quoted state during the callback.

[jmuehlner]

Yeah, that would definitely work. Or getting rid of the quoted local variable and using the one from the struct pointer could work. The comment says that the local variables are for there for performance reasons, but it's unclear to me how much difference that would make.