16

This still seems to be a very common topic. Over time, and I guess that's a sign of how the tools matured, our answers and comments to all of these end up being:

It's very nice that you want to use ${text tool} on your structured data, but that's not a good choice because ${variation of the same explanation that regular expressions don't do the format justice or become exceedingly complex for quoted strings}. Instead, you should try using {xmlstarlet,jq,yq,mlr} or try to use your favorite scripting language's (e.g., {Python|perl}) libraries to deal with the data in a syntax-aware manner.

We're really dedicating a lot of work to reproducing these recommendation. Is there a good standard Q&A where we answered that in a sufficiently friendly and generic way, which we could agree to canonically set "duplicate of" to?


(NB: title is meant to be sarcastic; nobody argues that you can't parse a CSV with only integer columns safely with awk. The questions we get asked are typically something like "I have a CSV with URLs and unrestricted remarks in various non-ascii encodings, and my grep one-liner from the depths of the interwebs can't deal with it", and the amount of work qualifiedly answering these questions reliably exceeds the effort put into the question so solidly that I think avoiding duplicating the work would be a good idea.)

6
  • 1
    I suspect Kusalananda will have written this answer a few times. Maybe one of those can be edited to be made more general.
    – terdon Mod
    May 11, 2023 at 9:44
  • There are Qs&As like Is there a robust command line tool for processing csv files? but they don’t feel like good duplicate targets. May 11, 2023 at 15:13
  • 1
    The problem is that if there were one such canonical question+answer, it would be the duplicate of very few questions, as the only thing that would be "duplicated" is the explanation of why it may be better to use dedicated tools to process structured documents. The actual issue that any one question may present would likely not be a duplicate of that "canonical" question.
    – Kusalananda Mod
    May 11, 2023 at 19:11
  • To continue on my previous comment: It would be the "Why is Kali Linux so hard to set up? Why won't people help me?" thing all over again, i.e. a QA that can't be used as a duplicate as its answer doesn't answer any real-life questions, and we should just try to help those people instead or close their questions as unclear (or as dupes of other real questions), whichever is more appropriate.
    – Kusalananda Mod
    May 11, 2023 at 22:50
  • @Kusalananda oh, that's an excellent point. Goal is still to help, indeed, so we need to address the actual question. Hm. that alone would make your answers worth an answer to this question; but I wonder whether we at least have a "common denominator" Q&A that we could use in an answering context, e.g. "For why trying to do this with awk, sed and grep is not going to end up being a good investment of anyone's time, please read [[ this answer ]]. Now, regarding your question, mlr has the --do-what-I-want flag…" May 12, 2023 at 18:23
  • Related? Seems to be the converse of: unix.meta.stackexchange.com/questions/2708/… May 18, 2023 at 7:02

1 Answer 1

15

Summary of long answer: I suggest that a canonical question+answer about using dedicated tools to process structured documents may not be useful for avoiding duplicates because the actual issues presented in questions are often unique. Rather than pointing to a duplicate question+answer, users should be helped directly with their individual issues and referred to a Meta question that describes the limitations of standard Unix tools for processing structured documents, and possible alternatives.

There is also a tangentially related rant at the end.


The problem is that if there were one such canonical question+answer, it would be the duplicate of very few questions, as the only thing that would be "duplicated" is the explanation of why it may be better to use dedicated tools to process structured documents. The actual issue that any one question may present would likely not be a duplicate of that canonical question and answer.

It would be the Why is Kali Linux so hard to set up? Why won't people help me? question all over again, i.e. a question that can't be used as a duplicate as its answer doesn't answer any real-life questions. Instead, we should try to help those people, or close their questions as unclear or as dupes of other real questions, whichever is more appropriate. When answering these questions, we may use the potentially misguided choice of tools, making sure the restrictions, conditions, and potential issues are clear, or we could do as I usually do and ignore the user's pick of tools. Or you do both of the above, which will become tedious in the long run if you tend to answer many of these types of questions.

Regarding the issue with Kali Linux, the "Why won't people help me?" question still serves a purpose. Not as a question that we close dupes against, but as a reference that we can point to when the topic of why it may be a good idea to try another Linux distribution in place of Kali arises (because a user is either in too deep water with Kali or are trying to use the distribution to do things it clearly not designed to do). We still help the user, but we can also provide a hint that they may want to consider another platform for their work, and why. Such a hint can be provided by linking the Kali question on our Meta site.

So, (and I think this would be a really good idea) we could potentially have a Meta question asking, "Why am I told not to parse structured document formats with line-oriented tools?" (or "Why won't people help me sed-uce my JSON docs?" something even more awk-wardly catchy), with appropriate answers that describe the basic issues with using the standard Unix line-based text processing tools, when these tools are perfectly fine to use (because they are sometimes the correct choice if we know enough about the input), and what the alternatives are, etc.


Tangent, no longer really an answer to the Meta-question:

One of the issues I personally have with text-processing answers in general, is that they are sometimes written as one-off answers, i.e., "Run this one-liner and it will do what you need!". Even though answers like these may answer the specific question in the form that it happened to be posed, we have no clue where the code will be used or what changes will be retro-fitted to it, or what input some random future user will try to pass through it may look like.

It is, therefore, vital that we point out the limitations on answers that try to parse e.g. YAML or XML with tools like grep, sed, and awk, as a way of educating users about the fragility and proper use of these standard Unix tools.

I don't want any medical practitioner to use journal systems based on parsing XML with awk, or a car manufacturer filtering JSON equipment test data with sed, just because some lazy programmer lifted a one-liner throwaway solution from a "peer reviewed" Unix site.

Users who don't think the code in their answers here on U&L will be used in production systems must think again.

6
  • 1
    We could still have a canonical "Why is it a bad idea to parse structured documents with a naive parser?" question and you could post a nice, detailed answer explaining why this is a bad idea. It might not be useful as a dupe target, but it would still be useful as a reference and something to point users to instead of just repeating the point in every answer.
    – terdon Mod
    May 16, 2023 at 17:43
  • @terdon This is pretty much what I propose, yes. Although I don't quite see that I would have any monopoly on answering such a question...
    – Kusalananda Mod
    May 16, 2023 at 22:55
  • Thanks for the rather comprehensive, and well-formulated, and backed with experience, answer! Fully agree, the goal is to help, and if we can keep the "you shouldn't be doing what you're trying to do" part short that's in everyone's interest. Focussing on closing a question as duplicate is wrong – it's not the desirable way to point people at "canonical" resources. Nice! May 18, 2023 at 11:25
  • 1
    Regarding the tangent: yes. That. I try to, as often as my time allows, to explain command strings and command line options (gotten a lot of positive feedback on my ASCII-art arrow/explanation schemes), exactly because it's way less useful for future readers, and can be outright dangerous, to just drop uncommented code. Plus, if we really had a lot of good generalizable answers, then pointing out that a new question is really a duplicate and linking to it becomes actually more helpful then writing another specific answer. May 18, 2023 at 11:28
  • RE: automotive software engineers parsing critical data structures with unsuitable tools: The specific of this is no topic for polite conversation (and much less for public conversation), but having friends working in that sector will cause headaches when you hear what their average colleagues are allowed to check in as code. May 18, 2023 at 11:31
  • @MarcusMüller "you should not be doing what you are trying to do" == "U S N B D W U A T T D" ? Or "you should not attempt what you are trying to attempt" == "U S N A W U A T T A" ? Definitely not useful as tag, but an acronym that could be helpful in comments (and chat). May 24, 2023 at 17:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .