Do we have a go-to reference Q&A for "I want to do {grep,sed,awk} on {xml,json,yaml,csv with text}, how do I shoot myself in the foot?" questions?

Question

This still seems to be a very common topic. Over time, and I guess that's a sign of how the tools matured, our answers and comments to all of these end up being:

It's very nice that you want to use ${text tool} on your structured data, but that's not a good choice because ${variation of the same explanation that regular expressions don't do the format justice or become exceedingly complex for quoted strings}. Instead, you should try using {xmlstarlet,jq,yq,mlr} or try to use your favorite scripting language's (e.g., {Python|perl}) libraries to deal with the data in a syntax-aware manner.

We're really dedicating a lot of work to reproducing these recommendation. Is there a good standard Q&A where we answered that in a sufficiently friendly and generic way, which we could agree to canonically set "duplicate of" to?

---

(NB: title is meant to be sarcastic; nobody argues that you can't parse a CSV with only integer columns safely with awk. The questions we get asked are typically something like "I have a CSV with URLs and unrestricted remarks in various non-ascii encodings, and my grep one-liner from the depths of the interwebs can't deal with it", and the amount of work qualifiedly answering these questions reliably exceeds the effort put into the question so solidly that I think avoiding duplicating the work would be a good idea.)

Answer 1

Summary of long answer: I suggest that a canonical question+answer about using dedicated tools to process structured documents may not be useful for avoiding duplicates because the actual issues presented in questions are often unique. Rather than pointing to a duplicate question+answer, users should be helped directly with their individual issues and referred to a Meta question that describes the limitations of standard Unix tools for processing structured documents, and possible alternatives.

There is also a tangentially related rant at the end.

---

The problem is that if there were one such canonical question+answer, it would be the duplicate of very few questions, as the only thing that would be "duplicated" is the explanation of why it may be better to use dedicated tools to process structured documents. The actual issue that any one question may present would likely not be a duplicate of that canonical question and answer.

It would be the Why is Kali Linux so hard to set up? Why won't people help me? question all over again, i.e. a question that can't be used as a duplicate as its answer doesn't answer any real-life questions. Instead, we should try to help those people, or close their questions as unclear or as dupes of other real questions, whichever is more appropriate. When answering these questions, we may use the potentially misguided choice of tools, making sure the restrictions, conditions, and potential issues are clear, or we could do as I usually do and ignore the user's pick of tools. Or you do both of the above, which will become tedious in the long run if you tend to answer many of these types of questions.

Regarding the issue with Kali Linux, the "Why won't people help me?" question still serves a purpose. Not as a question that we close dupes against, but as a reference that we can point to when the topic of why it may be a good idea to try another Linux distribution in place of Kali arises (because a user is either in too deep water with Kali or are trying to use the distribution to do things it clearly not designed to do). We still help the user, but we can also provide a hint that they may want to consider another platform for their work, and why. Such a hint can be provided by linking the Kali question on our Meta site.

So, (and I think this would be a really good idea) we could potentially have a Meta question asking, "Why am I told not to parse structured document formats with line-oriented tools?" (or "Why won't people help me sed-uce my JSON docs?" something even more awk-wardly catchy), with appropriate answers that describe the basic issues with using the standard Unix line-based text processing tools, when these tools are perfectly fine to use (because they are sometimes the correct choice if we know enough about the input), and what the alternatives are, etc.

---

Tangent, no longer really an answer to the Meta-question:

One of the issues I personally have with text-processing answers in general, is that they are sometimes written as one-off answers, i.e., "Run this one-liner and it will do what you need!". Even though answers like these may answer the specific question in the form that it happened to be posed, we have no clue where the code will be used or what changes will be retro-fitted to it, or what input some random future user will try to pass through it may look like.

It is, therefore, vital that we point out the limitations on answers that try to parse e.g. YAML or XML with tools like grep, sed, and awk, as a way of educating users about the fragility and proper use of these standard Unix tools.

I don't want any medical practitioner to use journal systems based on parsing XML with awk, or a car manufacturer filtering JSON equipment test data with sed, just because some lazy programmer lifted a one-liner throwaway solution from a "peer reviewed" Unix site.

Users who don't think the code in their answers here on U&L will be used in production systems must think again.