Overlooked dev tools: jq

Denis Lebedev |

I'm constantly looking for ways to improve my development workflow. I couldn't say that I am obsessed with the concept of efficiency, rather I am just a lazy person! If something could be done easier (with help of new tools or automation) and rough cost-benefit analysis shows that it is worth doing I'd give it a try.

Recent examples:

  • mergepbx to not bother with project file merges.
  • Sourcery to generate some of the code I'm really not keen to write over and over again. More on this tool next time.

The task

Currently I'm working on a fairly complicated feature which involves processing and interpreting deeply nested JSON payload. Even though there is a bit of documentation I prefer to see the data myself and infer as much of requirements directly from it. Luckily, API developers have created a test endpoint which returns all the supported "forms". It's quite big payload:

wc -l payload.json
> 7481

{
    sections: {
        ...
        type: "..."
        alternatives: {
            ...
            fields: {
                ...
                group: {
                    ...
                }
            }
        }
    }
}

First assumption I wanted to verify is if sections have more than 1 nested alternatives and if they do I wanted to quickly find this section to inspect visually (just knowing its type should be enough for this).

Rather than just browsing 7k lines json and believe that I haven't missed something I decided to use jq.

jq

jq is described "a lightweight and flexible command-line JSON processor".

After quickly studying the manual, I've figured how to get the answer:

jq '.sections[] | .type + ", alternatives: " + (.alternatives | length | tostring)' payload.json

"ssn, alternatives: 1"
"identity_upload, alternatives: 1"
"business_owners, alternatives: 2"
...
"address_upload, alternatives: 1"

Good! Then I wanted to know how many fields each section had:

jq '.sections[] | .type + ", fields: " + (.alternatives[] |.fields | length | tostring)' payload.json

"ssn, fields: 2"
"identity_upload, fields: 3"
"business_owners, fields: 6"
"business_owners, fields: 1"
...
"address_upload, fields: 2"

I've spent not more than 10 minutes writing these predicates. Well, I haven’t figured yet how to make more complex queries involving select(.type == ""), but I was able to analyse my assumptions about the payload structure. I am certain next time jq would come in handy again, say, to determine how many distinct values a field can have, etc.

Show Comments