Complete Guide to Filtering Objects in JSON Arrays Based on Inner Array Values Using jq

Technical Analysis of jq Filtering for Nested Array Objects

When processing JSON data, it is often necessary to filter arrays of objects based on values in nested arrays. jq, as a powerful JSON processing tool, provides flexible filtering mechanisms to address such problems.

Problem Scenario Analysis

Consider the following JSON input data containing an array of objects, where each object has an Id field and a Names nested array:

[
  {
    "Id": "cb94e7a42732b598ad18a8f27454a886c1aa8bbba6167646d8f064cd86191e2b",
    "Names": [
      "condescending_jones",
      "loving_hoover"
    ]
  },
  {
    "Id": "186db739b7509eb0114a09e14bcd16bf637019860d23c4fc20e98cbe068b55aa",
    "Names": [
      "foo_data"
    ]
  },
  {
    "Id": "a4b7e6f5752d8dcb906a5901f7ab82e403b9dff4eaaeebea767a04bac4aada19",
    "Names": [
      "jovial_wozniak"
    ]
  },
  {
    "Id": "76b71c496556912012c20dc3cbd37a54a1f05bffad3d5e92466900a003fbb623",
    "Names": [
      "bar_data"
    ]
  }
]

The goal is to filter out all objects from the array whose Names field does not contain the string "data", and output their Id values separated by newlines.

Core Filtering Technique

The correct jq filter should use the map function combined with select and contains:

. - map(select(.Names[] | contains("data"))) | .[] .Id

The working principle of this filter can be divided into three steps:

1. Array Difference Operation: . - map(...) removes elements meeting the condition from the original array
2. Conditional Filtering: select(.Names[] | contains("data")) selects objects containing the "data" string
3. Result Extraction: .[] .Id iterates through the remaining array and extracts the Id field

Syntax Details Analysis

In the select expression, the pipe symbol | must be used to pass array elements to the contains function. This is an important rule in jq syntax, ensuring correct data flow.

Incorrect writing: select(.Names[] contains("data"))

Correct writing: select(.Names[] | contains("data"))

Array Structure Preservation Methods

If maintaining the output as a complete JSON array rather than a stream is required, array wrapping or the map function can be used:

[ .[] | select(.Names[] | contains("data")) ]

Or equivalently:

map(select(.Names[] | contains("data")))

Both methods will produce a complete JSON array containing the filtered objects.

Practical Application Extensions

Filtering technology based on nested arrays has wide applications in complex JSON processing. For example, when processing log data, sorting and filtering based on timestamp arrays may be necessary:

{ log: { foo: "bar", entries: [
  { "s": "2015-02-19T11:02:31.869Z", "d": "ONE" },
  { "s": "2015-02-19T11:02:32.315Z", "d": "TWO" },
  { "s": "2015-02-19T11:02:32.295Z", "d": "THREE" },
  { "s": "2015-02-19T11:02:32.571Z", "d": "FOUR" }
]}}

Similar filtering techniques combined with sorting functions can be used to handle such nested structures.

Best Practices Summary

Mastering nested array filtering in jq requires understanding several key concepts: correct usage of pipe operators, conditional expressions in select functions, and various array manipulation methods. By practicing these techniques, various complex JSON data filtering requirements can be efficiently handled.