Using jq for Structural JSON File Comparison: Solutions Ignoring Key and Array Order

In software development, data analysis, and system management, comparing JSON files for structural consistency is a common requirement. Traditional text comparison tools like diff fail to handle differences in JSON object key order or array element order, which can lead to false positives. For example, consider the following two JSON files:

{
  "People": ["John", "Bryan"],
  "City": "Boston",
  "State": "MA"
}

{
  "People": ["Bryan", "John"],
  "State": "MA",
  "City": "Boston"
}

Semantically, these files are identical, but a simple text comparison would show differences. To address this, we can leverage the powerful capabilities of jq to achieve structural comparison by recursively sorting arrays and ignoring key order.

Core Solution: Recursive Array Sorting with jq

jq is a lightweight and flexible command-line JSON processor that includes built-in comparison operators, naturally ignoring object key order. However, array order still affects comparison results. To overcome this limitation, we need to sort all arrays before comparing. Here is a complete command based on the best answer:

jq --argfile a a.json --argfile b b.json -n 'def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?); ($a | (post_recurse | arrays) |= sort) as $a | ($b | (post_recurse | arrays) |= sort) as $b | $a == $b'

The core of this command lies in the custom function post_recurse, which recursively traverses all arrays in the JSON structure and applies the sort function to them. Using the --argfile parameter, we load the two JSON files as variables $a and $b, then compare them with the == operator after sorting. If it returns true, the files are structurally identical; otherwise, it returns false.

Technical Details and Implementation Principles

Recursive array sorting is key to this solution. In earlier versions of jq, the simple construct (.. | arrays) |= sort might not handle all edge cases, such as nested arrays or null values. Therefore, we adopt the post_recurse function, which ensures the recursive process correctly traverses all levels. This function is defined as follows:

def post_recurse(f): def r: (f | select(. != null) | r), .; r; def post_recurse: post_recurse(.[]?);

Here, post_recurse traverses the JSON tree by recursively applying function f, skipping null values to avoid infinite loops. Then, (post_recurse | arrays) |= sort applies the sorting operation to all array elements.

Comparison with Other Tools

Besides jq, other tools are available for JSON comparison. For instance, the jd tool offers a -set option that ignores array order:

jd -set A.json B.json

If no output is produced, the files are identical; otherwise, differences are shown with paths. However, jd may be less flexible than jq, especially when dealing with complex recursive structures. Another approach is to use diff with jq --sort-keys, but this only ignores key order and not array order.

Practical Applications and Performance Considerations

In practical applications, this comparison method is suitable for automated testing, data validation, and configuration management. For example, in continuous integration pipelines, this command can be integrated to ensure consistency in JSON configuration files. Performance-wise, for large JSON files, recursive sorting may add computational overhead. To optimize, consider using the --compact-output option to reduce output formatting costs or combine with tools like cmp for quick binary comparisons.

In summary, through jq's recursive array sorting capabilities, we can efficiently compare JSON files for structural equivalence, ignoring differences in key and array order. This method is not only accurate but also easy to integrate into existing workflows, providing developers and system administrators with a powerful toolset.