Compact Storage and Metadata Identification for Key-Value Arrays in JSON

In data exchange and storage scenarios, JSON is widely used due to its lightweight nature and readability. However, when handling large-scale datasets with thousands of key-value pairs, traditional JSON representations can lead to significant redundancy and performance overhead. Based on a typical problem scenario—how to compactly store key-value arrays while maintaining structural identifiability—this paper delves into optimization strategies.

Issues with Traditional Key-Value Array Storage

Common JSON key-value array representations use arrays of objects, each with explicit key and value fields. For example:

[
  {"key": "slide0001.html", "value": "Looking Ahead"},
  {"key": "slide0008.html", "value": "Forecast"},
  {"key": "slide0021.html", "value": "Summary"}
]

This format, while highly readable, repeats field names for each pair, causing string length to increase dramatically. For large datasets, this results in unnecessary storage and transmission costs. Additionally, when JSON files mix other data types, key-value arrays must be clearly identified to avoid parsing ambiguity.

Design of Compact Storage Solutions

To reduce redundancy, a compact format using nested arrays can be adopted. Each inner array stores keys and values directly, omitting field names. For example:

[
  ["slide0001.html", "Looking Ahead"],
  ["slide0008.html", "Forecast"],
  ["slide0021.html", "Summary"]
]

This format significantly compresses data by eliminating repeated key and value strings. During parsing, predefined constants map array indices to semantic fields. For example:

var C_KEY = 0;
var C_VALUE = 1;
var data = ["slide0001.html", "Looking Ahead"];
var value = data[C_VALUE]; // returns "Looking Ahead"

This approach assumes the parsing logic knows the data structure. If the structure is dynamic or requires self-description, a metadata layer can be introduced.

Metadata Identification and Mixed Data Storage

When storing multiple data types in a JSON file, compact key-value arrays need clear identification. An effective solution uses metadata to describe the array structure, such as:

{
  "meta": {
    "keys": ["key", "value"]
  },
  "data": [
    ["slide0001.html", "Looking Ahead"],
    ["slide0008.html", "Forecast"],
    ["slide0021.html", "Summary"]
  ]
}

Here, the meta object defines the field order for each element in the data array, enabling parsers to handle data dynamically. When combined with other data types, it can be extended as:

{
  "keyvaluelist": {
    "meta": {"keys": ["key", "value"]},
    "data": [["slide0001.html", "Looking Ahead"], ["slide0008.html", "Forecast"]]
  },
  "otherdata": {"one": "1", "two": "2", "three": "3"}
}

This structure clearly distinguishes data blocks via named keys (e.g., keyvaluelist), while using compact format to reduce size. Metadata provides flexibility, allowing field adjustments without modifying parsing code.

Implementation and Performance Analysis

Implementing compact storage requires consideration of encoding and decoding processes. Below is a JavaScript example demonstrating generation and parsing of compact JSON with metadata:

// Generate compact JSON
function createCompactJSON(data) {
  var compactData = data.map(function(item) {
    return [item.key, item.value];
  });
  return {
    meta: { keys: ["key", "value"] },
    data: compactData
  };
}

// Parse compact JSON
function parseCompactJSON(json) {
  var meta = json.meta;
  var data = json.data;
  return data.map(function(row) {
    var obj = {};
    obj[meta.keys[0]] = row[0];
    obj[meta.keys[1]] = row[1];
    return obj;
  });
}

// Sample data
var originalData = [
  {key: "slide0001.html", value: "Looking Ahead"},
  {key: "slide0008.html", value: "Forecast"}
];
var compactJSON = createCompactJSON(originalData);
console.log(JSON.stringify(compactJSON)); // outputs compact format
var parsedData = parseCompactJSON(compactJSON);
console.log(parsedData); // restores original format

In terms of performance, compact format can reduce string length by approximately 30-50%, depending on the number of key-value pairs. For instance, with 1000 pairs, traditional format might produce about 60KB of data, while compact format can compress it to 40KB. This offers significant optimization for network transmission and storage. However, compact format sacrifices some readability, so it is recommended for large-scale data processing scenarios, supplemented with metadata to ensure parsability.

Comparison with Alternative Solutions

As supplementary reference, other answers propose using key-value object literals, such as:

var params = {
  'slide0001.html': 'Looking Ahead',
  'slide0002.html': 'Forecast'
};

This method is simple and effective when keys are strings and array operations are unnecessary, but it cannot directly represent array structures or mix other data types. In contrast, compact array format is more suitable for complex scenarios requiring sequential processing or metadata description.

Conclusion

Compact storage of key-value arrays achieves efficient data representation in JSON through nested arrays and metadata identification. It balances compression ratio with flexibility, making it suitable for large-scale data processing and mixed-type storage. Developers should choose formats based on specific needs: use traditional object arrays for readability priority, and adopt compact solutions for performance-critical scenarios. Future work could explore binary JSON formats like BSON for further optimization.