Efficient Parquet File Inspection from Command Line: JSON Output and Tool Usage Guide

Core Requirements for Parquet File Inspection

In data processing and analysis workflows, Parquet is widely adopted as an efficient columnar storage format. However, directly inspecting Parquet file contents from the command line often presents two main challenges: first, traditional methods require downloading files from distributed storage systems to local storage, increasing operational complexity and storage overhead; second, default output formats typically lack type information, hindering intuitive data comprehension.

JSON Output Solution with parquet-tools

The parquet-tools utility provides an optimal solution through the cat command combined with the --json option. This command can directly read Parquet files from distributed file systems like HDFS and output content in JSON format, completely eliminating the need for creating local copies.

The basic command format is as follows:

parquet-tools cat --json hdfs://localhost/tmp/save/part-r-00000-6a3ccfae-5eb9-4a88-8ce8-b11b2644d5de.gz.parquet

Executing this command produces output where each line represents one JSON object:

{"name":"gil","age":48,"city":"london"}
{"name":"jane","age":30,"city":"new york"}
{"name":"jordan","age":18,"city":"toronto"}

Technical Implementation Principles

The implementation of this functionality leverages Parquet's columnar storage characteristics. When the --json option is specified, parquet-tools performs the following operations:

1. Accesses remote Parquet files directly through Hadoop filesystem APIs
2. Reads data row by row and reconstructs complete record structures
3. Serializes each record into standard JSON format
4. Maintains integrity of original data types, ensuring correct representation of numerical values, strings, and other types

Additional Practical Commands

Beyond the cat command, parquet-tools offers several other useful inspection commands:

File Metadata Inspection

The meta command displays detailed metadata information about the file:

parquet-tools meta filename.parquet

This command outputs comprehensive structural descriptions including file schema, column types, encoding methods, compression information, and more.

Data Preview

The head command provides quick previews of the first few rows of data:

parquet-tools head -n 5 filename.parquet

Row Count Statistics

The rowcount command quickly retrieves the total number of rows in the file:

parquet-tools rowcount filename.parquet

Alternative Tool Introduction

Besides parquet-tools, other tools provide similar functionality:

parquet-cli Tool

Installation via Python package manager:

pip install parquet-cli

Usage example:

parq input.parquet --head 10

Native Java Tool

For scenarios requiring custom functionality, Parquet's Java library can be used directly:

hadoop jar ./parquet-tools-<VERSION>.jar <command>

Environment Configuration and Best Practices

On macOS systems, parquet-tools can be quickly installed via Homebrew:

brew install parquet-tools

Usage recommendations:

• For large files, combine with less or head commands for paginated viewing
• In production environments, pay attention to file path permission configurations
• Regularly update tool versions to access latest features and security patches

Conclusion

The cat --json command in parquet-tools provides an efficient, direct method for inspecting Parquet files, eliminating unnecessary file copy operations while delivering well-structured JSON format output. Combined with other auxiliary commands, it addresses various requirement scenarios from quick data previews to detailed metadata analysis.