Parquet

Implementation of the parquet class.

ParquetParser(*args, compression='snappy', **kwargs)

Bases: FormatManager

Class for parsing Parquet files.

PARAMETER	DESCRIPTION
file_path	Path to the Parquet file. TYPE: str

Initialize the ParquetParser class.

VALID_COMPRESSION = ('snappy', 'zstd', 'gzip', 'lz4', 'brotli', 'none')

compression = compression

schema = self.find_schema()

convert_to_pandas(column_names=None)

Convert data to a pandas DataFrame. Args: column_names: Columns to convert to a pandas DataFrame. Returns: A pandas DataFrame containing the data from the specified columns in the Parquet file

convert_to_table(column_names=None)

Convert data to a PyArrow Table.

PARAMETER	DESCRIPTION
column_names	Columns to convert to a PyArrow Table. TYPE: str | list | None DEFAULT: None

RETURNS	DESCRIPTION
Table	A PyArrow Table containing the data from the specified columns in the Parquet file

create()

Create a Parquet file using PyArrow.

filter_data(condition, column, lazy=False)

Filter data based on a column value. Args: condition: the Condition to filter the data. column: Column to filter the data. lazy: if true retuns a Scanner object instead of a table. Default is False. Returns: pandas DataFrame (Lazy=False) or ds.Scanner (Lazy=True) containing the filtered data.

find_schema()

Find the schema of a Parquet file using PyArrow.

RETURNS	DESCRIPTION
schema	pyarrow.Schema TYPE: Schema | None

open_dataset(path, schema=None)

Open a directory of Parquet files as a unified lazy dataset. Args: path: Directory (or single file) containing .parquet files. schema: Optional schema to enforce.

RETURNS	DESCRIPTION
Dataset	A PyArrow Dataset object representing the Parquet files at the specified path.

open_writer(schema=None, compression=None)

Open a ParquetWriter for incremental / streaming writes.

The caller is responsible for closing the writer (use as a context manager or call .close()).

PARAMETER	DESCRIPTION
schema	Arrow schema for the file. Falls back to self.schema if not provided. TYPE: Schema | None DEFAULT: None
compression	Compression codec. Falls back to the instance default set at init. TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
ParquetWriter	An open pq.ParquetWriter.

read(columns=None, filters=None, lazy=False)

Read a Parquet file using PyArrow and offers two apporaches - memory intensive approach when lazy is false, where the entire file is loaded into memory as a PyArrow Table. - When lazy is true, it returns a PyArrow Dataset object that allows for more efficient querying and filtering Args: columns: Columns to read from the Parquet file. filters: Filters to apply when reading the Parquet file. Accepts either a list of tuples (e.g., [('column_name', '==', value)]) or a PyArrow expression. e.g., ds.field('column_name') == value. lazy: if true retuns a Scanner object instead of a table. Default is False. Returns: pa.Table (lazy=False) or ds.Scanner (lazy=True)

read_metadata()

Read only Parquet file metadata (schema + row counts). No row data is loaded, making this quick stats. Returns: Parquet FileMetaData object.

to_csv(output_path)

Convert data to a CSV file. Args: output_path: Path to the output CSV file.

write(data, group_size=None, compression=None)

Write data to a Parquet file. Args: data: Data to write.it accepts a PyArrow Table, a pandas DataFrame, or a list of dictionaries group_size: number of rows per group in the Parquet file (default is None, which means no grouping). compression: Compression codec (default "snappy").