awswrangler.s3.store_parquet_metadata

awswrangler.s3.store_parquet_metadata(path: str, database: str, table: str, filters: Union[List[Tuple], List[List[Tuple]], None] = None, dataset: bool = False, use_threads: bool = True, description: Optional[str] = None, parameters: Optional[Dict[str, str]] = None, columns_comments: Optional[Dict[str, str]] = None, compression: Optional[str] = None, mode: str = 'overwrite', catalog_versioning: bool = False, boto3_session: Optional[boto3.session.Session] = None) → Tuple[Dict[str, str], Optional[Dict[str, str]], Optional[Dict[str, List[str]]]]

Infer and store parquet metadata on AWS Glue Catalog.

Infer Apache Parquet file(s) metadata from from a received S3 prefix or list of S3 objects paths And then stores it on AWS Glue Catalog including all inferred partitions (No need of ‘MCSK REPAIR TABLE’)

The concept of Dataset goes beyond the simple idea of files and enable more complex features like partitioning and catalog integration (AWS Glue Catalog).

Note

On append mode, the parameters will be upsert on an existing table.

Note

In case of use_threads=True the number of threads that will be spawned will be get from os.cpu_count().

Parameters
  • path (Union[str, List[str]]) – S3 prefix (e.g. s3://bucket/prefix) or list of S3 objects paths (e.g. [s3://bucket/key0, s3://bucket/key1]).

  • database (str) – Glue/Athena catalog: Database name.

  • table (str) – Glue/Athena catalog: Table name.

  • filters (Union[List[Tuple], List[List[Tuple]]], optional) – List of filters to apply, like [[('x', '=', 0), ...], ...].

  • dataset (bool) – If True read a parquet dataset instead of simple file(s) loading all the related partitions as columns.

  • use_threads (bool) – True to enable concurrent requests, False to disable multiple threads. If enabled os.cpu_count() will be used as the max number of threads.

  • description (str, optional) – Glue/Athena catalog: Table description

  • parameters (Dict[str, str], optional) – Glue/Athena catalog: Key/value pairs to tag the table.

  • columns_comments (Dict[str, str], optional) – Glue/Athena catalog: Columns names and the related comments (e.g. {‘col0’: ‘Column 0.’, ‘col1’: ‘Column 1.’, ‘col2’: ‘Partition.’}).

  • compression (str, optional) – Compression style (None, snappy, gzip, etc).

  • mode (str) – ‘overwrite’ to recreate any possible existing table or ‘append’ to keep any possible existing table.

  • catalog_versioning (bool) – If True and mode=”overwrite”, creates an archived version of the table catalog before updating it.

  • boto3_session (boto3.Session(), optional) – Boto3 Session. The default boto3 session will be used if boto3_session receive None.

Returns

The metadata used to create the Glue Table. columns_types: Dictionary with keys as column names and vales as data types (e.g. {‘col0’: ‘bigint’, ‘col1’: ‘double’}). / partitions_types: Dictionary with keys as partition names and values as data types (e.g. {‘col2’: ‘date’}). / partitions_values: Dictionary with keys as S3 path locations and values as a list of partitions values as str (e.g. {‘s3://bucket/prefix/y=2020/m=10/’: [‘2020’, ‘10’]}).

Return type

Tuple[Dict[str, str], Optional[Dict[str, str]], Optional[Dict[str, List[str]]]]

Examples

Reading all Parquet files metadata under a prefix

>>> import awswrangler as wr
>>> columns_types, partitions_types, partitions_values = wr.s3.store_parquet_metadata(
...     path='s3://bucket/prefix/',
...     database='...',
...     table='...',
...     dataset=True
... )