awswrangler.lakeformation.read_sql_table

awswrangler.lakeformation.read_sql_table(table: str, database: str, transaction_id: Optional[str] = None, query_as_of_time: Optional[str] = None, catalog_id: Optional[str] = None, categories: Optional[List[str]] = None, safe: bool = True, map_types: bool = True, use_threads: bool = True, boto3_session: Optional[Session] = None) → Any

Extract all rows from AWS Glue Table (Transaction ID or time travel timestamp). Return Pandas DataFrame.

Note

ORDER BY operations are not honoured. i.e. sql=”SELECT * FROM my_table ORDER BY my_column” is NOT valid

Note

Pass one of transaction_id or query_as_of_time, not both.

Note

This function has arguments which can be configured globally through wr.config or environment variables:

  • catalog_id

  • database

Check out the Global Configurations Tutorial for details.

Parameters

  • table (str) – AWS Glue table name.

  • database (str) – AWS Glue database name

  • transaction_id (str, optional) – The ID of the transaction at which to read the table contents. Cannot be specified alongside query_as_of_time

  • query_as_of_time (str, optional) – The time as of when to read the table contents. Must be a valid Unix epoch timestamp. Cannot be specified alongside transaction_id

  • catalog_id (str, optional) – The ID of the Data Catalog from which to retrieve Databases. If none is provided, the AWS account ID is used by default.

  • categories (Optional[List[str]], optional) – List of columns names that should be returned as pandas.Categorical. Recommended for memory restricted environments.

  • safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

  • map_types (bool, default True) – True to convert pyarrow DataTypes to pandas ExtensionDtypes. It is used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema.

  • use_threads (bool) – True to enable concurrent requests, False to disable multiple threads. When enabled, os.cpu_count() is used as the max number of threads.

  • boto3_session (boto3.Session(), optional) – Boto3 Session. The default boto3 session is used if boto3_session receives None.

Returns

Pandas DataFrame.

Return type

pd.DataFrame

Examples

>>> import awswrangler as wr
>>> df = wr.lakeformation.read_sql_table(
...     table="my_table",
...     database="my_db",
...     catalog_id="111111111111",
... )

>>> import awswrangler as wr
>>> df = wr.lakeformation.read_sql_table(
...     table="my_table",
...     database="my_db",
...     transaction_id="1b62811fa3e02c4e5fdbaa642b752030379c4a8a70da1f8732ce6ccca47afdc9",
... )

>>> import awswrangler as wr
>>> df = wr.lakeformation.read_sql_table(
...     table="my_table",
...     database="my_db",
...     query_as_of_time="1611142914",
...     use_threads=True,
... )