Warning

This is a legacy version, please check out the latest one here.

DataFrames on AWS

• Install
  • Setting Up Lambda Layer
• Read the Tutorials
• Examples
  • Pandas
    • Writing Pandas Dataframe to S3 + Glue Catalog
    • Writing Pandas Dataframe to S3 as Parquet encrypting with a KMS key
    • Reading from AWS Athena to Pandas
    • Reading from AWS Athena to Pandas in chunks (For memory restrictions)
    • Reading from AWS Athena to Pandas with the blazing fast CTAS approach
    • Reading from Glue Catalog (Parquet) to Pandas
    • Reading from S3 (Parquet) to Pandas
    • Reading from S3 (CSV) to Pandas
    • Reading from S3 (CSV) to Pandas in chunks (For memory restrictions)
    • Reading from CloudWatch Logs Insights to Pandas
    • Typical Pandas ETL
    • Loading Pandas Dataframe to Redshift
    • Extract Redshift query to Pandas DataFrame
    • Loading Pandas Dataframe to Aurora (MySQL/PostgreSQL)
    • Extract Aurora query to Pandas DataFrame (MySQL)
  • PySpark
    • Loading Pyspark Dataframe to Redshift
    • Register Glue table from Dataframe stored on S3
    • Flatten nested PySpark DataFrame
  • General
    • Deleting a bunch of S3 objects
    • Get CloudWatch Logs Insights query results
    • Load partitions on Athena/Glue table (repair table)
    • Create EMR cluster
    • Athena query to receive the result as python primitives (Iterable[Dict[str, Any])
    • Load and unzip SageMaker model
    • Load and unzip SageMaker job output
• Diving Deep
  • Relational Databases (SQL) - (Oracle, PostgreSQL, MySQL, Microsoft SQL Server, etc)
  • Parallelism, Non-picklable objects and GeoPandas
  • Pandas with null object columns (UndetectedType exception)
  • Athena to Pandas Flow (PARALLEL/CTAS)
  • Pandas to Redshift Flow
  • Spark to Redshift Flow
• awswrangler package
  • Submodules
    • awswrangler.athena module
    • awswrangler.aurora module
    • awswrangler.cloudwatchlogs module
    • awswrangler.data_types module
    • awswrangler.dynamodb module
    • awswrangler.emr module
    • awswrangler.exceptions module
    • awswrangler.glue module
    • awswrangler.pandas module
    • awswrangler.redshift module
    • awswrangler.s3 module
    • awswrangler.sagemaker module
    • awswrangler.session module
    • awswrangler.spark module
    • awswrangler.utils module
  • Module contents
• Contributing
  • Steps
• License