awswrangler.athena.run_spark_calculation

awswrangler.athena.run_spark_calculation(code: str, workgroup: str, session_id: str | None = None, coordinator_dpu_size: int = 1, max_concurrent_dpus: int = 5, default_executor_dpu_size: int = 1, additional_configs: dict[str, Any] | None = None, spark_properties: dict[str, Any] | None = None, idle_timeout: int = 15, boto3_session: Session | None = None) → dict[str, Any]

Execute Spark Calculation and wait for completion.

Parameters:

  • code (str) – A string that contains the code for the calculation.

  • workgroup (str) – Athena workgroup name. Must be Spark-enabled.

  • session_id (str, optional) – The session id. If not passed, a session will be started.

  • coordinator_dpu_size (int, optional) – The number of DPUs to use for the coordinator. A coordinator is a special executor that orchestrates processing work and manages other executors in a notebook session. The default is 1.

  • max_concurrent_dpus (int, optional) – The maximum number of DPUs that can run concurrently. The default is 5.

  • default_executor_dpu_size (int, optional) – The default number of DPUs to use for executors. The default is 1.

  • additional_configs (Dict[str, Any], optional) – Contains additional engine parameter mappings in the form of key-value pairs.

  • spark_properties (Dict[str, Any], optional) – Contains SparkProperties in the form of key-value pairs.Specifies custom jar files and Spark properties for use cases like cluster encryption, table formats, and general Spark tuning.

  • idle_timeout (int, optional) – The idle timeout in minutes for the session. The default is 15.

  • boto3_session (boto3.Session(), optional) – Boto3 Session. The default boto3 session will be used if boto3_session receive None.

Returns:

Calculation response

Return type:

Dict[str, Any]

Examples

>>> import awswrangler as wr
>>> df = wr.athena.run_spark_calculation(
...     code="print(spark)",
...     workgroup="...",
... )