pyspark.sql.SparkSession.registerProgressHandler

SparkSession.registerProgressHandler(handler)

Register a progress handler to be called when a progress update is received from the server.

New in version 4.0.

Parameters

handlerProgressHandler

A callable that follows the ProgressHandler interface. This handler will be called on every progress update.

Examples

>>> def progress_handler(stages, inflight_tasks, done):
...     print(f"{len(stages)} Stages known, Done: {done}")
>>> spark.registerProgressHandler(progress_handler)
>>> res = spark.range(10).repartition(1).collect()  
3 Stages known, Done: False
3 Stages known, Done: True
>>> spark.clearProgressHandlers()