pyspark.BarrierTaskContext.allGather

BarrierTaskContext.allGather(message='')

This function blocks until all tasks in the same stage have reached this routine. Each task passes in a message and returns with a list of all the messages passed in by each of those tasks.

New in version 3.0.0.

Warning

In a barrier stage, each task much have the same number of allGather() calls, in all possible code branches. Otherwise, you may get the job hanging or a SparkException after timeout.

Notes

This API is experimental

pyspark.RDDBarrier.mapPartitionsWithIndex pyspark.BarrierTaskContext.attemptNumber