Slow Writes

The Shuffle Write time is visible as follows:

Spark UI --> Stages Tab --> Stages Detail --> Event timeline.

Symptom: my spark job is spending more time writing files to disk on shuffle writes.

Some potential causes: