Slow writes on S3

Using the default file output committer with S3a results in double data writes (sad times!). Use a newer cloud committer such as the "S3 magic committer" or a committer specialized for your hadoop cluster.

Alternatively, write to Apache Iceberg, Delta.io, or Apache Hudi.

Reference links

S3 Magic Committer blog and Hadoop documentation

EMRFS S3-optimized Committer