Slow writes on S3
Using the default file output committer with S3a results in double data writes (sad times!). Use a newer cloud committer such as the "S3 magic committer" or a committer specialized for your hadoop cluster.
Alternatively, write to Apache Iceberg, Delta.io, or Apache Hudi.
Reference links
S3 Magic Committer blog and Hadoop documentation