Slow writes due to Too many small files

Sometimes a partitioning approach works fine for a small dataset, but can cause a surprisingly large number of partitions for a slighly larger dataset. Check out The Small File Problem in context of HDFS.