S3 Multipart Uploads
When uploading large objects into S3, you should not use a single upload command: A single upload command, like when you do from the console, uses a single TCP connection and if this connection fails for some reason, you've got to start the whole upload again. Plus, single TCP connections sometimes suffer from bandwidth limitations so your upload takes longer than necessary. For this reason Amazon supports multi-part uploads into S3.
AWS currently has the following restrictions, that determine whether multi-part uploads are possible and/or required, and that determine the optimum part size:
- A single S3 object cannot exceed 5 TB.
- A single upload cannot exceed 5 GB.
- The maximum number of parts in a multi-part upload is 10,000.
This means that, for instance, the part size for a 5 TB upload need to be between 500 MB and 5 GB). In practice, the part size will be determined by the bandwidth and reliability of your internet connection: Smaller parts require less overhead in case of a dropped connection, but have a relatively larger overhead. Also, since all parts are encrypted while in transit (as part of the https protocol), your CPUs capability to encrypt the data may also be a bottleneck. If possible, choose an encryption algorithm that is supported in hardware by your CPU (e.g. AES-NI).
A multipart upload has three stages:
- Perform the API call "create-multipart-upload". This announces your intention to AWS to perform a multipart upload, and assigns an Upload ID.
- Perform the multipart uploads. You need to supply the Upload ID and the part number. You will be given an ETag when each part upload finishes. Obviously all the parts can and should be uploaded in parallel.
- Complete the multipart upload with the "complete-multipart-upload" API call. When you perform this call you need to supply a "parts" datastructure: A JSON document which tells S3 the identification and order of all the individual parts.
Performing the parallel uploads and keeping track of all the ETag and other identifiers requires quite a bit of programming. This is complicated by the fact that you need to check all MD5 checksums as well, to verify the transfer was not corrupted. The script below is a Bash shell script which demonstrates how this can be done. The script first generates a large file (200 MB) and then uploads this in chunks of 5 MB.
#!/bin/bash
# Global vars
FILE=/tmp/bigfile.$$
PREFIX=/tmp/bigfile.$$.part.
ETAGSFILE=/tmp/bigfile.$$.etags
FILESIZE=200
CHUNKSIZE=5
#
# Note on chunksize:
# - Minimum chunksize is 5 MB
# - There is no maximum size
# - The maximum number of chunks is 10.000
# - The maximum object size is 5 TB
BUCKET=upload.demo.wlid.nl
KEY=bigfile.$$
# Cleanup stuff
cleanup()
{
echo "Cleanup called. Killing all upload jobs..."
kill $(jobs -p)
wait
# Cancelling the multipart upload is only possible after all parts uploads
# have finished. This may take a while but we have no way to check...
sleep 10
echo "Cancelling the multipart upload..."
aws s3api abort-multipart-upload --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOADID"
echo "Cleaning up all files..."
rm -f $FILE
rm -f ${PREFIX}*
rm -f $ETAGSFILE
rm -f ${ETAGSFILE}.sorted
echo "Done."
exit
}
trap cleanup 1 2 3 9 15
# Function to initiate a parts transfer, and to add the part id to the parts file.
# This function will run in the background as a whole
upload_part ()
{
# Parameters:
# $1 is name of the chunk to upload
# $2 is the part number
echo "Calculating checksum for chunk $1..."
local MD5_CHUNK=$( md5 -q $1 )
echo "Checksum for chunk $1 is $MD5_CHUNK"
echo "Going to upload chunk $1 as part $2..."
local ETAG=$( aws s3api upload-part --bucket $BUCKET --key $KEY --upload-id $UPLOADID --part-number $2 --body $1 --output text )
# Note that the $ETAG variable contains double quotes at the beginning and end of the string.
echo "$2 $ETAG" >> $ETAGSFILE
echo "Upload for chunk $1, part $2 finished. ETag is $ETAG."
if [ "$ETAG" == "\"$MD5_CHUNK\"" ]
then
echo "Checksum for chunk $1, part $2 is correct"
else
echo "Checksum for chunk $1, part $2 is NOT correct"
fi
}
#
# Main body of code starts here
#
# Create the demo file. This will contain $FILESIZE MB of random data
echo "Creating a $FILESIZE MB demo file..."
dd if=/dev/urandom of=$FILE bs=1m count=$FILESIZE
# Calculate the MD5 sum of the whole file
# Code commented out - irrelevant
#echo "Calculating the MD5 checksum..."
#MD5_WHOLE=$(md5 -q $FILE)
#echo "Checksum is $MD5_WHOLE"
# Create the multipart upload chunks
echo "Splitting the demo file in $CHUNKSIZE MB chunks..."
split -b ${CHUNKSIZE}m $FILE $PREFIX
CHUNKS=$( ls ${PREFIX}* )
# Create the multipart upload request
echo "Creating the multipart upload job..."
UPLOADID=$( aws s3api create-multipart-upload --bucket $BUCKET --key $KEY --output text | awk '{print $3}' )
echo "Upload ID is $UPLOADID"
# Fire up the individual multipart uploads in parallel
set $CHUNKS
COUNTER=1
echo "Starting all upload threads..."
while [ "$#" -gt 0 ]
do
upload_part $1 $COUNTER &
(( COUNTER++ ))
shift
done
echo "Finished starting up the threads."
CHUNK_COUNT=$(( $COUNTER - 1 ))
wait
echo "All threads finished. "
# Sort the parts file
cat $ETAGSFILE | sort -n >> ${ETAGSFILE}.sorted
# Create the parts data structure
PARTS="{\"Parts\": ["
while read PART ETAG
do
PARTS="${PARTS} {\"ETag\": $ETAG, \"PartNumber\": $PART },"
done < ${ETAGSFILE}.sorted
PARTS="${PARTS%,}"
PARTS="${PARTS} ]}"
# Completing the multipart upload
echo "Completing the multipart upload..."
ETAG_RECEIVED=$(aws s3api complete-multipart-upload --bucket $BUCKET --key $KEY --upload-id $UPLOADID --multipart-upload "$PARTS" --output text | awk '{print $2}' | tr -d '"')
# Check the checksum
ETAG_EXPECTED="$(cat ${ETAGSFILE}.sorted | cut -d' ' -f2 | tr -d '"' | xxd -r -p - | md5)-${CHUNK_COUNT}"
echo "ETAG expected: $ETAG_EXPECTED"
echo "ETAG received: $ETAG_RECEIVED"
if [ "$ETAG_EXPECTED" == "$ETAG_RECEIVED" ]
then
echo "File uploaded successfully."
else
echo "Something went wrong."
fi
# Cleanup
trap '' 1 2 3 9 15
rm -f $FILE
rm -f ${PREFIX}*
rm -f $ETAGSFILE
rm -f ${ETAGSFILE}.sorted
While testing the script above, I found another script useful. This script aborts all multipart uploads to a certain bucket.
#!/bin/bash BUCKET=upload.demo.wlid.nl aws s3api list-multipart-uploads --bucket $BUCKET --output text | grep UPLOADS | while read text timestamp key type uploadid do echo "Going to abort upload id $uploadid for key $key" aws s3api abort-multipart-upload --bucket $BUCKET --key $key --upload-id $uploadid done