WLID AWS Demo Site

HTTP upload to S3

What's going on here?

S3 offers the ability to upload files via the HTTP POST protocol. The basic way to do this is to create an HTML form inside your HTML page. This HTML form code will look roughly like this:

<form action="https://upload-demo-wlid-nl.s3.eu-central-1.amazonaws.com/" method="post" enctype="multipart/form-data">
      <input type="hidden" name="bucket" value="upload-demo-wlid-nl">
      <input type="hidden" name="acl" value="private">
      <input type="hidden" name="key" value="${filename}">
      <input type="hidden" name="success_action_redirect" value="https://www.demo.wlid.nl/http-upload-successful.html">
      <input name="file" type="file"> <input type="submit" value="Upload"> 
</form>

The form data tells S3 in which bucket the file will need to be stored, under what name, and what ACL it should get. It also tells S3 which HTML file to display after a successful upload.

The above only works for public-write buckets. Which means that everybody can store anything in the bucket and retrieve it. That is typically something you don't want. So I do not recommend setting up your upload like above!

It is perfectly possible to secure your buckets against unwanted uploads, but you've got to put in some work.

1. Create an S3 bucket

This is easy. An upload bucket is just a regular bucket without any special characteristics. If you want your users to be able to download the data after uploading, you may want to set the ACLs to public read, and/or you may want to host a website from this bucket. And if it's not a public-read bucket, you can setup an event that, for instance, triggers an SNS message to a topic, or kicks of a Lambda script. Also, you may want to setup a lifecycle policy that deletes all uploads after a set number of days.

The upload is done over an https connection. The X.509 certificate for this connection is an AWS-owned wildcard certificate for *.s3.region.amazonaws.com. Wildcard certificates only work one level deep, so this certificate does apply to a bucket name such as upload-demo-wlid-nl but does not work for upload.demo.wlid.nl. So make sure your bucket name does not contain any dots.

For this demo I have setup a bucket "upload-demo-wlid-nl".

2. Setup an IAM user account, with a policy that allows uploads to the bucket

For security reasons you will want to give this IAM user limited privileges. For this demo I setup a user "S3UploadDemo", and I downloaded the security credentials.

I also created a policy "S3UploadPolicy" and attached this to the "S3UploadDemo" user:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::upload-demo-wlid-nl/*"
        }
    ]
}

3. Create the HTTP upload policy

If we would just rely on the IAM user policy, then the user would be able to upload files anywhere in the bucket, and these files can be of any type or size. Also, we would need to incorporate the IAM credentials (access key and secret key) in the HTML document somehow, and that's considered very bad practice, even if the policy attached to the user is extremely limited.

What you're going to do to solve both issues, is to create a "POST Policy" that specifies exactly what and where the user is allowed to upload. You are then going to sign this policy with the secret key of the IAM user we created earlier. The form data will thus contain essentially three things:

The access key of the IAM user.
The policy itself
The signature (which is actually the hash of the policy, signed with the IAM users secret key)

The "POST policy" starts off as a text file on your local computer. It will look like this:

{
  "expiration": "2050-01-01T00:00:00Z",
  "conditions": [
    {"bucket": "upload-demo-wlid-nl"},
    {"acl": "private"},
    ["starts-with", "$key", ""],
    {"x-amz-date": "20500101T000000Z"},
    {"x-amz-algorithm": "AWS4-HMAC-SHA256"},
    {"x-amz-credential": "AKIAJQOWQ5WGN7XKYQZQ/20500101/eu-central-1/s3/aws4_request"},
    {"success_action_redirect": "https://www.demo.wlid.nl/http-upload-successful.html"},
    ["content-length-range", 0, 1048576]
  ]
}

I called this file "http-upload-policy.json". It is relatively limited: It specifies a bucket to use, the acl to apply and a maximum content size of 1 MB. It also needs to specify a success URL, which is the page shown to the user after a successful upload. And there are a few other limitations included, which are essentially limits to what the user can put in the HTML form data. But there are a lot more things you can set in a policy. View the Amazon Documentation for more information.

4. Sign the policy

This is where it becomes tricky. Signing the policy is a multi-step process. The full documentation is here. As you can see, there are quite a few steps involved: One base64 encoding, five HMAC_SHA256 operations and a hex encoding. It is very daunting and the documentation is not very clear. You also need to be very careful in dealing with line endings, escape characters and other weird things, as the encryption is very sensitive.

Based on the examples given by AWS, I have written a little Python helper script which performs all the calculations for you, and which spits out the hidden HTML <input> elements that go inside your HTML form. You can download the script here.

Here's what running the script looks like:

$ ./generate-signature.python http-upload-policy.json 'AKIAJQOWQ5WGN7XKYQZQ' 'yOuDiDNOtReAlLyEXpeCtMYSeCReTkEyHeRE/DIdYoU' '20500101' 'eu-central-1' 's3'
<input type="hidden" name="X-Amz-Algorithm" value="AWS4-HMAC-SHA256">
<input type="hidden" name="X-Amz-Date" value="20500101T000000Z">
<input type="hidden" name="X-Amz-Credential" value="AKIAJQOWQ5WGN7XKYQZQ/20500101/eu-central-1/s3/aws4_request">
<input type="hidden" name="policy" value="ewogICJleHBpcmF0aW9uIjogIjIwMjAtMDEtMDFUMDA6MDA6MDBaIiwKICAiY29uZGl0aW9ucyI6IFsgCiAgICB7ImJ1Y2tldCI6ICJ1cGxvYWQuZGVtby53bGlkLm5sIn0sIAogICAgeyJhY2wiOiAicHJpdmF0ZSJ9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgIiJdLAogICAgeyJ4LWFtei1kYXRlIjogIjIwMjAwMTAxVDAwMDAwMFoifSwKICAgIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwKICAgIHsieC1hbXotY3JlZGVudGlhbCI6ICJBS0lBSlFPV1E1V0dON1hLWVFaUS8yMDIwMDEwMS9ldS1jZW50cmFsLTEvczMvYXdzNF9yZXF1ZXN0In0sCiAgICB7InN1Y2Nlc3NfYWN0aW9uX3JlZGlyZWN0IjogImh0dHA6Ly93d3cuZGVtby53bGlkLm5sL2h0dHAtdXBsb2FkLXN1Y2Nlc3NmdWwuaHRtbCJ9LAogICAgWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsIDAsIDEwNDg1NzZdCiAgXQp9Cg==">
<input type="hidden" name="X-Amz-Signature" value="7cbe71154a72de7534948984fcb6c2d6113cdfc80650c4dd82de2e4cd7490081">

4. Create the HTML upload page

Now you can create the HTML page containing the form. The form itself is really a combination of what I wrote earlier to upload to a public-write bucket, plus the output from the Python script. Make sure that any hidden fields within the form definition are allowed according to the policy. For instance, if you were to specify your key in the HTML form as "${filename}" (which means don't do any filename translation) but your policy specifies ["starts-with", "$key", "upload/"] then this will likely lead to an error.

Here is the full HTML form that is embedded in this page:

<form action="https://upload-demo-wlid-nl.s3.eu-central-1.amazonaws.com/" method="post" enctype="multipart/form-data">
      <input type="hidden" name="bucket" value="upload-demo-wlid-nl">
      <input type="hidden" name="acl" value="private">
      <input type="hidden" name="key" value="${filename}">
      <input type="hidden" name="success_action_redirect" value="https://www.demo.wlid.nl/http-upload-successful.html">
      <input type="hidden" name="X-Amz-Algorithm" value="AWS4-HMAC-SHA256">
      <input type="hidden" name="X-Amz-Date" value="20500101T000000Z">
      <input type="hidden" name="X-Amz-Credential" value="AKIAJQOWQ5WGN7XKYQZQ/20500101/eu-central-1/s3/aws4_request">
      <input type="hidden" name="policy" value="ewogICJleHBpcmF0aW9uIjogIjIwMjAtMDEtMDFUMDA6MDA6MDBaIiwKICAiY29uZGl0aW9ucyI6IFsgCiAgICB7ImJ1Y2tldCI6ICJ1cGxvYWQuZGVtby53bGlkLm5sIn0sIAogICAgeyJhY2wiOiAicHJpdmF0ZSJ9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgIiJdLAogICAgeyJ4LWFtei1kYXRlIjogIjIwMjAwMTAxVDAwMDAwMFoifSwKICAgIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwKICAgIHsieC1hbXotY3JlZGVudGlhbCI6ICJBS0lBSlFPV1E1V0dON1hLWVFaUS8yMDIwMDEwMS9ldS1jZW50cmFsLTEvczMvYXdzNF9yZXF1ZXN0In0sCiAgICB7InN1Y2Nlc3NfYWN0aW9uX3JlZGlyZWN0IjogImh0dHA6Ly93d3cuZGVtby53bGlkLm5sL2h0dHAtdXBsb2FkLXN1Y2Nlc3NmdWwuaHRtbCJ9LAogICAgWyJjb250ZW50LWxlbmd0aC1yYW5nZSIsIDAsIDEwNDg1NzZdCiAgXQp9Cg==">
      <input type="hidden" name="X-Amz-Signature" value="7cbe71154a72de7534948984fcb6c2d6113cdfc80650c4dd82de2e4cd7490081">
      <input name="file" type="file"> <input type="submit" value="Upload"> 
</form>

Note that the access key is embedded in the page, but the secret key is not. You only need the secret key when signing the policy.

5. Create the upload successful page

Create an HTML page that will be displayed if the upload was successful.