Uploading files directly to Contentful

With the growth of our customer base, we saw a growing demand for more complex workflows around asset management. Data migration from old systems was a common pain point for our users, as our implementation proposal required them to rely on yet another service on which to host their assets. In our quest for an excellent developer experience, this was something we wanted to tackle, and therefore we started tracking and analyzing current usage. We quickly confirmed our hypothesis: around 40% of all asset uploads come from users working directly with our CMA and hosting assets on third-party services.

Unsatisfied with the current situation, we decided to build this feature right into the platform, to make it even easier to use Contentful in a future-proof way for all your content management needs — no matter if that content is pure text or binary data like images, videos or documents. The team did a fantastic job, and it took only two weeks from idea to prototype, and another two weeks to release it for you.

How it works

Before talking about the technical details of the solution, let’s take a look at how easy it is to use this new service in your known Contentful workflow.

To upload a file, just send a POST request with a binary file in the request body to upload.contentful.com.

1
2
3
4
5

$ curl -i -XPOST \
  -H "Content-Type:application/octet-stream" \
  -H "Authorization: Bearer <accessToken>" \
  --data-binary "@/Users/ben/my-cute-cat.jpg" \
  https://upload.contentful.com/spaces/<spaceId>/uploads

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

HTTP/1.1 201 Created
cache-control: no-cache
Content-Type: application/json; charset=utf-8
Date: Tue, 21 Feb 2017 07:49:24 GMT
Server: Contentful
Strict-Transport-Security: max-age=15768000
vary: accept-encoding
X-Content-Type-Options: nosniff
X-Contentful-Request-Id: ec0bec70a00b5178362660a23f4b9739
Content-Length: 432
Connection: keep-alive

{
  "sys": {
    "id": "73DfxdBnwyhQNy95A8dvSf",
    "type": "Upload",
    "createdAt": "2017-02-21T07:49:25.000Z",
    "expiresAt": "2017-02-23T00:00:00.000Z",
    "space": {
      "sys": {
        "type": "Link",
        "linkType": "Space",
        "id": "qa65bkvd5q1q"
      }
    },
    "createdBy": {
      "sys": {
        "type": "Link",
        "linkType": "User",
        "id": "1QaAgxMYKvdas32K4v319F"
      }
    }
  }
}

Now you can create an asset as you normally would do, but instead of passing an external URL to the upload field, you are feeding to the CMA a link object with the ID of the newly introduced uploadFrom field to it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

curl -i -XPOST \
  -H "Authorization: Bearer <accessToken>" \
  https://api.contentful.com/spaces/<spaceId>/assets \
  -d '
  {
    "fields": {
      "file": {
        "en-US": {
          "uploadFrom": {
            "sys": {
              "type": "Link",
              "linkType": "Upload",
              "id": "73DfxdBnwyhQNy95A8dvSf"
            }
          }
        }
      }
    }
  }'

After the asset's creation, the workflow will be the same as ever, so if you're using the CMA directly remember to process and then publish it!

We are also updating our official SDKs to harness this new functionality. If, like many, your language of choice is Javascript, here's a look at how you'll be able to use this new feature with ease:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

const sdk = require('contentful-management');
const fs = require('fs');
const spaceId = '<spaceId>';
const filePath = '/Users/ben/my-cute-cat.jpg';
const fileName = 'my-cute-cat';
const contentType = 'image/jpg';
const accessToken = '<accessToken>';

const sdkClient = sdk.createClient({
  spaceId: spaceId,
  accessToken: accessToken
});

sdkClient.getSpace(spaceId).then((space) => {
  console.log('uploading...');
  return space.createUpload({
    file: fs.readFileSync(filePath),
    contentType,
    fileName
  })
  .then((upload) => {
    console.log('creating asset...');
    return space.createAsset({
      fields: {
        title: {
          'en-US': fileName
        },
        file: {
          'en-US': {
            fileName: fileName,
            contentType: contentType,
            uploadFrom: {
              sys: {
                type: 'Link',
                linkType: 'Upload',
                id: upload.sys.id
              }
            }
          }
        }
      }
    })
    .then((asset) => {
      console.log('prcessing...');
      return asset.processForLocale('en-US', { processingCheckWait: 2000 });
    })
    .then((asset) => {
      console.log('publishing...');
      return asset.publish();
    })
    .then((asset) => {
      console.log(asset);
      return asset;
    })
  })
  .catch((err) => {
    console.log(err);
  });
});

Another tool we just updated to support this new feature is our excellent .NET SDK. Here's a quick look at this new implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

var bytes = File.ReadAllBytes("c:\example\yourfile.txt");
var managementAsset = new ManagementAsset();
managementAsset.SystemProperties = new SystemProperties();
managementAsset.SystemProperties.Id = "<newAssetId>";
managementAsset.Title = new Dictionary<string, string> {
    { "en-US", "New asset" }
};
managementAsset.Files = new Dictionary<string, File>
{
    { "en-US", new File() {
        ContentType = "text/plain",
        FileName = "your-file.txt"
    }}
};
await client.UploadFileAndCreateAsset(managementAsset, bytes);

Of course, if you are already hosting your assets somewhere public, you might want to still use the old CMA workflow. That's perfectly fine! The new direct file upload system is just a more convenient way, and it will live alongside the previous implementation, so as to satisfy everyone's needs.

For all the details on how to use Contentful to work with assets, our Content Management API Documentation covers everything you need to know.

How we built it

After the problem space was clear and we knew what we wanted to build, we started to research possible solutions. First we looked into tus.io and Amazon S3’s Pre-Signed URLs feature. The S3 way sounded nice, as especially with Transfer Acceleration, you can achieve very fast upload rates. However, we were forced to drop it because of lackluster developer experience: all AWS APIs are completely different from ours, as they use XML instead of JSON and they provide rather unintuitive error messages. The other alternative, Tus.io, is an open source protocol for resumable uploads, and has plenty of features. Unfortunately, after we analyzed usage patterns related to assets, we realized that we would have needed only a tiny fraction of what tus.io’s is capable of, but we would have had to deal with the overhead of running it in our infrastructure. After some investigation, we decided against both solutions and rolled our own microservice. In general, we try hard not to reinvent the wheel, as we'd much rather stand on the shoulders of giants, so to speak. However, we realized that in this situation the benefits of a custom solution far outweighed its costs.

The service would live in our infrastructure, but we needed to completely separate it from any other systems that we have, as handling long running requests with large payloads is fundamentally different from the fast and small JSON requests/response cycle we normally optimize for. With this setup, we were free to finely tune the new endpoints for handling streaming of binary data, longer timeouts, increased buffer sizes and payload limits.

Another upside of this setup was that we stayed in full control of the API design and how it interacts with the user, so that it will feel immediately familiar to all Contentful users. By having the entire life cycle of an upload process in our control, we can also provide a more convenient and secure experience. As you may have noticed in the example above, the service is just returning arbitrary IDs, which you can only use in the asset endpoints of the CMA to protect your uploads from any abuse. Of course, all new endpoints are protected, and users can only access them with a valid CMA access token.

To achieve all this in a fast way, we started to move forward with our Kubernetes production cluster. This service was an excellent opportunity to get more experience with the setup and still be able to scale to customer demand very quickly. For this reason, all upload endpoints are labeled as beta right now. You can expect a deep dive post on the topic of our Kubernetes setup coming up soon.

If you feel like trying out this new service, jump over to the documentation and upload some files. The team would love to hear your feedback on this. Happy uploading!