Skip to main content

File Storage

This page will guide you through everything file storage related that the server provides.

The server comes with built-in support for connecting to an AWS S3 or Minio file storage service, as well as providing a fileStorage export to make interactions with these services easier.

Basic concepts​

File storage has the following moving parts:

  • file server: This could either be a Minio server or an AWS server.
  • fileStorage export: A utility exported by @coko/server that provides methods to interact with the file server.
  • File model: A model provided by @coko/server to represent files in your application's database.

The basic flow of uploading a file would look like this:

  • Call the fileStorage utility's upload method with the file and filename as arguments
  • This will store the files to the remote file server and return an array of storedObjects
  • Each storedObject contains a key, which is the unique identifier of the object on the file server
  • Store the result in the files table in your local database

It is worth noting that images will get converted to multiple sizes (small, medium & original) by the fileStorage utility. This will result in three objects stored on the file server, but only one entry in your local database (which will contain the keys for all sizes).

Since the result of the upload method could be mutliple objects for images, we always return an array to maintain consistency in the returned values.

Connecting to a file server​

To get started, you will need a file server running. This could either be a Minio or AWS server. @coko/server will handle the connection to the file server, as long as you pass the following environment variables.

# Used to build the url of the file server. Port is optional.
S3_PROTOCOL
S3_HOST
S3_PORT

# You will need to have set up a user and a bucket on your file server beforehand.
S3_ACCESS_KEY_ID
S3_SECRET_ACCESS_KEY
S3_BUCKET

Development with Minio​

Whether you are using AWS or Minio in production, it can be useful to bring up a local Minio server during development, so that you can develop your app without needing an active file server running in the cloud.

To run Minio locally, you need to add some code to your development docker-compose file that will:

  • Create a Minio server with a root user
  • As root, create a non-root user (we will use the non-root user to connect to minio)
  • Create a bucket that the non-root user has access to

You will need some additional environment variables:

MINIO_ROOT_USER
MINIO_ROOT_PASSWORD
MINIO_CONSOLE_PORT

Then in your development docker-compose file:

version: '3'

services:
  # Create the file server and give it a root user
  file_hosting:
    image: minio/minio
    ports:
      - ${S3_PORT:-9000}:9000
      - ${MINIO_CONSOLE_PORT:-9001}:9001
    volumes:
      - minio_storage:/data
    environment:
      - MINIO_ROOT_USER=${MINIO_ROOT_USER:-admin}
      - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-superSecretAdminPassword}
    command: server --console-address ":${MINIO_CONSOLE_PORT:-9001}" /data
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:9000/minio/health/live']
      interval: 30s
      timeout: 20s
      retries: 3

  # Create non-root user and bucket that we'll connect to
  createbucket:
    image: minio/mc
    depends_on:
      file_hosting:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host add cokoServer http://file_hosting:${S3_PORT:-9000} ${MINIO_ROOT_USER:-admin} ${MINIO_ROOT_PASSWORD:-superSecretAdminPassword};
      /usr/bin/mc admin user add cokoServer/ ${S3_ACCESS_KEY_ID:-nonRootUser} ${S3_SECRET_ACCESS_KEY:-nonRootPassword};
      /usr/bin/mc admin user enable cokoServer/ ${S3_ACCESS_KEY_ID:-nonRootUser};
      /usr/bin/mc mb cokoServer/${S3_BUCKET:-uploads};
      /usr/bin/mc admin policy set cokoServer/ readwrite user=${S3_ACCESS_KEY_ID:-nonRootUser};
      exit 0;
      "

  # Pass the values of what you created above to your server as environment variables
  server:
    environment:
      - S3_PROTOCOL: http
      - S3_HOST: file_hosting
      - S3_PORT: ${S3_PORT:-9000}
      - S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:-nonRootUser}
      - S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:-nonRootPassword}
      - S3_BUCKET: ${S3_BUCKET:-uploads}
    ... # the rest of your server service configuration

volumes:
  minio_storage:

Enable the provided File model​

The server comes with a built-in File model.

You can enable it in your app by adding the following line in your components.js file.

// config/components.js

module.exports = [
  ...,
  '@coko/server/src/models/file'
]

You can check the implementation and the properties of the model here.

fileStorage export​

The server exports the fileStorage utility.

const { fileStorage } = require("@coko/server");

The utility provides the following methods:

upload​

Uploads your file to the connected file server. It accepts a file read stream and a filename as arguments. If the file is an image it will automatically create the different image sizes.

await fileStorage.upload(filestream, filename);

Returns an array of stored objects.

getUrl​

Given an object key, it returns a url for that object.

await fileStorage.getUrl(objectKey);

Note that by default, each url is valid for one day. This is configurable through the expiresIn option (the value is in seconds).

await fileStorage.getUrl(objectKey, { expiresIn: 172800 }); // expires in two days

deleteFiles​

Given an array of object keys, the corresponding objects get deleted from the file server.

await fileStorage.deleteFiles([objectKeyOne, objectKeyTwo]);

healthcheck​

Will tell you whether the server is successfully connected to the file server.

await fileStorage.healthcheck();

Note that this function runs once by default when the server first tries to connect to the file server with the provided environment variables.

list​

Lists all files currently stored in the bucket.

await fileStorage.list();

download​

Given an object key and a local path, this will grab the file from the file server and write it locally to the provided path.

await fileStorage.download(key, path);

Helpers​

The fileStorage utility and the File model work independently of each other. In other words, the utility will never touch your database, and the model will never communicate with the file server. This allows for maximum flexibility.

However, since we expect the combination of the two to be quite common, we provide the following helper functions that cover the most common use cases.

createFile​

Uploads a file to the file server and then saves it to the database.
In its simplest form, it looks like this:

const { createFile } = require("@coko/server");

await createFile(fileStream, filename);

You can see the full list of arguments here.

deleteFiles​

Given an array of file ids from the application database, this deletes the given files, both from the file server and the application database.

const { deleteFiles } = require("@coko/server");

await deleteFiles([id1, id2]);