Object Storage

As introduced in File Storage, Mayan EDMS takes ownership of document files and stores them using the configured storage backend.

Besides the default filesystem storage backend, another supported one is the s3boto3 backend provided by the Django Storages project. This backend allows Mayan EDMS to use any S3 (Simple Storage Service) compatible service as storage for documents, caches and tempoary files.

This backend works not only with AWS, but any other S3-compatible API such as Minio. IBM Cloud Object Storage and others.

Using S3 storage it is possible to scale Mayan EDMS to millions of documents without the need to resize the document storage volume.

The following steps will configure Mayan EDMS to use a S3 style storage for document file storage.

Warning

If migrating from a different storage backend, first make a backup before beginning this process.

Ensure Mayan EDMS is stopped first and that every file from the existing storage directory is copied to the new storage location with the corresponding tool like s3cmd.

Note

The configuration below only updates the document file storage. You will need to add additional variables if you also want to update the location of the Caches, Temporary location, Shared folders, etc. Anything that is currently configured with the django.core.files.storage.FileSystemStorage backend can have its backend changed, but this is not advisable for caches and temporary files locations due to performance reasons.

Docker Deployments

Add the following environment variables to your docker-compose.yml file in the environment section:

MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage
MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS: "{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private'}"

If deploying Mayan EDMS without Docker Compose, add the environment variable to the complete docker run command:

-e MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND=storages.backends.s3boto3.S3Boto3Storage \
-e MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS="{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private'}"

Once added and the values changed to match the desired bucket, run:

docker-compose restart

For non Compose docker deployments, remove and re-deploy the container if not using docker-compose.

Note

The default_acl value is needed to prevent the backend from writing to S3 with a public-readable ACL. This is blocked by default on some providers such as AWS resulting in permission denied errors when trying to upload a document

Note

The default_acl option has been removed from django-storages in version 1.10 (2020-08-30). Instead use object_parameters with the options mentioned in https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_object and update your MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS setting accordingly.

Direct Deployments

For environments installed using the direct deployment method, edit the supervisor file at /etc/supervisor/conf.d/mayan-edms.conf and add these two extra settings above the MAYAN_DATABASES option, ensuring that the lines remain indented as before.

An example database option is shown for guidance; do not replace the existing MAYAN_DATABASES line, add the new variables above it.

MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND=storages.backends.s3boto3.S3Boto3Storage,
MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS="{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private'}",
MAYAN_DATABASES="{default: {ENGINE: django.db.backends.postgresql, HOST: 127.0.0.1, NAME: mayan, PASSWORD: mayanuserpass, USER: mayan}}"

To complete the change, restart Supervisor on all systems running Mayan EDMS:

sudo systemctl restart supervisor

Non-AWS S3 Deployments

If you are using Minio or another non-AWS based S3-compatible storage solution (such as Ceph with a S3 Gateway), there will need to be additional variables entries needed for the MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS line.

The default_acl configuration may also need to be ommitted if the platform does not support AWS-style ACLs, but try first with the default_acl value set for security:

MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS="{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private','endpoint_url':'https://address-of-s3-server:9000','verify':'False'}"

Note

The verify option validates for trusted TLS certificates. Change this to True if your S3-compatible server is using trusted signed certificates. Leave it at false for self-signed certificates, and don’t forget to change the https to http for non-TLS setups

You may also need to add the variables region_name and/or location depending on your S3 server configuration.

endpoint_url: “https://<my internal NAS FQDN>:9000”, verify: False

Settings

    • DOCUMENTS_FILE_STORAGE_BACKEND

      Path to the Storage subclass to use when storing document files.

      Default:

      django.core.files.storage.FileSystemStorage

    • DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS

      Arguments to pass to the DOCUMENT_FILE_STORAGE_BACKEND.

      Default:

      location: /builds/mayan-edms/mayan-edms/mayan/media/document_file_storage