Object Storage¶
As introduced in File Storage, Mayan EDMS takes ownership of document files and stores them using the configured storage backend.
Besides the default filesystem storage backend, another supported one is
the s3boto3
backend provided by the Django Storages project. This
backend allows Mayan EDMS to use any S3 (Simple Storage Service) compatible
service as storage for documents, caches and tempoary files.
This backend works not only with AWS, but any other S3-compatible API such as Minio. IBM Cloud Object Storage and others.
Using S3 storage it is possible to scale Mayan EDMS to millions of documents without the need to resize the document storage volume.
The following steps will configure Mayan EDMS to use a S3 style storage for document file storage.
Warning
If migrating from a different storage backend, first make a backup before beginning this process.
Ensure Mayan EDMS is stopped first and that every file from the existing storage directory is copied to the new storage location with the corresponding tool like
s3cmd
.Note
The configuration below only updates the document file storage. You will need to add additional variables if you also want to update the location of the Caches, Temporary location, Shared folders, etc. Anything that is currently configured with the
django.core.files.storage.FileSystemStorage
backend can have its backend changed, but this is not advisable for caches and temporary files locations due to performance reasons.
Docker Deployments¶
Add the following environment variables to your docker-compose.yml file
in the environment
section:
MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND: storages.backends.s3boto3.S3Boto3Storage MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS: "{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private'}"
If deploying Mayan EDMS without Docker Compose, add the environment variable to the complete docker run command:
-e MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND=storages.backends.s3boto3.S3Boto3Storage \ -e MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS="{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private'}"
Once added and the values changed to match the desired bucket, run:
docker-compose restart
For non Compose docker deployments, remove and re-deploy the container if not using docker-compose.
Note
The
default_acl
value is needed to prevent the backend from writing to S3 with a public-readable ACL. This is blocked by default on some providers such as AWS resulting in permission denied errors when trying to upload a documentNote
The
default_acl
option has been removed from django-storages in version 1.10 (2020-08-30). Instead useobject_parameters
with the options mentioned in https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_object and update yourMAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS
setting accordingly.
Direct Deployments¶
For environments installed using the direct deployment method, edit the
supervisor file at /etc/supervisor/conf.d/mayan-edms.conf
and add these two extra settings above the MAYAN_DATABASES
option,
ensuring that the lines remain indented as before.
An example database option is shown for guidance; do not
replace the existing MAYAN_DATABASES
line, add the new variables above it.
MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND=storages.backends.s3boto3.S3Boto3Storage, MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS="{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private'}", MAYAN_DATABASES="{default: {ENGINE: django.db.backends.postgresql, HOST: 127.0.0.1, NAME: mayan, PASSWORD: mayanuserpass, USER: mayan}}"
To complete the change, restart Supervisor on all systems running Mayan EDMS:
sudo systemctl restart supervisor
Non-AWS S3 Deployments¶
If you are using Minio or another non-AWS based S3-compatible
storage solution (such as Ceph with a S3 Gateway), there will
need to be additional variables entries needed for the
MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS
line.
The default_acl
configuration may also need to be
ommitted if the platform does not support AWS-style ACLs,
but try first with the default_acl
value set for
security:
MAYAN_DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS="{'access_key':'yourS3accesskey','secret_key':'yourS3secretkey','bucket_name':'yourS3bucketname','default_acl':'private','endpoint_url':'https://address-of-s3-server:9000','verify':'False'}"
Note
The
verify
option validates for trusted TLS certificates. Change this toTrue
if your S3-compatible server is using trusted signed certificates. Leave it at false for self-signed certificates, and don’t forget to change the https to http for non-TLS setups
You may also need to add the variables region_name
and/or location
depending on your S3 server configuration.
endpoint_url: “https://<my internal NAS FQDN>:9000”, verify: False
Settings¶
DOCUMENTS_FILE_STORAGE_BACKEND
Path to the Storage subclass to use when storing document files.
Default:
django.core.files.storage.FileSystemStorage
DOCUMENTS_FILE_STORAGE_BACKEND_ARGUMENTS
Arguments to pass to the DOCUMENT_FILE_STORAGE_BACKEND.
Default:
location: /builds/mayan-edms/mayan-edms/mayan/media/document_file_storage