Version 4.4

Released: January 16, 2023

Status: Stable

Changes

ACLs

A column showing the permission count of each ACL was added.

Authentication

Support was added for OpenID Connect (OIDC) authentication. This authentication is handled by a new named authentication_oidc.

Cabinets

The API permission layout was updated to match that of the HTTP views when creating cabinets.

The create permission is now required to create parent as well as child cabinets. This change replaces requiring the edit permission to create child cabinets via the HTTP views.

The parent cabinet is now added as the action object to the cabinet creation event when a child cabinet is created.

A new cabinet deleted event was added. This event is committed when a child cabinet is deleted. The parent cabinet is recorded as the action object for the event.

Dependencies

Several dependency versions were updated:

Celery from 5.1.2 to 5.2.7.
extract msg from 0.34.3 to 0.36.4.
djangorestframework from 3.13.1 to 3.14.0.
drf-yasg from 1.20.0 to 1.21.4.
sentry-sdk from 1.5.8 to 1.21.1.
Pillow from 9.2.0 to 9.4.0.

Docker

The Docker image tags were updated:

Docker from 20-dind to 20.10.21-dind
ElasticSearch from 7.17.0 to 7.17.7
PostgreSQL from 12.11-alpine to 13.11.2-alpine
Redis from 6.2-alpine to 7.0.5-alpine

Docker BuildKit is now enabled when building images. A dedicated Docker build RUN cache was added.

Docker Compose

The HTTP port exposed by the frontend is now configurable via the .env file.

The frontend and all_in_one profiles HTTP and Traefik configuration were unified.

Support for Let’s Entry TLS termination was added to the all_in_one profile.

The RabbitMQ administration HTTP port is now configurable via the .env file.

The Traefik dashboard, HTTP and HTTPS entrypoints ports are now configurable via the .env file.

The volume location of the Traefik Let’s Encrypt certificate is now configurable as well.

Support was added for Let’s Encrypt DNS challenge.

A Keycloak Docker Compose service was added for OIDC based SSO.

The Docker Compose service image names are now configurable.

Documents

Hebrew was added to the default list of document languages. This makes it easier to support OCR for Hebrew documents.

The document file creation code was split into smaller units. This reduces the complexity of the several conditional statements. This code change also increases the resilience of the upload code and guarantees document files are created even if there are critical errors during the uploaded file introspection.

A new permission was added to change the type of a document. When support for changing the type of a document was added, it was considered a property and controlled via the document property edit permission.

Since changing the type of a documents now causes a cascade of other changes, it was isolated as an individual class of event along with its own permission.

The new document change type permission is required for the document being changed and for the document type to which the document will be changed into.

The registration of the document models error logging was moved to the documents app.

Document downloads

Document file download code was moved to a new app called document downloads. All existing document file download permission and events are migrated automatically.

Support was added for document file bulk downloads.

Document exports

The code to export document versions was move into its own app named document_exports. Existing export events and permissions are migrated automatically.

Document parsing

The document parsing verbose_name attribute was updated to be the help_text attribute. Under the document type view, automatic document parsing is now called “Auto parsing”.

Downloads

Data from file and download content creation or examination now defaults to byte format instead of unicode.

Download files are now associated to a specific users and not to a parent object. Delete, download, and view permissions were added to allow users to share a download file.

A column showing the download file size was added.

Events

Object event attributes are restored when the event is ignored. This removes the need to set the event attributes at the start and at the end of a code context.

File caching

The compatibility of the file caching storage system with S3 object storage was improved and works with more implementations.

Columns showing the cache partitions and partition file totals were added. This helps determine how effective is a cache maximum size value by showing how many objects and files the cache size limit is able to yield.

When purging caches and cache partitions, the user initiating the request is now tracked in the event.

File metadata

A new file metadata driver was added to extract information from .msg files.

The file metadata value fields was changed from a character field with a maximum length of 255 characters to a text field. In theory this allows adding file attributes values of unlimited size.

Update the file metadata model verbose_name attribute to be the help_text attribute. In the user interface, this field now appears as “Auto process”.

Logging

Error log registration now register error log permissions too by default for the attached object.

The exception ImproperlyConfigured is now when a model is registered for error logging more than once.

A new setting was added to disable logging message ANSI color codes. The setting is named LOGGING_DISABLE_COLOR_FORMATTER and defaults to False.

Metadata

The document metadata value fields was changed from a character field with a maximum length of 255 characters to a text field. In theory this allows adding document metadata values of unlimited size.

Permissions

The invalid permission error handling was improved. Instead of returning an error that stop execution when an invalid permission identifier is requested, the permission model will return a one line text indicating that the permission name is invalid. This error message will be displayed in place of the intended permission label.

This change allows uninterrupted functionality even when there are invalid or non migrated permissions.

This behavior was also extended to cover invalid permission namespace requests.

A troubleshooting section was added expanding the explaining.

Permission caching was improved. Custom caching code was removed and upstream Django caching utilities are now used.

OCR

As OCR is done per page, the OCR error logging was moved from the document version to the document version page.

The OCR backend code is now executed under a new method called _execute to avoid subclasses not calling the super class.

The base backend class now prepares the image to be processed for OCR and passes the file object to the subclass.

The OCR finished event commit was from the task to the manager. This ensures it is called regardless of the code path and simplifies the task function.

Update the document version OCR verbose_name attribute to be the help_text attribute. This field now appears in the user interface as “Auto OCR”.

Platform

Support was added to allow platform client backends to register tool links.

The default Sentry client sample rate was changed from 0.05 to 0.005.

REST API

Removed injected objects on API views. Each API view now needs to query for the object explicitly. This is change makes the REST API less efficient and performant but was made to mirror how upstream Django REST framework works and expects projects to be coded.

The view object is now passed to the action object serializer.

Explicit labels were added to serializer fields.

Swagger schema model definition introspection was fixed.

The REST API was refactored to have views to behave like user interface views and returning querysets either via the source_queryset property or the get_source_queryset method. This change unifies queryset and permission handling between HTTP and REST API views. API views are now prevented from overriding the queryset return methods and allows the SchemaInspectionAPIViewMixin mixin to work in all instances.

The search and sources actions API views were updated to a dummy model serializer during Swagger introspection. This is required because the actual model or action is not known at the time of introspection.

Search

The third stage of the search system refactor was completed.

Search syntax for pre processing was added and all backends were updated to work with it.

The scoped search syntax was simplified.

More than two operands are now supported per operator. Example, __OR_1_2_3.

A new NOT operator was added.

Support for virtual fields was added. These allow searching for fields that do not exist in the database but are either a combination of fields or a dynamic runtime data.

A new virtual field was added that matches any field of an object. This field is emulated for search backends that do not support it natively.

All search is now conducted using an internal scope system. User search queries are converted this internal structure reducing code complexity and allowing adding more search dialects in the future.

Support was added for search data types. Data types supported are numbers, booleans, text, dates, and raw terms. Raw terms are passes as-is to the search backend without processing.

Support was added for text, accents and hyphen normalization as well as case folding.

The “Search” facet link is now named “Basic search”.

The “Match All” form field was replaced with a radio box to allow supporting search again persistence and also the default state of the field.

A search explainer was added. The search explainer output is showing as a sub title on the search results view.

A search bar was added to the top menu to allow access to search features regardless of the current view.

Settings

More details are now shown when a bootstrap setting parsing error occurs.

The navigation of the settings apps was improved.

Support for setting choices was added.

A new HTML widget was added to better format setting values.

A few feature called composable settings was added. This feature adds templating support to bootstrap settings allowing the settings to work with values that are not defined until runtime such as Kubernetes operator secrets.

Template names are the same as the bootstrap setting but include the SETTING_TEMPLATE_ prefix. Environment variables, Python global and config file values are available to the setting template.

Base settings initialization was improved and converted to replace usage of os.path with pathlib.Path to do path manipulation.

The default secret key value is now only used if the secret key file is not found not just if the secret key file is present but unreadable.

Errors when reading the secret key file are no longer obscured.

Statistics

The statistics app navigation was improved.

A doughnut chart type statistic was added as well as a pie chart type statistic.

A column displaying the chart type per statistic was added.

The library chartjs was updated from version 2.8.0 to 3.9.1.

The chart templates were unified. The ChartJS templates were also unified.

Statistics modules are not automatically loaded and no longer require a dummy star import.

Support was added for passing full chart context not just plot data.

The URL layout was improved.

Three new document pie chart statistics were added:

document count per document type

document file count per document type

document file page count per document type

Storage

Django Storages and boto3 Python libraries are included by default. This makes it possible to use S3 type object storages out of the box.

Templating

The home_view setting was removed from the default Template context. Template instances need to include their own context using the new context argument.

Testing

The test document IDs are now tracked. A list of the test document IDs is retained in number and string format.

Management command testing was standardized via the ManagementCommandTestMixin.

User interface

Views and actions are now collapsed by default. This reduces the visual complexity of the views. These are presented as two new table columns. Menus with a single action are unrolled into a button.

Icons were added for collapsed views and actions menus.

Views and actions markup was separated into their own respective partial templates. This reduces duplication and improves usability of the markup.

The z-index value of the sidebar was reduced to avoid menus to display behind it.

Simulated horizontal rulers were added to the body of tables.

Slim dropdown menus were added.

Card navigation was moved to the card footer.

Click events were fixed and no longer pass through views and action dropdown caret icons.

The card words “Actions” and “Views” are now translatable.

Reduced the table padding to increase data area. Minor spacing and margin tweaks were also added.

Support sorting tabled by multiple columns was added.

A third state to column sorting was added. The sorting states are now: ascending, descending, none.

A new setting was added to disable automatic upload after dragging files to the DropZone widget. The setting is named VIEWS_SHOW_DROPZONE_SUBMIT_BUTTON and defaults to False.

Other

Fix the function interfaces when calling get_mayan_object_permissions to get_mayan_view_permissions to override an API view permission layout.
Update navigation permission check to short circuit check when the current user has not logged in yet.
Use the optimized version (+=) of the + operator.
New reusable view mixin, MultipleExternalObjectViewMixin.
Isolate compressed file MIME type matching exception catching to the pertinent code.
Short circuit the source column source object resolution code to support the list template showing columns even on empty lists. Unlike models and queryset, empty lists of class instances won’t display any columns.
Download GPG keys in binary mode.
Add libfuse2, libsasl2-dev, and libldap2-dev to the development setup makefile target.
Convert pagination template into a partial.
Remove Docker mirror configuration from the GitLab CI file. This is up to the runner to configure.
Add documentation directives to insert setting or setting namespace instances.
Improve search and object storage documentation by adding automated setting references.
Encapsulate mozilla-django-oidc settings as Mayan authentication backend arguments.
Add support for OpenID Connect Discovery (https://openid.net/specs/openid-connect-discovery-1_0.html).
Code style refactor and cleanup:
- Strip trailing commas.
- Sort arguments, dictionary keys and class methods.
- Unroll nested contexts.
- Separate model data and business logic code.
- Move add or remove code to models. Directly and as added methods to external models.
- Pass the user to action methods instead of injecting the user as the event actor. Injecting the user as the event actor will be done only on immediate methods that do not allow arguments or data layer model methods with well defined upstream arguments.
- Add keyword arguments.
- Rename mixins modules to be more explicit.
- Normalize the UploadWizard class step_post_upload_process method arguments.
- Remove many instances of force_text.
- Move several upload_to functions to their corresponding app’s utils module.
- Promote private _user argument to an official argument.
Update the makefile to enable the Sentry client if the SENTRY_DSN value is passed to the runserver, runserver-plus, or staging-frontend targets.
Moved management command names to the literals module of each app.

Removals

Management commands

Remove deprecated management commands:
- checkdependencies replaced by dependencies_check.
- checkversion replaced by dependencies_check_version.
- createautoadmin replaced by autoadmin_create.
- generaterequirements replaced by dependencies_generate_requirements.
- initialsetup replaced by common_initial_setup.
- installdependencies replaced by dependencies_install.
- mountindex replaced by mirroring_mount_index.
- performupgrade replaced by common_perform_upgrade.
- platformtemplate replaced by platform_template.
- preparestatic replaced by appearance_prepare_static.
- purgelocks replaced by lock_manager_purge_locks.
- purgepermissions replaced by permissions_purge.
- purgeperiodictasks replaced by task_manager_purge_periodic_tasks.
- purgestatistics replaced by statistics_purge.
- revertsettings replaced by settings_revert.
- savesettings replaced by settings_save.
- showsettings replaced by settings_show.
- showversion replaced by dependencies_show_version.

Backward incompatible changes