9 OpenEMM Updates and Upgrades

With OpenEMM we make a difference between updates (changing to a new minor release, like from 20.10.000.050 to 20.10.000.100) and upgrades (changing to a new major release, like from 20.04 to 20.10).

If you plan to do an upgrade, we strongly recommend to update to the lastet minor release of your OpenEMM version, first. Only after that, execute the upgrade to the new major version. This makes sure, that the delta between the two major versions is as little as possible and the old version is well prepared for the upgrade.

If you apply an upgrade to your production environment, make sure to create a backup with OST → Database Maintenance → Backup MariaDB database before, because each upgrade also updates schema and content of your database. You may need the database backup in case of a roll back (see section Rollbacks for OpenEMM below for details).

9.1 Updating and Upgrading OpenEMM

Before you upgrade OpenEMM to a new major version, check the required software stack in section Software Stack first. If you have to update a component of the software stack like Java, Python or MariaDB, make sure to stop OpenEMM before by executing

/home/openemm/bin/openemm.sh stop

to avoid any disruption or misbehaviour of OpenEMM after the update of the software component.

The OpenEMM Wiki at https://wiki.openemm.org shows the latest available versions of the openemm code package at the top.

You can download the latest versions of the packages with menu Install or update package from AGNITAS Website of OST. Please answer “no” to all downloads offers until the openemm code package is offered.

An update of the openemm code package will also update the schema of the OpenEMM database, if necessary. Please keep in mind that you have to start OST as user root, if you want to update the openemm code package.

Do not forget to restart OpenEMM with menu Restart OpenEMM after you have installed a new openemm code package to activate it. But do the restart at an convenient time: Do not restart OpenEMM during a dispatch of a mailing, or right after the dispatch (due to the brief downtime of the redirect service which would cause missed openings and click redirects).

When the upgrade is finished and OpenEMM has been restarted, log into the GUI of OpenEMM and check the status of the automated background processes in tab Job list of sub-menu System status in menu Administration.

9.2 Do not Skip Major Releases when Upgrading

If you upgrade an OpenEMM version prior to 24.10, do not skip an upgrade of a major version, i.e. if you use OpenEMM 23.10 and you plan to upgrade to OpenEMM 24.10, do not skip the upgrade to version 24.04! You have to install and start the intermediate version at least once. This ensures that possible migration tasks included in the intermediate OpenEMM version are executed, so that the database and configuration of OpenEMM remains in a consistent state.

To make sure that you are able to upgrade to each major release of OpenEMM, update to the runtime version of the next major release and after that download and update the openemm code package. Do not update the runtime package to a new major release without updating and executing the openemm code package, too!

If you have skipped updating openemm code packages while updating the runtime package, you have to use the switch menu to switch back to an older runtime version (see section Rollbacks for OpenEMM below for details).

Even if you do not want to use an intermediate version, you have to start this version once (like 24.04 in the above-mentioned case). This makes sure that the startup process in the OpenEMM code is executed and initiates all pending migration tasks contained in the skipped OpenEMM version.

For example, the startup process of OpenEMM 20.04 migrates all configuration properties from configuration file emm.properties into the database (among other things). This migration is important, because OpenEMM 20.10 reads its configuration properties from the database.

9.3 Special Advice for Upgrades to Version 22.10

OpenEMM 22.10 needs two more Python packages on all servers with a backend service. If you installed Python 3.8 from the repository (see section Python 3 for OpenEMM), use

pip3.8 install msgpack
pip3.8 install websockets

If you compiled Python yourself, use

python3 -m pip install msgpack
python3 -m pip install websockets

Please note that this is the last version supporting MySQL. If you want to migrate from MySQL to MariaDB, we recommend this procedure:

stop OpenEMM and MySQL
export the database content to have a backup just in case, for instance using command mysqldump:
```
mysqldump -aCceQx --hex-blob --routines --triggers -u root -p -r openemm.sql openemm
```
change your DBMS from MySQL 5.7 to MariaDB 10.2 (because these versions offer the maximum of compatibility: https://mariadb.com/kb/en/upgrading-from-mysql-to-mariadb)
run mariadb-upgrade to migrate the DB structure from MySQL to MariaDB
change file dbcfg in directory /opt/agnitas.com/etc for MariaDB
start MariaDB
start OpenEMM

Make sure that the database user used by OpenEMM has the required rights. If the user name is openemm, we recommend to create the user this way:

CREATE USER 'openemm'@'localhost' IDENTIFIED BY '<password>';
GRANT ALL PRIVILEGES ON openemm.* TO openemm;
GRANT SUPER ON *.* TO openemm;
FLUSH PRIVILEGES;

If the OpenEMM database does not work correctly, import it from your backup instead (for instance using command mysql).

Now, that you are running OpenEMM with MariaDB 10.2, which is no longer supported, we recommend to update to a later version of MariaDB. See section MariaDB for the version we use ourselves for the OpenEMM public cloud, and see section Upgrading MariaDB for advice on how to upgrade a MariaDB version.

9.4 Special Advice for Upgrades to Version 23.04

Please note that OpenEMM 23.04 does no longer support MySQL 5.7. See the section before for the process to migrate from MySQL to MariaDB.

The EMM database schema for workflows has changed several years ago. With OpenEMM 23.10 we will remove the code to support the legacy format. To check if you are affected, beginning with OpenEMM 23.04, OST will check your workflows at startup time. If OST detects an inactive legacy workflow, you will see message

“Your EMM database contains inactive legacy workflows [list of workflow names]. Please delete those workflows since they will no longer work with EMM 23.10 and later.”

If OST detects an active legacy workflow, you will see message

“Your EMM database contains active legacy workflows [list of workflow names] which can not be migrated. Please rebuild these workflows from scratch and delete the legacy workflows since they will no longer work with EMM 23.10 and later. If you need help, please contact support.”

Please re-build those workflows before you upgrade OpenEMM to version 23.10 or later.

If you want to check manually, if you have legacy workflows in your database, check in table workflow_reaction_tbl if it contains entries with field is_legacy_mode set to value 1.

Also, the report icon will no longer be supported in workflows beginning with OpenEMM 23.10. Therefore, OST will check all workflows for this icon, too. If a workflow is found, you should remove the report icon before you update to OpenEMM 23.10.

9.5 Special Advice for Upgrades to Version 23.10

OpenEMM 23.10 needs an update of Python lib paramiko and three new Python libs. Execute

python3 -m pip install -U 'paramiko>=3.2.0'
python3 -m pip install aiosmtplib
python3 -m pip install asyncinotify
python3 -m pip install asyncssh

to update and install the required libs.

Once you have finished the upgrade to OpenEMM 23.10, you can remove the requirement of root access for the mailloop service. To do this, stop Postfix on the server running the mailloop service as user root with

systemctl stop postfix

and stop the mailloop service as user mailloop with

mailloop.sh stop

Now, change line

relay_domains = /home/mailloop/var/run/relay.domains

in configuration file main.cf in directory /etc/postfix to

relay_domains = hash:/home/mailloop/var/run/relay.domains

and restart the mailloop service as user mailloop with

mailloop.sh start

Finally restart Postfix as user root with

systemctl start postfix

Please note that OpenEMM 23.10 does no longer use deprecated wkhtmltox, but NPM package puppeteer to render thumbnail images and PDF documents. Therefore, the installation of nodejs and a one-time execution of npm is necessary. See section Rendering of Thumbnail Images and PDF Documents for details.

9.6 Special Advice for Upgrades to Version 24.04 and 24.10

Please make sure to switch to Java 17 as soon as possible, because Java 11 will be no longer supported by OpenEMM 25.04.

Also, we recommend to switch to Python 3.11 as soon as possible, because Python version 3.11 will be a minimum requirement of OpenEMM 25.04.

9.7 Special Advice for Upgrades to Version 25.04

OMT (OpenEMM Maintenance Tool) was renamed to OST (OpenEMM Support Tool).

OpenEMM 25.04 requires Java 17 and Python 3.11 or later. Please make sure to have both versions available on your EMM servers.

MariaDB has to be at least at version 10.6.10 to work with Python 3.11. We use MariaDB version 10.11.7 for our production environment.

OpenEMM 25.04 does not support deprecated wkhtmltox any longer, but uses NPM package puppeteer to render thumbnail images and PDF documents.

After the upgrade, you must delete file derby-10.14.<patch>.jar in directory /home/openemm/tomcat/lib (<patch> is the placeholder for the exact version).

We recommend to switch from nodejs 18 to version 20:

RHEL:

dnf module reset nodejs
dnf module enable nodejs:20
dnf install nodejs

SLES:

zypper remove nodejs18
zypper install nodejs20

If you use Tomcat version 10.1.42 or later, you may have problems to import multiple files and see error message "FileCountLimitExceededException" in log file catalina.out of Tomcat. To avoid this problem, set parameters maxParameterCount and maxPartCount of the Connector configuration in configuration file server.xml to value 1024 each.

When you are finished with the EMM upgrade, log into the GUI as user emm-master and check in menu Administration, sub-menu System Status in tab Show Jobqueue that all jobs are working.

9.8 Special Advice for Upgrades to Version 25.11

If you use Tomcat version 10.1.42 or later, you may have problems to import multiple files and see error message "FileCountLimitExceededException" in log file catalina.out of Tomcat. To avoid this problem, set parameters maxParameterCount and maxPartCount of the Connector configuration in configuration file server.xml to value 1024 each.

Please make sure to switch to Java 21 as soon as possible, because Java 17 will no longer be supported by future versions of OpenEMM.

When you are finished with the OpenEMM upgrade, log into the GUI as user emm-master and check in menu Administration, sub-menu System Status in tab Show Jobqueue that all jobs are working.

9.9 Rollbacks for OpenEMM

If you are not happy with a version you updated or upgraded to, you can roll back OpenEMM to a former version or roll forward to a later version with menu Switch OpenEMM version. This menu lists the active versions of the EMM services available on the current server so that you can switch separately

the runtime environment RUNTIME (including OST itsself)
the openemm frontend (consisting of GUI service EMM, statistics service STATISTICS and webservices service WS)
the openemm backend (consisting of various services packaged into BACKEND-OPENEMM)
the documentation package MANUAL (including all context sensitive help pages)

After you have selected a specific service you get a list of all versions available on the server so that you can chose the version you want to activate.

Warning: If you switch back the openemm frontend or backend not to an older minor version (last 3 digits of the version number), but to an older major version (like going back from 26.05 to 25.11), be advised that the database can not be rolled back. This is, because the upgrade may have deleted or overwritten existing data in your OpenEMM database and this data can not be restored out of thin air. Therefore, you must restore your database backup (see section OpenEMM Updates and Upgrades) first.

If you need to know which services and versions of OpenEMM were active at what time in the past, menu Show update history of OST provides a list of all available services, their versions and the exact startup times.

Do not forget to restart OpenEMM with menu Restart OpenEMM after you have switched the frontend or backend version of OpenEMM.

If you get offered updates only to the latest version of OpenEMM, you have to switch back the runtime version. For instance, a runtime version 25.11 only offers you updates to versions of OpenEMM 25.11. But as soon as you have upgraded to a runtime version 26.05, you get offered upgrades to OpenEMM 26.05 as well.

9.10 Templates and Web Forms

You do not have to start from scratch when producing mailings or creating web forms in OpenEMM. At https://www.agnitas.de/en/download-center/ you can download templates and web forms which you can import into OpenEMM. Make sure to replace in web forms any placeholders for a company ID with value "1" if this is not done automatically during import.

9.11 Preparations before Updating MariaDB

If you plan to update your version of MariaDB, due to a bug in older versions of MariaDB (MDEV-19292), you may get an error "rowsize too large" afterwards which prevents you from using your OpenEMM database after the update of MariaDB.

Statement from MariaDB regarding this problem:

"Prior to MariaDB 10.2.26, MariaDB 10.3.17, and MariaDB 10.4.7, MariaDB doesn't properly calculate the row sizes while executing DDL. In these versions, unsafe tables can be created, even if InnoDB strict mode is enabled."

To check if you are affected by this bug, please execute SQL statement

SELECT count(*) FROM information_schema.innodb_sys_tables WHERE name LIKE 'openemm/%' AND ROW_FORMAT = 'Compact';

If the result of this SQL statement is greater than 0, your OpenEMM database uses tables using the format which may cause "unsafe" tables. In this case, we strongly recommend to convert all database tables to format "dynamic" with statement

ALTER TABLE <table name> ROW_FORMAT = DYNAMIC;

before upgrading your version of MariaDB (after you made a backup of your database, of course)!

To simplify this task for you, OST offers an automated conversion in menu Database Maintenance, sub-menu Check MariaDB table format. Alternatively, here is a little script that does the conversion of all tables (please execute as database user root):

DELIMITER $$
create procedure tmp_convert_row ()
Begin
   DECLARE done INT DEFAULT FALSE; DECLARE dbtable varchar(100);
   declare tab_curse cursor for select SUBSTRING(name, 9) from information_schema.innodb_sys_tables where name like 'openemm/%' and ROW_FORMAT = 'Compact';
   DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
    open tab_curse;
     read_loop: LOOP
        fetch tab_curse into dbtable;
    IF done THEN LEAVE read_loop; END IF;
    SET @SQLText = CONCAT('alter table ', dbtable, ' ROW_FORMAT=DYNAMIC');
        PREPARE stmt FROM @SQLText; EXECUTE stmt; DEALLOCATE PREPARE stmt;
   end loop;
   close tab_curse;
END$$
DELIMITER ;
call tmp_convert_row ();

9.12 Upgrading MariaDB

MariaDB recommends these steps to upgrade a MariaDB version:

Modify the repository configuration, so the system's package manager installs the required version.
Stop MariaDB.
Uninstall the old version of MariaDB.
Install the new version of MariaDB.
Make any desired changes to configuration options in option files, such as my.cnf. This includes removing any options that are no longer supported.
Start MariaDB.
Run mariadb-upgrade.