11 OpenEMM Administration
11.1 Automated Startup
If you want OpenEMM to automatically launch at server reboot, you can use a systemd unit file for that purpose. Create a new file openemm.service in directory /etc/systemd/system/ with the following content (please note that the After properties belong into one line):
[Unit]
Description=OpenEMM startup script
After=var-run.mount network.target local-fs.target time-sync.target postfix.service
[Service]
User=openemm
Group=openemm
Type=oneshot
RemainAfterExit=true
LimitNOFILE=16384
ExecStart=/home/openemm/bin/openemm.sh start
ExecStop=/home/openemm/bin/openemm.sh stop
TimeoutSec=300
StandardOutput=journal+console
[Install]
WantedBy=multi-user.target
After deploying this file, reload the systemd-daemon and enable the openemm service with
At next server reboot, OpenEMM will be started automatically.
11.2 Jobqueue Monitoring
The OpenEMM JobQueue executes several cron-job-like background processes of OpenEMM, called JobWorkers. You can see a subset of the JobQueue data in the GUI of OpenEMM: Log in and go to menu Administration → Sub menu System Status → tab Show Jobqueue for details. This view is helpful to check the status of those JobWorkers.
You should make sure that you have configured an email address for error messages (see DB info below), because some JobWorkers like AutoOptimization or WorkflowReactionHandler take care of automated processes and should always be active. For other JobWorkers it makes sense to check their status at least from time to time, because some hanging jobs like cleanup processes will have no immediate effects but can drag down performance of EMM over time.
If you do not want to check the status of EMM jobs in the frontend, you can also access the EMM database directly. For MariaDB, the SQL statement is
SELECT id, description, lastresult, running, nextstart, laststart
FROM Job_queue_tbl
WHERE deleted = 0 AND
( (lastresult != 'Ok') -- has errors
OR (nextstart < CURRENT_TIMESTAMP() - INTERVAL 10 MINUTE) -- overdue
OR ( (running = 1) AND
(laststart < CURRENT_TIMESTAMP() - INTERVAL 4 HOUR)) -- (too) long running
);
To automate the jobqueue check, you may run this statement from a shell script and analyze its ouput with a monitoring software like Icinga.
Table job_queue_tbl lists all JobWorkers periodically executed by OpenEMM's JobQueue. You can control the execution times of the cleanup jobs via their entries in job_queue_tbl because the mass deletion of information can place serious strain on your database resources. Here are some information on the data fields of the job_queue_tbl:
- ID: id of job
- CREATED: creation date of job entry
- LASTSTART: last time the job was started
- RUNNING: status of execution (>0 = currently running)
- LASTRESULT: result of last execution (OK or NULL if no errors happend, or error message)
- STARTAFTERERROR: flag for start after error (>0 = yes)
- LASTDURATION: run time of latest execution of the job in milli seconds
- NEXTSTART: next time the job will launch
- RUNCLASS: Java class triggered by the job (example: org.agnitas.util.quartz.DBCleanerJobWorker executes the DB cleanup job)
- EMAILONERROR: email address to notify in case of an error in a job
- DELETED: flag to indicate if this job was deleted (>0 = deleted)
- INTERVAL: schedule for job execution, some format and examples:
- Format: ***M, example: **00 (daily execution on the hour)
- Format: ***0;***M, example: ***0;***5 (execution every 5 minutes)
- Format: HHMM, example: 1200 (daily execution at noon)
- Format: DaDaDa:HHMM, example: Su:1200 (execution on Sundays at noon), MoTuWeThFr:1800 (execution from Monday to Friday at 18:00)
11.3 Database Backup
For MariaDB there exist plenty of strategies for database backups and tons of books and Internet resources on that subject. However, if you run only a medium MariaDB database with a few GByte of data and if you can live with an interruption of services of very few minutes, you may simply use tool mysqldump:
Executed at the command line, this statement copies a database dump in a very robust format into text file openemm.sql. The database dump can be imported back into an empty database openemm simply with
Menu Database Maintenance of OST offers a backup and restore of the OpenEMM database based on these commands.
If you run a bigger MariaDB database which should not be stopped during backup time, we recommend the use of the tool Percona XtraBackup.
11.4 Generic Database Tuning
80% of all application performance problems are really database performance problems. If you run a big OpenEMM installation and you are not satisfied with the application's performance, here are some database tuning tips you should try.
If certain tenants of your OpenEMM database hold a long list of recipients, you may speed up database operations like calculating statistics significantly with a combined index on several fields of table customer_1_binding_tbl.
We recommend the following two indices in case they do not exist yet:
CREATE INDEX custbind1$mlid_user_cuid$idx ON customer_1_binding_tbl (mailinglist_id, user_status, customer_id);
CREATE INDEX custbind1$user_mlid_cuid$idx ON customer_1_binding_tbl (user_status, user_type, mailinglist_id, customer_id);
If you use any other recipient profile field than email for duplicate checks in imports, you should put an index on this field in customer_1_tbl to significantly speed up file imports:
Replace placeholder <fieldname> with the name of the recipient profile field you use for duplicate checks.
11.5 MariaDB Database Tuning
InnoDB is the default storage engine of MariaDB. While InnoDB supports row locking and real transactions for better crash protection, the internal data structure is more complex compared to simpler storage engines like MyISAM. This leads to larger table sizes, slower writes, slower full table scans and slower handling of BLOBs and CLOBs. Also, backup and recovery via mysqldump/mysql is slower.
To get the best performance from MariaDB and because the configuration parameters of storage engine InnoDB can make a big difference, you should define at least the following properties in section [mysqld] of MariaDB's configuration file my.cnf (usually found in directory /etc):
- innodb_buffer_pool_size: default value is only 128 MByte, which is way too small for bigger databases with lots of InnoDB tables. If your EMM database runs on a dedicated server, innodb_buffer_pool_size should be set to 50% of the RAM size of this DBMS server if this server has less than 32 GByte RAM. If the DBMS server has 32 GByte RAM or more, set innodb_buffer_pool_size to 75% of the server RAM size.
- innodb_log_file_size: default value is 96 MByte. Its value should be set to 25% of the size of innodb_buffer_pool_size, but not higher than 256 MByte to limit recovery time after a database crash.
- innodb_file_per_table: By default the InnoDB engine saves all table data into system tablespace file ibdata1 in directory /var/lib/mysql. Set innodb_file_per_table to value 1 to force InnoDB to create a separate file for each database table.
- innodb_lock_wait_timeout: Default value is 50 (seconds). For big databases this value is too small. We recommend to use 1800 (30 minutes) or even 3600 (60 minutes) for this value.
- innodb_rollback_on_timeout: When the timeout defined in innodb_lock_wait_timeout is reached, by default only the blocking statement is rolled back, not the entire transaction. This could lead to data inconsistencies. Therefore, we recommend to set innodb_rollback_on_timeout to value 1 to enforce rollbacks of the entire transactions.
Restart MariaDB to activate these changes.
You can check if the new values are active with
within MariaDB.
If you want to know more about the InnoDB configuration parameters, we recommend to have a look at https://mariadb.com/kb/en/innodb-system-variables.
11.6 OpenEMM Recovery after Database Downtime
If the OpenEMM database was down or the database connection was lost (for whatever reason), please follow these steps to achieve the maximum data integrity:
- Stop OpenEMM. If the database downtime was planned, stop OpenEMM before database shutdown, of course. You stop OpenEMM by executing
- After the OpenEMM database is up and running, re-start OpenEMM with
11.7 Stopping the Sending in Case of Emergency
A mail dispatach can be stopped in the GUI of OpenEMM. But if you want to dig deeper and stop the sending of a certain mailing at the command line level, it is important to understand the process of its generation, distribution and dispatch:
Meta mail packages are generated by backend. These files are located in directory
and one part of the file name is the ID of the mailing. If files with the ID of the mailing to be stopped, are removed, they are no longer distributed by the backend for dispatch to its mailer service. However, to avoid internal conflicts, you have to stop process pickdist before deleting the files with
and you have to re-start pickdist afterwards with
If the backend has already distributed meta mail packages for processing by its mailer service, you will find the packages which have not been processed yet or which are in process right now in directory
If a file is already in process, deleting it does not help. You may check if a file is in process with
If you see process xmlback with a file name as argument, this file is already in process, i.e. process xmlback generates the final email files out of the meta mail package.
Otherwise you may delete the file(s). In this case you should stop process pickdist before deleting the files with
and you have to re-start pickdist afterwards with
If a meta mail package is in process by the mailer service, there is no easy way to delete the email files generated by these packages. You may stop MTA Postfix, check the mail queues and manually delete the files of all emails belonging to the mailing to be stopped.
Please be aware that the statistics for a certain mailing in the GUI of OpenEMM may not be correct if you manually delete meta mail packages of this mailing or email files of this mailing from the MTA queues.
11.8 Out of Memory
If you work with big lists and experience an error message like
Java.lang.OutOfMemoryError: Java heap space
you have to allocate more memory to the Java Virtual machine (JVM). You can increase the minimum and maximum memory in file emm.sh.additional.properties (which overwrites settings of emm.sh) in directory /home/console/bin or /home/rdir/bin by increasing the values of parameters -Xms for minimum and -Xmx for maximum memory in variable JAVA_OPTS_EXTERNAL. If you have allocated all memory available and the error remains, you should increase your server RAM to at least 2 GByte (better: 4 GByte) and modify the parameters accordingly.
11.9 Log Rotation
To prevent the Tomcat log from filling up the hard disk of the OpenEMM server, you may install a log rotation to get rid of old log files. Create file tomcat-openemm in directory /etc/logrotate.d with this suggested content:
/home/openemm/logs/catalina.out {
copytruncate
daily
rotate 7
compress
dateext
size 10k
missingok
sharedscripts
postrotate
########################################################################
# zip files older than 180 min and delete access_logs older than 90 days
########################################################################
find /home/openemm/logs/access -name "*.log" -mmin +180 -exec gzip -9 {} \;
find /home/openemm/logs/access -name "*.log.gz" -mtime +90 -exec rm {} \;
find /home/openemm/logs -name "*.gz" -mtime +10 -exec rm {} \;
find /home/openemm/logs -name "*manager*.log" -mmin +180 -exec rm {} \;
find /home/openemm/logs -name "localhost*.log" -mmin +180 -exec rm {} \;
#################################################
# delete files older than 5 days in logs/webapps/
#################################################
find /home/openemm/logs/webapps -type f -mtime +5 -exec rm {} \;
###############################
# Delete old ARCHIVES > 10 days
###############################
find /home/openemm/var/spool/ARCHIVE -type d -mtime +10 2>/dev/null | xargs -r rm -fr
##################################
# Delete old logfiles from backend
##################################
find /home/openemm/var/log -name "*.log" -mtime +30 -exec rm {} \;
endscript
}
For redirect servers you should replace value 10k in line 6 to 1M and replace console with rdir in all paths. You also may shorten the various retention times to your individual needs.
11.10 Changing Security-Related Files
OpenEMM passwords are saved in the database not only encrypted but salted as well. The salt file with the file extension salt is located in directory /home/openemm/conf/keys.
If you want to change the salt file, please do it before you start operating OpenEMM, because otherwise all saved passwords will not work any longer. For generating a new salt just save a string of letters, digits and other characters of the ASCII character set (decimal values 33 to 126) with a maximum length of 32 characters in a simple text file and name the new file like the old salt file. You may change the file name in configuration file emm.properties of the corresponding service.
To prevent access of the statistics service by a random server, the statistics service is accessible via HTTPS protocol only, uses a private key and grants access only to those servers providing the corresponding public key. This public key is provided by the GUI service. The private key in file birt_private.pem is located in directory /home/openemm/conf/keys/. The corresponding public key in file birt_public.pem is located in the same directory, since in OpenEMM GUI and statistics service operate on the same server.
If you think that the keys are not safe (enough) for your purpose, you may replace them and restart OpenEMM.