Skip to content

12 EMM Architecture

12.1 General Architecture

EMM consists of several independent services for scalability. It runs on top of a well proven Open Source software stack without any further dependencies to any commercial software:

  • Red Hat Enterprise Linux (RHEL) or Suse Linux Enterprise Server (SLES) or compatible
  • Postfix
  • MariaDB
  • Java OpenJDK
  • Apache Tomcat
  • Python 3
  • Node.js

You may also use a commercial Oracle RDBMS instead of MariaDB.

The following image shows the software stack EMM is running on (white background) and the logical code layers of EMM and major frameworks and libraries used by EMM (grey background):

We try hard to keep the software stack EMM relies on as lean as possible. This will minimize the maintenance efforts of updating software stack components due to new security risks or required support of new hardware or software, or replacing components altogether due to their end of life.

12.2 Open Platform Design

A software like EMM must not operate as an island, but it should be capable to be integrated with third party systems like a CRM or CMS software, an e-commerce shop, an ERP platform or a business intelligence software. Therefore, EMM was designed to be a very open and flexible platform with lots of interfaces and extension capabilities.

EMM offers not only a highly customizable graphical interface (GUI) for its users, but also an easy to use URL API für URI tunneling and an (optional) rich SOAP webservice interface (including methodes for bulk operations) as well as an (optional) RESTful API to be used by third-party software. For outbound communication EMM provides an (optional) callback API for webhooks to notify third-party software about various events.

Additionally, for ETL purposes EMM provides a time-driven automated data import/export to exchange data files with third-party systems. You may also use this feature to export raw data of EMM.

You may also use integration platforms Make or N8N to connect and integrate EMM with other applications. The EMM Inhouse Wiki offers further information on EMM connectors for these platforms.

Furthermore, you are able to extend EMM's database schema with additional customized data tables which can be filled and read via the automated data import/export.

To process and enrich emails from legacy applications, EMM offers an (optional) SMTP API. The SMTP API accepts emails and is able to rebuild and resend those mails with HTML design, added headers and footers, a tracking pixel to track mail openings as well as trackable links. After dispatch EMM takes care of bounce management and mailbox provider feedback.

The variety of EMM's APIs ensures, that EMM can be integrated as a headless system in any existing environment.

12.3 Architecture Components

The EMM software is not a monolith, but a modulith, divided into several independent backend and frontend services (software) to be able to run on different servers (hardware). The break down of the software into different modules opposed to a single monolith allows for better performance, offers scalability and higher availability (see sections below). Communication between the various EMM services is done via the database, i.e. the database is the hub of the application.

This image shows an abstraction of a typical EMM deployment:

This image shows an abstraction of a typical EMM deployment

To gain a better understanding of the purpose of the various backend and frontend services, this is a summary of their main jobs:

12.3.1 EMM Backend Services

  • Merger: This service is active on the merger server, reads mail-related data from the database and generates meta mail packages for processing by the mailer service (mailer). The merger service also collects reponses and bounce data from the mailer services and mailloop service to update the database with results from the mail sendings. A merger service implementation is also deployed on the frontend and redirect servers to create mail previews and to send out trigger-based mails.
  • Mailer: This service is active on the mail servers and generates individual emails out of the meta mail packages and sends them out into the Internet (no database connection required). It also registers instant bounce messages. A mailer service implementation is also deployed on the frontend and redirect servers to create mail previews and to send out trigger-based mails.
  • Mailloop: This service is active on the filter server and processes feedback like bounce messages, autoresponder mails and mail replies by recipients. It registers the results of the processed feedback to be used by the merger service, it sends out autoresponder mails defined in the EMM GUI and it forwards all mail replies not filtered out to a predefined destination (feedback address).
  • Backend Utilities: A collection of backend scripts, deployed on the frontend and redirect servers.

The backend services consist mainly of Python scripts, launched by shell scripts, a C program (xmlback) and a Java package for the merger service. Separation of concerns leads to one script for one job with heavy logging for more robustness and to ease maintenance.

The processing of the meta mail packages to generate the individual emails is written in C (file xmlback), because this part of the code is very performance sensitive.

12.3.2 EMM Frontend Services

  • GUI: This service is active on the/a frontend server and provides the browser-based user interface of EMM.
  • Statistics: This service is active on the/a frontend server and generates statistics (tables with numerical values as well as visual charts and diagrams) for the EMM GUI and creates reports in PDF and CSV format for download and to be sent as email attachments.
  • Webservices: This service is active on the/a frontend server and provides the webservice interface of EMM.
  • REST: This service is active on the/a frontend server and provides the RESTful interface of EMM.
  • Redirect: This service is active on the redirect servers and receives requests from mail openers, link clickers and tracking points on websites, writes the info to the database, redirects link click requests to their final destination (web pages) and reads/writes cookies of the mail recipients for web tracking.
    Optionally, the redirect frontend tracks views of WPNs and clicks in WPNs. It also registers any subscriptions and unsubscriptions from websites offering WPNs.

12.3.3 Major Components of the Software Stack

The frontend services consist of Java web applications, including JSPs and Javascript code, running on top of the web container Tomcat and Jarkata EE libs. The frontend services employ Java framework Spring for dependency injection (IoC), Spring MVC to implement the MVC pattern and Tiles to modularize the user interface.

Spring MVC is used because it integrates nicely with Spring and Tiles. Since Spring MVC is comparatively thin and does not abstract away too much from the classic request-response cycle (opposed to a component-based MVC framework like JSF) it is very easy to integrate Ajax functionality and Javascript code.

Javascript code is mainly used to improve the usability of the GUI and to reduce the number of full request-response cycles. This helps improving the responsiveness and usability of the GUI. EMM does not use a JS framework but Vanilla JS for performance reasons.

For performance reasons, EMM does not decouple the GUI from the service code via a REST API, but the GUI communicates directly with the service code, which is part of the GUI service and runs in the web container of Tomcat.

Also, EMM does not use an O/R mapper like Hibernate due to some negative experience with lacking performance, cache consistency problems and obscure SQL statements generated by these frameworks. EMM uses Spring JDBC templates with prepared SQL statements and its own caching instead to achieve maximum database performance.

SQL statements are customized and optimized for DBMS Oracle and MariaDB. We have chosen the open source DBMS MariaDB over PostgreSQL, because MariaDB usually needs less resources and offers better performance for read-heavy applications like EMM and EMM does not need the more sophisticated features of PostgreSQL. However, support of PostgreSQL is in the works!

EMM does not use separate databases for online transaction processing (OLTP) and analytical processing (OLAP). This has the advantage that all statistics and analytical reports can be created in realtime based on realtime data. The direct use of SQL, like described in the paragraphs before, makes it possible to archieve a decent performance with minimal waiting time even for statistics and reports based on bigger data sets.

The statistics service of EMM delivers realtime data for the GUI, the report files and the webservices. For the GUI and report files, it uses the BIRT framework and its Webviewer because BIRT supports all kind of data sources, uses design templates, generates various output formats and its Report Designer integrates nicely with the Eclipse IDE, used for the development of the EMM frontend. The Webviewer of BIRT renders tables and charts as HTML or PDF and the runtime engine also provides its results as CSV formatted report files. Other file formats like MS Excel would be possible too.

The webservices of EMM are based on the Spring WS framework. Spring WS eases the development of new webservices because it uses the contract first pattern: An XSD file (XML schema) defines request and response signatures of the webservice methods, and a XML configuration file provides data for the endpoint mapping, data for automatic WSDL generation, etc. Based on these two configuration files the framework auto-generates the required domain classes.

For maximum performance, EMM relies on an SMTP server to send out mails and to accept bounces and replies. EMM uses Postfix for that task by default, because Postfix is a proven, secure, and fast MTA (Mail Transfer Agent), and Postfix is able to encrypt emails out of the box (if the receiving MTA is able to process encypted mails). And opposed to Sendmail, Postfix offers easier configuration, better performance and better logging.

To process responses received by Postfix, EMM uses a plugin based on Sendmail's milter API. To be able to handle several domains in parallel, EMM uses Postfix' transport maps (in cooperation wit relay domains).

Logging of EMM can be configured via configuration file log4j.properties for maintenance purposes.

12.4 Architecture for Scalability

While EMM is able to run on a single server, it can demonstrate its full strength as a multi-server platform for high scalability through scale-out and for high availability through redundancy. A scale-out to a higher number of servers is possible due to EMM's code architecture, consisting of various independent services.

Typical deployments of EMM consist of 3 to 30 servers – mainly depending on the number of mails to be sent per minute. Usually, the DBMS server and the merger service are represented by one dedicated server each (sometimes with an optional DBMS standby server and sometimes with a second merger service on another server). The mailloop service either runs on a dedicated machine or on a shared server. Depending on the availability demands, the frontend services are deployed on one or two servers. The mailer services and redirect services typically run on multiple servers to increase mail output and input processing (like tracking of mail openings and processing link clicks).

12.5 Architecture for Availability

The redirect service should always be implemented on several servers using round-robin DNS for load balancing. For high availability we recommend to implement a pair of load balancers in front of the servers offering redirect services. The load balancers take care that servers not responding fast enough are no longer served with requests. If you use a load balancer for servers running redirect services, make sure that the load balancer forwards the external IP addresses in the requests to the redirect services. Otherwise, some mailing statistics and reipient data (like the op-in IP address) could be distorted.

We recommend using Apache HTTP Server for load balancing with Tomcat module mod_jk and the tool keepalived on each load balancer to check each others availability. Additionally, we offer an auto-restart option which very often helps if a redirect service is not available. It can be activated for times not covered by your support team. (Please contact the AGNITAS support team for implementation support at support@agnitas.com.)

For high availability environments we recommend to operate a standby database for a quick failover in case of hardware problems, because the database is a single point of failure. (Please contact the AGNITAS support team for implementation support.)

For high availability of the GUI service and/or webservices we recommend to operate it on two different servers at the same time with a load balancer in front, serving only one of these two servers. In this case you also have the option to implement a ZDU capability (Zero Downtime Updates) to update the EMM code without interrupting the service for the user. (Please contact the AGNITAS support team for implementation support.)

For high availability of the central merger service we offer the possibility to operate two servers with merger services in parallel with an integrated failover capability, i.e. if one merger service fails, the other one takes over. (Please contact the AGNITAS support team for implementation support.)

To monitor the hardware EMM is running on, we recommend to implement a monitoring system like Icinga which monitors important parameters like CPU and memory consumption, CPU load, I/O traffic and disk space. Especially disk space can be an issue since memory consumption of EMM can grow quite quickly with tracking info and log content when you send lots of emails. See sections Automatic File Cleanup, Automatic Datzabase Cleanup and Log Rotation for tipps on how to reduce disk space.

If DBMS MariaDB is chosen to run the database of EMM, EMM uses the database engine InnoDB. This database engine is the default engine since MariaDB/MySQL 5.5 and offers maximum data integrity because of its auto repair feature and different recovery modes in case of an unplanned server outage.

12.6 Architecture for Security

AGNITAS, the developer of EMM, is certified according to ISO 27001 (IT data security), ISO 27017 (cloud data security) and ISO 27018 (cloud data privacy). Because of this, we put high emphasis on information security in our software code, development processes and operations. Therefore, all passwords in the EMM database are strongly encrypted and salted (see section Changing Security-Related Files).

Since the GUI service has to be accessible from the outside, it provides several security features: EMM requires complex passwords, frequent password changes, it limits and logs login tries, uses 2FA (two factor authentication) and binds the EMM session to an initial session Cookie. All these security features are configurable in the user interface or database.

Once logged in into EMM, access to its features is controlled by a permission system with customizable user profiles and more than 250 single user rights (please see the user manual for details).

The GUI service never trusts input from clients. Therefore, all input is screened on the server-side based on an internal whitelist, i.e. input which is not explicitly defined as acceptable, will be rejected. XSS and CSRF protection are implemented as well.

GUI service and statistics service both use prepared SQL statements to prevent users from injecting malformed code into existing SQL statements, which could be used to manipulate data or to spy out confidential data from EMM's database.

The redirect service has to be accessible from the outside, but offers only a few servlets as access points, and all input is screened on the server-side based on an internal whitelist.

The SOAP webservices, which also have to be accessible from the outside, use WSSE (webservice security extension) for authentication. Please see the SOAP webservices API documentation for further details.

EMM is periodically penetration tested by independent companies like Deutsche Telekom and TÜV and all relevant findings are addressed by the devlopment team.