Template:Dovecot:Main Page Information

DOVECOT PRO

Dovecot Pro Overview

Dovecot Pro provides a reliable and scalable Mail solution to customers with enterprise level stability and scalability on top of the open source version. This product is built based on the experience the Dovecot Team has made by working for many years with the largest ISPs of the world. Dovecot Pro license enables Object Storage Plugins supports various object storages.

Dovecot Pro supports easy change from many existing IMAP and POP3 servers by allowing transparent migration of users in to Dovecot platform.

Dovecot Pro Product Components

Main Dovecot Pro Product components are:

• Dovecot Pro Mail
• Dovecot Object Storage
• Dovecot Full Text Search
• Dovecot Pro Vault
• Dovecot Migration Framework

Dovecot Pro Mail

Overview

Depending the size of the installation Dovecot mail system contains servers in different roles: Proxies, Directors, Backends. Stateless design in Dovecot allows any of the components to be lost, shut down for maintenance or upgrade, without affecting the service availability. All the users on the same site will still receive service, only capacity providing the service is diminished, meaning that some speed of the service might be lost if system is under heavy load at that time. Dovecot architecture is possible to scale horizontally and vertically. The recommended network topology has:

• Dovecot Proxy in public network,
• Dovecot Director and Dovecot Backend in private network,
• Object storage typically in Storage network.

Backends can use independent local storages or extensible-shared storage, like NFS or object storage. All Dovecot components include the same application but are configured to become different components. The Dovecot components are explained below in more detail.

Dovecot Proxy

Dovecot Proxies act as a frontend IMAP/POP3/LMTP servers for client connections. Dovecot Proxies make the user database lookup to LDAP to validate the user and to lookup the user parameters. In multi-site installations, Dovecot Proxy servers’ main function is to forward user to the load balancer in front of the Dovecot Director of the site where the user is located. Dovecot proxy also decrypts TLS/SSL sessions, which are more CPU consuming than the user lookup and traffic forwarding.

Dovecot Director

Directors listen to IMAP/POP3/LMTP protocols and balance load and provide high-availability for the Dovecot Backends.

The main difference between a regular load balancer and Dovecot Director is that the director makes sure that a single user is never accessed by different Backends at the same time. This is needed in order to avoid user data corruption. Dovecot Directors are taking care of this as they are sharing information of the users and user sessions while the user sessions are active. Each Dovecot Director knows which Backend is taking care of existing user session for any user. In front of Dovecot Directors there needs to be a load balancer to provide High Availability (HA) for them. Dovecot Directors are stateless meaning that any one of them can be switched off for maintenance without user noticing. This also makes updating procedures for the Dovecot components possible without interrupting the user session, thus making the operation transparent to user.

Dovecot Backend

Dovecot Backend does all the hard work of reading and writing mails to storage and handling all of the IMAP/POP3/LMTP protocols. Dovecot Backend is connected to mail storage, typically filesystem or cloud storage, where user mails and mail indexes are stored.

As an user is connecting to Dovecot for reading the mails the user’s mail indexes are fetched from the mail storage. The mail indexes are updated as long as the session is valid. This makes fetching of the user’s mail perform fast. As the session validity expires or the user logs off, the updated index is stored back to storage waiting for the next login. Dovecot Backends are stateless, making it possible for user connections to be connected to any Backend.

Dovecot Object Storage

Overview

Dovecot supports different object storages as mail storage, where the mails can be stored to cloud services such as but not limited to Amazon S3, or Microsoft Azure or to local “in house” object storage olution such as but not limited to Scality, Ceph or Openstack Swift or using mixture of both (storing spam to cloud, actual mails to local object storage). Scality is well integrated to Dovecot and allows use of generic low cost x86 servers instead of NAS/SAN devices.

If mails are stored to Object Storage, they can be accessed from any Backend, which are totally stateless as well as the other servers. All sessions of the same user are directed to the same Backend by Dovecot Director to prevent data corruption. As the user session reaches the Dovecot Backend the user’s indexes are loaded from object storage to local cache of the Backend. As the session is over, the indexes are loaded back to object storage. In the unlucky event of Backend failure the next Backend can continue where the other Backend left off since there are no locking problems (in comparison to NFS) and the indexes can be merged and updated by Dovecot Backend as soon as the servers are back on-line.

NFS storage is also supported, but this does not require Dovecot Object Storage plug-in. More about how to setup NFS storage can be found from section 8.4 Object Storage Cluster setup.

Object Storage Advantages

Dovecot Object Storage usage is optimized for cloud technologies. The Dovecot Object Storage Plugin enables your Dovecot email service to store the email data into cost efficient cloud storage solution. Object storage plugin keeps track of which users' and mailboxes' index files have been opened locally and uploads / downloads them to object storage as necessary. Object Storage is using the same Dovecot index files, which are already proven to be stable and efficient. The object storage plugin consists of three major components:

• The actual obox plugin, which implements an sdbox-like mailbox format, which is accessed via the lib-fs API (instead of POSIX filesystem API). Each message is stored in its own file (object). The index files are still accessed using the POSIX FS API.

• lib-fs Backends that implement support for various object storages, such as Amazon S3, Windows Azure and Dropbox. There is also a "fscache" Backend that implements local filesystem cache for mail objects.

• metacache processes, which implement synchronizing Dovecot index files (and other metadata files, like Sieve scripts) between the local filesystem and object storage.

Dovecot Full Text Search

Overview

As the amount and importance of information stored in email messages is increasing in people’s everyday lives, searching through those messages is becoming ever more important. At the same time mobile clients add their own restrictions for what can be done on the client side.

The ever-diversifying client software also tests the limits of the IMAP protocol and current server implementations. When an indexing Backend is not present, searches fall back on slow sequential searches through all message headers or text. Thus efficient and feature rich server side searching has grown in importance.

A two-part reimplementation of the Dovecot indexing and search architecture has been designed. This is to provide more customizability for searching and a more unified range of features for all existing FTS plugins. Additionally a renewed Dovecot native implementation of the full FTS stack is provided for better performance and scaling for large mail volumes.

Full text indexing and search

Triggers for Full Text Search (FTS) indexing are configurable. It can be started on demand when searching, automatically when new messages arrive or as a batch job. Full text search has the following features:

• Indexes can be stored in the Object Storage
• Smaller indexes compared to all current search Backends to improve search

performance

• Avoid indexing duplicate data by using word stemming and normalization and skipping

bad character and stop words

• Part of Dovecot mail server Backend, no 3rd party software needed
• No extra java virtual machine needed for search sing word stemming and normalizat
• Substring search for partial word search

Dovecot's standard IMAP SEARCH TEXT/BODY parameters use the FTS indexes. Searches through message headers benefit from Dovecot's fast message index cache implementation, which often contains the necessary information. Optionally header searches can also be done from FTS indexes.

Search Parameters

The following are specified as the search parameters to be used in the webmail User Interface to be passed on to the Dovecot Search Backend:

Details of the possible search queries can be found from here: http://wiki.dovecot.org/Tools/Doveadm/SearchQuery

Dovecot Pro Vault

Overview

The storage for Dovecot Vault is based on an existing cluster based on Obox. Emails are stored in a read-only namespace. Apart from the regular Obox configuration of the storage cluster, the namespace is configured as hidden and the containing mailbox is automatically created.

Read-only access to Vault e-mails

All mails stored in the archive namespace have to be read only. This is achieved using the “ACL” plugin to prevent deleting emails.

Incoming e-mails

Incoming e-mails are delivered to Dovecot via LMTP. The “Vault” plugin performs the job of storing the emails in the archive namespace first, and if that succeeds it begins the actual mail delivery, including running any Sieve scripts.

Outgoing e-mails

The SMTP submission server (e.g. Postfix) is used to catch outgoing e-mails and then BCCed to Dovecot’s LMTP via another port. LMTP will, then, deliver these e-mails to the archive namespace, as usual. The LMTP service for the outgoing e-mails will be configured to execute a sieve filtering script to store the email to the archive namespace and add the “\Seen” flag to it.

Vault Encryption

Emails stored in the archive namespace can be encrypted using the “mail-crypt” plugin. Encryption will be done using Elliptic Curve Cryptography.

Dovecot Migration Framework

Overview

Dovecot migration framework is a framework of controlling mailbox content migration from a centralized management system. Framework provides centralized repository to limit number of Backends running mailbox dsync processes and number of parallel dsync processes each Backend is allowed to run. It also gathers statistics from each Backend. Framework provides JSON/RESTAPI interface for integration into customers existing environment and management tools. Framework also optionally sends statistics information to Graphite server for progress visualisation.

The Migration Framework can be used for both Content and User Migration phases.

Key features

• Queuing and distribution of accounts to migrate with locking for multiple Backends running the actual migration.
• Centralized status display
• Centralized statistics collection
• Centralized control of number of Backends doing the actual content copy
• Centralized control on number of parallel copies each Backend is allowed to run
• Centralized log collection
• Progress visualization with 3rd party Graphite possible