This chapter describes how Postfix works, what each piece of the system does, and how these components relate to each other. After going through this material, you should have an understanding of Postfix as a whole, so that you can focus on individual goals.

Postfix consists of a small number of programs that interact with user processes (sendmail, postqueue, postsuper, and so on) and a larger number of programs that run in the background. Only the programs that run in the background are controlled by the master daemon. The master daemon's job is to determine what work there is to do and dispatch the appropriate program to do the work. This modular design allows for a higher level of security because each program runs with the lowest privilege set needed to fulfill its task.

You can think of the whole Postfix system as a router. This may sound strange at first, but remember that a router's job is to look at an IP packet, determine the destination IP address (and possibly the source), and then choose the right interface to route the packet toward its destination. Postfix does the same thing with mail (see Figure 5-1), looking at the destination of a message (the envelope recipient) and the source (the envelope sender) to determine the application that will move the message closer to its final destination.

Postfix works like a router

Now let's look more closely at the system. A real router usually accepts IP packets from multiple interfaces, routing them back out through the interfaces. The same is true for Postfix; it accepts messages from multiple sources and then passes the mail on to multiple destinations. A message's origin may be the local sendmail binary or an SMTP or QMQP connection. The destination can be a local mailbox, outgoing SMTP or LMTP, a pipe into a program, and more. Figure 5-2 shows this view of Postfix.

A Postfix "router" accepts and establishes all kinds of connections

The origin and destination of a message seem clear enough, but how does Postfix pick a delivery method given a destination? A router uses routing tables that match IP addresses to networks to determine a path. Postfix does the same thing with email addresses.

In Postfix, lookup tables are called maps. Postfix uses maps not only to find out where to send mail, but also to impose restrictions on clients, senders, and recipients, and to check certain patterns in email content. Figure 5-3 shows where the maps--to name but a few, aliases, virtual, and transport are shown--fit in.

Maps are the lookup tables of the Postfix "router"

Figure 5-4 shows an overview of the Postfix daemons and how they fit together.

Figure 5-4: The relationships between the Postfix daemons

Postfix polls all queues in the directory specified by the queue_directory parameter in your main.cf file. The queue directory is usually /var/spool/postfix. Each queue has its own subdirectory with a name identifying the queue. All messages that Postfix handles stay in these directories until Postfix delivers them. You can determine the status of a message by its queue: incoming, maildrop, deferred, active, hold, or corrupt.

Maps are files and databases that Postfix uses to look up information. Maps have many different purposes, but they all have one thing in common--a left-hand side (LHS, or key) and a right-hand side (RHS, or value).

To use a map, you specify a key and get the associated value as a result.

Note: The keys and values here come from various files and would not make sense in one file. The preceding list is just an illustration to show that all map entries take the same basic form.

Map Types

Postfix can use many different kinds of maps. The formats available depend on the way Postfix was compiled on your particular system. To find out what formats your Postfix supports, run postconf -m on the command line. You should get a list of map types:

# postconf -m
btree
cdb
cidr
environ
hash
ldap
mysql
nis
pcre
proxy
regexp
sdbm
static
tcp
unix

Indexed Maps (hash, btree, dbm, and So On)

Indexed maps are binary databases built from regular text files with commands such as newaliases, postalias, and postmap. The binary maps have an indexed format so that Postfix can quickly retrieve the value associated with a key. As a further performance improvement, the Postfix daemons open these maps when starting up, and they do not re-read them unless they notice a change in the map files in the filesystem. To reload a map, a daemon exits and a new one is started by the master daemon.

The most common indexed maps are built from the aliases, virtual, transport, relocated, and sasl_passwd text files. You can identify a map file because its name is the original file with a suffix that also tells you the index format. For example, an aliases map file built with the postalias command is named aliases.db.

Postfix queries entries in a predefined order specified in the manual pages for the tables (access(5), transport(5), virtual(5), aliases(5) and canonical(5)). In other words, each map lookup actually consists of a series of single queries (derived from the original query) on single keys in the indexed map.

Linear Maps (PCRE, regexp, CIDR, and Flat Files)

Linear maps are regular text files. Postfix reads these files from top to bottom, making them different from indexed maps. This difference is quite important, because the first match in the file determines the action that Postfix will take. Postfix ignores any later entries, whether they match the query or not.

Consider the following regexp map, where a john.doe@example.com lookup returns OK, because the first line matches.

/john\.doe@example\.com/ OK
/example\.com/ REJECT

However, if you swap the lines in the regexp map, the other entry matches first, so the same john.doe@example.com lookup returns REJECT:

/example\.com/ REJECT
/john\.doe@example\.com/ OK

You do not need to convert linear maps to a binary form (in fact, you can't do it). The Postfix daemons read them at startup and do not notice any changes to the map until they are restarted. Typical Postfix linear maps include header_checks, body_checks, and mime_header_checks (see Chapter 9).

This can cause a significant slowdown, especially if you have extensive *_checks parameters that use regexp or PCRE (Perl-compatible regular expression) type maps in order to prevent spam from entering the system. When this happens, it's usually time to hand complex spam filtering to an external application.

To make the Postfix daemons notice changes in linear maps, run postfix reload. If the timing is not critical, you can set the max_use parameter to define a time-to-live for daemons. As soon as a daemon has processed the number of tasks specified in that parameter, it quits and is restarted by master. Upon restart, it re-reads all required maps.

Databases (MySQL, PostgreSQL, LDAP)

Postfix treats a database just like an indexed map. The result of a database query is Match (along with the value returned by the query) or No match. The principal difference between a database map and an indexed map is that you do not need to restart a daemon when there is a change to the database. Postfix does not assume that the postmaster is the only person who can alter the database.

The drawback to this approach is that the database may not be able to handle the number of queries gracefully, because Postfix needs to perform at least three queries for each lookup in a map (see the "See How Postfix Queries Maps" section that follows). Under heavy load, the database backend could stop working, and your mail service would be vulnerable to a self-induced meltdown or a denial-of-service attack. This possibility should not prevent you from using database backends, but you should be aware of the risk.

Database lookups can become a problem for systems with a heavy load, but this isn't the only issue to consider--latency can be another problem. Database queries have a higher latency than indexed maps because Postfix must connect to the database backend, send the query, and then wait for the result. With an indexed map, Postfix has only to consult data that is already loaded in memory.

If your database becomes a bottleneck, and you do not have an excessively large map, you can insert a map between the database and Postfix. That is, you can create an indexed map from a complete database query, and then run Postfix with that map. You need to remember to update the map as often as necessary, but the proxymap daemon can be used to significantly reduce the number of concurrent connections.

Determining the Number of Simultaneous Connections to a Database

Postfix daemons (smtpd, smtp, and so on) run with a process limit (set by default_process_limit) of 100 simultaneous processes. Running at peak load, there would be 100 concurrent smtpd daemons, each querying the database backend for one access(5) lookup (e.g., because we use a map for checking if the client is in our personal blacklist and should then be denied from sending mail to us).

Remember that one lookup results in at least three queries, so the number of simultaneous queries to the database would be at least default_process_limit * 3 (which, in the default configuration, would be 300 queries), while the number of simultaneous connections is default_process_limit. This is only the number of queries and connections for smtpd daemons; other daemons, such as local and qmgr, may be working on other jobs, adding to the number of open connections and simultaneous queries.

How Postfix Queries Maps

Maps can be used for various tasks. Postfix has table-driven mechanisms that use maps (see access(5), aliases(5), canonical(5), and transport(5)). These maps can use different lookup mechanisms (LDAP, NIS, SQL, btree, hash, regexp, cdb, cidr, pcre, and so on). Note that the lookup order described below only applies to access(5) type maps.

This order of lookups implies that Postfix performs several lookups for each query, which isn't really a problem unless you're using high-latency maps like SQL or LDAP maps (and, of course, you should expect that a lot of lookups will need multiple queries). This is just one thing to remember before you put all your maps into LDAP and then complain on the postfix-users mailing list that "Postfix is slow. . . ."

Postfix supports information sources that are not built on top of Postfix and that aren't even under your direct control, such as blacklists (DNSBL and RHSBL lists), DNS-based lists, and other external sources. Blacklists are almost exclusively used in smtpd_*_restrictions parameters in order to reject mail coming from clients or senders listed in DNSBL- or RHSBL-style lists (see Chapter 7).

As with any external query, these lookups can fail due to connectivity problems, denial-of-service attacks against the blacklist servers, and other problems. In case of a timeout or other failure, Postfix may still accept mail (bypassing a possible restriction), but it will log an appropriate warning to the mail log.

Postfix ships with a number of command-line utilities to assist you with administration tasks. Although they perform different functions (such as querying maps, examining queue files, dequeuing and requeuing messages, and changing the configuration), they all have one thing in common--their names start with "post."

postfix

The postfix command stops, starts, and reloads the configuration with the stop, start, and reload options.

postalias

The postalias command creates an indexed alias map from an alias file. It works just like the postmap command (described shortly), but it pays special attention to the notation in an alias file (where a colon separates the key and value). postalias must be used on alias files.

postcat

The postcat command displays the content of a message in a mail queue.

To read a message in a mail queue, you need its queue ID. Run mailq for a list of queue IDs. For example, the queue ID of the following message is F2B9715C0B3:

# mailq
F2B9715C0B3 2464 Mon Oct 13 15:29:39 markus.herrmann@example.com
            (connect to mail.example.com[217.6.113.151]: Connection timed out)
            torsten.hecke@example.net
-- 2 Kbytes in 1 Requests.

After obtaining a queue ID, use it as an option to postcat to see the contents of the queue file:

# postcat -q F2B9715C0B3

postmap

The postmap command's primary purpose is to build indexed maps from flat files. For example, to build /etc/postfix/virtual.db from /etc/postfix/virtual, run the following command.

# postmap hash:/etc/postfix/virtual

The postmap command can do more. Among its most useful features is the ability to test any kind of map that your Postfix installation supports. This is extremely helpful when debugging a configuration where lookups to the maps appear to fail, and you are unsure whether the key and value are actually visible to Postfix.

Debugging an Entry in a Lookup Table

To determine whether Postfix can find an entry in a map, use postmap -q. For example, the following command returns the value assigned to the key <sender@example.com> in the map /etc/postfix/sender_access (type hash):

# postmap -q sender@example.com hash:/etc/postfix/sender_access
OK

It's important to note that postmap does not look for the terms <sender@>, <example.com>, and <com>, even though these terms are in the access(5) manual page. You need to perform those lookups manually:

# postmap -q sender@ hash:/etc/postfix/sender_access
# postmap -q example.com hash:/etc/postfix/sender_access
# postmap -q com hash:/etc/postfix/sender_access

postdrop

The postdrop command reads mail from the standard input and drops the result into the maildrop directory. This program works in conjunction with the sendmail utility.

postkick

The postkick command sends a request to a Postfix daemon through a local transport channel, making Postfix interprocess communication accessible to shell scripts and other programs.

Requeuing a Message

The following advanced postkick example shows how to requeue a message for immediate redelivery:

# cat queueidlist | postsuper -r -
postkick public pickup W

This sequence of commands moves all selected messages listed in queueidlist to the maildrop queue with the postsuper -r - command, where the pickup daemon would process them like any other piece of mail. By doing this, you reset the content filter to the setting appropriate for local submission and add an extra Received: header.

The postkick command requests an immediate maildrop queue scan. Otherwise, the messages would stay in the maildrop queue for a maximum of 60 seconds. The pickup daemon submits the message to the cleanup daemon, where it gets a new queueid and is deposited into the incoming queue. The whole point is to move the message to the active queue as quickly as possible.

postlock

The postlock command gives you exclusive access to mbox files that Postfix writes, and then it runs a command while holding the lock. The lock you get from postlock is compatible with the Postfix local delivery agent. Postfix does not touch the file while your command executes. Here is an example:

# postlock /var/mail/user from

postlog

The postlog command allows external programs, such as shell scripts, to write messages to the mail log. This is a Postfix-compatible logging interface; by default, it logs the text from the command line as a single record. Here's a very simple example:

# postlog This is a test
postlog: This is a test
# grep "This is a test" /var/log/mail.log
Feb 20 11:50:16 mail postlog: This is a test

postqueue

The postqueue command is a user interface to Postfix queues, giving you functionality that is traditionally available with the sendmail command.

postsuper

The postsuper command maintains jobs inside Postfix queues. Unlike postqueue, this command is restricted to the superuser, and it can run while Postfix is down. Some postsuper features are needed to check the queue before daemon processes are started. Table 5-1 shows what the postsuper command can do.

One of the most frequent uses of postsuper is deleting a message from the mail queue with postsuper -d queueid. Doing this manually is tedious, especially when deleting many files. The following Perl script (delete_from_mailq) makes it easier:

#!/usr/bin/perl
$REGEXP = shift || die "no email-address given (regexp-style, e.g. bl.*\@yahoo.com)!";
@data = qx</usr/sbin/postqueue -p>;
for (@data) {
	if (/^(\w+)(\*|\!)?\s/) {
		$queue_id = $1;
	}
	if($queue_id) {
		if (/$REGEXP/i) {
			$Q{$queue_id} = 1;
			$queue_id = "";
		}
	}
}
#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;
foreach (keys %Q) {
	print POSTSUPER "$_\n";
};
close(POSTSUPER);

Here's how you'd use it:

# mailq
C73A015C095     7509 Mon Oct 13 14:56:17 MAILER-DAEMON
	(connect to mx5.ancientaward.com[64.156.166.211]: Connection refused)
 
National_Nosepicking_Month@mx5.ancientaward.com

Notice that the sender is identified as <MAILER-DAEMON> here. To remove these bounces, run delete-from-mailq as root:

# delete-from-mailq MAILER-DAEMON
postsuper: C73A015C095: removed
postsuper: Deleted: 1 message

About the Authors:

Ralf Hildebrandt and Patrick Koetter are active and well-known figures in the Postfix community. Hildebrandt is a systems engineer for T-NetPro, a German telecommunications company, and Koetter runs his own company consulting and developing corporate communication for customers in Europe and Africa. Both have spoken about Postfix at industry conferences and contribute regularly to a number of open source mailing lists.

No Starch Press has more information on the book here: www.nostarch.com/postfix.htmThe authors maintain an errata and download page for the book, which you might want to check out at: www.postfix-book.com/errata.html