$Cambridge: exim/doc/doc-txt/NewStuff,v 1.8 2004/11/10 10:29:56 ph10 Exp $ New Features in Exim -------------------- This file contains descriptions of new features that have been added to Exim, but have not yet made it into the main manual (which is most conveniently updated when there is a relatively large batch of changes). The doc/ChangeLog file contains a listing of all changes, including bug fixes. Version 4.44 ------------ 1. There is a new build-time option called CONFIGURE_GROUP which works like CONFIGURE_OWNER. It specifies one additional group that is permitted for the runtime configuration file when the group write permission is set. 2. The "control=submission" facility has a new option /retain_sender. This has the effect of setting local_sender_retain true and local_from_check false for the incoming message in which it is encountered. 3. $recipients is now available in the predata ACL (oversight). 4. The value of address_data from a sender verification is now available in $sender_address_data in subsequent conditions in the ACL statement. Note: this is just like $address_data. The value does not persist after the end of the current ACL statement. If you want to preserve it, you can use one of the ACL variables. 5. The redirect router has two new options: forbid_sieve_filter and forbid_exim_filter. When filtering is enabled by allow_filter, these options control which type(s) of filtering are permitted. By default, both Exim and Sieve filters are allowed. 6. A new option for callouts makes it possible to set a different (usually smaller) timeout for making the SMTP connection. The keyword is "connect". For example: verify = sender/callout=5s,connect=1s If not specified, it defaults to the general timeout value. 7. The new variables $sender_verify_failure and $recipient_verify_failure contain information about exactly what failed. In an ACL, after one of these failures, the relevant variable contains one of the following words: qualify the address was unqualified (no domain), and the message was neither local nor came from an exempted host; route routing failed; mail routing succeeded, and a callout was attempted; rejection occurred at or before the MAIL command (that is, on initial connection, HELO, or MAIL); recipient the RCPT command in a callout was rejected; postmaster the postmaster check in a callout was rejected. The main use of these variables is expected to be to distinguish between rejections of MAIL and rejections of RCPT. 8. The command line option -dd behaves exactly like -d except when used on a command that starts a daemon process. In that case, debugging is turned off for the subprocesses that the daemon creates. Thus, it is useful for monitoring the behaviour of the daemon without creating as much output as full debugging. Version 4.43 ------------ 1. There is a new Boolean global option called mua_wrapper, defaulting false. This causes Exim to run an a restricted mode, in order to provide a very specific service. Background: On a personal computer, it is a common requirement for all email to be sent to a smarthost. There are plenty of MUAs that can be configured to operate that way, for all the popular operating systems. However, there are MUAs for Unix-like systems that cannot be so configured: they submit messages using the command line interface of /usr/sbin/sendmail. In addition, utility programs such as cron submit messages this way. Requirement: The requirement is for something that can provide the /usr/sbin/sendmail interface and deliver messages to a smarthost, but not provide any queueing or retrying facilities. Furthermore, the delivery to the smarthost should be synchronous, so that if it fails, the sending MUA is immediately informed. In other words, we want something that in effect converts a command-line MUA into a TCP/SMTP MUA. Solutions: There are a number of applications (for example, ssmtp) that do this job. However, people have found them to be lacking in various ways. For instance, some sites want to allow aliasing and forwarding before sending to the smarthost. Using Exim: Exim already had the necessary infrastructure for doing this job. Just a few tweaks were needed to make it behave as required, though it is somewhat of an overkill to use a fully-featured MTA for this purpose. Setting mua_wrapper=true causes Exim to run in a special mode where it assumes that it is being used to "wrap" a command-line MUA in the manner just described. If you set mua_wrapper=true, you also need to provide a compatible router and transport configuration. Typically there will be just one router and one transport, sending everything to a smarthost. When run in MUA wrapping mode, the behaviour of Exim changes in the following ways: (a) A daemon cannot be run, nor will Exim accept incoming messages from inetd. In other words, the only way to submit messages is via the command line. (b) Each message is synchonously delivered as soon as it is received (-odi is assumed). All queueing options (queue_only, queue_smtp_domains, control=queue, control=freeze in an ACL etc.) are quietly ignored. The Exim reception process does not finish until the delivery attempt is complete. If the delivery was successful, a zero return code is given. (c) Address redirection is permitted, but the final routing for all addresses must be to the same remote transport, and to the same list of hosts. Furthermore, the return_address must be the same for all recipients, as must any added or deleted header lines. In other words, it must be possible to deliver the message in a single SMTP transaction, however many recipients there are. (d) If the conditions in (c) are not met, or if routing any address results in a failure or defer status, or if Exim is unable to deliver all the recipients successfully to one of the hosts immediately, delivery of the entire message fails. (e) Because no queueing is allowed, all failures are treated as permanent; there is no distinction between 4xx and 5xx SMTP response codes from the smarthost. Furthermore, because only a single yes/no response can be given to the caller, it is not possible to deliver to some recipients and not others. If there is an error (temporary or permanent) for any recipient, all are failed. (f) If more than one host is listed, Exim will try another host after a connection failure or a timeout, in the normal way. However, if this kind of failure happens for all the hosts, the delivery fails. (g) When delivery fails, an error message is written to the standard error stream (as well as to Exim's log), and Exim exits to the caller with a return code value 1. The message is expunged from Exim's spool files. No bounce messages are ever generated. (h) No retry data is maintained, and any retry rules are ignored. (i) A number of Exim options are overridden: deliver_drop_privilege is forced true, max_rcpt in the smtp transport is forced to "unlimited", remote_max_parallel is forced to one, and fallback hosts are ignored. The overall effect is that Exim makes a single synchronous attempt to deliver the message, failing if there is any kind of problem. Because no local deliveries are done and no daemon can be run, Exim does not need root privilege. It should be possible to run it setuid=exim instead of setuid=root. See section 48.3 in the 4.40 manual for a general discussion about the advantages and disadvantages of running without root privilege. 2. There have been problems with DNS servers when SRV records are looked up. Some mis-behaving servers return a DNS error or timeout when a non-existent SRV record is sought. Similar problems have in the past been reported for MX records. The global dns_again_means_nonexist option can help with this problem, but it is heavy-handed because it is a global option. There are now two new options for the dnslookup router. They are called srv_fail_domains and mx_fail_domains. In each case, the value is a domain list. If an attempt to look up an SRV or MX record results in a DNS failure or "try again" response, and the domain matches the relevant list, Exim behaves as if the DNS had responded "no such record". In the case of an SRV lookup, this means that the router proceeds to look for MX records; in the case of an MX lookup, it proceeds to look for A or AAAA records, unless the domain matches mx_domains. 3. The following functions are now available in the local_scan() API: (a) void header_remove(int occurrence, uschar *name) This function removes header lines. If "occurrence" is zero or negative, all occurrences of the header are removed. If occurrence is greater than zero, that particular instance of the header is removed. If no header(s) can be found that match the specification, the function does nothing. (b) BOOL header_testname(header_line *hdr, uschar *name, int length, BOOL notdel) This function tests whether the given header has the given name. It is not just a string comparison, because whitespace is permitted between the name and the colon. If the "notdel" argument is TRUE, a FALSE return is forced for all "deleted" headers; otherwise they are not treated specially. For example: if (header_testname(h, US"X-Spam", 6, TRUE)) ... (c) void header_add_at_position(BOOL after, uschar *name, BOOL topnot, int type, char *format, ...) This function adds a new header line at a specified point in the header chain. If "name" is NULL, the new header is added at the end of the chain if "after" is TRUE, or at the start if "after" is FALSE. If "name" is not NULL, the headers are searched for the first non-deleted header that matches the name. If one is found, the new header is added before it if "after" is FALSE. If "after" is true, the new header is added after the found header and any adjacent subsequent ones with the same name (even if marked "deleted"). If no matching non-deleted header is found, the "topnot" option controls where the header is added. If it is TRUE, addition is at the top; otherwise at the bottom. Thus, to add a header after all the Received: headers, or at the top if there are no Received: headers, you could use header_add_at_position(TRUE, US"Received", TRUE, ' ', "X-xxx: ..."); Normally, there is always at least one non-deleted Received: header, but there may not be if received_header_text expands to an empty string. (d) BOOL receive_remove_recipient(uschar *recipient) This is a convenience function to remove a named recipient from the list of recipients. It returns TRUE if a recipient was removed, and FALSE if no matching recipient could be found. The argument must be a complete email address. 4. When an ACL "warn" statement adds one or more header lines to a message, they are added at the end of the existing header lines by default. It is now possible to specify that any particular header line should be added right at the start (before all the Received: lines) or immediately after the first block of Received: lines in the message. This is done by specifying :at_start: or :after_received: (or, for completeness, :at_end:) before the text of the header line. (Header text cannot start with a colon, as there has to be a header name first.) For example: warn message = :after_received:X-My-Header: something or other... If more than one header is supplied in a single warn statement, each one is treated independently and can therefore be placed differently. If you add more than one line at the start, or after the Received: block, they will end up in reverse order. Warning: This facility currently applies only to header lines that are added in an ACL. It does NOT work for header lines that are added in a system filter or in a router or transport. 5. There is now a new error code that can be used in retry rules. Its name is "rcpt_4xx", and there are three forms. A literal "rcpt_4xx" matches any 4xx error received for an outgoing SMTP RCPT command; alternatively, either the first or both of the x's can be given as digits, for example: "rcpt_45x" or "rcpt_436". If you want (say) to recognize 452 errors given to RCPT commands by a particular host, and have only a one-hour retry for them, you can set up a retry rule of this form: the.host.name rcpt_452 F,1h,10m Naturally, this rule must come before any others that would match. These new errors apply to both outgoing SMTP (the smtp transport) and outgoing LMTP (either the lmtp transport, or the smtp transport in LMTP mode). Note, however, that they apply only to responses to RCPT commands. 6. The "postmaster" option of the callout feature of address verification has been extended to make it possible to use a non-empty MAIL FROM address when checking a postmaster address. The new suboption is called "postmaster_ mailfrom", and you use it like this: require verify = sender/callout=postmaster_mailfrom=abc@x.y.z Providing this suboption causes the postmaster check to be done using the given address. The original "postmaster" option is equivalent to require verify = sender/callout=postmaster_mailfrom= If both suboptions are present, the rightmost one overrides. Important notes: (1) If you use a non-empty sender address for postmaster checking, there is the likelihood that the remote host will itself initiate a callout check back to your host to check that address. As this is a "normal" callout check, the sender will most probably be empty, thus avoiding possible callout loops. However, to be on the safe side it would be best to set up your own ACLs so that they do not do sender verification checks when the recipient is the address you use for postmaster callout checking. (2) The caching arrangements for postmaster checking do NOT take account of the sender address. It is assumed that either the empty address, or a fixed non-empty address will be used. All that Exim remembers is that the postmaster check for the domain succeeded or failed. 7. When verifying addresses in header lines using the verify=header_sender option, Exim behaves by default as if the addresses are envelope sender addresses from a message. Callout verification therefore tests to see whether a bounce message could be delivered, by using an empty address in the MAIL FROM command. However, it is arguable that these addresses might never be used as envelope senders, and could therefore justifiably reject bounce messages (empty senders). There is now an additional callout option for verify=header_sender that allows you to specify what address to use in the MAIL FROM command. You use it as in this example: require verify = header_sender/callout=mailfrom=abcd@x.y.z Important notes: (1) As in the case of postmaster_mailfrom (see above), you should think about possible loops. (2) In this case, as in the case of recipient callouts with non-empty senders (the use_sender option), caching is done on the basis of a recipient/sender pair. 8. If you build Exim with USE_READLINE=yes in Local/Makefile, it will try to load libreadline dynamically whenever the -be (test expansion) option is used without command line arguments. If successful, it will then use readline() for reading the test data. A line history is supported. By the time Exim does this, it is running as the calling user, so this should not cause any security problems. Security is the reason why this is NOT supported for -bt or -bv, when Exim is running as root or exim, respectively. Note that this option adds to the size of the Exim binary, because the dynamic loading library is not otherwise included. On my desktop it adds about 2.5K. You may need to add -ldl to EXTRA_LIBS when you set USE_READLINE=yes. 9. Added ${str2b64:} to the expansion operators. This operator converts an arbitrary string into one that is base64 encoded. 10. A new authenticator, called cyrus_sasl, has been added. This requires the presence of the Cyrus SASL library; it authenticates by calling this library, which supports a number of authentication mechanisms, including PLAIN and LOGIN, but also several others that Exim does not support directly. The code for this authenticator was provided by Matthew Byng-Maddick of A L Digital Ltd (http://www.aldigital.co.uk). Here follows draft documentation: xx. THE CYRUS_SASL AUTHENTICATOR The cyrus_sasl authenticator provides server support for the Cyrus library Implementation of the RFC 2222 "Simple Authentication and Security Layer". It provides a gatewaying mechanism directly to the Cyrus interface, so if your Cyrus library can do, for example, CRAM-MD5, then so can the cyrus_sasl authenticator. By default it uses the public name of the driver to determine which mechanism to support. Where access to some kind of secret file is required, for example in GSSAPI or CRAM-MD5, it is worth noting that the authenticator runs as the exim user, and that the Cyrus SASL library has no way of escalating privileges by default. You may also find you need to set environment variables, depending on the driver you are using. xx.1 Using cyrus_sasl as a server The cyrus_sasl authenticator has four private options. It puts the username (on a successful authentication) into $1. server_hostname Type: string* Default: $primary_hostname This option selects the hostname that is used when communicating with the library. It is up to the underlying SASL plug-in what it does with this data. server_mech Type: string Default: public_name This option selects the authentication mechanism this driver should use. It allows you to use a different underlying mechanism from the advertised name. For example: sasl: driver = cyrus_sasl public_name = X-ANYTHING server_mech = CRAM-MD5 server_set_id = $1 server_realm Type: string Default: unset This is the SASL realm that the server is claiming to be in. server_service Type: string Default: "smtp" This is the SASL service that the server claims to implement. For straigthforward cases, you do not need to set any of the authenticator's private options. All you need to do is to specify an appropriate mechanism as the public name. Thus, if you have a SASL library that supports CRAM-MD5 and PLAIN, you might have two authenticators as follows: sasl_cram_md5: driver = cyrus_sasl public_name = CRAM-MD5 server_set_id = $1 sasl_plain: driver = cyrus_sasl public_name = PLAIN server_set_id = $1 11. There is a new global option called tls_on_connect_ports. Its value must be a list of port numbers; the most common use is expected to be tls_on_connect_ports = 465 Setting this option has the same effect as -tls-on-connect on the command line, but only for the specified ports. It applies to all connections, both via the daemon and via inetd. You still need to specify all the ports for the daemon (using daemon_smtp_ports or local_interfaces or the -X command line option) because this option does not add an extra port -- rather, it specifies different behaviour on a port that is defined elsewhere. The -tls-on-connect command line option overrides tls_on_connect_ports, and forces tls-on-connect for all ports. 12. There is a new ACL that is run when a DATA command is received, before the data itself is received. The ACL is defined by acl_smtp_predata. (Compare acl_smtp_data, which is run after the data has been received.) This new ACL allows a negative response to be given to the DATA command itself. Header lines added by MAIL or RCPT ACLs are not visible at this time, but any that are defined here are visible when the acl_smtp_data ACL is run. 13. The "control=submission" ACL modifier has an option "/domain=xxx" which specifies the domain to be used when creating From: or Sender: lines using the authenticated id as a local part. If the option is supplied with an empty domain, that is, just "/domain=", Exim assumes that the authenticated id is a complete email address, and it uses it as is when creating From: or Sender: lines. 14. It is now possible to make retry rules that apply only when the failing message has a specific sender. In particular, this can be used to define retry rules that apply only to bounce messages. The syntax is to add a new third item to a retry rule, of the form "senders=
". The retry timings themselves then become the fourth item. For example: * * senders=: F,1h,30m would match all bounce messages. If the address list contains white space, it must be enclosed in quotes. For example: a.domain timeout senders="x@b.dom : y@c.dom" G,8h,10m,1.5 When testing retry rules using -brt, you can supply a sender using the -f command line option, like this: exim -f "" -brt user@dom.ain If you do not set -f with -brt, a retry rule that contains a senders list will never be matched. 15. Two new control modifiers have been added to ACLs: "control = enforce_sync" and "control = no_enforce_sync". This makes it possible to be selective about when SMTP synchronization is enforced. The global option smtp_enforce_sync now specifies the default state of the switch. These controls can appear in any ACL, but the most obvious place to put them is in the ACL defined by acl_smtp_connect, which is run at the start of an incoming SMTP connection, before the first synchronization check. 16. Another two new control modifiers are "control = caseful_local_part" and "control = caselower_local_part". These are permitted only in the ACL specified by acl_smtp_rcpt (i.e. during RCPT processing). By default, the contents of $local_part are lower cased before ACL processing. After "control = caseful_local_part", any uppercase letters in the original local part are restored in $local_part for the rest of the ACL, or until "control = caselower_local_part" is encountered. However, this applies only to local part handling that takes place directly in the ACL (for example, as a key in lookups). If a "verify = recipient" test is obeyed, the case-related handling of the local part during the verification is controlled by the router configuration (see the caseful_local_part generic router option). This facility could be used, for example, to add a spam score to local parts containing upper case letters. For example, using $acl_m4 to accumulate the spam score: warn control = caseful_local_part set acl_m4 = ${eval:\ $acl_m4 + \ ${if match{$local_part}{[A-Z]}{1}{0}}\ } control = caselower_local_part Notice that we put back the lower cased version afterwards, assuming that is what is wanted for subsequent tests. 17. The option hosts_connection_nolog is provided so that certain hosts can be excepted from logging when the +smtp_connection log selector is set. For example, you might want not to log SMTP connections from local processes, or from 127.0.0.1, or from your local LAN. The option is a host list with an unset default. Because it is consulted in the main loop of the daemon, you should strive to restrict its value to a short inline list of IP addresses and networks. To disable logging SMTP connections from local processes, you must create a host list with an empty item. For example: hosts_connection_nolog = : If the +smtp_connection log selector is not set, this option has no effect. 18. There is now an acl called acl_smtp_quit, which is run for the QUIT command. The outcome of the ACL does not affect the response code to QUIT, which is always 221. Thus, the ACL does not in fact control any access. For this reason, the only verbs that are permitted are "accept" and "warn". The ACL can be used for tasks such as custom logging at the end of an SMTP session. For example, you can use ACL variables in other ACLs to count messages, recipients, etc., and log the totals at QUIT time using one or more "logwrite" modifiers on a "warn" command. You do not need to have a final "accept", but if you do, you can use a "message" modifier to specify custom text that is sent as part of the 221 response. This ACL is run only for a "normal" QUIT. For certain kinds of disastrous failure (for example, failure to open a log file, or when Exim is bombing out because it has detected an unrecoverable error), all SMTP commands from the client are given temporary error responses until QUIT is received or the connection is closed. In these special cases, the ACL is not run. 19. The appendfile transport has two new options, mailbox_size and mailbox_ filecount. If either these options are set, it is expanded, and the result is taken as the current size of the mailbox or the number of files in the mailbox, respectively. This makes it possible to use some external means of maintaining the data about the size of a mailbox for enforcing quota limits. The result of expanding these option values must be a decimal number, optionally followed by "K" or "M". 20. It seems that there are broken clients in use that cannot handle multiline SMTP responses. Can't people who implement these braindead programs read? RFC 821 mentions multiline responses, and it is over 20 years old. They must handle multiline responses for EHLO, or do they still use HELO? Anyway, here is YAWFAB (yet another workaround for asinine brokenness). There's a new ACL switch that can be set by control = no_multiline_responses If this is set, it suppresses multiline SMTP responses from ACL rejections. One way of doing this would have been just to put out these responses as one long line. However, RFC 2821 specifies a maximum of 512 bytes per response ("use multiline responses for more" it says), and some of the responses might get close to that. So I have implemented this by doing two very easy things: (1) Extra information that is normally output as part of a rejection caused by sender verification failure is omitted. Only the final line (typically "sender verification failed") is now sent. (2) If a "message" modifier supplies a multiline response, only the first line is output. The setting of the switch can, of course, be made conditional on the calling host. 21. There is now support for the libradius library that comes with FreeBSD. This is an alternative to the radiusclient library that Exim already supports. To use the FreeBSD library, you need to set RADIUS_LIB_TYPE=RADLIB in Local/Makefile, in addition to RADIUS_CONFIGURE_FILE, and you probably also need -libradius in EXTRALIBS. Version 4.42 ------------ 1. The "personal" filter test is brought up-to-date with recommendations from the Sieve specification: (a) The list of non-personal From: addresses now includes "listserv", "majordomo", and "*-request"; (b) If the message contains any header line starting with "List=-" it is treated as non-personal. 2. The Sieve functionality has been extended to support the "copy" and "vacation" extensions, and comparison tests. 3. There is now an overall timeout for performing a callout verification. It defaults to 4 times the callout timeout, which applies to individual SMTP commands during the callout. The overall timeout applies when there is more than one host that can be tried. The timeout is checked before trying the next host. This prevents very long delays if there are a large number of hosts and all are timing out (e.g. when the network connections are timing out). The value of the overall timeout can be changed by specifying an additional sub-option for "callout", called "maxwait". For example: verify = sender/callout=5s,maxwait=20s 4. Changes to the "personal" filter test: (1) The list of non-personal local parts in From: addresses has been extended to include "listserv", "majordomo", "*-request", and "owner-*", taken from the Sieve specification recommendations. (2) If the message contains any header line starting with "List-" it is treated as non-personal. (3) The test for "circular" in the Subject: header line has been removed because it now seems ill-conceived. 5. The autoreply transport has a new option called never_mail. This is an address list. If any run of the transport creates a message with a recipient that matches any item in the list, that recipient is quietly discarded. If all recipients are discarded, no message is created. Version 4.40 ------------ The documentation is up-to-date for the 4.40 release. What follows here is a brief list of the new features that have been added since 4.30. 1. log_incoming_interface affects more log lines. 2. New ACL modifier "control = submission". 3. CONFIGURE_OWNER can be set at build time to define an alternative owner for the configuration file, in addition to root and exim. 4. Added expansion variables $body_zerocount, $recipient_data, and $sender_data. 5. The time of last modification of the "new" subdirectory is now used as the "mailbox time last read" when there is a quota error for a maildir delivery. 6. The special item "+ignore_unknown" may now appear in host lists. 7. The special domain-matching patterns @mx_any, @mx_primary, and @mx_secondary can now be followed by "/ignore=". 8. New expansion conditions: match_domain, match_address, match_local_part, lt, lti, le, lei, gt, gti, ge, and new expansion operators time_interval, eval10, and base62d. 9. New lookup type called "iplsearch". 10. New log selectors ident_timeout, tls_certificate_verified, queue_time, deliver_time, outgoing_port, return_path_on_delivery. 11. New global options smtp_active_hostname and tls_require_ciphers. 12. Exinext has -C and -D options. 13. "domainlist_cache" forces caching of an apparently variable list. 14. For compatibility with Sendmail, the command line option -prval:sval is equivalent to -oMr rval -oMs sval. 15. New callout options use_sender and use_postmaster for use when verifying recipients. 16. John Jetmore's "exipick" utility has been added to the distribution. 17. The TLS code now supports CRLs. 18. The dnslookup router and the dnsdb lookup type now support the use of SRV records. 19. The redirect router has a new option called qualify_domain. 20. exigrep's output now also includes lines that are not related to any particular message, but which do match the pattern. 21. New global option write_rejectlog. If it is set false, Exim no longer writes anything to the reject log. ****