| 1 | $Cambridge: exim/doc/doc-txt/NewStuff,v 1.46 2005/05/25 09:58:16 fanf2 Exp $ |
| 2 | |
| 3 | New Features in Exim |
| 4 | -------------------- |
| 5 | |
| 6 | This file contains descriptions of new features that have been added to Exim, |
| 7 | but have not yet made it into the main manual (which is most conveniently |
| 8 | updated when there is a relatively large batch of changes). The doc/ChangeLog |
| 9 | file contains a listing of all changes, including bug fixes. |
| 10 | |
| 11 | |
| 12 | Exim version 4.52 |
| 13 | ----------------- |
| 14 | |
| 15 | TF/01 Support for checking Client SMTP Authorization has been added. CSA is a |
| 16 | system which allows a site to advertise which machines are and are not |
| 17 | permitted to send email. This is done by placing special SRV records in |
| 18 | the DNS, which are looked up using the client's HELO domain. At this |
| 19 | time CSA is still an Internet-Draft. |
| 20 | |
| 21 | Client SMTP Authorization checks are performed by the ACL condition |
| 22 | verify=csa. This will fail if the client is not authorized. If there is |
| 23 | a DNS problem, or if no valid CSA SRV record is found, or if the client |
| 24 | is authorized, the condition succeeds. These three cases can be |
| 25 | distinguished using the expansion variable $csa_status, which can take |
| 26 | one of the values "fail", "defer", "unknown", or "ok". The condition |
| 27 | does not itself defer because that would be likely to cause problems |
| 28 | for legitimate email. |
| 29 | |
| 30 | The error messages produced by the CSA code include slightly more |
| 31 | detail. If $csa_status is "defer" this may be because of problems |
| 32 | looking up the CSA SRV record, or problems looking up the CSA target |
| 33 | address record. There are four reasons for $csa_status being "fail": |
| 34 | the client's host name is explicitly not authorized; the client's IP |
| 35 | address does not match any of the CSA target IP addresses; the client's |
| 36 | host name is authorized but it has no valid target IP addresses (e.g. |
| 37 | the target's addresses are IPv6 and the client is using IPv4); or the |
| 38 | client's host name has no CSA SRV record but a parent domain has |
| 39 | asserted that all subdomains must be explicitly authorized. |
| 40 | |
| 41 | The verify=csa condition can take an argument which is the domain to |
| 42 | use for the DNS query. The default is verify=csa/$sender_helo_name. |
| 43 | |
| 44 | This implementation includes an extension to CSA. If the query domain |
| 45 | is an address literal such as [192.0.2.95], or if it is a bare IP |
| 46 | address, Exim will search for CSA SRV records in the reverse DNS as if |
| 47 | the HELO domain was e.g. 95.2.0.192.in-addr.arpa. Therefore it is |
| 48 | meaningful to say, for example, verify=csa/$sender_host_address - in |
| 49 | fact, this is the check that Exim performs if the client does not say |
| 50 | HELO. This extension can be turned off by setting the main |
| 51 | configuration option dns_csa_use_reverse = false. |
| 52 | |
| 53 | If a CSA SRV record is not found for the domain itself, then a search |
| 54 | is performed through its parent domains for a record which might be |
| 55 | making assertions about subdomains. The maximum depth of this search is |
| 56 | limited using the main configuration option dns_csa_search_limit, which |
| 57 | takes the value 5 by default. Exim does not look for CSA SRV records in |
| 58 | a top level domain, so the default settings handle HELO domains as long |
| 59 | as seven (hostname.five.four.three.two.one.com) which encompasses the |
| 60 | vast majority of legitimate HELO domains. |
| 61 | |
| 62 | The dnsdb lookup also has support for CSA. Although dnsdb already |
| 63 | supports SRV lookups, this is not sufficient because of the extra |
| 64 | parent domain search behaviour of CSA, and (as with PTR lookups) |
| 65 | dnsdb also turns IP addresses into lookups in the reverse DNS space. |
| 66 | The result of ${lookup dnsdb {csa=$sender_helo_name} } has two |
| 67 | space-separated fields: an authorization code and a target host name. |
| 68 | The authorization code can be "Y" for yes, "N" for no, "X" for explicit |
| 69 | authorization required but absent, or "?" for unknown. |
| 70 | |
| 71 | PH/01 The amount of output produced by the "make" process has been reduced, |
| 72 | because the compile lines are often rather long, making it all pretty |
| 73 | unreadable. The new style is along the lines of the 2.6 Linux kernel: |
| 74 | just a short line for each module that is being compiled or linked. |
| 75 | However, it is still possible to get the full output, by calling "make" |
| 76 | like this: |
| 77 | |
| 78 | FULLECHO='' make -e |
| 79 | |
| 80 | The value of FULLECHO defaults to "@", the flag character that suppresses |
| 81 | command reflection in "make". When you ask for the full output, it is |
| 82 | given in addition to the the short output. |
| 83 | |
| 84 | TF/02 There have been two changes concerned with submission mode: |
| 85 | |
| 86 | Until now submission mode always left the return path alone, whereas |
| 87 | locally-submitted messages from untrusted users have the return path |
| 88 | fixed to the user's email address. Submission mode now fixes the return |
| 89 | path to the same address as is used to create the Sender: header. If |
| 90 | /sender_retain is specified then both the Sender: header and the return |
| 91 | path are left alone. |
| 92 | |
| 93 | Note that the changes caused by submission mode take effect after the |
| 94 | predata ACL. This means that any sender checks performed before the |
| 95 | fix-ups will use the untrusted sender address specified by the user, not |
| 96 | the trusted sender address specified by submission mode. Although this |
| 97 | might be slightly unexpected, it does mean that you can configure ACL |
| 98 | checks to spot that a user is trying to spoof another's address, for |
| 99 | example. |
| 100 | |
| 101 | There is also a new /name= option for submission mode which allows you |
| 102 | to specify the user's full name to be included in the Sender: header. |
| 103 | For example: |
| 104 | |
| 105 | accept authenticated = * |
| 106 | control = submission/name=${lookup {$authenticated_id} \ |
| 107 | lsearch {/etc/exim/namelist} } |
| 108 | |
| 109 | The namelist file contains entries like |
| 110 | |
| 111 | fanf: Tony Finch |
| 112 | |
| 113 | And the resulting Sender: header looks like |
| 114 | |
| 115 | Sender: Tony Finch <fanf@exim.org> |
| 116 | |
| 117 | TF/03 The control = fakereject ACL modifier now has a fakedefer counterpart, |
| 118 | which works in exactly the same way except it causes a fake SMTP 450 |
| 119 | response after the message data instead of a fake SMTP 550 response. |
| 120 | You must take care when using fakedefer because it will cause messages |
| 121 | to be duplicated when the sender retries. Therefore you should not use |
| 122 | fakedefer if the message will be delivered normally. |
| 123 | |
| 124 | TF/04 There is a new ratelimit ACL condition which can be used to measure |
| 125 | and control the rate at which clients can send email. This is more |
| 126 | powerful than the existing smtp_ratelimit_* options, because those |
| 127 | options only control the rate of commands in a single SMTP session, |
| 128 | whereas the new ratelimit condition works across all connections |
| 129 | (concurrent and sequential) to the same host. |
| 130 | |
| 131 | The syntax of the ratelimit condition is: |
| 132 | |
| 133 | ratelimit = <m> / <p> / <options> / <key> |
| 134 | |
| 135 | If the average client sending rate is less than m messages per time |
| 136 | period p then the condition is false, otherwise it is true. |
| 137 | |
| 138 | The parameter p is the smoothing time constant, in the form of an Exim |
| 139 | time interval e.g. 8h for eight hours. A larger time constant means it |
| 140 | takes Exim longer to forget a client's past behaviour. The parameter m is |
| 141 | the maximum number of messages that a client can send in a fast burst. By |
| 142 | increasing both m and p but keeping m/p constant, you can allow a client |
| 143 | to send more messages in a burst without changing its overall sending |
| 144 | rate limit. Conversely, if m and p are both small then messages must be |
| 145 | sent at an even rate. |
| 146 | |
| 147 | The key is used to look up the data used to calcluate the client's |
| 148 | average sending rate. This data is stored in a database maintained by |
| 149 | Exim in its spool directory alongside the retry database etc. For |
| 150 | example, you can limit the sending rate of each authenticated user, |
| 151 | independent of the computer they are sending from, by setting the key |
| 152 | to $authenticated_id. The default key is $sender_host_address. |
| 153 | |
| 154 | Each ratelimit condition can have up to two options. The first option |
| 155 | specifies what Exim measures the rate of, and the second specifies how |
| 156 | Exim handles excessively fast clients. |
| 157 | |
| 158 | The per_mail option means that it measures the client's rate of sending |
| 159 | messages. This is the default if none of the per_* options is specified. |
| 160 | |
| 161 | The per_conn option means that it measures the client's connection rate. |
| 162 | |
| 163 | The per_byte option limits the sender's email bandwidth. Note that it |
| 164 | is best to use this option in the DATA ACL; if it is used in an earlier |
| 165 | ACL it relies on the SIZE parameter on the MAIL command, which may be |
| 166 | inaccurate or completely missing. You can follow the limit m in the |
| 167 | configuration with K, M, or G to specify limits in kilobytes, |
| 168 | megabytes, or gigabytes respectively. |
| 169 | |
| 170 | The per_cmd option means that Exim recomputes the rate every time the |
| 171 | condition is processed, which can be used to limit the SMTP command rate. |
| 172 | The alias per_rcpt is provided for use in the RCPT ACL instead of per_cmd |
| 173 | to make it clear that the effect is to limit the rate at which recipients |
| 174 | are accepted. Note that in this case the rate limiting engine will see a |
| 175 | message with many recipients as a large high-speed burst. |
| 176 | |
| 177 | If a client's average rate is greater than the maximum, the rate |
| 178 | limiting engine can react in two possible ways, depending on the |
| 179 | presence of the strict or leaky options. This is independent of the |
| 180 | other counter-measures (e.g. rejecting the message) that may be |
| 181 | specified by the rest of the ACL. The default mode is leaky, which |
| 182 | avoids a sender's over-aggressive retry rate preventing it from getting |
| 183 | any email through. |
| 184 | |
| 185 | The strict option means that the client's recorded rate is always |
| 186 | updated. The effect of this is that Exim measures the client's average |
| 187 | rate of attempts to send email, which can be much higher than the |
| 188 | maximum. If the client is over the limit it will be subjected to |
| 189 | counter-measures until it slows down below the maximum rate. |
| 190 | |
| 191 | The leaky option means that the client's recorded rate is not updated |
| 192 | if it is above the limit. The effect of this is that Exim measures the |
| 193 | client's average rate of successfully sent email, which cannot be |
| 194 | greater than the maximum. If the client is over the limit it will |
| 195 | suffer some counter-measures, but it will still be able to send email |
| 196 | at the configured maximum rate, whatever the rate of its attempts. |
| 197 | |
| 198 | As a side-effect, the ratelimit condition will set the expansion |
| 199 | variables $sender_rate containing the client's computed rate, |
| 200 | $sender_rate_limit containing the configured value of m, and |
| 201 | $sender_rate_period containing the configured value of p. |
| 202 | |
| 203 | Exim's other ACL facilities are used to define what counter-measures |
| 204 | are taken when the rate limit is exceeded. This might be anything from |
| 205 | logging a warning (e.g. while measuring existing sending rates in order |
| 206 | to define our policy), through time delays to slow down fast senders, |
| 207 | up to rejecting the message. For example, |
| 208 | |
| 209 | # Log all senders' rates |
| 210 | warn |
| 211 | ratelimit = 0 / 1h / strict |
| 212 | log_message = \ |
| 213 | Sender rate $sender_rate > $sender_rate_limit / $sender_rate_period |
| 214 | |
| 215 | # Slow down fast senders |
| 216 | warn |
| 217 | ratelimit = 100 / 1h / per_rcpt / strict |
| 218 | delay = ${eval: 10 * ($sender_rate - $sender_rate_limit) } |
| 219 | |
| 220 | # Keep authenticated users under control |
| 221 | deny |
| 222 | ratelimit = 100 / 1d / strict / $authenticated_id |
| 223 | |
| 224 | # System-wide rate limit |
| 225 | defer |
| 226 | message = Sorry, too busy. Try again later. |
| 227 | ratelimit = 10 / 1s / $primary_hostname |
| 228 | |
| 229 | # Restrict incoming rate from each host, with a default rate limit |
| 230 | # set using a macro and special cases looked up in a table. |
| 231 | defer |
| 232 | message = Sender rate $sender_rate exceeds \ |
| 233 | $sender_rate_limit messages per $sender_rate_period |
| 234 | ratelimit = ${lookup {$sender_host_address} \ |
| 235 | cdb {DB/ratelimits.cdb} \ |
| 236 | {$value} {RATELIMIT} } |
| 237 | |
| 238 | |
| 239 | Version 4.51 |
| 240 | ------------ |
| 241 | |
| 242 | PH/01 The format in which GnuTLS parameters are written to the gnutls-param |
| 243 | file in the spool directory has been changed. This change has been made |
| 244 | to alleviate problems that some people had with the generation of the |
| 245 | parameters by Exim when /dev/random was exhausted. In this situation, |
| 246 | Exim would hang until /dev/random acquired some more entropy. |
| 247 | |
| 248 | The new code exports and imports the DH and RSA parameters in PEM |
| 249 | format. This means that the parameters can be generated externally using |
| 250 | the certtool command that is part of GnuTLS. |
| 251 | |
| 252 | To replace the parameters with new ones, instead of deleting the file |
| 253 | and letting Exim re-create it, you can generate new parameters using |
| 254 | certtool and, when this has been done, replace Exim's cache file by |
| 255 | renaming. The relevant commands are something like this: |
| 256 | |
| 257 | # rm -f new.params |
| 258 | # touch new.params |
| 259 | # chown exim:exim new.params |
| 260 | # chmod 0400 new.params |
| 261 | # certtool --generate-privkey --bits 512 >new.params |
| 262 | # echo "" >>new.params |
| 263 | # certtool --generate-dh-params --bits 1024 >> new.params |
| 264 | # mv new.params params |
| 265 | |
| 266 | If Exim never has to generate the parameters itself, the possibility of |
| 267 | stalling is removed. |
| 268 | |
| 269 | PH/02 A new expansion item for dynamically loading and calling a locally- |
| 270 | written C function is now provided, if Exim is compiled with |
| 271 | |
| 272 | EXPAND_DLFUNC=yes |
| 273 | |
| 274 | set in Local/Makefile. The facility is not included by default (a |
| 275 | suitable error is given if you try to use it when it is not there.) |
| 276 | |
| 277 | If you enable EXPAND_DLFUNC, you should also be aware of the new redirect |
| 278 | router option forbid_filter_dlfunc. If you have unprivileged users on |
| 279 | your system who are permitted to create filter files, you might want to |
| 280 | set forbid_filter_dlfunc=true in the appropriate router, to stop them |
| 281 | using ${dlfunc to run code within Exim. |
| 282 | |
| 283 | You load and call an external function like this: |
| 284 | |
| 285 | ${dlfunc{/some/file}{function}{arg1}{arg2}...} |
| 286 | |
| 287 | Once loaded, Exim remembers the dynamically loaded object so that it |
| 288 | doesn't reload the same object file in the same Exim process (but of |
| 289 | course Exim does start new processes frequently). |
| 290 | |
| 291 | There may be from zero to eight arguments to the function. When compiling |
| 292 | a local function that is to be called in this way, local_scan.h should be |
| 293 | included. The Exim variables and functions that are defined by that API |
| 294 | are also available for dynamically loaded functions. The function itself |
| 295 | must have the following type: |
| 296 | |
| 297 | int dlfunction(uschar **yield, int argc, uschar *argv[]) |
| 298 | |
| 299 | Where "uschar" is a typedef for "unsigned char" in local_scan.h. The |
| 300 | function should return one of the following values: |
| 301 | |
| 302 | OK Success. The string that is placed in "yield" is put into |
| 303 | the expanded string that is being built. |
| 304 | |
| 305 | FAIL A non-forced expansion failure occurs, with the error |
| 306 | message taken from "yield", if it is set. |
| 307 | |
| 308 | FAIL_FORCED A forced expansion failure occurs, with the error message |
| 309 | taken from "yield" if it is set. |
| 310 | |
| 311 | ERROR Same as FAIL, except that a panic log entry is written. |
| 312 | |
| 313 | When compiling a function that is to be used in this way with gcc, |
| 314 | you need to add -shared to the gcc command. Also, in the Exim build-time |
| 315 | configuration, you must add -export-dynamic to EXTRALIBS. |
| 316 | |
| 317 | TF/01 $received_time is a new expansion variable containing the time and date |
| 318 | as a number of seconds since the start of the Unix epoch when the |
| 319 | current message was received. |
| 320 | |
| 321 | PH/03 There is a new value for RADIUS_LIB_TYPE that can be set in |
| 322 | Local/Makefile. It is RADIUSCLIENTNEW, and it requests that the new API, |
| 323 | in use from radiusclient 0.4.0 onwards, be used. It does not appear to be |
| 324 | possible to detect the different versions automatically. |
| 325 | |
| 326 | PH/04 There is a new option called acl_not_smtp_mime that allows you to scan |
| 327 | MIME parts in non-SMTP messages. It operates in exactly the same way as |
| 328 | acl_smtp_mime |
| 329 | |
| 330 | PH/05 It is now possible to redefine a macro within the configuration file. |
| 331 | The macro must have been previously defined within the configuration (or |
| 332 | an included file). A definition on the command line using the -D option |
| 333 | causes all definitions and redefinitions within the file to be ignored. |
| 334 | In other words, -D overrides any values that are set in the file. |
| 335 | Redefinition is specified by using '==' instead of '='. For example: |
| 336 | |
| 337 | MAC1 = initial value |
| 338 | ... |
| 339 | MAC1 == updated value |
| 340 | |
| 341 | Redefinition does not alter the order in which the macros are applied to |
| 342 | the subsequent lines of the configuration file. It is still the same |
| 343 | order in which the macros were originally defined. All that changes is |
| 344 | the macro's value. Redefinition makes it possible to accumulate values. |
| 345 | For example: |
| 346 | |
| 347 | MAC1 = initial value |
| 348 | ... |
| 349 | MAC1 == MAC1 and something added |
| 350 | |
| 351 | This can be helpful in situations where the configuration file is built |
| 352 | from a number of other files. |
| 353 | |
| 354 | PH/06 Macros may now be defined or redefined between router, transport, |
| 355 | authenticator, or ACL definitions, as well as in the main part of the |
| 356 | configuration. They may not, however, be changed within an individual |
| 357 | driver or ACL, or in the local_scan, retry, or rewrite sections of the |
| 358 | configuration. |
| 359 | |
| 360 | PH/07 $acl_verify_message is now set immediately after the failure of a |
| 361 | verification in an ACL, and so is available in subsequent modifiers. In |
| 362 | particular, the message can be preserved by coding like this: |
| 363 | |
| 364 | warn !verify = sender |
| 365 | set acl_m0 = $acl_verify_message |
| 366 | |
| 367 | Previously, $acl_verify_message was set only while expanding "message" |
| 368 | and "log_message" when a very denied access. |
| 369 | |
| 370 | PH/08 The redirect router has two new options, sieve_useraddress and |
| 371 | sieve_subaddress. These are passed to a Sieve filter to specify the :user |
| 372 | and :subaddress parts of an address. Both options are unset by default. |
| 373 | However, when a Sieve filter is run, if sieve_useraddress is unset, the |
| 374 | entire original local part (including any prefix or suffix) is used for |
| 375 | :user. An unset subaddress is treated as an empty subaddress. |
| 376 | |
| 377 | PH/09 Quota values can be followed by G as well as K and M. |
| 378 | |
| 379 | PH/10 $message_linecount is a new variable that contains the total number of |
| 380 | lines in the header and body of the message. Compare $body_linecount, |
| 381 | which is the count for the body only. During the DATA and |
| 382 | content-scanning ACLs, $message_linecount contains the number of lines |
| 383 | received. Before delivery happens (that is, before filters, routers, and |
| 384 | transports run) the count is increased to include the Received: header |
| 385 | line that Exim standardly adds, and also any other header lines that are |
| 386 | added by ACLs. The blank line that separates the message header from the |
| 387 | body is not counted. Here is an example of the use of this variable in a |
| 388 | DATA ACL: |
| 389 | |
| 390 | deny message = Too many lines in message header |
| 391 | condition = \ |
| 392 | ${if <{250}{${eval: $message_linecount - $body_linecount}}} |
| 393 | |
| 394 | In the MAIL and RCPT ACLs, the value is zero because at that stage the |
| 395 | message has not yet been received. |
| 396 | |
| 397 | PH/11 In a ${run expansion, the variable $value (which contains the standard |
| 398 | output) is now also usable in the "else" string. |
| 399 | |
| 400 | PH/12 In a pipe transport, although a timeout while waiting for the pipe |
| 401 | process to complete was treated as a delivery failure, a timeout while |
| 402 | writing the message to the pipe was logged, but erroneously treated as a |
| 403 | successful delivery. Such timeouts include transport filter timeouts. For |
| 404 | consistency with the overall process timeout, these timeouts are now |
| 405 | treated as errors, giving rise to delivery failures by default. However, |
| 406 | there is now a new Boolean option for the pipe transport called |
| 407 | timeout_defer, which, if set TRUE, converts the failures into defers for |
| 408 | both kinds of timeout. A transport filter timeout is now identified in |
| 409 | the log output. |
| 410 | |
| 411 | |
| 412 | Version 4.50 |
| 413 | ------------ |
| 414 | |
| 415 | The documentation is up-to-date for the 4.50 release. |
| 416 | |
| 417 | **** |