Notes on the Sieve implementation for Exim Exim Filter Versus Sieve Filter Exim supports two incompatible filters: The traditional Exim filter and the Sieve filter. Since Sieve is a extensible language, it is important to understand "Sieve" in this context as "the specific implementation of Sieve for Exim". The Exim filter contains more features, such as variable expansion, and better integration with the host environment, like external processes and pipes. Sieve is a standard for interoperable filters, defined in RFC 5228, with multiple implementations around. If interoperability is important, then there is no way around it. Exim Implementation The Exim Sieve implementation offers the core as defined by RFC 5228, the "encoded-character" extension (RFC 5228), the "envelope" test (RFC 5228), the "fileinto" action (5228), the "copy" parameter (RFC 3894), the "vacation" action (5230), the "notify" action (draft-ietf-sieve-notify-12) with mailto URIs (draft-ietf-sieve-notify-mailto-05), the "i;ascii-numeric" comparator (RFC 2244) and the subaddress parameter (RFC 5233). The Sieve filter is integrated in Exim and works very similar to the Exim filter: Sieve scripts are recognized by the first line containing "# sieve filter". When using "keep" or "fileinto" to save a mail into a folder, the resulting string is available as the variable $address_file in the transport that stores it. The following routers and transport show a typical use of Sieve: begin routers localuser_verify: driver = accept domains = +localdomains local_part_suffix = "-*" local_part_suffix_optional check_local_user require_files = $home/.forward verify_only = true localuser_deliver: driver = redirect domains = +localdomains local_part_suffix = "-*" local_part_suffix_optional sieve_subaddress = "${sg{$local_part_suffix}{^-}{}}" sieve_useraddress = "$local_part" check_local_user require_files = $home/.forward file = $home/.forward check_ancestor allow_filter file_transport = localuser reply_transport = vacation sieve_vacation_directory = $home/mail/vacation verify = false begin transports localuser: driver = appendfile file = ${if eq{$address_file}{inbox} \ {/var/mail/$local_part} \ {${if eq{${substr_0_1:$address_file}}{/} \ {$address_file} \ {$home/mail/$address_file} \ }} \ } delivery_date_add envelope_to_add return_path_add mode = 0600 vacation: driver = autoreply Absolute files are stored where specified, relative files are stored relative to $home/mail and "inbox" goes to the standard mailbox location. To enable "vacation", sieve_vacation_directory is set to the directory where vacation databases are held (don't put anything else in that directory) and point reply_transport to an autoreply transport. Setting the Sieve useraddress and subaddress allows to use the subaddress extension. RFC Compliance Exim requires the first line to be "# sieve filter". Of course the RFC does not enforce that line. Don't expect examples to work without adding it, though. RFC 5228 requires using CRLF to terminate the end of a line. The rationale was that CRLF is universally used in network protocols to mark the end of the line. This implementation does not embed Sieve in a network protocol, but uses Sieve scripts as part of the Exim MTA. Since all parts of Exim use \n as newline character, this implementation does, too. You can change this by defining the macro RFC_EOL at compile time to enforce CRLF being used. The folder specified by "fileinto" must not contain the character sequence ".." to avoid security problems. RFC 5228 does not specify the syntax of folders apart from keep being equivalent to fileinto "INBOX". This implementation uses "inbox" instead. Sieve script errors currently cause that messages are silently filed into "inbox". RFC 5228 requires that the user is notified of that condition. This may be implemented in future by adding a header line to mails that are filed into "inbox" due to an error in the filter. The automatic replies generated by "vacation" do not contain an updated "references" header field. Semantics Of Keep The keep command is equivalent to fileinto "inbox": It saves the message and resets the implicit keep flag. It does not set the implicit keep flag; there is no command to set it once it has been reset. Semantics Of Fileinto RFC 5228 does not specify if "fileinto" tries to create a mail folder, in case it does not exist. This implementation allows to configure that aspect using the appendfile transport options "create_directory", "create_file" and "file_must_exist". See the appendfile transport in the Exim specification for details. Allof And Anyof Test RFC 5228 does not specify if these tests use shortcut/lazy evaluation. Exim uses shortcut evaluation. Action Reordering RFC 5228 does not specify if actions may be executed out of order. Exim may execute them out of order, e.g. messages may be filed to folders or forwarded in a different order than specified, because those actions only setup delivery, but do not execute it themselves. Sieve Syntax And Semantics RFC 5228 uses a generic grammar as syntax for commands and tests and performs many checks during semantic analysis. Syntax is specified by grammar rules, semantics by natural language. The intention is to provide a framework for the syntax that describes current commands as well as future extensions, and describing commands by semantics. The following replacement for section 8.2 gives a grammar for specific commands of this implementation, thus removing most of the semantic analysis. Since the parser can not parse unsupported extensions, the result is strict error checking of any executed and not executed code until "stop" is executed or the end of the script is reached. 8.2. Grammar The grammar is specified in ABNF with two extensions to describe tagged arguments that can be reordered and grammar extensions: { } denotes a sequence of symbols that may appear in any order. Example: options = a b c start = { options } is equivalent to: start = ( a b c ) / ( a c b ) / ( b a c ) / ( b c a ) / ( c a b ) / ( c b a ) The symbol =) is used to append to a rule: start = a start =) b is equivalent to start = a b The basic Sieve commands are specified using the following grammar, which language is a subset of the generic grammar above. The start symbol is "start". address-part = ":localpart" / ":domain" / ":all" comparator = ":comparator" string match-type = ":is" / ":contains" / ":matches" string = quoted-string / multi-line string-list = "[" string *("," string) "]" / string address-test = "address" { [address-part] [comparator] [match-type] } string-list string-list test-list = "(" test *("," test) ")" allof-test = "allof" test-list anyof-test = "anyof" test-list exists-test = "exists" string-list false-test = "false" true=test = "true" header-test = "header" { [comparator] [match-type] } string-list string-list not-test = "not" test relop = ":over" / ":under" size-test = "size" relop number block = "{" commands "}" if-command = "if" test block *( "elsif" test block ) [ "else" block ] stop-command = "stop" { stop-options } ";" stop-options = keep-command = "keep" { keep-options } ";" keep-options = discard-command = "discard" { discard-options } ";" discard-options = redirect-command = "redirect" { redirect-options } string ";" redirect-options = require-command = "require" { require-options } string-list ";" require-options = test = address-test / allof-test / anyof-test / exists-test / false-test / true-test / header-test / not-test / size-test command = if-command / stop-command / keep-command / discard-command / redirect-command commands = *command start = *require-command commands The extensions "envelope" and "fileinto" are specified using the following grammar extension. envelope-test = "envelope" { [comparator] [address-part] [match-type] } string-list string-list test =/ envelope-test fileinto-command = "fileinto" { fileinto-options } string ";" fileinto-options = command =/ fileinto-command The extension "copy" is specified as: fileinto-options =) ":copy" redirect-options =) ":copy" The i;ascii-numeric Comparator RFC 2244 describes this comparator and specifies that non-numeric strings are considered equal with an ordinal value higher than any numeric string. Although not stated explicitly, this includes the empty string. A range of at least 2^31 is required. This implementation does not limit the range, because it does not convert numbers to binary representation before comparing them. The vacation extension The extension "vacation" is specified using the following grammar extension. vacation-command = "vacation" { vacation-options } vacation-options = [":days" number] [":subject" string] [":from" string] [":addresses" string-list] [":mime"] [":handle" string] command =/ vacation-command Semantics Of ":mime" The draft does not specify how strings using MIME entities are used to compose messages. As a result, different implementations generate different mails. The Exim Sieve implementation splits the reason into header and body. It adds the header to the mail header and uses the body as mail body. Be aware, that other implementations compose a multipart structure with the reason as only part. Both conform to the specification (or lack thereof). Semantics Of Not Using ":mime" Sieve scripts are written in UTF-8, so is the reason string in this case. This implementation adds MIME headers to indicate that. This is not required by the vacation draft, which does not specify how the UTF-8 reason is processed to compose the resulting message. Default Subject RFC 5230 specifies that the default message subject is "Auto: " plus the old subject. Using this subject is dangerous, because many mailing lists verify addresses by sending a secret key in the subject of a message, asking to reply to the message for confirmation. Using the default vacation subject confirms any subscription request of this kind, allowing to subscribe a third party to any mailing list, either to annoy the user or to declare spam as legitimate mail by proving to use opt-in. Rate Limiting Responses In absence of a handle, this implementation hashes the reason, ":subject" option, ":mime" option and ":from" option and uses the hex string representation as filename within the "sieve_vacation_directory" to store the recipient addresses for this vacation parameter set. The draft specifies that sites may define a minimum ":days" value than 1. This implementation uses 1. The maximum value MUST greater than 7, and SHOULD be greater than 30. This implementation uses a maximum of 31. Vacation recipient address databases older than 31 days are automatically removed. Users do not have to remove them manually when modifying their scripts. Don't put anything but vacation databases in that directory or you risk that it will be removed, too! Global Reply Address Blacklist The draft requires that each implementation offers a global black list of addresses that will never be replied to. Exim offers this as option "never_mail" in the autoreply transport. The enotify extension The extension "enotify" is specified using the following grammar extension. notify-command = "notify" { notify-options } notify-options = [":from" string] [":importance" <"1" / "2" / "3">] [":options" 1*(string-list / number)] [":message" string] command =/ notify-command valid_notify_method = "valid_notify_method" test =/ valid_notify_method Only the mailto URI scheme is implemented.