Add log selector sender_verify_fail.

[exim.git] / doc / doc-txt / README.SIEVE

$Cambridge: exim/doc/doc-txt/README.SIEVE,v 1.9 2005/11/21 10:09:13 ph10 Exp $

              Notes on the Sieve implementation for Exim

Exim Filter Versus Sieve Filter

Exim supports two incompatible filters: The traditional Exim filter and
the Sieve filter. Since Sieve is a extensible language, it is important
to understand "Sieve" in this context as "the specific implementation
of Sieve for Exim".

The Exim filter contains more features, such as variable expansion, and
better integration with the host environment, like external processes
and pipes.

Sieve is a standard for interoperable filters, defined in RFC 3028,
with multiple implementations around. If interoperability is important,
then there is no way around it.


Exim Implementation

The Exim Sieve implementation offers the core as defined by draft
3028bis-4 (next version of RFC 3028 that fixes specification mistakes),
the "envelope" (3028bis), the "fileinto" (3028bis), the "copy" (RFC 3894)
and the "vacation" (draft-ietf-sieve-vacation-04.txt) extension, the
"i;ascii-numeric" comparator (RFC 2244).

The Sieve filter is integrated in Exim and works very similar to the
Exim filter: Sieve scripts are recognized by the first line containing
"# sieve filter".  When using "keep" or "fileinto" to save a mail into a
folder, the resulting string is available as the variable $address_file
in the transport that stores it.  The following routers and transport
show a typical use of Sieve:

begin routers

localuser_verify:
  driver = accept
  domains = +localdomains
  local_part_suffix = "-*"
  local_part_suffix_optional
  check_local_user
  require_files = $home/.forward
  verify_only = true

localuser_deliver:
  driver = redirect
  domains = +localdomains
  local_part_suffix = "-*"
  local_part_suffix_optional
  sieve_subaddress = "${sg{$local_part_suffix}{^-}{}}"
  sieve_useraddress = "$local_part"
  check_local_user
  require_files = $home/.forward
  file = $home/.forward
  check_ancestor
  allow_filter
  file_transport = localuser
  reply_transport = vacation
  sieve_vacation_directory = $home/mail/vacation
  verify = false

begin transports

localuser:
  driver = appendfile
  file = ${if eq{$address_file}{inbox} \
              {/var/mail/$local_part} \
              {${if eq{${substr_0_1:$address_file}}{/} \
                    {$address_file} \
                    {$home/mail/$address_file} \
              }} \
         }
  delivery_date_add
  envelope_to_add
  return_path_add
  mode = 0600

vacation:
  driver = autoreply

Absolute files are stored where specified, relative files are stored
relative to $home/mail and "inbox" goes to the standard mailbox location.
To enable "vacation", sieve_vacation_directory is set to the directory
where vacation databases are held (don't put anything else in that
directory) and point reply_transport to an autoreply transport.
Setting the Sieve useraddress and subaddress allows to use the subaddress
extension.


RFC Compliance

Exim requires the first line to be "# sieve filter".  Of course the RFC
does not enforce that line.  Don't expect examples to work without adding
it, though.

RFC 3028 requires using CRLF to terminate the end of a line.
The rationale was that CRLF is universally used in network protocols
to mark the end of the line.  This implementation does not embed Sieve
in a network protocol, but uses Sieve scripts as part of the Exim MTA.
Since all parts of Exim use \n as newline character, this implementation
does, too.  You can change this by defining the macro RFC_EOL at compile
time to enforce CRLF being used.

Sieve scripts can not contain NUL characters in strings, but mail
headers could contain MIME encoded NUL characters, which could never
be matched by Sieve scripts using exact comparisons.  For that reason,
this implementation extends the Sieve quoted string syntax with \0
to describe a NUL character, violating \0 being the same as 0 in
RFC 3028.

The folder specified by "fileinto" must not contain the character
sequence ".." to avoid security problems.  RFC 3028 does not specify the
syntax of folders apart from keep being equivalent to fileinto "INBOX".
This implementation uses "inbox" instead.

Sieve script errors currently cause that messages are silently filed into
"inbox".  RFC 3028 requires that the user is notified of that condition.
This may be implemented in future by adding a header line to mails that
are filed into "inbox" due to an error in the filter.

The automatic replies generated by "vacation" do not contain an updated
"references" header field.


Semantics Of Keep

The keep command is equivalent to fileinto "inbox": It saves the
message and resets the implicit keep flag.  It does not set the
implicit keep flag; there is no command to set it once it has
been reset.


Semantics Of Fileinto

RFC 3028 does not specify if "fileinto" tries to create a mail folder,
in case it does not exist.  This implementation allows to configure
that aspect using the appendfile transport options "create_directory",
"create_file" and "file_must_exist".  See the appendfile transport in
the Exim specification for details.


Allof And Anyof Test

RFC 3028 does not specify if these tests use shortcut/lazy evaluation.
Exim uses shortcut evaluation.


Action Reordering

RFC 3028 does not specify if actions may be executed out of order.
Exim may execute them out of order, e.g. messages may be filed to
folders or forwarded in a different order than specified, because
those actions only setup delivery, but do not execute it themselves.


Wildcard Matching

RFC 3028 is not exactly clear if comparators act on unicode characters
or on octets containing their UTF-8 representation.  As it turns out,
many implementations go the second way.  This does not make a difference
but for wildcard matching and octet-wise comparison.  Working on unicode
means a dot matches a character.  Working on UTF-8 means the dot matches
a single octet of a multi-octet sequence.  For octet-wise comparisons,
working on UTF-8 means arbitrary byte sequences in headers can not be
matches, as they are rarely correct UTF-8 sequences and can thus not be
expressed as string literal.  This implementation works on unicode, but
this may be changed in case RFC3028bis specifies this issue safe and sound.


Sieve Syntax And Semantics

RFC 3028 confuses syntax and semantics sometimes.  It uses a generic
grammar as syntax for commands and tests and performs many checks during
semantic analysis.  Syntax is specified by grammar rules, semantics
by natural language, despite the latter often talking about syntax.
The intention was to provide a framework for the syntax that describes
current commands as well as future extensions, and describing commands
by semantics.

The following replacement for section 8.2 gives two grammars, one for
the framework, and one for specific commands, thus removing most of the
semantic analysis.  Since the parser can not parse unsupported extensions,
the result is strict error checking of any executed and not executed code
until "stop" is executed or the end of the script is reached.

8.2. Grammar

The atoms of the grammar are lexical tokens.  White space or comments may
appear anywhere between lexical tokens, they are not part of the grammar.
The grammar is specified in ABNF with two extensions to describe tagged
arguments that can be reordered and grammar extensions: { } denotes a
sequence of symbols that may appear in any order.  Example:

  options = a b c
  start   = { options }

is equivalent to:

  start   =  ( a b c ) / ( a c b ) / ( b a c ) / ( b c a ) / ( c a b ) / ( c b a )

The symbol =) is used to append to a rule:

  start =  a
  start =) b

is equivalent to

  start =  a b

All Sieve commands, including extensions, MUST be words of the following
generic grammar with the start symbol "start".  They SHOULD be specified
using a specific grammar, though.

   argument        = string-list / number / tag
   arguments       = *argument [test / test-list]
   block           = "{" commands "}"
   commands        = *command
   string          = quoted-string / multi-line
   string-list     = "[" string *("," string) "]" / string
   test            = identifier arguments
   test-list       = "(" test *("," test) ")"
   command         = identifier arguments ( ";" / block )
   start           = command

The basic Sieve commands are specified using the following grammar, which
language is a subset of the generic grammar above.  The start symbol is
"start".

  address-part     =  ":localpart" / ":domain" / ":all"
  comparator       =  ":comparator" string
  match-type       =  ":is" / ":contains" / ":matches"
  string           =  quoted-string / multi-line
  string-list      =  "[" string *("," string) "]" / string
  address-test     =  "address" { [address-part] [comparator] [match-type] }
                      string-list string-list
  test-list        =  "(" test *("," test) ")"
  allof-test       =  "allof" test-list
  anyof-test       =  "anyof" test-list
  exists-test      =  "exists" string-list
  false-test       =  "false"
  true=test        =  "true"
  header-test      =  "header" { [comparator] [match-type] }
                      string-list string-list
  not-test         =  "not" test
  relop            =  ":over" / ":under"
  size-test        =  "size" relop number
  block            =  "{" commands "}"
  if-command       =  "if" test block *( "elsif" test block ) [ "else" block ]
  stop-command     =  "stop" { stop-options } ";"
  stop-options     =
  keep-command     =  "keep" { keep-options } ";"
  keep-options     =
  discard-command  =  "discard" { discard-options } ";"
  discard-options  =
  redirect-command =  "redirect" { redirect-options } string ";"
  redirect-options =
  require-command  =  "require" { require-options } string-list ";"
  require-options  =
  test             =  address-test / allof-test / anyof-test / exists-test
                      / false-test / true-test / header-test / not-test
                      / size-test
  command          =  if-command / stop-command / keep-command
                      / discard-command / redirect-command
  commands         =  *command
  start            =  *require-command commands

The extensions "envelope" and "fileinto" are specified using the following
grammar extension.

  envelope-test    =  "envelope" { [comparator] [address-part] [match-type] }
                      string-list string-list
  test             =/ envelope-test

  fileinto-command =  "fileinto" { fileinto-options } string ";"
  fileinto-options =
  command          =/ fileinto-command

The extension "copy" is specified as:

  fileinto-options =) ":copy"
  redirect-options =) ":copy"


The i;ascii-numeric Comparator

RFC 2244 describes this comparator and specifies that non-numeric strings
are considered equal with an ordinal value higher than any numeric string.
Although not stated explicitly, this includes the empty string.  A range
of at least 2^31 is required.  This implementation does not limit the
range, because it does not convert numbers to binary representation
before comparing them.


The vacation extension

The extension "vacation" is specified using the following grammar
extension.

  vacation-command =  "vacation" { vacation-options } <reason: string>
  vacation-options =  [":days" number]
                      [":subject" string]
                      [":from" string]
                      [":addresses" string-list]
                      [":mime"]
                      [":handle" string]
  command          =/ vacation-command


Semantics Of ":mime"

The draft does not specify how strings using MIME entities are used
to compose messages.  As a result, different implementations generate
different mails.  The Exim Sieve implementation splits the reason into
header and body.  It adds the header to the mail header and uses the body
as mail body.  Be aware, that other imlementations compose a multipart
structure with the reason as only part.  Both conform to the specification
(or lack thereof).


Semantics Of Not Using ":mime"

Sieve scripts are written in UTF-8, so is the reason string in this
case.  This implementation adds MIME headers to indicate that.  This
is not required by the vacation draft, which does not specify how
the UTF-8 reason is processed to compose the resulting message.


Default Subject

The draft specifies that the default message subject is "Auto: " plus
the old subject.  Using this subject is dangerous, because many mailing
lists verify addresses by sending a secret key in the subject of a
message, asking to reply to the message for confirmation.  Using the
default vacation subject confirms any subscription request of this kind,
allowing to subscribe a third party to any mailing list, either to annoy
the user or to declare spam as legitimate mail by proving to use opt-in.


Rate Limiting Responses

In absence of a handle, this implementation hashes the reason,
":subject" option, ":mime" option and ":from" option and uses the hex
string representation as filename within the "sieve_vacation_directory"
to store the recipient addresses for this vacation parameter set.

The draft specifies that sites may define a minimum ":days" value than 1.
This implementation uses 1.  The maximum value MUST greater than 7,
and SHOULD be greater than 30.  This implementation uses a maximum of 31.

Vacation recipient address databases older than 31 days are automatically
removed.  Users do not have to remove them manually when modifying their
scripts.  Don't put anything but vacation databases in that directory
or you risk that it will be removed, too!


Global Reply Address Blacklist

The draft requires that each implementation offers a global black list
of addresses that will never be replied to.  Exim offers this as option
"never_mail" in the autoreply transport.