Michael Haardt's patch to update Sieve to RFC3028bis.
[exim.git] / doc / doc-txt / README.SIEVE
CommitLineData
5ea81592 1$Cambridge: exim/doc/doc-txt/README.SIEVE,v 1.5 2005/06/17 10:47:05 ph10 Exp $
495ae4b0
PH
2
3 Notes on the Sieve implementation for Exim
4
5Exim Filter Versus Sieve Filter
6
7Exim supports two incompatible filters: The traditional Exim filter and
8the Sieve filter. Since Sieve is a extensible language, it is important
9to understand "Sieve" in this context as "the specific implementation
10of Sieve for Exim".
11
12The Exim filter contains more features, such as variable expansion, and
13better integration with the host environment, like external processes
14and pipes.
15
16Sieve is a standard for interoperable filters, defined in RFC 3028,
17with multiple implementations around. If interoperability is important,
18then there is no way around it.
19
20
21Exim Implementation
22
5ea81592
PH
23The Exim Sieve implementation offers the core as defined by RFC 3028bis,
24the "envelope" (RFC 3028), the "fileinto" (RFC 3028), the "copy" (RFC
253894) and the "vacation" (draft-ietf-sieve-vacation-02.txt) extension,
495ae4b0
PH
26the "i;ascii-numeric" comparator, but not the "reject" extension.
27Exim does not support MDMs, so adding it just to the sieve filter makes
28little sense.
29
30The Sieve filter is integrated in Exim and works very similar to the
31Exim filter: Sieve scripts are recognized by the first line containing
32"# sieve filter". When using "keep" or "fileinto" to save a mail into a
33folder, the resulting string is available as the variable $address_file
34in the transport that stores it. A suitable transport could be:
35
36localuser:
37 driver = appendfile
38 file = ${if eq{$address_file}{inbox} \
39 {/var/mail/$local_part} \
40 {${if eq{${substr_0_1:$address_file}}{/} \
41 {$address_file} \
42 {$home/$address_file} \
43 }} \
44 }
45 delivery_date_add
46 envelope_to_add
47 return_path_add
48 mode = 0600
49
50Absolute files are stored where specified, relative files are stored
51relative to $home and "inbox" goes to the standard mailbox location.
52
53To enable "vacation", set sieve_vacation_directory for the router to
54the directory where vacation databases are held (don't put anything
55else in that directory) and point reply_transport to an autoreply
56transport.
57
58
59RFC Compliance
60
61Exim requires the first line to be "# sieve filter". Of course the RFC
62does not enforce that line. Don't expect examples to work without adding
63it, though.
64
65RFC 3028 requires using CRLF to terminate the end of a line.
66The rationale was that CRLF is universally used in network protocols
67to mark the end of the line. This implementation does not embed Sieve
68in a network protocol, but uses Sieve scripts as part of the Exim MTA.
69Since all parts of Exim use \n as newline character, this implementation
70does, too. You can change this by defining the macro RFC_EOL at compile
71time to enforce CRLF being used.
72
73Exim violates RFC 2822, section 3.6.8, by accepting 8-bit header names, so
74this implementation repeats this violation to stay consistent with Exim.
75This is in preparation to UTF-8 data.
76
77Sieve scripts can not contain NUL characters in strings, but mail
78headers could contain MIME encoded NUL characters, which could never
79be matched by Sieve scripts using exact comparisons. For that reason,
80this implementation extends the Sieve quoted string syntax with \0
81to describe a NUL character, violating \0 being the same as 0 in
82RFC 3028. Even without using \0, the following tests are all true in
83this implementation. Implementations that use C-style strings will only
84evaulate the first test as true.
85
86Subject: =?iso-8859-1?q?abc=00def
87
88header :contains "Subject" ["abc"]
89header :contains "Subject" ["def"]
90header :matches "Subject" ["abc?def"]
91
92Note that by considering Sieve to be a MUA, RFC 2047 can be interpreted
93in a way that NUL characters truncating strings is allowed for Sieve
94implementations, although not recommended. It is further allowed to use
95encoded NUL characters in headers, but that's not recommended either.
96The above example shows why. Good code should still be able to deal
97with it.
98
99RFC 3028 states that if an implementation fails to convert a character
100set to UTF-8, two strings can not be equal if one contains octects greater
101than 127. Assuming that all unknown character sets are one-byte character
102sets with the lower 128 octects being US-ASCII is not sound, so this
103implementation violates RFC 3028 and treats such MIME words literally.
104That way at least something could be matched.
105
106The folder specified by "fileinto" must not contain the character
107sequence ".." to avoid security problems. RFC 3028 does not specifiy the
108syntax of folders apart from keep being equivalent to fileinto "INBOX".
109This implementation uses "inbox" instead.
110
111Sieve script errors currently cause that messages are silently filed into
112"inbox". RFC 3028 requires that the user is notified of that condition.
113This may be implemented in future by adding a header line to mails that
114are filed into "inbox" due to an error in the filter.
115
116
d1d97a76 117Strings Containing Header Names Or Envelope Elements
495ae4b0
PH
118
119RFC 3028 does not specify what happens if a string denoting a header
d1d97a76
PH
120field or envelope element does not contain a valid name, e.g. it
121contains a colon for a header or it is not "from" or "to" for envelopes.
495ae4b0 122This implementation generates an error instead of ignoring the header
d1d97a76
PH
123field in order to ease script debugging, which fits in the common picture
124of Sieve.
495ae4b0
PH
125
126
127Header Test With Invalid MIME Encoding In Header
128
129Some MUAs process invalid base64 encoded data, generating junk.
130Others ignore junk after seeing an equal sign in base64 encoded data.
131RFC 2047 does not specify how to react in this case, other than stating
132that a client must not forbid to process a message for that reason.
133RFC 2045 specifies that invalid data should be ignored (appearantly
134looking at end of line characters). It also specifies that invalid data
135may lead to rejecting messages containing them (and there it appears to
136talk about true encoding violations), which is a clear contradiction to
137ignoring them.
138
139RFC 3028 does not specify how to process incorrect MIME words.
140This implementation treats them literally, as it does if the word is
141correct, but its character set can not be converted to UTF-8.
142
143
495ae4b0
PH
144Semantics Of Keep
145
146The keep command is equivalent to fileinto "inbox": It saves the
147message and resets the implicit keep flag. It does not set the
148implicit keep flag; there is no command to set it once it has
149been reset.
150
151
152Semantics of Fileinto
153
154RFC 3028 does not specify if "fileinto" tries to create a mail folder,
155in case it does not exist. This implementation allows to configure
156that aspect using the appendfile transport options "create_directory",
157"create_file" and "file_must_exist". See the appendfile transport in
158the Exim specification for details.
159
160
161Semantics of Redirect
162
163Sieve scripts are supposed to be interoperable between servers, so this
164implementation does not allow redirecting mail to unqualified addresses,
165because the domain would depend on the used system and on systems with
166virtual mail domains it is probably not what the user expects it to be.
167
168
169String Arguments
170
171There has been confusion if the string arguments to "require" are to be
5ea81592
PH
172matched case-sensitive or not. The comparator default is case-insensitive
173comparison, but "require" does not allow to specify a comparator, so
174this default does not apply. Lacking a clear specification, matching
175the strings exactly makes most sense. The same is valid for comparator
176names, also specified as strings.
495ae4b0
PH
177
178
179Sieve Syntax and Semantics
180
181RFC 3028 confuses syntax and semantics sometimes. It uses a generic
182grammar as syntax for actions and tests and performs many checks during
183semantic analysis. Syntax is specified as grammar rule, semantics
184with natural language, despire the latter often talking about syntax.
185The intention was to provide a framework for the syntax that describes
186current commands as well as future extensions, and describing commands
187by semantics. Since the semantic analysis is not specified by formal
188rules, it is easy to get that phase wrong, as demonstrated by the mistake
189in RFC 3028 to forbid "elsif" being followed by "elsif" (which is allowed
190in Sieve, it's just not specified correctly).
191
192RFC 3028 does not define if semantic checks are strict (always treat
193unknown extensions as errors) or lazy (treat unknown extensions as error,
194if they are executed), and since it employs a very generic grammar,
195it is not unreasonable for an implementation using a parser for the
196generic grammar to indeed process scripts that contain unknown commands
197in dead code. It is just required to treat disabled but known extensions
198the same as unknown extensions.
199
200The following suggestion for section 8.2 gives two grammars, one for
201the framework, and one for specific commands, thus removing most of the
202semantic analysis. Since the parser can not parse unsupported extensions,
203the result is strict error checking. As required in section 2.10.5, known
204but not enabled extensions must behave the same as unknown extensions,
205so those also result strictly in errors (though at the thin semantic
206layer), even if they can be parsed fine.
207
2088.2. Grammar
209
210The atoms of the grammar are lexical tokens. White space or comments may
211appear anywhere between lexical tokens, they are not part of the grammar.
212The grammar is specified in ABNF with two extensions to describe tagged
213arguments that can be reordered and grammar extensions: { } denotes a
214sequence of symbols that may appear in any order. Example:
215
216 start = { a b c }
217
218is equivalent to:
219
220 start = ( a b c ) / ( a c b ) / ( b a c ) / ( b c a ) / ( c a b ) / ( c b a )
221
222The symbol =) is used to append to a rule:
223
224 start = a
225 start =) b
226
227is equivalent to
228
229 start = a b
230
231All Sieve commands, including extensions, MUST be words of the following
232generic grammar with the start symbol "start". They SHOULD be specified
233using a specific grammar, though.
234
235 argument = string-list / number / tag
236 arguments = *argument [test / test-list]
237 block = "{" commands "}"
238 commands = *command
239 string = quoted-string / multi-line
240 string-list = "[" string *("," string) "]" / string
241 test = identifier arguments
242 test-list = "(" test *("," test) ")"
243 command = identifier arguments ( ";" / block )
244 start = command
245
246The basic Sieve commands are specified using the following grammar, which
247language is a subset of the generic grammar above. The start symbol is
248"start".
249
250 address-part = ":localpart" / ":domain" / ":all"
251 comparator = ":comparator" string
252 match-type = ":is" / ":contains" / ":matches"
253 string = quoted-string / multi-line
254 string-list = "[" string *("," string) "]" / string
255 address-test = "address" { [address-part] [comparator] [match-type] }
256 string-list string-list
257 test-list = "(" test *("," test) ")"
258 allof-test = "allof" test-list
259 anyof-test = "anyof" test-list
260 exists-test = "exists" string-list
261 false-test = "false"
262 true=test = "true"
263 header-test = "header" { [comparator] [match-type] }
264 string-list string-list
265 not-test = "not" test
266 relop = ":over" / ":under"
267 size-test = "size" relop number
268 block = "{" commands "}"
269 if-command = "if" test block *( "elsif" test block ) [ "else" block ]
270 stop-command = "stop" { stop-options } ";"
271 stop-options =
272 keep-command = "keep" { keep-options } ";"
273 keep-options =
274 discard-command = "discard" { discard-options } ";"
275 discard-options =
276 redirect-command = "redirect" { redirect-options } string ";"
277 redirect-options =
278 require-command = "require" { require-options } string-list ";"
279 require-options =
280 test = address-test / allof-test / anyof-test / exists-test
281 / false-test / true-test / header-test / not-test
282 / size-test
283 command = if-command / stop-command / keep-command
284 / discard-command / redirect-command
285 commands = *command
286 start = *require-command commands
287
288The extensions "envelope" and "fileinto" are specified using the following
289grammar extension.
290
291 envelope-test = "envelope" { [comparator] [address-part] [match-type] }
292 string-list string-list
293 test =/ envelope-test
294
295 fileinto-command = "fileinto" { fileinto-options } string ";"
296 fileinto-options =
297 command =/ fileinto-command
298
299The extension "copy" is specified as:
300
301 fileinto-options =) ":copy"
302 redirect-options =) ":copy"
303
304
305The i;ascii-numeric Comparator
306
307RFC 2244 describes this comparator and specifies that non-numeric strings
308are considered equal with an ordinal value higher than any numeric string.
309Although not stated explicitly, this includes the empty string. A range
310of at least 2^31 is required. This implementation does not limit the
311range, because it does not convert numbers to binary representation
312before comparing them.
313
314
315The vacation extension
316
317The extension "vacation" is specified using the following grammar
318extension.
319
320 vacation-command = "vacation" { vacation-options } <reason: string>
321 vacation-options = [":days" number]
495ae4b0 322 [":subject" string]
f656d135
PH
323 [":from" string]
324 [":addresses" string-list]
495ae4b0 325 [":mime"]
f656d135 326 [":handle" string]
495ae4b0
PH
327 command =/ vacation-command
328
329
330Semantics Of ":mime"
331
f656d135
PH
332The draft does not specify how strings using MIME entities are used
333to compose messages. As a result, different implementations generate
334different mails. The Exim Sieve implementation splits the reason into
335header and body. It adds the header to the mail header and uses the body
336as mail body. Be aware, that other imlementations compose a multipart
337structure with the reason as only part. Both conform to the specification
338(or lack thereof).
495ae4b0
PH
339
340
341Semantics Of Not Using ":mime"
342
343Sieve scripts are written in UTF-8, so is the reason string in this
344case. This implementation adds MIME headers to indicate that. This
345is not required by the vacation draft, which does not specify how
346the UTF-8 reason is processed to compose the resulting message.
347
348
495ae4b0
PH
349Default Subject
350
5ea81592
PH
351The draft specifies that the default message subject is "Auto: " plus
352the old subject. Using this subject is dangerous, because many mailing
353lists verify addresses by sending a secret key in the subject of a
354message, asking to reply to the message for confirmation. Using the
355default vacation subject confirms any subscription request of this kind,
356allowing to subscribe a third party to any mailing list, either to annoy
357the user or to declare spam as legitimate mail by proving to use opt-in.
495ae4b0
PH
358
359
360Rate Limiting Responses
361
f656d135
PH
362In absence of a handle, this implementation hashes the reason,
363":subject" option, ":mime" option and ":from" option and uses the hex
364string representation as filename within the "sieve_vacation_directory"
365to store the recipient addresses for this vacation parameter set.
495ae4b0
PH
366
367The draft specifies that sites may define a minimum ":days" value than 1.
368This implementation uses 1. The maximum value MUST greater than 7,
369and SHOULD be greater than 30. This implementation uses a maximum of 31.
370
371Vacation recipient address databases older than 31 days are automatically
372removed. Users do not have to remove them manually when modifying their
373scripts. Don't put anything but vacation databases in that directory
374or you risk that it will be removed, too!
375
376
377Global Reply Address Blacklist
378
379The draft requires that each implementation offers a global black list
380of addresses that will never be replied to. Exim offers this as option
381"never_mail" in the autoreply transport.