From 76146973f89f0e9265d85827285b9258910a56d7 Mon Sep 17 00:00:00 2001 From: Jeremy Harris Date: Sun, 30 Mar 2014 21:48:32 +0100 Subject: [PATCH] More care with headers add/remove lists. Bug 1452 As a side-effect, playing games with newlines no longer gives an altered message body/ Testcase 0324 is questionable (though passing) --- doc/doc-docbook/spec.xfpt | 54 ++++++++------ doc/doc-txt/ChangeLog | 7 ++ src/src/functions.h | 1 + src/src/readconf.c | 24 ++++-- src/src/routers/rf_get_munge_headers.c | 99 ++++++++++++------------- src/src/string.c | 44 +++++++++++ src/src/transport.c | 76 ++++++++++--------- test/confs/0096 | 3 +- test/confs/0097 | 5 +- test/confs/0166 | 6 ++ test/mail/0046.userx | Bin 3497 -> 3495 bytes test/mail/0166.userx | 4 + test/mail/0166.usery | 4 + test/mail/0351.userx | 1 - test/mail/0412.CALLER | 4 - test/scripts/0000-Basic/0481 | 2 +- test/stderr/5400 | 2 +- 17 files changed, 205 insertions(+), 131 deletions(-) diff --git a/doc/doc-docbook/spec.xfpt b/doc/doc-docbook/spec.xfpt index c00469b0d..f7636f85a 100644 --- a/doc/doc-docbook/spec.xfpt +++ b/doc/doc-docbook/spec.xfpt @@ -16747,11 +16747,12 @@ and the discussion in chapter &<>&. -.option headers_add routers string&!! unset +.option headers_add routers list&!! unset .cindex "header lines" "adding" .cindex "router" "adding header lines" -This option specifies a string of text that is expanded at routing time, and -associated with any addresses that are accepted by the router. However, this +This option specifies a list of text headers, newline-separated, +that is associated with any addresses that are accepted by the router. +Each item is separately expanded, at routing time. However, this option has no effect when an address is just being verified. The way in which the text is used to add header lines at transport time is described in section &<>&. New header lines are not actually added until the @@ -16760,8 +16761,8 @@ header lines in string expansions in the transport's configuration do not &"see"& the added header lines. The &%headers_add%& option is expanded after &%errors_to%&, but before -&%headers_remove%& and &%transport%&. If the expanded string is empty, or if -the expansion is forced to fail, the option has no effect. Other expansion +&%headers_remove%& and &%transport%&. If an item is empty, or if +an item expansion is forced to fail, the item has no effect. Other expansion failures are treated as configuration errors. Unlike most options, &%headers_add%& can be specified multiple times @@ -16783,11 +16784,12 @@ avoided. The &%repeat_use%& option of the &%redirect%& router may be of help. -.option headers_remove routers string&!! unset +.option headers_remove routers list&!! unset .cindex "header lines" "removing" .cindex "router" "removing header lines" -This option specifies a string of text that is expanded at routing time, and -associated with any addresses that are accepted by the router. However, this +This option specifies a list of text headers, colon-separated, +that is associated with any addresses that are accepted by the router. +Each item is separately expanded, at routing time. However, this option has no effect when an address is just being verified. The way in which the text is used to remove header lines at transport time is described in section &<>&. Header lines are not actually removed until @@ -16796,8 +16798,8 @@ to header lines in string expansions in the transport's configuration still &"see"& the original header lines. The &%headers_remove%& option is expanded after &%errors_to%& and -&%headers_add%&, but before &%transport%&. If the expansion is forced to fail, -the option has no effect. Other expansion failures are treated as configuration +&%headers_add%&, but before &%transport%&. If an item expansion is forced to fail, +the item has no effect. Other expansion failures are treated as configuration errors. Unlike most options, &%headers_remove%& can be specified multiple times @@ -19797,10 +19799,11 @@ value that the router supplies, and also overriding any value associated with &%user%& (see below). -.option headers_add transports string&!! unset +.option headers_add transports list&!! unset .cindex "header lines" "adding in transport" .cindex "transport" "header lines; adding" -This option specifies a string of text that is expanded and added to the header +This option specifies a list of text headers, newline-separated, +which are (separately) expanded and added to the header portion of a message as it is transported, as described in section &<>&. Additional header lines can also be specified by routers. If the result of the expansion is an empty string, or if the expansion @@ -19821,18 +19824,20 @@ transports, the settings of &%message_prefix%& and &%message_suffix%& should be checked, since this option does not automatically suppress them. -.option headers_remove transports string&!! unset +.option headers_remove transports list&!! unset .cindex "header lines" "removing" .cindex "transport" "header lines; removing" -This option specifies a string that is expanded into a list of header names; +This option specifies a list of header names, colon-separated; these headers are omitted from the message as it is transported, as described in section &<>&. Header removal can also be specified by -routers. If the result of the expansion is an empty string, or if the expansion +routers. +Each list item is separately expanded. +If the result of the expansion is an empty string, or if the expansion is forced to fail, no action is taken. Other expansion failures are treated as errors and cause the delivery to be deferred. Unlike most options, &%headers_remove%& can be specified multiple times -for a router; all listed headers are added. +for a router; all listed headers are removed. @@ -31983,7 +31988,7 @@ they do not affect the values of the variables that refer to header lines. the transport cannot refer to the modified header lines, because such expansions all occur before the message is actually transported. -For both routers and transports, the result of expanding a &%headers_add%& +For both routers and transports, the argument of a &%headers_add%& option must be in the form of one or more RFC 2822 header lines, separated by newlines (coded as &"\n"&). For example: .code @@ -31993,10 +31998,10 @@ headers_add = X-added-header: added by $primary_hostname\n\ Exim does not check the syntax of these added header lines. Multiple &%headers_add%& options for a single router or transport can be -specified; the values will be concatenated (with a separating newline -added) before expansion. +specified; the values will append to a single list of header lines. +Each header-line is separately expanded. -The result of expanding &%headers_remove%& must consist of a colon-separated +The argument of a &%headers_remove%& option must consist of a colon-separated list of header names. This is confusing, because header names themselves are often terminated by colons. In this case, the colons are the list separators, not part of the names. For example: @@ -32005,11 +32010,12 @@ headers_remove = return-receipt-to:acknowledge-to .endd Multiple &%headers_remove%& options for a single router or transport can be -specified; the values will be concatenated (with a separating colon -added) before expansion. +specified; the arguments will append to a single header-names list. +Each item is separately expanded. -When &%headers_add%& or &%headers_remove%& is specified on a router, its value -is expanded at routing time, and then associated with all addresses that are +When &%headers_add%& or &%headers_remove%& is specified on a router, +items are expanded at routing time, +and then associated with all addresses that are accepted by that router, and also with any new addresses that it generates. If an address passes through several routers as a result of aliasing or forwarding, the changes are cumulative. diff --git a/doc/doc-txt/ChangeLog b/doc/doc-txt/ChangeLog index 6d9b28383..cc136b83c 100644 --- a/doc/doc-txt/ChangeLog +++ b/doc/doc-txt/ChangeLog @@ -66,6 +66,13 @@ JH/10 Bugzilla 1005: ACL "condition =" should accept values which are negative JH/11 Add dnsdb tlsa lookup. From Todd Lyons. +JH/12 Expand items in router/transport headers_add or headers_remove lists + individually rather than the list as a whole. Bug 1452. + + Required for reasonable handling of multiple headers_ options when + they may be empty; requires that headers_remove items with embedded + colons must have them doubled (or the list-separator changed). + Exim version 4.82 ----------------- diff --git a/src/src/functions.h b/src/src/functions.h index 395961530..35500a165 100644 --- a/src/src/functions.h +++ b/src/src/functions.h @@ -350,6 +350,7 @@ extern int stdin_feof(void); extern int stdin_ferror(void); extern int stdin_ungetc(int); extern uschar *string_append(uschar *, int *, int *, int, ...); +extern uschar *string_append_listele(uschar *, uschar, const uschar *); extern uschar *string_base62(unsigned long int); extern uschar *string_cat(uschar *, int *, int *, const uschar *, int); extern uschar *string_copy_dnsdomain(uschar *); diff --git a/src/src/readconf.c b/src/src/readconf.c index b41d22d2a..81e3bc267 100644 --- a/src/src/readconf.c +++ b/src/src/readconf.c @@ -1562,15 +1562,21 @@ switch (type) Because we only do this once, near process start-up, I'm prepared to let this slide for the time being, even though it rankles. */ } - else if (*str_target && (ol->type & opt_rep_str)) - { + else if (ol->type & opt_rep_str) + { uschar sep = Ustrncmp(name, "headers_add", 11)==0 ? '\n' : ':'; - saved_condition = *str_target; - strtemp = saved_condition + Ustrlen(saved_condition)-1; - if (*strtemp == sep) *strtemp = 0; /* eliminate trailing list-sep */ - strtemp = string_sprintf("%s%c%s", saved_condition, sep, sptr); - *str_target = string_copy_malloc(strtemp); - } + uschar * cp; + + /* Strip trailing whitespace and seperators */ + for (cp = sptr + Ustrlen(sptr) - 1; + cp >= sptr && (*cp == '\n' || *cp == '\t' || *cp == ' ' || *cp == sep); + cp--) *cp = '\0'; + + if (cp >= sptr) + *str_target = string_copy_malloc( + *str_target ? string_sprintf("%s%c%s", *str_target, sep, sptr) + : sptr); + } else { *str_target = sptr; @@ -4131,4 +4137,6 @@ while(next_section[0] != 0) (void)fclose(config_file); } +/* vi: aw ai sw=2 +*/ /* End of readconf.c */ diff --git a/src/src/routers/rf_get_munge_headers.c b/src/src/routers/rf_get_munge_headers.c index 76e87e9c7..c44ba70b8 100644 --- a/src/src/routers/rf_get_munge_headers.c +++ b/src/src/routers/rf_get_munge_headers.c @@ -13,7 +13,7 @@ * Get additional headers for a router * *************************************************/ -/* This function is called by both routers to sort out the additional headers +/* This function is called by routers to sort out the additional headers and header remove list for a particular address. Arguments: @@ -32,83 +32,78 @@ rf_get_munge_headers(address_item *addr, router_instance *rblock, header_line **extra_headers, uschar **remove_headers) { /* Default is to retain existing headers */ - *extra_headers = addr->p.extra_headers; -if (rblock->extra_headers != NULL) +if (rblock->extra_headers) { - header_line *h; - uschar *s = expand_string(rblock->extra_headers); + uschar * list = rblock->extra_headers; + int sep = '\n'; + uschar * s; + int slen; - if (s == NULL) - { - if (!expand_string_forcedfail) + while ((s = string_nextinlist(&list, &sep, NULL, 0))) + if (!(s = expand_string(s))) { - addr->message = string_sprintf("%s router failed to expand \"%s\": %s", - rblock->name, rblock->extra_headers, expand_string_message); - return DEFER; + if (!expand_string_forcedfail) + { + addr->message = string_sprintf("%s router failed to expand \"%s\": %s", + rblock->name, rblock->extra_headers, expand_string_message); + return DEFER; + } } - } - - /* Expand succeeded. Put extra header at the start of the chain because - further down it may point to headers from other routers, which may be - shared with other addresses. The output function outputs them in reverse - order. */ - - else - { - int slen = Ustrlen(s); - if (slen > 0) + else if ((slen = Ustrlen(s)) > 0) { - h = store_get(sizeof(header_line)); + /* Expand succeeded. Put extra headers at the start of the chain because + further down it may point to headers from other routers, which may be + shared with other addresses. The output function outputs them in reverse + order. */ + + header_line * h = store_get(sizeof(header_line)); /* We used to use string_sprintf() to add the newline if needed, but that causes problems if the header line is exceedingly long (e.g. adding something to a pathologically long line). So avoid it. */ if (s[slen-1] == '\n') - { - h->text = s; - } + h->text = s; else - { - h->text = store_get(slen+2); - memcpy(h->text, s, slen); - h->text[slen++] = '\n'; - h->text[slen] = 0; - } - - h->next = addr->p.extra_headers; + { + h->text = store_get(slen+2); + memcpy(h->text, s, slen); + h->text[slen++] = '\n'; + h->text[slen] = 0; + } + + h->next = *extra_headers; h->type = htype_other; h->slen = slen; *extra_headers = h; } - } } /* Default is to retain existing removes */ - *remove_headers = addr->p.remove_headers; -if (rblock->remove_headers != NULL) +/* Expand items from colon-sep list separately, then build new list */ +if (rblock->remove_headers) { - uschar *s = expand_string(rblock->remove_headers); - if (s == NULL) - { - if (!expand_string_forcedfail) + uschar * list = rblock->remove_headers; + int sep = ':'; + uschar * s; + uschar buffer[128]; + + while ((s = string_nextinlist(&list, &sep, buffer, sizeof(buffer)))) + if (!(s = expand_string(s))) { - addr->message = string_sprintf("%s router failed to expand \"%s\": %s", - rblock->name, rblock->remove_headers, expand_string_message); - return DEFER; + if (!expand_string_forcedfail) + { + addr->message = string_sprintf("%s router failed to expand \"%s\": %s", + rblock->name, rblock->remove_headers, expand_string_message); + return DEFER; + } } - } - else if (*s != 0) - { - if (addr->p.remove_headers == NULL) - *remove_headers = s; - else - *remove_headers = string_sprintf("%s : %s", addr->p.remove_headers, s); - } + else if (*s) + *remove_headers = string_append_listele(*remove_headers, ':', s); } return OK; diff --git a/src/src/string.c b/src/src/string.c index 6f54cc624..eb73fae3e 100644 --- a/src/src/string.c +++ b/src/src/string.c @@ -966,6 +966,50 @@ return buffer; #endif /* COMPILE_UTILITY */ +#ifndef COMPILE_UTILITY +/************************************************ +* Add element to seperated list * +************************************************/ +/* This function is used to build a list, returning +an allocated null-terminated growable string. The +given element has any embedded seperator characters +doubled. + +Arguments: + list points to the start of the list that is being built, or NULL + if this is a new list that has no contents yet + sep list seperator charactoer + ele new lement to be appended to the list + +Returns: pointer to the start of the list, changed if copied for expansion. +*/ + +uschar * +string_append_listele(uschar * list, uschar sep, const uschar * ele) +{ +uschar * new = NULL; +int sz = 0, off = 0; +uschar * sp; + +if (list) + { + new = string_cat(new, &sz, &off, list, Ustrlen(list)); + new = string_cat(new, &sz, &off, &sep, 1); + } + +while (sp = Ustrchr(ele, sep)) + { + new = string_cat(new, &sz, &off, ele, sp-ele+1); + new = string_cat(new, &sz, &off, &sep, 1); + ele = sp+1; + } +new = string_cat(new, &sz, &off, ele, Ustrlen(ele)); +new[off] = '\0'; +return new; +} +#endif /* COMPILE_UTILITY */ + + #ifndef COMPILE_UTILITY /************************************************* diff --git a/src/src/transport.c b/src/src/transport.c index 549da4694..d4495393b 100644 --- a/src/src/transport.c +++ b/src/src/transport.c @@ -626,19 +626,9 @@ header_line *h; /* Then the message's headers. Don't write any that are flagged as "old"; that means they were rewritten, or are a record of envelope rewriting, or were removed (e.g. Bcc). If remove_headers is not null, skip any headers that -match any entries therein. Then check addr->p.remove_headers too, provided that -addr is not NULL. */ - -if (remove_headers) - { - uschar *s = expand_string(remove_headers); - if (!s && !expand_string_forcedfail) - { - errno = ERRNO_CHHEADER_FAIL; - return FALSE; - } - remove_headers = s; - } +match any entries therein. It is a colon-sep list; expand the items +separately and squash any empty ones. +Then check addr->p.remove_headers too, provided that addr is not NULL. */ for (h = header_list; h != NULL; h = h->next) if (h->type != htype_old) { @@ -656,7 +646,15 @@ for (h = header_list; h != NULL; h = h->next) if (h->type != htype_old) uschar buffer[128]; while ((s = string_nextinlist(&list, &sep, buffer, sizeof(buffer)))) { - int len = Ustrlen(s); + int len; + + if (i == 0) + if (!(s = expand_string(s)) && !expand_string_forcedfail) + { + errno = ERRNO_CHHEADER_FAIL; + return FALSE; + } + len = Ustrlen(s); if (strncmpic(h->text, s, len) != 0) continue; ss = h->text + len; while (*ss == ' ' || *ss == '\t') ss++; @@ -731,36 +729,40 @@ if (addr) } } -/* If a string containing additional headers exists, expand it and write -out the result. This is done last so that if it (deliberately or accidentally) -isn't in header format, it won't mess up any other headers. An empty string -or a forced expansion failure are noops. An added header string from a -transport may not end with a newline; add one if it does not. */ +/* If a string containing additional headers exists it is a newline-sep +list. Expand each item and write out the result. This is done last so that +if it (deliberately or accidentally) isn't in header format, it won't mess +up any other headers. An empty string or a forced expansion failure are +noops. An added header string from a transport may not end with a newline; +add one if it does not. */ if (add_headers) { - uschar *s = expand_string(add_headers); - if (s == NULL) - { - if (!expand_string_forcedfail) - { errno = ERRNO_CHHEADER_FAIL; return FALSE; } - } - else - { - int len = Ustrlen(s); - if (len > 0) + int sep = '\n'; + uschar * s; + + while ((s = string_nextinlist(&add_headers, &sep, NULL, 0))) + if (!(s = expand_string(s))) { - if (!sendfn(fd, s, len, use_crlf)) return FALSE; - if (s[len-1] != '\n' && !sendfn(fd, US"\n", 1, use_crlf)) - return FALSE; - DEBUG(D_transport) + if (!expand_string_forcedfail) + { errno = ERRNO_CHHEADER_FAIL; return FALSE; } + } + else + { + int len = Ustrlen(s); + if (len > 0) { - debug_printf("added header line(s):\n%s", s); - if (s[len-1] != '\n') debug_printf("\n"); - debug_printf("---\n"); + if (!sendfn(fd, s, len, use_crlf)) return FALSE; + if (s[len-1] != '\n' && !sendfn(fd, US"\n", 1, use_crlf)) + return FALSE; + DEBUG(D_transport) + { + debug_printf("added header line:\n%s", s); + if (s[len-1] != '\n') debug_printf("\n"); + debug_printf("---\n"); + } } } - } } /* Separate headers from body with a blank line */ diff --git a/test/confs/0096 b/test/confs/0096 index 1ee9b746f..a182a9303 100644 --- a/test/confs/0096 +++ b/test/confs/0096 @@ -21,7 +21,8 @@ localuser: driver = accept local_parts = userx headers_add = "${if def:h_x-rbl-warning: {Added: xxxx}fail}" - headers_remove = "${if def:h_x-rbl-warning: {subject}fail}" + # Colon-sep list! + headers_remove = "${if def::h_x-rbl-warning:: {subject}fail}" transport = local_delivery diff --git a/test/confs/0097 b/test/confs/0097 index c8e773fc2..ad88aee09 100644 --- a/test/confs/0097 +++ b/test/confs/0097 @@ -21,7 +21,8 @@ localuser: driver = accept local_parts = userx headers_add = "${if def:h_x-rbl-warning: {Added: by router}}" - headers_remove = "${if def:h_x-rbl-warning: {subject}}" + # Colon-sep list! + headers_remove = "${if def::h_x-rbl-warning:: {subject}}" transport = local_delivery @@ -35,7 +36,7 @@ local_delivery: envelope_to_add file = DIR/test-mail/$local_part headers_add = "${if def:h_tadd: {Added: by transport}}" - headers_remove = "${if def:h_tadd: {tadd}}" + headers_remove = "${if def::h_tadd:: {tadd}}" return_path_add user = CALLER diff --git a/test/confs/0166 b/test/confs/0166 index b94331b44..7d2f06611 100644 --- a/test/confs/0166 +++ b/test/confs/0166 @@ -30,6 +30,9 @@ user: headers_add = X-Delivered-To: $local_part@$domain retry_use_local_part transport = local_delivery + headers_add = X-rtr-hdr: 1 + headers_add = ${if bool{false} {X-rtr-hdr: 2}} + headers_add = X-rtr-hdr: 3 # ----- Transports ----- @@ -41,6 +44,9 @@ local_delivery: envelope_to_add file = DIR/test-mail/$local_part user = CALLER + headers_add = X-tpt-hdr: 1 + headers_add = ${if bool{false} {X-tpt-hdr: 2}} + headers_add = X-tpt-hdr: 3 # End diff --git a/test/mail/0046.userx b/test/mail/0046.userx index cf7f26ea0aaf8117643a45f20cabe30af61b41be..05c078ec0c7820deb23f21ea850840e434de2a15 100644 GIT binary patch delta 15 XcmZ1}y delta 15 XcmZ23y;6F^TAs;kc#0>>@pb|LGX(|p diff --git a/test/mail/0166.userx b/test/mail/0166.userx index 3c58c0da1..d07630976 100644 --- a/test/mail/0166.userx +++ b/test/mail/0166.userx @@ -12,5 +12,9 @@ X-Delivered-To: b@test.ex X-Delivered-To: c@test.ex X-Delivered-To: d@test.ex X-Delivered-To: userx@test.ex +X-rtr-hdr: 1 +X-rtr-hdr: 3 +X-tpt-hdr: 1 +X-tpt-hdr: 3 diff --git a/test/mail/0166.usery b/test/mail/0166.usery index 529767368..8076ae65d 100644 --- a/test/mail/0166.usery +++ b/test/mail/0166.usery @@ -11,5 +11,9 @@ X-Delivered-To: a@test.ex X-Delivered-To: bb@test.ex X-Delivered-To: e@test.ex X-Delivered-To: usery@test.ex +X-rtr-hdr: 1 +X-rtr-hdr: 3 +X-tpt-hdr: 1 +X-tpt-hdr: 3 diff --git a/test/mail/0351.userx b/test/mail/0351.userx index 50bfd7d6c..41b34c002 100644 --- a/test/mail/0351.userx +++ b/test/mail/0351.userx @@ -19,7 +19,6 @@ Resent-From: CALLER_NAME Found: yes Found2: yes FOUND-found2: !! - TO: userx@test.ex, usery@test.ex -------------------------------- diff --git a/test/mail/0412.CALLER b/test/mail/0412.CALLER index 260252beb..410121dae 100644 --- a/test/mail/0412.CALLER +++ b/test/mail/0412.CALLER @@ -7,7 +7,6 @@ From: CALLER_NAME Message-Id: Date: Tue, 2 Mar 1999 09:44:33 +0000 Found: no - FROM: CALLER_NAME -------------------------------- REPLY_ADDRESS: CALLER_NAME @@ -25,7 +24,6 @@ From: CALLER_NAME Message-Id: Date: Tue, 2 Mar 1999 09:44:33 +0000 Found: no - FROM: CALLER_NAME -------------------------------- REPLY_ADDRESS: CALLER_NAME @@ -43,7 +41,6 @@ From: CALLER_NAME Message-Id: Date: Tue, 2 Mar 1999 09:44:33 +0000 Found: no - FROM: CALLER_NAME -------------------------------- REPLY_ADDRESS: usery@test.ex @@ -61,7 +58,6 @@ Message-Id: Sender: CALLER_NAME Date: Tue, 2 Mar 1999 09:44:33 +0000 Found: no - FROM: -------------------------------- REPLY_ADDRESS: diff --git a/test/scripts/0000-Basic/0481 b/test/scripts/0000-Basic/0481 index d1a9a4a70..e1b8574db 100644 --- a/test/scripts/0000-Basic/0481 +++ b/test/scripts/0000-Basic/0481 @@ -1,4 +1,4 @@ -# multiple remove_headers and trailing colons +# multiple remove_headers in routers, and trailing colons exim -odi userx Remove-Me: this header is to be removed Another: This is another header diff --git a/test/stderr/5400 b/test/stderr/5400 index 29ff83b42..1c9b0d83a 100644 --- a/test/stderr/5400 +++ b/test/stderr/5400 @@ -207,7 +207,7 @@ X-hdr-rtr: qqq added header line(s): X-hdr-rtr-new: +++ --- -added header line(s): +added header line: X-hdr-tpt-new: new --- ----------- done cutthrough headers send ------------ -- 2.25.1