#ident "@(#)smail:$Id: ToDo,v 1.157 2017/10/15 21:11:43 woods Exp $" Move to GitHub Things that should be done before the next minor release (patches are, of course, gratefully accepted!): Critical BUGS: -------------- - check why queue_only changes its default value with the recent changes - check that expand_string() isn't used while a previous call's output is still referenced - sometimes it seems as if bounces that are deferred on delivery are apparently lost!!! This should not be possible though because the transport doesn't know anything about whether the message is a bounce message or not. 2017/10/12 01:38:33: [28525] [m1e2WCX-00r8deC] Deferred TO: ROUTER:bind_hosts TRANSPORT:inet_zone_bind_smtp ERROR:(ERR151) The inet_zone_bind_smtp transport reports a temporary failure. The remote SMTP server at 'spamex1.dm.egate.net' said: 451-198.96.117.51 is not yet authorized to send delivery status reports to 451 . Please try later. 2017/10/12 01:38:33: [14460] [m1e23KJ-00r8h4C] Returned error BY:[14564] FOR: ORIG-FOR: TO: 2017/10/12 01:38:33: [14460] [m1e23KJ-00r8h4C] Completed. - don't bounce on errors from Cyrus deliver!!! [[FIXED??? (with pipe driver error code changes)]] 2017/10/12 01:12:04: [1381] [m1dyD4S-00r7ShC] Failed TO: ORIG-TO: DIRECTOR:user_with_mbox TRANSPORT:cyrus_deliver ERROR:(ERR144) transport cyrus_deliver: child PID# [11252] returned status EX_OSFILE (72) (see also the entry below about giving transport drivers FDs to write to the logs) - safe signal handling: http://evbergen.home.xs4all.nl/unix-signals.html - "retry time not yet reached" is the WRONG error here! But how can this happen!?!?!? 2007/04/29 18:40:03: [6666] [m1Hgq8h-003T2zC] Failed TO: ROUTER:bind_hosts TRANSPORT:inet_zone_bind_smtp ERROR:(ERR174) retry/smtp/67.28.113.16: retry time not yet reached - still the restart fails on occasion: 2007/07/25 00:00:28: [20342] left behind 1 un-killable smtpd processes 2007/07/25 00:00:28: [20342] SIGHUP handled, restarting: execv(/usr/pkg/libexec/smail/smail, '/usr/pkg/libexec/smail/smail -bd -q5m') here is another example where the client was idle, SIGHUP(?) was sent to the daemon, then the client sent a newline, and when the first logentry appears the client also sees: "421 4.3.2 centrally.weird.com Service no longer available, closing channel." and the connection drops. The parent daemon then complains that it appears to have disappeared: 2011/11/17 16:08:04: [21671] SMTP connection terminated by signal SIGHUP after while talking with once.weird.com(S010 60026bb6c284e.ok.shawcable.net)[24.67.98.78] 2011/11/17 16:08:04: [24101] smtpd PID# [21671] seems to have disappeared before it could be sent signal 1! 2011/11/17 16:08:04: [24101] left behind 1 un-killable smtpd processes - the following occurred during a testing session with "DEBUG 99" enabled where a reload restart was triggered. Initially a 550 error had been delivered in response to a MAIL command: 000 [6024] read_smtp_command() called. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. 000 [6024] smtp_reload_sig(SIGHUP) called.... 000 [6024] read_smtp_command() got EOF with EINTR. then after a short delay the normal timeout occurred: 000 [6024] smtp_receive_timeout_sig(SIGALRM) called.... 421 4.3.2 centrally.weird.com SMTP command timeout, closing channel 000 write_log(SYS): 2012/09/20 05:05:56: [6024] SIGALRM received while talking with woods@localhost[127.0.0.1] Connection closed by foreign host. it looks like read_smtp_command() should check to see if any(?) signal handler had been triggered and abort on EINTR instead of retrying. [[FIXED??? (src/smtprecv.c: rev 1.340)]] - maybe the clearing message_id problem for multiple transactions in SMTP can be fixed? - make sure logsumm is forward-compatible with the logs for every message from verify_ptr, as well as the new smtp_reject_hosts pattern too. - figure out how to deal with bounces being sent from a mail-hub client via the submission port XXX maybe whoson can be used some way to authorize the client to send bounces? - implement RFC 6409 (submission) adaptations (timeouts, AUTH, etc.) - make sure use of any of the destructive string manipulation functions (e.g. srip_*()) are not mixed with STR() macros, since the former do not maintain the length attribute for the latter. - consider making the STR() macro add a NUL iff necessary. Important Bugs: --------------- - why does Smail not see the "domain" entry in /etc/resolv.conf unless it comes second??? [[NOT-SMAIL??? bind_compute_domain() uses res_init(3) and then returns a copy of _res.defdname]] - fix the doc for $received_field -- or just rewrite it all in C. document the string expansion of the Error director message text. - document the pitfalls of rejecting null-return-path transactions with multiple recipients, e.g. when outbound mail is sent with a return path that points back to an alias that expands to multiple addresses at some common remote site. If such mail bounces the common remote site may see an incoming transaction with a null-return-path and multiple recipients. See example in RFC 3461 section 5.2.2 (d). - fix smtplib.c so that SMTP error responses indicate which command triggered them. - don't show PID when verb level is high and only a newline is printed. - consider changing the format of the SMTP transport debug messages to look more like the sendmail/vrfy output. - implement max_load_ave - auth_domains should allow expansion. It could also maybe default to "$hostnames:$more_hostnames", e.g. for an internal mailbox server that routes all other mail off to a smarthost gateway -- but then it would have to be documented with a reminder to be sure to unset it if no local delivery at all is supposed to happen. Which is more common, a host with no local delivery always routing everying to a smarthost gateway; or a host that handles internal delivery for all domains it is a member of and routes only unknown domains to the smarthost gateway? [auth_domains really should be called something more like smarthost_auth_hostnames, and/or it should be a private attribute of the smarthost router (though conceivably it could be used by the some other router someday)] - try to make sure check_sender() verifies the deliverability of all local addresses and acceptability of all remote addresses. verify_sender() needs to be refactored, gain a new API, and be moved into someplace more appropriate, so that the same logic can be used in main.c and in smtprecv.c - how should we stop (local) MUA clients from abusively using a local sender address which should never be seen except from valid remotes, e.g. , etc. - consider adding a lookup db for valid non-local sender addresses so that local users can only use these explicitly recorded, and hopefully verified to be valid, sender addresses, in hopes of stopping backscatter bounce attacks from within the local network. - get rid of the angle brackets in DEBUG() output when handling SMTP envelope addresses: <>: full match found to 'planix.com'. <>: planix.com matched by bind_hosts router: parse_address: LOCAL routed <> --> at planix.com - the transaction summary looks confusing when Succeed entries are not printed for some that had temporary delivery failures. Maybe none of the deferrals should be shown in a bounce message? |------------------------- Failed addresses follow: ---------------------| address 'uk-abuse@yahoo-inc.com' failed: inet_zone_bind_smtp transport reports unknown user: 550 5.1.1 ... User unknown |------------------------- Transaction log summary follows: -------------| defer: reason: (ERR172) inet_zone_bind_smtp transport failure: no connection to remote SMTP server: 421-smtp-in01.aci.on.ca Too many connections! 421 Try again later. [2005/12/29 19:26:56: [20297]] defer: reason: (ERR172) inet_zone_bind_smtp transport failure: no connection to remote SMTP server: 421-smtp-in01.aci.on.ca Too many connections! 421 Try again later. [2005/12/29 19:26:56: [20297]] fail: reason: (ERR156) inet_zone_bind_smtp transport reports unknown user: 550 5.1.1 ... User unknown [2005/12/29 19:26:56: [20297]] note that only the first is a failed address. - steal code from BIND to enumerate all the local interface addresses so that we can include each one explicitly in smtp_remote_allow since "localnet" is too broad. (or just add listen_name, if that's set) - there is some rare crash happening in smtpd after a HELO questionable operand [could this just be the "MAIL FROM:" heap overflow in preparse_address_1()?] - apparently we can get a NULL when the mail domain has a CNAME but no MX for the target (at least on the first query, i.e. until the local caching server caches the found records for the domain): 01/23/2004 08:33:22: [17724] DNS MX error: MX domain '(null)' has a target host 'extenda.net' which is an invalid CNAME (for 'ds2.domainspa.com') $ host -t mx extenda.net *** extenda.net has no MX record (Authoritative answer) $ host -a extenda.net extenda.net NS dns1.name-services.com extenda.net NS dns2.name-services.com extenda.net NS dns3.name-services.com extenda.net NS dns4.name-services.com extenda.net NS dns5.name-services.com extenda.net SOA dns1.name-services.com info.name-services.com ( 2002050701 ;serial number (version) 3600 ;slave refresh period (1 hour) 120 ;slave retry interval (2 minutes) 86400 ;slave expire time (1 day) 3600 ;negative response TTL (1 hour) ) extenda.net CNAME ds2.domainspa.com $ host -t mx ds2.domainspa.com *** ds2.domainspa.com has no MX record (Authoritative answer) this should of course be impossible in the first place (CNAME with other data), but some nameservers are just too stupid for words In any case maybe the log should be fixed to read more sensibly? (dunno if this happens when the CNAME is valid, but still points to a name with no MX -- that should cause 974 processing, and if there's also no A RR then that's a hard error) - use new flags, and maybe a new field or two, to mark when the retry file error and most recent msglog defer error have been added for a given address. Otherwise we get two copies of the previous message, as below: |------------------------- Failed addresses follow: ---------------------| address: hostmaster@[213.248.29.9] failed: transport inet_zone_bind_smtp: connect(mail.domainreservationsite.com:smtp): Connection timed out Previous retry error: transport inet_zone_bind_smtp: connect(mail.domainreservationsite.com:smtp): Connection timed out Retry duration (1day) has been exceeded. No further delivery attempts will be made. Retry duration (1day) has been exceeded. No further delivery attempts will be made. |------------------------- Transaction log summary follows: -------------| defer: reason: (ERR151) temporary failure from inet_zone_bind_smtp transport: 454 : Relay access denied [01/16/2004 14:14:34: [18044]] defer: reason: (ERR148) transport inet_zone_smtp: connect([213.248.29.9]:smtp): Connection refused [01/16/2004 11:15:10: [7171]] defer: reason: (ERR151) temporary failure from inet_zone_smtp transport: 454 : Relay access denied [01/16/2004 14:14:34: [18044]] fail: reason: (ERR148) transport inet_zone_bind_smtp: connect(mail.domainreservationsite.com:smtp): Connection timed out Previous retry error: transport inet_zone_bind_smtp: connect(mail.domainreservationsite.com:smtp): Connection timed out Retry duration (1day) has been exceeded. No further delivery attempts will be made. Retry duration (1day) has been exceeded. No further delivery attempts will be made. [01/16/2004 14:14:30: [18044]] Note also the problem with retry files for domain literals "merging" with retry files for the domain itself. - what about retry-file errors for addresses that go to the same transport and next_host? shouldn't they all have the same text merged/recorded? - why is "smail:" being added to paniclog entries, especially during a normal queue run? It can not happen unless only_testing is set and that's only set in certain command-line driven operations! - continue to make sure all foreign data is safely formatted wherever it's used: + is str_c_quote() sufficient to avoid violating SMTP message response rules and RFC 822 header text rules? + maybe responses logged in msglog need to be quoted (%v) too? Sometimes broken servers respond with 8'th-bit-high chars.... - DO NOT RUN SMAIL AS ROOT!!!! + the main smail/sendmail binary will be set-group-ID to 'smail' (not 'mail') (and owned by root, of course, but not setuid, just setgid) + the queues will be writable by the group 'smail' (not "mail") and owned (and also writable) by the user 'smail'. + the smtp daemon will call setuid(nobody) && setgid(smail), and perhaps also re-exec itself, to permanently and absolutely 100% guaranteed give up all the privileges of its invoker after it has a socket descriptor already bound to port 25, but before it accepts any connections (it will have to be started by root, of course, on many types of systems) + the smtp daemon will record its real_uid as "smail" (still using local_sender=$nobody of course) in the queue file) + $trusted_users will have to include "smail" so that sender addresses in queue files written by the smtp daemon will be trusted as valid. + checkerr will be run (e.g. from cron) as smail:smail + local delivery to mailbox spool (/var/mail) files will be done only by a separate setgid-'mail' agent ala *BSD mail.local, with kernel locking where possible, *.lock files only where absolutely necessary. + initial local mailbox spool (/var/mail) file creation, if necessary, will be done by a tiny setuid-root helper on systems that have a root-only chown(2) [one exists somewhere already -- search the net]; they will be owned by the user, group 'mail', and be mode 660. The spool directory will be mode 555 if kernel locking is possible, else 575 & group 'mail' if necessary for *.lock files (and in all cases owned by root, of course). the helper will _only_ create an empty spool file if one does not already exist, and it will determine the pathname to use based on the user name given to it on its command line. + all mail readers will be expected to either use a small helper to safely copy the mailbox spool file to a private place (like movemail), which will use kernel file locking when possible or be setgid-'mail' and use *.lock files if necessary (and hopefully copy it away to a private place before the user does his/her reading); mail readers (including movemail) will be strongly "encouraged" to keep the zero-length spool file after emptying it; and the use of setgid-'mail' for readers will be very Very VERY strongly warned against (anything more complex than the old V7 /bin/mail has had many security bugs because of such set-ID hacks). + .forward files will have to be world readable (or at least readable by the group 'smail'), *or* "Forward to" support can be used (but never both). + delivery to files and pipes would normally be done either as nobody:nogroup or nobody:smail, and there'll be no more pipe_as_user or pipe_as_sender or user or group private attributes for the pipe transport. However nothing will prevent the pipe transport from execing a set-group-id (or set-user-id) wrapper binary though. The latter would be sensible for something like Cyrus-IMAPd where the setgid-cyrus "deliver" wrapper would be protected in a directory accessible only by the "smail" group (and of course it would verify that it was invoked as "nobody:smail" before exec'ing the real "deliver"). + access to locked files will only be attempted for a limited amount of time and messages will be left in the queue if delivery is unsuccessful because of the presence of any pre-existing a lock - deal with un-qualified local hostnames when there's no qualify file in some more sane way.... [is this still necessary now that the command-line parameters are turned into a "field" for processing?] - qualify addresses differently depending on where the mail arrived from -- e.g. if not from a domain in smtp_remote_allow then add a bogus domain like "address.invalid". - flesh out the test_headers code in main.c - use ftruncate() to remove partially written messages in appendfile.c if ERR_135 [if possible]. - fix aliasfile parser to allow case sensitive aliases. [also keep in mind the lists director also uses "lists/${lc:user}"] - fix expand_string() et al to always return newly allocated storage, and then make sure that storage is freed when it's done with. - fix expand_string_to() to ignore a backslash-escaped '$', but to remove the backslash.... - fix read_entry() to either always truncate the last newline, or else at least have an option to allow it to do so. - turn down the verbose logging of failed locks, if known other smail process holds lock.... eg: 02/28/96 12:07:36: open_spool: /local/var/spool/smail/input/0trpIB-00076nC: lock failed: Permission denied Unfortunately this will probably require re-writing the spool locking functions to use pid-in-a-lock-file mechanisms. [effectively fixed in 3.2.1 for systems that return EAGAIN if lock_fd() meets another lock?] - the logic in notify() (and error_delivery()) is far too hairy. classify_addr() seems to be more on the right track, but it's still a bit wonky. For example there is no "Returned" log entry written when an error is sent for an alias that has an owner -- only the owner entry is noted (Error sent) entry is written. [first write a spec on how address structures get linked together, then make sure the rest of the code adheres to the spec!] - do error checking and reporting in driver "cache" functions, and check for those errors in the cache_*() functions and their callers. Once that's done the calls to read_retry_file() can be migrated to cache_transport(). - the no-delivery option (-N) doesn't prevent delivery of a bounce. Should it? write_log(SYS): [m1FUpFP-002IhbC] Received FROM:root PROGRAM:smail SIZE:101 DELAY:8 write_log(SYS): [m1FUpFP-002IhbC] Failed TO: ERROR:(ERR101) unknown target host domain name notify(): returning to sender: notify_new_message: sending to: - The maximum retry duration must be done on the target domain name while the retry lock file for interval calculation must be done on the IP address of the destination host. The code in transport.c should check if the retry duration has been exceeded by doing a match_retry_domain(addr->target) while a transport driver should only check if the retry interval has past for the given target host address it's about to connect to. I.e. the transport.c code will check a retry interval against a domain name and the transport drivers will create the retry interval lock based on the target host address. TCPSMTP-like transport drivers must not attempt to connect to lower-precedence MX hosts if all the best-pref MX hosts are currently locked by other processes and instead should simply defer delivery. Lesser-preference hosts should only be tried if there is a connect error (that's not one which forces an immediate bounce) with _all_ higher-pref hosts. This way we can connect simultaneously to every host address given for any target domain, but we won't retry any given host unless the retry duration has been exceeded; and finally we'll bounce the message if the retry duration for the target domain has been exceeded. - modify the interpretation of the retry_interval so that the delay between the first two or three attempts can be specified separately from the subsequent delays, and modify the default retry interval to allow two connection attempts in the first hour the message is in the queue, and then back off to once every two or three hours for all subsequent retries. - maybe the retry file should also allow for hostname REs too? If so then are they all REs (how do we maintain the leading dot compatability?) or should we just use the quoted string trick ala aliases? - modify the host retry locking mechanism to allow for multiple concurrent deliveries to a given target host. Perhaps each sender locks the retry file only temporarily, not for the whole duration of its delivery attempt, and if the file is new (or empty?) or contains only fewer than the concurrency limit of "pid N" lines then it appends its own "pid N" line to the end of the file, unlocks it and goes on (perhaps first checking that all the existing "pid N" lines refer to existing running processes, rewriting the file to omit any dead ones). If the file exists but contains an error number and message then it does the normal retry duration timeout first. The concurrency limit could be (optionally) specified in the retry configuration so that different destinations can be given different concurrency limits. - make sure we don't even start to use a spool directory unless the minimum reserved space is available -- this will hopefully allow messages in that directory to continue to be processed (i.e. for their msglog files, and retry files, to be written to) - be careful about never filling the filesystem for logfile and paniclog too (just defer connections if there's a write error for the "accepted connection" log message? Can we be even more careful, as well?) - only create one "def.*" set of files, perhaps in the top level, or perhaps in the "conf" subdirectory. - update RFC references for SMTP et al in messages and text. - think about giving transport drivers (esp. ones that run as another user), open FDs for the logs. The lines here between tagged with "sendmail:" only show up in the full transaction log summary in a bounce message (as hinted by the write_log line above): stderr: <"|/usr/pkg/bin/qpage -m -l 0 -p woods"> orig-to: stderr output from pipe: "/usr/pkg/bin/qpage" "-m" "-l" "0" "-p" "woods" open_child(): execve(/usr/pkg/bin/qpage, envp): No such file or directory. write_log(SYS|PANIC|TTY): [m1S1Oh8-002Y1FC] open_child(): exec(/usr/pkg/bin/qpage) failed (possible local configuration error): No such file or directory sendmail: [22317] cannot open panic log /var/log/smail/paniclog: Permission denied sendmail: [22317] cannot open system log /var/log/smail/logfile: Permission denied sendmail: 2012/02/25 15:58:18: [22317] [m1S1Oh8-002Y1FC] open_child(): exec(/usr/pkg/bin/qpage) failed (possible local configuration error): No such file or directory 2012/02/25 15:58:18: [22317] [m1S1Oh8-002Y1FC] open_child(): exec(/usr/pkg/bin/qpage) failed (possible local configuration error): Permission denied 2012/02/25 15:58:18: [22317] [m1S1Oh8-002Y1FC] open_child(): exec(/usr/pkg/bin/qpage) failed (possible local configuration error): Permission denied error: transport pipe: child PID# [22317] returned status EX_OSFILE (72) [2012/02/25 15:58:18: [2580]] fail: <"|/usr/pkg/bin/qpage -m -l 0 -p woods"> orig-to: reason: (ERR144) transport pipe: child PID# [22317] returned status EX_OSFILE (72) - connection counting is probably not working correctly -- have seen "accepted connection 1 from" while other connections are still open. - something odd might be happening with parsing here (with the '!'), and also the message is not clear whether the error is about the recipient address or the sender address:: 2020/04/01 16:57:28: [269] remote RCPT TO:: o1678950229.outbound-mail.sendgrid.net[167.89.50.229]: recipient 'weird!@weird.com', for sender 'bounces+14767294-558f-weird!=weird.com@sendgrid.net', not deliverable: (ERR_164) localhost MX check: DNS error: DNS lookup for MX for 'weird': Unknown host Incomplete Features: -------------------- - log the remote queue-id from the end-of-DATA response - bootstrap a defs.mk and include that so we can use normal make macro expansion instead of having to source defs.sh in every command-line. - consider shipping empty .depend files so that we can "include" them in the Makefiles too. - we need a ${foraddrs: operator for the "for" subfield of recieved_field and other places where address lists should be iterated over. (or should we simply mimic the existing, ugly, ``$('' and ``$)'' syntax provided for the pipe transport cmd attribute?) Note it should probably iterate over the list backwards, or else the recipient list should be reversed immediately by whatever sets it. Note this is only needed if received_field remains as a config variable -- if its complexity is internalized into C code then all we need is a flag (possibly per-transport) saying whether or not to include (all or some of) the recipient addresses in the output (and possibly a flag saying whether or not to include a comment identifying the server software too). - This raises the question of addresses from the BCC header ending up in the Received header sent to a non-BCC'ed recipient. We should follow the advice of RFC 2821: "Since this rule is often violated in practice, and cannot be enforced, sending SMTP systems that are aware of "bcc" use MAY find it helpful to send each blind copy as a separate message transaction containing only a single RCPT command." We should also use a per-address flag to indicate whether the address is one of the BCC recipients or not and only they would be omitted from the received header "for RECIP[, RECIP2] ..." sub-field (unless the current recipient is a BCC'ed recipient; and maybe if the current recipient is the sender then all recipients could be listed, though this could still end up being security leak since the sender may then forward the copy he/she receives without realizing that its received header reveals the whole BCC list). For transport protocols such as SMTP which allow multiple transactions per connection the transport drivers sort the recipient list based on this flag and then use it to determine whether they should start a new transaction, while still connected, for the BCC'ed recipient's. Other transports that cannot do multiple transactions per "job", but which can do multiple recipients per transaction (e.g. UUCP) would do all the visible recipients first and then defer (with a retry duration of zero) the (remaining) BCC recipient(s) to be done in a separate transaction (which might even somehow be triggered immediately). - when receiving via SMTP a message which has a blank BCC header we could give it our best shot to find the apparent list of recipient addresses in the visible TO and CC headers, then after matching them with the SMTP recipients only list the visible ones in the "for" subfield, _except_ when delivering a message to what seems to be a BCC'ed recipient, in which case we would also include that recipient's address in the "for" subfield. [This does seem to be a lot of work to do though just to accomodate MTAs that don't follow the RFC 2821 advice of sending BCC'ed recipients in a separate transaction.] - instead of having to verify gTLD validity using messy REs in the smtp_*_reject_hostnames rules consider doing a DNS lookup in the root zone to be sure that it exists. - add "delivered-to:" and "x-original-to:" headers ala Postfix. Can these both be done with an "append_header" in the local transports? Is that the right way to implement them? append_header="X-Original-To: ${top:input_addr}" NOTE: postfix apparently prepends this header..... that might help sort multiple copies out by interleaving them with received headers. one would have to "remove_header=delivered-to", but doing it through this mechanism wouldn't allow for loop detection, which is the whole purpose of "delivered-to" in the first place. hmmm..... remove_header="delivered-to", append_header="Delivered-To: ${input_addr}" - smtp_sender_reject* should be just sender_reject* and should apply to command-line addresses and header addresses alike. - separate out the use of quote() into quote_value() and quote_addr(), the latter expecting in_addr and producing RFC 821 quoted addresses. - invent "%a" for str_printf() for producing RFC 821 addresses. - checkerr should RCS the dead-mail.senders list, if RCS commands can be found.... - dead-mail.egrep support needs to be re-written to use the same specification and matching mechanisms as body_checks, and by default maybe body_checks and body_checks_always should be used (in addition to?) the dead-mail.egrep list. Too bad pcregrep(1) doesn't have an option for doing whole-file matching and for taking a list of REs in the "/RE/OPTIONS" format.... - implement ":redirect:" for the error director to return a proper SMTP-level redirect response message. - consider implementing optional min/max limits in struct attr_table, especially for t_int and t_interval - invent and implement a syntax for match_pcre_list() that can take into account the client source address, esp. for negated expressions. Perhaps something like this using a "flag" setting: :! /^Subject: blah/im c='192.168.0.2:172.16.10.10'; allow these guys to say "blah" :/^Subject: blah/im; but don't allow anyone else to say "blah" Note the use of single quotes to hide the ':' in the c= list. Also potentially useful for preventing MUA clients and other known abusers from abusing local system addresses: smtp_sender_reject = :/MAILER-DAEMON@(mail\.)?example\\.com/i c='192.168.168.0/24'; You are not allowed to (ab)use internal addresses! (keeping in mind that this prevents them from sending _to_ those matching addresses also, since this check is done for RCPT as well) - can the same kind of thing be done for match_re_list(), i.e. match_hostname()? Or should match_hostname() just switch to using the full quoted-RE syntax, i.e. switch to using match_pcre_list()? This would allow restricting local sender addresses to local users in an alternate manner to that of the smtp_local_sender_restrict feature: smtp_sender_reject_hostnames = :! /${foreach:hostname:${rxquote:value}|}EMPTY/i c='localnet:192.168.0/24'; allow these guys :! /${foreach:more_hostnames:${rxquote:value}|}EMPTY/i c='$smtp_remote_allow'; allow these guys too :/${foreach:hostname:${rxquote:value}|}EMPTY/i; your sender address is restricted to local clients! :/${foreach:more_hostnames:${rxquote:value}|}EMPTY/i; your sender address is restricted to local clients! Note the use of a variable expansion in the 2nd "c=" setting. Note though that the smtp_local_sender_allow feature could only be replicated by doing the above with smtp_sender_reject, but the resulting expressions could turn out to be very complex and not easily suited for expansion with ${foreach: Similarly this would also allow use of smtp_hello_reject_hostnames to reject MX'ed-for hostnames from non-local hosts, e.g. in the case where the MX'ed-for host does connect to the MX server in order to deliver its own outbound mail. - consider implementing whole_header_checks{_always} so that each RE matches the whole header portion just as with body_checks, which would allow much more exacting patterns matching multiple fields to be used. - think about an option, or another set of *_header_checks_* vars, that would match the canonical header entries, i.e. un-folded headers. - think about how to pre-compile other RE lists. - implement line-wrapping in format_pcre_list(). - write an expression pretty-printer for the likes of received_field. - may need more testing of comment handling in bsearch and lsearch alias files, though so far so good! - smtp_bad_mx_targets should match "faked" MXes too! - think about logging the sender_host_port value too. - pd/Makefile should check if the pathalias driver is included before diving into the pd/pathalias directory. - consider making all file readers check that the file's last-modified time hasn't changed while the file was being read so as to prevent problems when people use software which modifies files in place instead of safely writing all changes to a new file and then using rename(). - SIGTERM (SIGHUP?) in delivery_signals() should set a flag to cause smail to exit between calls to transport drivers. In between such calls, the state will be consistent and safe, the message will be secure in the queue, and it is okay to call close_spool(). - all signal handlers should only set flags (and do debug logging???) (this would mean relying on interruptable calls, and detecting them and checking all flags at those places) - implement an outbound ACL to prevent delivery to destinations listed in either smtp_output_reject_hostnames, smtp_output_reject_ipaddr, smtp_output_reject_dnsbl (addrs), or smtp_output_reject_rhsbl (hostnames). Maybe these can be checked during verify too so that they'll even fail at RCPT TO: time, though that would then require that at least the domain/host forms be excepted if they are "islocalhost()". We may need smtp_output_reject_except_{host,ipaddr} lists too/anyway. This will allow the like of inputs.relays.osirusoft to be used to block even command-line users from sending to a known open relay. - implement a client-source-address based ACL to allow restricted EXPN usage (smtp_expn_allow?). - implement more generic ACLs for at least directors (and maybe routers, and what about transports too?). E.g. this could be used to only allow certain alias files, or lists, or the magic Cyrus noquota director, to be used only if the message originated from the local host, a local sender, or from some other trusted client address. - implement smtp transport driver attributes to allow the sending_name (and thus the greeting name) to be specified uniquely for a given transport (overriding the global setting, if any). - make sure RFC-[2]821 local-part quoting isn't too aggressive and re-quoting things that don't need re-quoting. [OK now?] - make sure that quoting is done properly on addresses returned by the rewrite router - check to be sure we use IANA registered protocol names, etc. in received headers and such places. - don't allow bogus A RR's to "match" (0, 255.255.255.255, 127/8, RFC-1918 addresses etc.). Probably need to provide a config variable that contains a list of "bogus" addresses, something like smtp_bad_mx_targets but more general for all A RRs. Perhaps names given in HELO would skip this check if the client address matches in smtp_remote_allow or something. What about names for internal MX hosts though? How do we know if they're "internal"? - think about making mailq's '-E' work for runq too. [unfreezemail effectively does this though] - 'runq -E' though could also include an option to allow re-writing of the recipient address -- though perhaps this is best left to real experts who know how to safely edit queue files directly..... - add an option to 'mailq -E' that'll only match if the undelivered addresses are local or not to make it easier to find and unfreeze large batches of messages which might now be locally deliverable (this could be especially useful for Cyrus IMAP sites where many messages get frozen because of quota restrictions). [this is partly done now in a hackish way by the tempfail retry logic in util/checkerr, and there's also the REGEX-capable 'unfreezemail -u ADDR'] - Normally we don't want to restrict users who are not using our mail server to relay their outgoing messages but rather only to receive (and probably re-route remotely again) their incoming messages (i.e. virtual domain users). Such users will not usually be using clients listed in smtp_remote_allow. For finer grained smtp_local_sender_restrict control we should be able to tell the difference between an address routed via $hostnames or $more_hostnames and one routed via some outside router like one using the 'rewrite' driver. Maybe we need a full list of domains for which we do anti-spoof checks. In the mean time if you host virtual domains like this then you'd best not enable smtp_local_sender_restrict. - think about how to include the "ORIG-TO:" field from the "Delivered" log entry in the received header -- for SMTP this should be the original envelope recipient address for this particular delivery (what do we do when a delivery has collapsed multiple addresses when avoiding duplicate delivery? It looks like only the "first" will appear in the log, whatever "first" means....) - properly fix all the other director drivers to have RE-capable prefix and suffix attributes (like what was done recently for the user driver) - add a "cmd" attribute to the smartuser driver (and a corresponding smart_user_cmd config variable) such that dynamic lookups can be done (e.g. forwarding address verification to an internal server). - think about adding a verify_command to the pipe transport driver so that VRFY and RCPT TO: commands, as well as '-bv', can actually check if a delivery will succeed. This is primarily most useful when local delivery is done via a pipe driver and where the delivery agent has some easy way to report if delivery might fail due to quota violations or other problems (eg. with Cyrus IMAP). Note the "error" director, in combination with the "cmd" attribute mentioned above for the smartuser driver driver, may eliminate some need for this feature since lookups may now expand to the ":error:message text" form that the error director will match. E.g. perhaps a smartuser director could be configured with: smart_user_cmd="|/$lib_dir/cyrchkquota ${shquote:user}" and cyrchkquota would interactively check the current mailbox usage for $user and either return $user again, or return a form matched by the error director, e.g. ":error:$user is over quota!" (note this would impose and imply a process invocation on every "RCPT TO:" or similar verification activity) - think about inventing a new router driver that is like the queryprogram director driver but also is able to rewrite the local-part of an address as well -- it would be just like the rewrite router driver, but instead of static file lookups it could do dynamic lookups. It will definitely need to support the a "required" private attribute to allow its magic to be more efficiently restricted. - add new "quota" and "quota_threshold_warn" attributes to the appendfile transport driver both of which would be run through expand_string() so they could do file lookups. A non-zero "quota" would simply enforce the quota with a failure if file size would exceed the value after delivery. "quota_threshold_warn" would trigger delivery of a warning message to the same in_addr that triggered the current delivery if the size of the file crosses the given threshold after delivery completes. If "quota" is also set, the threshold may be specified as a percentage of "quota' by following the value with a percent sign. The warning message could be configurable with a "quota_warn_msg" attribute or similar. [idea from exim] [see BDB's contrib/patch.appendfile-maxsize for the partial implementation] - how can we do quotas for local delivery through the pipe driver, e.g. to a mail.local(8) LDA? Perhaps some trick with the queryprogram router driver could be used to check the local user's mailbox for quota conditions and it could re-route them to something that ends up at the error driver if they're over their limit. - this happens sometimes when multi-homed with multiple IP subnets on the same segment and connecting to a peer's "alternate" address.... 07/07/1999 17:47:12: [4931] remote EHLO: questionable operand: 'becoming.weird.com': from root@becoming.weird.com source [204.29.161.180]: Remote address PTR lookup failed (Unknown host). This would probably be fixed by always greeting with the name matching our actual source address [getsockname() and then gethostbyaddr() or getnameinfo()], and would re-invent/remove the meaning of 'primary_name'. - think about allowing $listen_name to be set on command line too [if this is used for more than one domain then you'll need separate config files anyay, so just use -C; but if you are using this to avoid having SMTP on some interfaces then this info may be easier to manage in one place in the /etc/rc* files or whatever]. - do something to make aliasfile parsing identical across lookup protos. (related to 'db lookup parser' bug above?) - Put the following in default.c for SVR4's local, pipe, & file transports: remove_header="Content-Length", append_header="${if !header:Content-Type :Content-Type: text}", append_header="Content-Length: $body_size", - think about how to integrate checkerr and savelog so that security violations can be snarfed from logfile just after it is cycled. Perhaps a new over-all maintenance script (smailmaint?) could do the work and there would only be one crontab entry necessary. Note that there's no need to use the antiquated savelog on systems that have a newsyslog(1) capable of not compressing the .0 file (eg. my version!). [syslog logging would also change all of this since then security violations will get higher priority from syslog if the admin so desires...] - add an "always" attribute to the directors drivers, esp. aliasfile. - add 'senders' and 'senders_except' attributes to directors and routers to implement restricted aliases, transports, etc. - implement lookup drivers/protocols through drivertab.c [already partly supported in mkdrivtab.sh] - think about allowing hostnames in match_ip() by doing a reverse lookup on the address and matching the resulting PTR(s) with any hostname patterns [regex's too, or just glob(3), or just domain suffixes?]. This would allow hostnames with dynamically assigned IPs to be added to lists like smtp_rbl_except or even smtp_remote_allow. Remember to always do the safe thing when no PTR is found -- i.e. return a code saying that a test was not possible (either temporary error indicator if DNS times out, or permanent if authoritative NXDOMAIN) and let the calling code can do the "safe" thing (eg. reject a relay attempt). - consider some way of specifying use of SPF records for a domain in lists like smtp_rbl_except (even though SPF is evil and stupid, it does at least offer a standard way for a domain to say how its outgoing mail might usually originate) (this of course also requires an SPF parser to be added/written, sigh.) - think about making smtp_remote_allow and other users of match_ip() and match_re_list() capable of specifying a file lookup mechanism in a list element: smtp_remote_allow="localnet:10/8:192.168/16:\ ${lookup:sender_host_addr:ipsearch{ /etc/smail/remote.allow}:$value}" where "ipsearch" iterates the [new] match_ip() function over all the values in the file. (does this mean keeping the double compare?) (the file should probably be cached in-core and treated as a list if it's not too big). See next item too about how to specify the variable containing the value being searched for instead of magically knowing what it is as in the above example. - think about adding a new magic variable name like "key" that the caller of expand_string() can set in a dummy addr structure so that expansion of ${lookup in a list-style variable can be done reliably without having to know what variable is used for matching (eg. with $sender_host_addr in the previous item and $sender in the next one) - think about propogating local_name from parent addr to cur so that it is easily accessible, e.g. in the pipe transport. - think about fixing parsing of all list-style variables so that they can all always optionally include an element that is run through expand_string(), in particular so that ${lookup can be used. The first trick here is in making sure there's some way to always specify the value being searched for in the list, and making sure each variable's definition documents the expected lookup variable(s). The second trick is making sure the expansion results in something useful so that the caller makes sense of the lookup result. For hostname and IP# lookups this should be as simple as expanding the the searched-for key if the lookup succeeds, or the key prefixed by '!' if not. + eg. this would allow collapsing smtp_sender_reject and smtp_sender_reject_db into just the former. In this particular example the caller would have to arrange to have the expected lookup variable set to the appropriate value: smtp_sender_reject:".*@[^@]*\\.localdomain;bogus domain!:\ ${lookup:sender:lsearch{dead-mail.senders} then {$sender;$value} else {!$sender}" - think about adding 'DNS' and 'RDNS' db search protocols for ${lookup. - think about how to get more detailed errors from ${lookup so that expand_string() doesn't just end up with an empty value. This would help pass DB_AGAIN and FILE_AGAIN errors out to the caller so that messages can be deferred (or failed) just like when a lookup fails in a router or director. Maybe a generic callback such as chk_expansion_error() would suffice? - Think about splitting lsearch and USE_LSEARCH_REGEXCMP into a plain old lsearch and a new "re_search" (is this a bad name? ;-) [JPR suggests "grep", how how about "grepsearch"?] for straight RE linear searches. We could turn off icase then too since PRCE has "(?i)". Think about not using double quotes to trigger the RE match in "grepsearch", but rather doing it for every key value. Think about a combined lsearch+grepsearch that would do what lsearch+REGEXCMP does now with the double-quote trigger. - fix the error messages in config file parsing to include at least the line number, and anything else helpful, not just: 05/07/1997 15:40:59: [13914] /etc/smail/config: parse error: unexpected end of attribute and not even just: 01/11/2004 11:27:39: [20640] /etc/smail/config: parse error: unexpected end of quoted string for variable more_hostnames The problem is that read_entry() can suck up more than one line at a time. If we had a global (or callback) to record the line number of the last line read then at least we'd do as well as GCC and similar. Note also that the newlines are gone by the time parse_entry() and parse_config() get ahold of the next chunk of text so they still couldn't point to the exact line of the error, but rather just the starting line of the entry they're working on. - the config file syntax needs some way to allow for appending to an existing value -- e.g. the default, or the value from the primary config when reading the secondary config, etc. ("+=" and maybe that's for prepending and "=+" is for appending?) - think about changing the "var" portion of the eq{ et al condition operators to be a fully expanded value, not just a variable name (which would make the eqic{ et al operators effectively redundant). - consider more high-level profiling and resource utilization support to find out where Smail spends the most of its time. For example would it make sense to optionally record more firmly the routing and transport selection in the spool file so that it doesn't have to be re-calculated? (re-calcuating routing has the advantage that stuff stuck in the queue can be delivered if the configation can be changed appropriately) - add support for Kiem-Phong Vo Vmalloc library, particularly debugging support [partly done]. Also add hooks to build with sfio (i.e. without the stdio layer). Is Vmalloc less prone to heap overflow/underflow exploits? - document ${eval: if it turns out to be useful anywhere but with -bP. [should test to see if it happens to work in the variable part of eq{ condition operators] - re-write aliasfile.c in the style of the fwdfile.c with a finish_*() function, etc. - make the startup log message more verbose (version, build, build date, release date, etc.) [use $smtp_banner ???] - figure out how to do the configuration for per-transport (or even per-target?) relaying control. - pass a flag to fill_attributes() so that it can print a more meaningful error message that indicates if an unknown attribute is expected to be either a generic attribute, or a driver-specific attribute (possibly either the word "generic" or the driver name). - the error message string returned by parse_header() doesn't indicate which header line the problem was with, never mind which specific address in the case of address parsing problems. - add configurable reserved space for spool_dirs ($min_spool_free?) (percent or absolute or either?) - consider trying to ensure all variables are run through expand_string() before their values are used. Much easier if expand_string() always returns newly allocated storage.... Optionally at least always run them through expand_string() in read_config_file() so that at least they could be built up in parts (especially for second_config_file). - if all vars are not expanded then consider especially visible_name being expanded, and/or having a default value, i.e. to have the same value as smart_path: visible_name=${if def:smart_path {$smart_path}} - keep a static copy of all default settings, or at least all defaults for config attributes, so that a command-line option, or a magic parameter to '-bP' can be used to print just the changed values. For now though it would be easiest to add a flag to each config attribute so that "changed" attributes can be displayed. I.e. if an attribute is read from a config file then it will be noted as "changed", even if the resulting value is the same as the default. Hmmmm... mabye if the config reader compares the new and current values it can actually set the flag properly without needing to keep static copies of anything! Maybe this can even be made to work for other config entries such as directors and routers and transports too! A verbosity flag could also control whether the defaults are printed as commented-out entries too! - for all string type attributes, especially those that are lists, allow a magic '$DEFAULT' expansion to include the original default value by referencing the static copy of the default setting. - add a way to supress warnings for smtp_helo_broken_allow. - checkerr should maybe try to find the original message-id for double bounces and look for related log entries for it too, then we could see right in its report the original source of the failing message. - look for more places where xprintf(), dprintf(), and str_printf() could use new '%S' (like %*s in printf(3)) could be used. - add more to the API defined in list.c/list.h and make more use of it. - think about how to safely and portably typedef uid_t and gid_t [autobuild?] - consider implementing "HELP command". - think about some way to support SysVr3.x's broken SIGCLD [really?]. - add new types to smailconf.c: enums, arrays of int, char*, long, etc. (also validate format of arrays of int & long to make sure all values are allowable and all ranges are correctly specified). - document the following useful remove_header transport attributes: Disposition-Notification-.* Read-Receipt-To Registered-Mail-Reply-Requested-By Return-Receipt-Requested Return-Receipt-To X-Confirm-Reading-To - allow rlimit settings to be changed by config -- in particular we should at least always accommodate message_buf_size. - we should probably not deliver any message which has no data (e.g. "." is sent immediately after "DATA" in an SMTP transaction). - consider making use of ENHANCEDSTATUSCODES in the smtp transport. - make sure that any other deprecated config attributes are marked with FA_FMT_DEPRECATED. - consider giving , and perhaps , and maybe some of the other RFC 2142 standard mailboxes, the same attention as so that they cannot be accidentally not accepted as locally deliverable. - consider avoiding smtp_greeting_delay if the peer address is in some list of addresses (maybe smtp_hello_broken_allow, though the main idea would be to speed up message reception from very busy peers) - consider adding a "Warning" header to each message that was accepted by way of a bypass list, perhaps with some/all configurable content. Note that we'd need a way to detect the matching of an explicitly negated expression in the various regex lists so that we could treat them as bypasses too. - consider allowing for rejection of clients when the MX for the greeting host matches any RHSBL (since this might indicate we won't be able to contact their postmaster reliably). - improve SMTP error responses to recommend contacting the recipient "by other means" whenever this is the best alternative. - see also: https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml New Features: ------------- These are primarily things that should wait for the next major release. - implement hooks to feed blocklistd with abuse info - implement message postmarks using DKIM (DomainKeys Identified Mail) draft-ietf-dkim-overview draft-ietf-dkim-base draft-ietf-dkim-ssp-requirements RFC 4686 - consider implementing support for Postfix-compatible policy daemons. - consider allowing full expand_string() expansion of the right-hand side of aliases (what are the security implications of allowing expansions to create multiple addresses, recursive eval, etc.?) - consider an option to return warning messages (notify.c) to (local-only?) senders when a (remote) delivery attempt is unsuccessful because of a temporary error. Where would this be done? Do temporary errors percolate back up into the transport invoker? Should they? - someone somewhere has AUTH patches for a version of smail that masquerades as "sendmail": home.axman.com, gomer.august.net 15:08 [10] $ telnet home.axman.com 25 Trying 216.87.129.60... Connected to home.axman.com. Escape character is '^]'. 220 axman.com sendmail ready at Wed, 21 Jul 2004 14:08:19 -0500 (CDT) HELP 250-The following SMTP commands are recognized: 250- 250- HELO hostname - startup and give your hostname 250- EHLO hostname - startup with extension info 250- MAIL FROM: - start transaction from sender 250- RCPT TO: - name recipient for message 250- EXPN
- expand mailing list address 250- DATA - start text of mail message 250- RSET - reset state, drop transaction 250- NOOP - do nothing 250- DEBUG [level] - set debugging level, default 1 250- HELP - produce this help message 250- QUIT - close SMTP connection 250- 250-The normal sequence of events in sending a message is to state the 250-sender address with a 'MAIL FROM:' command, give the recipients with 250-as many 'RCPT TO:' commands as are required (one address per command) 250-and then to specify the mail message text after the DATA command. 250 Multiple messages may be specified. End the last one with a QUIT. DEBUG 250 Debugging level: 1 EHLO building.weird.com 250-home.axman.com Hello building.weird.com (building.weird.com from address [204.92.254.24]), here is what we support: 250-SIZE 10240000 250-8BITMIME 250-PIPELINING 250-EXPN 250-AUTH LOGIN PLAIN 250 HELP quit 221 home.axman.com closing connection write_log:[8532] remote got QUIT from building.weird.com(building.weird.com) [204.92.254.24]. Connection closed by foreign host. - Implement LMTP support. It is like SMTP, except that - the client sends "LHLO client.host.domain" as its greeting - after the end of the message, a return code for every accepted "RCPT TO:" is returned by the server. - With LMTP in use the address verification has to go all the way through to doing an LMTP VRFY command to hopefully get over-quota notification right away. For efficiencly this will require caching the LMTP connection across SMTP transactions so there's only ever one per client SMTP connection. For even better efficiency this same connection should be used for final delivery if that happens using the same process that queued the incoming message(s). - Implement STARTTLS for port#25 and consider supporting SMTPS (SMTP over TLS/SSL on port 465) as well. Consider LibreSSL and PolarSSL (aka mbed TLS). - Implement SASL (with SASL options to require SSL?). - consider supporting the Postfix "XADDR" SMTP command (maybe only for connections where peer_is_localhost gets set?) - Easier debugging of SMTPD access restrictions. The SMTP command "XADDR client-address client-hostname" changes Postfix's idea of the remote client name and address, so that you can pretend to connect from anywhere on the Internet. - Think about a config variable (maybe $log_events?) that could control which items are logged and which are not [or wait for syslog support?] - consider implementing simple subscriber-only lists by verifying that the sender address is in the distribution list (if the director driver of the recipient is a forwardfile? or if the director's name is "lists"?) (what about supporting no-distrib lists for alternate sender addresses?) - implement RFC 3461 ESMTP DSN (maybe not NOTIFY=DELAY though) - implement RFC 3464 "MIME Delivery Status Notifications" for bounces. - implement at least some of RFC 2852, DELIVERBY, to allow clients to specify their own retry duration and to request notifications and traces. [how? more stored command-line parameters in the queue file?] - write a minimal mailstats replacement (new log file format only) [real stats, not just what logsumm does] - implement '-R' -Rstring Go through the queue of pending mail and attempt to deliver any message with a reci- pient containing the specified string. This is useful for clearing out mail directed to a machine which has been down for awhile. - implement ETRN from RFC 1985 ala the above (patch already available, but needs some performance enhancements and support for '-R'). - implement other standards-track SMTP extensions.... - possible make the daemon children change their ps command line text to show what they are currently doing (on systems where this is possible) - teach substitute() to recognize the variable names listed in conf_attributes, etc.(?) - consider deprecating the aliasinclude and forwardinclude director drivers in favour of a new "matchdriver" attribute in the lone genericinclude driver. This new attribute would work like the "matchdirector" attribute but would match all directors using the specified driver. The aliasinclude director entry would then use the genericinclude driver and have "matchdriver=aliasfile", etc. The code savings would be tiny but the documentation would be much cleaner! - it would be nice to have some kind of sanity checker that could work out conflicts between various directors -- i.e. some way to see if more than one director would match a given mailbox name. - implement an LDAP DB lookup protocol. - implement WHOSON via a DB lookup protocol. This would require variable expansion (eval) to be done for all those variables which can contain lists of IP/network values and thus which can currently use the magic "whoson" keyword, but this would also allow us to deprecate that magic while at the same time extending the mechanisms which can be used to look up values - consider adding an option to checkerr to turn off statistics reporting - consider adding a config var to hold more detailed local contact info (e.g. URL and/or phone #) that can be spewed in SMTP errors and bounce messages. - consider adding support for a "quarantine" queue like Sendmail's '-qQ' (unfortunately '-q' is already the queue_interval in Smail) - need some way to enforce a quota on the number of messages per day which a given client (as well as perhaps per sender address) can send. After the limit is reached then no more message would be accepted until some delay was reached. This might be implemented in the same fashion as the retry/smtp cache is implemented. Just create a file for each client and increment a counter there for each message accepted from the client. If the counter is too high then the last-modified date of the file vs. some delay variable determines when the next message can be accepted. Daily limits could be automatically reset with a simple script that removed all the files (older than some time) at midnight or whenever (people who send a lot of mail at midnight would have to rely on the delay mechanism to reset their counter). Perhaps some more complex windowing feature could be implemented to restrict the number of messages within a given time window (need an algorithm for that!). Miscellaneous: -------------- - use __func__ in debug, et al, messages??? - consider eliminating both $received_field and $from_field. - consider adding an "Auto-Submitted" field (RFC 3834?) to bounces. - consider ignoring blank lines in read_smtp_command().... - should we deprecate "localnet" support? it's not "CIDR" compatible and is just too dangerous to use safely in most environments these days of widely used CIDR-ized class-A networks unless we can also get the netmask of the receiving interface. - consider moving defer_delivery, fail_delivery, succeed_delivery, and error_delivery to log.c since they write log messages. - smtp pipelining causes the wrong error to be displayed with '-v1', but luckily the right error message is logged: delivery FAILED: fatal error from inet_zone_bind_smtp transport:\n503-5.5.1 'DATA' command must be preceded by 'RCPT TO:' command.\n503-5.5.1\n503-5.5.1 If you are seeing this message in a bounce, or in an alert box\n503-5.5.1 from your mailer client, etc., then your mailer software\n503-5.5.1 is not showing you the correct and most meaningful error message.\n503 5.5.1 Please report this error to those responsible for your mailer software. unlocking retry/smtp/205.207.148.251 and unlinking. write_log(SYS): Failed TO:com1324@aci.on.ca ORIG-TO:jschemmer@naturalwhite.com ROUTER:bind_hosts TRANSPORT:inet_zone_bind_smtp ERROR:(ERR152) fatal error from inet_zone_bind_smtp transport:\n559-5.0.0 Attempts to send to the address '' are being rejected.\n559-5.0.0 This address has not been accepted.\n559-5.0.0 Reason given: (ERR193) address '' failed: This mailbox has exceeded its allotted storage quota.\n559 5.0.0 A permanent failure has been logged. - remove nested includes from routers/bind.h and transports/tcpsmtp.h - do GC on xprintf() values passed to send_smtp_msg() in smtprecv.c - do GC on argv from read_message() in modes.c and smtprecv.c (implement free_argv() to free a vector of allocated char * arrays) - ensure that all (struct error *)->message strings are dynamically allocated so that free_error() can release them. - add documentation to each header describing what other headers it depends on (e.g. "log.h" needs "addr.h" for struct identify_addr) - investigate this weird log message fragment: ORIG-ID:<199604230758.AA13625@post.tandem.com\POS,$ZNET^U5> (possibly related: what'll happen if a message-ID header has other crap, and even continued lines, in it too?) - install ".so" (soelim) manual pages with their full longer names on systems with longnames (do we detect this feature dynamically, or do we rely on a configuration item?) [need to fix up xrefs too?] - we should add IsValid*() checking? from: [one place this should be done is in addr.c:check_target_and_remainder()] [[ or maybe re-implement it with a hand-written parser ]] - what about checking syntax of names retrieved from PTRs too? - read RFC-2821 and RFC-2822 even more carefully. - re-check use of all RFC 3463 ENHANCEDSTATUSCODES - consider implementing an "RFCS" command to list supported RFCs, etc. For example this result from Maillenium: RFCS 214- RFC -- description -- 214- 821 Simple Mail Transport Protocol 214- 822 Standard for ARPA Internet Text Messages 214- 1047 Duplicate Messages and SMTP 214- 1321 MD5 Message-Digest Algorithm 214- 1652 SMTP Service Extensions for 8bit-MIME transport 214- 1869 SMTP Service Extensions 214- 1870 SMTP Service Extensions for Message Size Declaration 214- 1891 SMTP Service Extensions for Delivery Status Notifications 214- 2195 IMAP/POP AUTHorize Extension for Simple Challenge/Response 214- 2197 SMTP Service Extensions for Command Pipelining 214- 2222 Simple Authentication and Security Layer (SASL) 214- 2246 The TLS Protocol 214- 2487 SMTP Service Extension for Secure SMTP over TLS 214- 2505 Anti-Spam Recommendations for SMTP MTAs 214- 2554 SMTP Service Extensions for Authentication 214- 2595 Using TLS with IMAP, POP3 and ACAP ( Auth Plain ) 214- 2821 Simple Mail Transport Protocol ( updates rfc821 ) 214- 2822 Internet Message Format 214 2852 Deliver By SMTP Extension - consider implementing a "VERS[ion]" command like Maillenium: VERS 250-version: V04.50c++ 250-compiled: 13-Jan-03 14:35:17 250-codebase: cpu_rs6000.os_aix.comp_ibm 250-developers: AT&T Labs (Middletown) - Maillennium 250-core_team: Steve Spear, Michael McGroary, Al Robinson 250-support: ALGOR LDAP(Open) MTA(switch) STUB 250-active: LDAP 250-up_since: Wed Jun 18 20:59:59 2003 250 snmp: compiled in / running Note though that ZMailer reports its version with the "VERB" command: VERB 250-2.0.0 ZMailer SMTP server 2.99.56 #1: Tue Apr 27 10:45:18 EDT 2004 250-2.0.0 Copyright 1990 Rayan S. Zachariassen 250 2.0.0 Copyright 1991-2003 Matti Aarnio - think about not stripping comments from aliases, etc., and providing GCOS info; esp. for EXPN and VRFY, perhaps re-using smtp_info to control. - Should the "real_user" director set ignore_alias_match? - consider allowing multiple whitespace characters to act as one when speparating words in a string parsed by expand_string(). - consider ignoring trailing whitespace on config entries that don't have quoted value settings. [already done?] - think about the possible benefits of having separate DBG_DRIVER types for each of the different kinds of drivers (router, director, transport). - clean up the duplication between COPY_STRING() and copy() -- maybe even call it smail_strdup()? - there may still be some minor memory leakage in 'mailq -s' - clean up remaining use of deprecated index(), bcopy(), whatever. - an interval of '1y' prints as '52w1d5h45m36s'. - consider a feature to have a dynamic list of hosts (i.e. a lookup db) that could be used when sorting MXs such that a 4xx response could be returned at RCPT time for any address in any domain that this host is secondary for. The list could then be regularly updated by an external script that tested whether the primary host was up or not and thus this secondary would only accept mail for the domain if the primary was down. Timing of the down check wouldn't have to be immediate since any messages deferred for a just recently down primary would still end up getting delivered eventually, either to this secondary if the next check happens before the next delivery attempt, or to the primary if the primary is up/reachable again before the next delivery attempt. Transient reachability problems between the primary and the client-SMTP are irrelevant since they will presumably go away before the delivery times out. This would allow one to implement true "backup"-MX service for longer-term customer outages without having to worry quite so much about filtering issues in these days of direct-to- secondary spamming. - note that when a user has a ~/.forward file that explicitly forwards their mail to their "local" address (e.g. so that it can be shared across NFS such that mail sent to them on a workstation will go to the central mail hub) the owner address gets changed to real-$user, which of course still routes to the same local address and so bounces due to over-quota problems on the mail hub will end up in the error queue. Maybe this is a good thing? - a client mailer of a hub should make visible_domain be the hub's domain.... - The "required" and "ignore" private attributes implemented by many/most router drivers should be deprecated and new common "only_match_domains" and "ignore_domains" attributes should be created in their place. (Note the bind driver borks "required" to have a non-standard meaning and has its own private match_domains hack instead.) - the typical BIND resolver does its debugging to stdout and smail doesn't trap stdout into the '-D' file so the output from the resolver still goes to the tty. The user could redirect to append to the same file of course.... - the "${strip:" meta-expander is really poorly named -- it should be named ${make_filename: or similar since that's what it really does. - investigate smail vs. MH and BCC/DCC. Note also that MH uses 'Dcc' instead of 'Bcc' for normal (direct) blind carbon and that this header may not be stripped when '-t' is used! - consider not stripping off angle brackets from addresses -- smail's parser doesn't need them, but they are a proper part of route-addrs and we would probably be better off always transforming addresses into true route-addr form. If so then watch out for places where config files and alias files, etc., use keys that don't have this form. Most places could optionally use that form, but lists files shouldn't. - consider trying to get backslash escaped mailboxes to work more like in sendmail where they prevent alias expansions. Need to come up with some justification for this behaviour first though.... - consider rejecting transactions with a null return path if the client host is matched in a list of hosts that typically send bogus bounces (or alternately in any of the RBL domains for such hosts). - Microsoft IDIOTS: 220 exchange1.ACC.WORKFORCE.COM Microsoft ESMTP MAIL Service, Version: 5.0.2195.1600 ready at Tue, 6 Feb 2001 13:05:01 -0800 EHLO proven.weird.com 250-exchange1.ACC.WORKFORCE.COM Hello [204.92.254.15] 250-TURN 250-ATRN 250-SIZE 250-ETRN 250-PIPELINING 250-DSN 250-ENHANCEDSTATUSCODES 250-8bitmime 250-BINARYMIME 250-CHUNKING 250-VRFY # it's _NOT_ optional! 250-X-EXPS GSSAPI NTLM LOGIN 250-X-EXPS=LOGIN 250-AUTH GSSAPI NTLM LOGIN 250-AUTH=LOGIN # bogus -- it's just "AUTH" 250-XEXCH50 250-X-LINK2STATE 250 OK # what's this BS!?!?!? quit 221 2.0.0 exchange1.ACC.WORKFORCE.COM Service closing transmission channel - Maillenium idiots too: 220 prserv.net - Maillennium ESMTP/MULTIBOX in2 #13 EHLO proven.weird.com 250-prserv.net 250-7BIT # bogus -- that's the default! 250-8BITMIME 250-DSN 250-HELP 250-NOOP # bogus -- this isn't an option! 250-PIPELINING 250-SIZE 10485760 250-VERS V04.50c++ # what the hell? 250 XMVP 2 - the bouncemail script doesn't always (e.g. with hostname lookup failure) Unlikely: --------- - add "write" methods to the DB interface. This might allow us to implement things like greylisting (and rate limiting too?) via an abstract interface: smtp_greylist_write = "${write:smtp_greylist_key:whoson}" smtp_greylist_lookup = "${lookup:smtp_greylist_key:whoson}" - consider adding some way to deal with VERP-style sender addresses (especially those conformant to RFC 3464) when doing greylisting so that every post from a list won't be greylisted. - consider adding some way to deal with client-SMTP pools so that mail from a given sender won't have to be greylisted every time a new sever from the pool is used as the sender's relay host. It seems at some large domains, e.g. AOL and Amazon, sometimes multiple servers even share the same queue, at least this is possible according to the notes published at - disable all of the whoson expansion checks when smtp_greylist_using_whoson is not set - figure out some way to tell how many greylisting entries are never used again -- suggesting they are responsible for stopping at least one unwanted transaction. (a count of 0 (and first_use == last_use) and first_use time > smtp_greylist_maxtime, but we need a "list" function for whosond before we can tally them up) - include the whoson source, or just the patch? [nope -- greylisting is stupid]