DEMYLTIFY: a milter library in Common Lisp

This is a Common Lisp library to implement milters. A milter is a Sendmail filter (hence the contraction); a daemon program that extends and augments the Sendmail functionality and implements features that are not provided by Sendmail itself, such as spam filtering, virus protection, mail archiving, mailing lists etc. Matter of fact, much of the logic behind Sendmail routing and access control could, in fact, be off loaded to a milter or a composition of milters.

Milters are usually C programs linked to the libmilter library, which comes with Sendmail. Interfacing to such library is not always an option, especially for many Lisp systems.

The libmilter library implements the milter protocol, the (de)serialisation of the data and the multi-threading. This is what demyltify does as well, in a more lispy style.

Milters written with demyltify don’t need to be multi-thread. It’s up to you whether, in the ON-CONNECTION callback, to spawn a new thread, fork a new process, or simply do nothing special to handle Sendmail connections.

How it works

The program calls START-MILTER passing a port number and a function. The milter library binds a socket to that port and waits for Sendmail connections.

For each connection, the milter library calls the callback function that was provided to START-MILTER, passing a socket. The callback, in turn, must call SERVER-LOOP with the context object that usually will contain further state data and milter options such as the protocol options.

On each event received from Sendmail, the library calls the relevant handler (method). Each event method accepts an event object and a context object, and returns an action object.

To install and compile

• link demyltify.asd into your ASDF system directory
• start your Lisp
• (asdf:oos 'asdf:load-op :demyltify)

Usage

To use this library, all you have to do is:

• write your own context class inheriting from MILTER-CONTEXT
• specialise the HANDLE-EVENT methods on your MILTER-CONTEXT class for all the events you care about (the default definition will simply let any mail through)
• call START-MILTER

The default options negotiation method will signal an error condition if the MTA doesn’t fully support the milter prerequisites. This is a sensible behaviour considering that, if the MTA doesn’t match the performed actions and required events of the milter, there is very little the milter can do about it; it will simply not work.

HANDLE-EVENT methods must return an action symbol or object which will be sent to the MTA. The action without arguments are specified as keywords. Those are

• :CONTINUE get on with the next event
• :ACCEPT accept the message
• :REJECT bounce the message
• :DISCARD silently ignore the message
• :PROGRESS hang on, the milter is performing some lengthy computation
• :TEMPORARY-FAILURE the message can’t be processed by the milter because of a temporary problem

This library is mostly stateless, so the program, if needs to, is responsible to save its state in the context object. To do that you are supposed to write your own context class which inherits from MILTER-CONTEXT and pass it to START-MILTER.

The lifetime of a context object is the same as the Sendmail connection. The user program has to make sure that it resets whatever state, in the context, that is message-specific, at every message boundary. Usually good places are the MAIL or the END-OF-MESSAGE / ABORT handlers.

START-MILTER is a procedure that never exits under normal circumstances. It enters a loop serving MTA connections on the specified socket. It is appropriate for the ON-CONNECTION function to fork or fire a new thread. You don’t need to use START-MILTER, if you want to write your own server function, go ahead, but for most practical purposes it does what you need to connect to Sendmail.

The event handlers

The event handlers are CLOS methods specialised on the event type and the context.

;; here we add up the byte count per message
(defmethod handle-event ((e event-body) (ctx my-context))
  (incf (ctx-byte-count ctx) (length (event-body-data e)))
  keep-going)

;; at the beginning of each message we reset the counter
(defmethod handle-event ((e event-mail) (ctx my-context))
  (setf (ctx-byte-count ctx) 0)
  keep-going)

The events a milter can handle are:

• EVENT-ABORT when Sendmail aborts the current message (others may follow)
• EVENT-BODY a chunk of the message body (passed the headers)
• EVENT-CONNECT when a client MTA connects to our Sendmail
• EVENT-DATA marks the beginning of the message body
• EVENT-DISCONNECT Sendmail wishes to disconnect but it will connect again later
• EVENT-END-OF-HEADERS to signal the end of the email’s headers part
• EVENT-END-OF-MESSAGE at the end of a message body
• EVENT-HEADER for each email header
• EVENT-HELLO when Sendmail sees a HELO from its client
• EVENT-MAIL when Sendmail receives a MAIL command from its client
• EVENT-QUIT when Sendmail asks the milter to lay down and die
• EVENT-RECIPIENT for each recipient on the email envelope
• EVENT-UNKOWN invalid SMTP command from Sendmail’s client

Internally the milter library handles the following events. In normal circumstances you shouldn’t bother with them:

• EVENT-DEFINE-MACRO definition of symbolic values that supplement other events
• EVENT-OPTIONS negotiation of event and actions between Sendmail and the milter

A context class derives from a MILTER-CONTEXT like this:

;; we specialise the context to add the byte count per message
(defclass my-context (milter-context)
  ((byte-count :accessor ctx-byte-count)))

Milter start

To start the milter you simply call start-milter and you pass the internet port and the connection callback. The callback will be called with a socket as argument and, in turn, it should call server-loop passing a milter context of your choice. Example:

(defun start-milter-loop (socket)
  (be context (make-instance 'my-context
                             :socket socket
                             :events '(:mail :body)
                             :actions '(:add-header))
    (server-loop context)))

(defun start-my-milter ()
  (let ((*log-file* #P"mymilter.log"))
    (start-milter 20025 #'start-milter-loop)))

Macros

Sendmail before some events passes some additional data to the milter. This data is in form of values associated to a symbolic name (macro) such as mail_host, _ (the connection host), rcpt_mailer, rcpt_host, etc. An association list, at the end of the day.

A milter may access these values with the GET-MACRO function, passing the current context and the macro name as a string. Example:

(let ((host (get-macro ctx "_")))
  (format t "Got connection from ~A~%" host))

In an EVENT-RECIPIENT handler method it might be used like this:

(defmethod handle-event ((event event-recipient) (ctx my-context))
  (push (make-recipient :address (extract-mail-address (event-recipient-address event))
			:mailer (get-macro ctx "rcpt_mailer")
			:host (get-macro ctx "rcpt_host"))
	(ctx-my-recipients ctx))
  :continue)

Sendmail configuration

To install a milter in Sendmail, in /etc/mail/sendmail.mc, you have to add a line like these:

INPUT_MAIL_FILTER(`filter1', `S=unix:/var/run/demyltify.socket, F=T')
INPUT_MAIL_FILTER(`filter2', `S=inet:20025@localhost, F=T')

and compile the .mc into a .cf file:

cd /etc/mail
make
make install restart

Then make sure you use the same address in the call of START-MILTER:

(start-milter #P"/var/run/demyltify.socket" #'my-connect-callback)
(start-milter 20025 #'my-start-milter-loop)

The F=T flag tells Sendmail to treat milter-related errors (ie milter not listening or crashing) as temporary. Read the Sendmail’s cf/README file if you need further details.

Sendmail does not start the milters. You have to do that yourself at boot time (anyhow, before Sendmail needs them to process a message).

See also

Some sample code is in the examples directory:

• simple.lisp is a milter that counts bytes in messages
• threaded.lisp is the threaded version of simple.lisp
• forked.lisp is the multi-process version of simple.lisp

The following pages could be useful to understand what a milter is and what it does:

• http://www.sendmail.com/partner/resources/development/milter_api/
• https://www.milter.org/developers/api/index

There is also a version of this library for Clojure, which is available on GitHub at http://github.com/fourtytoo/demyjtify

Gotchas

This work is based on an informal description of the undocumented Sendmail-milter protocol. This code may therefore be outdated right now, as the Sendmail folks don’t want you to mess with their protocol. They rather want you to use their pthread-based libmilter library in C. Although, in practice, it’s unlikely that this code will be invalidated by the next few Sendmail versions, you never know.

This code has been tested on SBCL, CMUCL and CLISP. Porting to other Lisp systems should be fairly easy.

Credits

Credit should be given to Todd Vierling (tv@pobox.com, tv@duh.org) for documenting the MTA/milter protocol and writing the first implementation in Perl.

License

Copyright © 2004-2015 Walter C. Pelissero

Distributed under the GNU Lesser General Public License either version 2 or (at your option) any later version.