How can I use Nagios to monitor ASSP?

Well, for starters you can check whether the service is running at all. Define a command like this:

define  command {
        command_name    check_snmp_proc_by_name
        command_line    $USER1$/check_snmp_process.pl -C public -2 -H $HOSTADDRESS$ -n $ARG1$ --warn=0,1 --critical=0,2
        }

Obviously you will need the relevant plugin installed, I think this one is pretty much standard on most distros.

Then use it like this:

define service {
                use                             normal-service
                host_name                       you-mail-server
                service_description             AntiSpam SMTP Proxy
                check_command                   check_snmp_proc_by_name!assp
                }

You will need to replace "assp" in the above with the actual name of the daemon running.

That's the simplest check. If you want to do more than that, you could obviously write your own shell script that tries to send a spammy email through the proxy and then checks that the message goes where it should go (depends on your system configuration). The nagios website has documentation on what sort of values this script needs to return to nagios. Most likely you will have to run that shell script on the server where the daemon runs (in order to be able to check the results). But you can use the check_by_ssh plugin to trigger it remotely and transfer the results back to your nagios server.

I suggest to monitor the overall result: Have a service send an email e.g. 15 min and check that it arrives. (or arrives not, if you send a spammy email).