Rob Reynolds - The Fervent Coder : SidePOP, HowTo

Lessons In Building An Email Parser

Rob Reynolds — Fri, 18 Dec 2009 09:11:00 GMT

The Classic Infinite Email Loop

When building an email parser, one must think about validating an email address that sends you a message. Why? Just because an email message has a from address does not already mean that it is a valid email address. When you're building an auto-responder, not validating an email can cause an email loop. Some of you can already see where I’m going with this.

I was building something in Bombali that would respond to emails when receiving them. When Bombali received a message from an address, Bombali would send a response. No checking on the address. In the world of computers, this is a classic mistake. In my head I hear the word "classic" as if it came from SNL and the "Classic Peg" skits. Classssssss-ic (with laughing and shoulder shrugging) Mistake. Right? Well this is all new to me, I’ve never been able to do this programmatically until recently. So Bombali received a message with a no reply from address. It sent a response. Then it got a response from the postmaster saying that this was an invalid address. I bet you’re guessing what happens next? Yes, Bombali responded to the postmaster. And then the postmaster responded back. Over a matter of about 30 minutes there was quite a bit of traffic back and forth until I deleted the emails before Bombali could check them. At least I was smart enough to set the email checking to every half minute.

How did I get past this? Validation. An authorized list form of validation. Interested? Read on dear reader.

SidePOP XmlConfigurator

I was exposing my configuration in the last post, so I built an XmlConfigurator that does this for you and gives you back a list of Email Watchers (renamed SidePOPRunner).

private void configure_mail_watcher()
{
    EmailWatcherConfigurator configurator = new SidePopXmlConfigurator();
    foreach (EmailWatcher emailWatcher in configurator.configure())
    {
        emailWatcher.MessagesReceived += runner_messages_received;
        emailWatcher.start();
    }
}

Subscriber for Receiving Email

The code above and the code below show how easy it is for your application to receive email. I map the email to a local object and then I pass that off to a function to parse and send a response.

private void runner_messages_received(object sender, MessageListEventArgs e)
{
    IEnumerable<SidePOPMailMessage> messages = e.Messages;

    foreach (SidePOPMailMessage message in messages)
    {
        Email mail_message = Map.from(message).to<Email>();
        parse_and_send_response(mail_message);
    }
}

So now I am receiving email, but I need to parse it do determine what to do. So let’s show the parser first before the method for sending a response.

Really Simple Email Parser

This is a really simple parser. It’s not by any means what some people would use in production and that’s fine. It works for what Bombali needs.

public MailQueryType parse(Email message, IList<IMonitor> monitors, IDictionary<string, ApprovalType> authorization_dictionary)
{
    MailQueryType query_type = MailQueryType.Authorizing;

    string user = message.from_address.to_lower();
    bool user_is_authorized = false;

    if (authorization_dictionary.ContainsKey(user))
    {
        ApprovalType user_is_approved = authorization_dictionary[user];
        if (user_is_approved == ApprovalType.Approved)
        {
            user_is_authorized = true;
        }
    }

    if (user_is_authorized)
    {
        query_type = MailQueryType.Help;
        string subject_and_body = message.subject + "|" + message.message_body;

        if (message_contains_status(subject_and_body)) query_type = MailQueryType.Status;
        if (message_contains_config(subject_and_body)) query_type = MailQueryType.Configuration;
        if (message_contains_down(subject_and_body)) query_type = MailQueryType.CurrentDownItems;
        if (message_contains_approve(subject_and_body)) query_type = MailQueryType.Authorized;
        if (message_contains_deny(subject_and_body)) query_type = MailQueryType.Denied;
        if (message_contains_version(subject_and_body)) query_type = MailQueryType.Version;
    }

    return query_type;
}

I have a MailQueryType so that I separate what how to respond from actually responding. What you can see is that I’m already checking to see if we have an approved user before I set a different MailQueryType. The mail parser just sends back how Bombali should respond to the message.

Sending a Response

In subscribing to receiving email, I passed the message to the method below. The second line of the method shows me passing the message off to the mail parser to find out how to respond. The switch could probably be replaced by a Strategy pattern, but for now it works and it’s all I need.

private void parse_and_send_response(Email mail_message)
{
    string respond_to = mail_message.from_address.to_lower();
    MailQueryType query_type = mail_processor.parse(mail_message, monitors, authorization_dictionary);
    Log.bound_to(this).Info("{0} received a message from {1} of type {2}.", ApplicationParameters.name, respond_to, query_type.ToString());

    string response_text = string.Empty;

    if (query_type == MailQueryType.Authorized || query_type == MailQueryType.Denied)
    {
        string[] body_words = mail_message.message_body.Split(' ');
        foreach (string body_word in body_words)
        {
            if (body_word.Contains("@"))
            {
                respond_to = body_word.Replace(Environment.NewLine, "");
                break;
            }
        }
    }

    switch (query_type)
    {
        case MailQueryType.Denied:
            authorization_dictionary.Add(respond_to, ApprovalType.Denied);
            return;
            break;
        case MailQueryType.Authorized:
            authorization_dictionary.Add(respond_to, ApprovalType.Approved);
            response_text = string.Format("Congratulations - you have been approved!{0}Send 'help' for options", Environment.NewLine);
            break;
        case MailQueryType.Help:
            response_text =
                string.Format(
                    "Options - send one:{0} help - this menu{0} status - up time{0} config - all monitors{0} down - current monitors in error{0} version - current version",
                    Environment.NewLine);
            break;
        case MailQueryType.Status:
            TimeSpan up_time_current = up_time.Elapsed;
            response_text = string.Format("{0} has been up and running for {1} days {2} hours {3} minutes and {4} seconds.", ApplicationParameters.name,
                                          up_time_current.Days, up_time_current.Hours, up_time_current.Minutes, up_time_current.Seconds);
            break;
        case MailQueryType.CurrentDownItems:
            response_text = string.Format("Services currently down:{0}", Environment.NewLine);
            foreach (IMonitor monitor in monitors)
            {
                if (monitor.who_to_notify_as_comma_separated_list.to_lower().Contains(respond_to))
                {
                    if (!monitor.status_is_good) response_text += string.Format("{0}{1}", monitor.name, Environment.NewLine);
                }
            }
            break;
        case MailQueryType.Authorizing:
            response_text =
                string.Format("Bombali notified admin to add you to approved list. If you are added, you will receive a response.");
            break;
        case MailQueryType.Version:
            response_text = string.Format("Bombali is currently running version {0}.", Version.get_version());
            break;
        default:
            response_text = string.Format("{0} has not been implemented yet. Please watch for updates.", query_type);
            break;
    }

    SendNotification
        .from(BombaliConfiguration.settings.email_from)
        .to(respond_to)
        .with_subject("Bombali Response")
        .with_message(response_text)
        .and_use_notification_host(BombaliConfiguration.settings.smtp_host);

    if (query_type == MailQueryType.Authorizing)
    {
        SendNotification
        .from(BombaliConfiguration.settings.email_from)
        .to(BombaliConfiguration.settings.administrator_email)
        .with_subject("Bombali Request")
        .with_message(string.Format("{0} reqests approval. Send approve/deny w/email address. Ex. 'deny bob@nowhere.com'", respond_to))
        .and_use_notification_host(BombaliConfiguration.settings.smtp_host);
    }

}

You’ll notice that I’m authorizing any new email addresses that send an email to Bombali before allowing them access to the application. I do that by having an administrator contact I can send a request for approve or deny to before allowing access. I also send an email back to the sender letting them know they are in the process of being approved. At most I would send one email to a bad address. As administrator, the email can be validated by a human. And I only respond to an authorized list. I could also build in something to validate the email by checking for a response from a post master telling me the address doesn’t exist.

So remember readers - always check your sources. Otherwise you might get in an infinite loop. And that would be bad.

How To Check Email Programmatically - SidePOP

Rob Reynolds — Tue, 08 Dec 2009 06:46:00 GMT

Background

Sending email has long been easy to do with the .NET Framework. There really hasn’t been a facility for checking email though. I recently had a need to be able to check email with and send a response with Bombali (a monitoring tool). I went searching for examples or a solution I could use. I found a few articles on this from a few years ago including the .NET POP3 MIME Client. I never found anything that was free and I never found anything that was easy to configure. Those who know me know I have a tinge of the NIH (not invented here), especially when there are no free alternatives. So I created my own based on the articles I had read. And in the spirit of chucknorris, I named it SidePOP!

Configuration

Configuration for receiving email should be easy. Feast your eyes on all that is required for configuration (and for those of you who dislike XML, it’s not required – you can configure all in code):

<configSections>
  <section name="sidepop" type="sidepop.configuration.SidePOPConfiguration, sidepop"/>
</configSections>

<!-- 110 is normal POP3 SSL uses port 995-->
<sidepop>
  <accounts>
    <add hostName="mail.somewhere" userName="yep" password="" minutesBetweenChecks=".1" />
    <add name="Number1" description="Main account" enabled="true" hostName="mail.somewhere.net" hostPort="110" useSSL="false" userName="" password="" minutesBetweenChecks="1" timeoutInMinutes="1"  />
  </accounts>
</sidepop>

You see there are two items here. The first one has the required items in it. The second one is the entire set of items to configure. Notice that we are asking for a password in plaintext. This is where you might think about encrypting this section somehow.

Code Setup

Configuring SidePOP to run is a little more verbose that I might like right now. Looking below you see that it’s still not that much code. I hope to set up a facility to automatically look at the configuration file and configure SidePOP (for those that like XML). If someone wants to send me a patch as well, that would be awesome.

private void configure_mail_watcher()
{
    foreach (AccountConfigurationElement account in SidePOPConfiguration.settings.accounts)
    {
        if (account.enabled)
        {
            SidePopRunner runner = new SidePopRunner(new DefaultPop3Client(account.hostName, account.hostPort,
                                     account.useSSL, account.userName,
                                     account.password), account.minutes_between_checks);
            runner.MessagesReceived += runner_messages_received;
            runner.run();
            Log.bound_to(this).Info("{0} is configured to watch for messages with user {1} at {2} every {3} minutes.", ApplicationParameters.name,
                                    account.userName, account.hostName, account.minutes_between_checks);
        }
    }
}

private void runner_messages_received(object sender, MessageListEventArgs e)
{
    IEnumerable<SidePOPMailMessage> messages = e.Messages;

    const string subject = "Bombali Response";
    TimeSpan up_time_current = up_time.Elapsed;
    string text_message = string.Format("{0} has been up and running for {1} days {2} hours {3} minutes and {4} seconds.", ApplicationParameters.name, up_time_current.Days, up_time_current.Hours, up_time_current.Minutes, up_time_current.Seconds);

    foreach (SidePOPMailMessage message in messages)
    {
        Log.bound_to(this).Info("{0} received a message from {1}. Responding that service is still running.",ApplicationParameters.name,message.From.Address);
    
        SendNotification.from(BombaliConfiguration.settings.email_from).to(message.From.Address).with_subject(
        subject).with_message(text_message).and_use_notification_host(BombaliConfiguration.settings.smtp_host);
    }
}

When I start the service, I call the method configure_mail_watcher(). Because I can have multiple accounts I can check with SidePOP, I am going to loop through the settings and check each one to ensure they are enabled. If so, then I am going to set up a SidePopRunner and subscribe to runner_messages_received (this may not work with multiple accounts, I haven’t yet tested that). Then I kick off the runner for that account.

I could instead subscribe to runner.MessageReceived (instead of runner.MessagesReceived) for getting an event notification for every message instead of batching it up to a list if I desired.

When a message is received, I find out how long the service has been up. I didn’t show it here, but up_time is a System.Diagnostics.Stopwatch. In the constructor I start it. Now I just capture how long it has been up. Then for each message, I send a response to the sender of the email address telling them how long the service has been up. To add more sophistication, I could very easily add a rules engine here to process the message based on a list of rules looking for keywords in the email.

That’s all I need to do. Now I’m up and ready to rock.

What It Looks Like

Here is what it looks like running with Bombali. I send the service a text message. This is what it logs.

2009-12-07 23:59:05,875 5 [INFO ] - Bombali received a message from someone @ somewhere.net. Responding that service is still running.
2009-12-07 23:59:05,875 5 [INFO ] - Sending email to someone @ somewhere.net with subject "Bombali Response" and message:
Bombali has been up and running for 0 days 0 hours 31 minutes and 22 seconds.

This is what I get from my phone.

Now I know that my service is still up and running with a simple text. All for the low price of a few lines of code and the sidePOP.dll library.

Conclusion

SidePOP gives me a level of sophistication that I didn’t have before. Now that I have access to get email, I could very easily add rules to do different things based on the content of the email received. How often have you wanted your application to check email and just didn’t want to pay the price for the products out there that can do this? Now it’s completely free with SidePOP.

Download SidePOP

NOTE: This is still early. Any part of the configuration could change. Be sure you take a look at the sample app to always have the newest configuration. Register any bugs you find here: http://code.google.com/p/sidepop/issues/list

WARNING: SidePOP will delete your email when it checks it. That’s how it can be sure it’s only dealing with new messages every time. Do not test on an account you care about. You’ve been warned.

SVN here: http://sidepop.googlecode.com/svn/trunk/

Downloads here: http://code.google.com/p/sidepop/downloads/list