Mail Theory

From Wiki99

Jump to: navigation, search

↑ Computers ↑
← prev: TCP/IP Server Setup next: Mail postfix and IMAP Setup →

Why Run Your Own Email Server?

Why would you want to run your own email server? You should be aware that running an email server requires some work, and it is something you really want to keep active all the time. If your web server, for example, goes down while you are on vacation, sure it's a little irritating for the fans who visit your web site, but it's not the end of the world. However if your email server goes down, this could result in real problems given how dependent many of us now are on email.

In the past there was more of a reason to run your own email server, but with GMail that is less of an issue. I would recommend that you read this article to understand what an email server entails, then ask yourself whether or not GMail would serve your needs. You can mix and match some of the ideas I provide below; for example one possibility would be to set up postfix on your server to handle outgoing email, but not to handle incoming email.

There are two main reasons to run your own server, and if your needs are simple you may only need to do half this work. This is reflected in the fact that an email server does not run one program but a few different programs, and you may not need to install all of them. These two main reasons are

control of outgoing mail (Send outgoing email from anywhere) and
control of incoming mail (Handle email consistently across multiple computers).

Along with these main reasons, you may more specialized interests, generally in the category of incoming mail, for example:

you may want multiple email addresses,
you may want to run your own spam filter.

Send outgoing email from anywhere

The first big reason you might want to run your own email system is to be able to send email from anywhere. (Specifically this refers to sending out email using Apple's mail program Mail.app. If you send out email using Gmail or some other web-based mail service like Yahoo or Hotmail, none of this applies to you.)

The problem is this: Most ISPs only allow you to send email over their network if you have an account with them. (This is to try to deal with spammers.)
Suppose you are travelling to another city and manage to get an internet connection, maybe at a friend's house, maybe through a public wireless access point. You should be able to download your mail into Mail.app. However when you try to send out a reply, you quite likely will get a message from Mail.app saying that the smtp server isn't working and asking if you would like to use an alternative smtp server. What is happening is that Mail.app is connecting to your ISP's smtp server and asking it to send out this piece of mail, but that server is configured to reject connections from any computer that is not directly connected to the ISP's network.

Now you may be able to deal with this problem in simpler ways than setting up your own email server.

You may, if you don't travel very much and don't use email much, just decide it's no big deal, and on the rare occasions that it happens you'll start up a Gmail or Yahoo mail session and reply to an urgent email message using some webmail service.
If you subscribe to .Mac you may not know it, but .Mac has a web interface to your email account, so you can switch to that and use that to reply, rather than using Mail.app.
You may have an ISP which is enlightened enough to provide authenticated smtp. (Earthlink does so. I don't know which other ISPs do so.)
In that case you simply need to fill in the appropriate fields in Mail.app's server settings (click the check box that says Use SSL, fill in the correct name for the mail server and your username and password) and now you can send email using that smtp server from almost anywhere. (Some networks are so stupid they block authenticated smtp packets. I've never seen this in the US, but I have seen it in Asia.)

But if these cases don't apply to you, maybe you want to set things up so that wherever you are and whatever network you are connected to, when you send out mail it goes out without complaint.

Handle email consistently across multiple computers

The second big reason for running your own mail server is that you want a consistent view of your mail across multiple computers. You want your work computer and your G5 tower and your portable all to show the same mail that you received, and sent.
Again if you only use webmail this is not a problem, but most of us don't use only webmail.
Again, if you are happy with .Mac this is not a problem. .Mac actually uses the same IMAP technology I am going to describe in detail, and is a very nice solution to the problem, if you are willing to pay the .Mac subscription fee and if your email fits within the 15MB or whatever it is these days that they provide for you. However if your email frequently includes large attachments, .Mac may not be quite practical for you.

Theory Of How Email Works On The Internet

With those motivations in mind, let's discuss email in the abstract before we get to details. Even more so than with our discussion of IP addresses and domain names, if you understand how all this stuff works, it's pretty easy to fix problem, while if you do not understand things, you can have a real problem just following a set of instructions for a machine that is setup just a little different from yours when something happens that doesn't match your list of instructions.

Before you do anything else, while reading this article, note the following:

DO NOT START A COPY OF MAIL.APP RUNNING, EITHER ON THE SERVER OR ON ANY OTHER COMPUTER THAT COMMUNICATES WITH THE SERVER UNTIL WE ARE COMPLETELY DONE.

If you do not follow the above instructions, the result will (probably) not be a complete disaster, but you will confuse yourself, and you will fill the folders on your hard drive that store email with various bits of garbage that waste space.

The email client

Sending email

How does an email client send mail? The answer is that it delivers each piece of email to a program running on some computer somewhere. That computer sees if the mail is addressed to that computer, and if not figures out from the email address some other computer to send it to. Usually that is the computer to whom the mail is addressed, but if not that second computer repeats the process. And so after bouncing through a few computers the mail reaches its final home.
(This is obviously pretty much exactly how packets travel through the internet. The difference for email is that email works at a more abstract level than TCP/IP. It uses addressing based on strings (like DNS) rather than numbers, and it delivers an entire email message, of arbitrary size, rather than a packet whose maximum size is limited. For this reason "internet" email can run over networks that do not TCP/IP. In the late 80s and the 90s this was important because many computers that did not use TCP/IP were nevertheless able to send mail to and receive mail from computers that did use TCP/IP. Nowadays, when pretty much every computer runs TCP/IP, this is much less important and the truth is that the details of the email protocol over the internet could probably be simplified a bit to remove some of the material that is only there to support older non-TCP/IP computers. But given that we have something that works and that everyone agrees upon, annd given the fighting that would occur if we tried to change it, we're probably stuck with what we have for at least the next twenty years or so.)

In a sense, the previous explanation is obvious. But what happens in more detail?

The protocol used to send mail is called smtp (simple mail transport protocol). Any computer that can receive and forward mail runs a background program that listens to port 25, the smtp port. When you set up an email account in Mail.app, one of the things you specify is the outgoing mail server, which usually has a name like smtp.comcast.net, and a port (which is usually 25), and probably a userID and a password so random spammers can't send mail out using that mail server.

For reasons we will get to, this combination of userID and password is not very good security. For that reason, as mentioned above, most ISPs also won't allow anyone to send out mail through their computers unless that person is also directly connected to their network. That way, if you are a spammer, because you are directly connected to their network they can shut down your account. If you were connecting from some random place in the world, they would have no way to stop you.

So it should now be obvious what happens. You ask your email client to send out a piece of mail. It sends a message to the computer and port you named when you set up the email account describing the piece of mail. (This computer the mail is sent to is called the smtp server because it is running a program, listening to port 25, that understands smtp). It's then the smtp server's job to figure out from the email address (including things like CC: and BCC:) how to get that piece of mail to where it is supposed to go. (If multiple addresses are specified, this may involve sending out multiple copies of the incoming message.)

Now remember one of the annoyances we wanted to fix was to be able to send out mail from our portable no matter where we were in the world and what network we were connected to. What we want to do is set up our personal email server to be able to receive mail from our portable anywhere in the world. That server, because it is permanently connected to comcast.net or whatever ISP we use, can then send the mail out to the comcast.net smtp server, and that comcast smtp server will figure out how to deliver it to rest of the world.

What we will do is a two-stage process, that I'll describe here briefly and in much more detail later.

First:

We need to have running on our server a program that listens for messages (incoming mail) on the smtp port, port 25, and sends those messages out to our ISP's smtp server.
The program we will use for this is called postfix, and we need to get it up and running and give it a small amount of configuration.
Part of the configuration is that we will set up postfix so that it will not send out any email that arrives at port 25, except email that originated from the server itself. This means that outside spammers can't get to send mail to the world through our server. They can certainly connect to our smtp server on port 25, but they cannot tell it to forward mail, because we have set up postfix to only forward mail (ie send mail to the outside world) coming from the server itself.

Second:
This seems a little strange, and not very useful.
If the only mail that the server can send to the outside world is mail that comes from the server itself, how does that help us when we're somewhere far away with our portable?

So the next part of the process is to set things up so that mail that we send from our portable gets to our server through what's called an ssh tunnel, and so looks to our server like that mail originated on the server.

Receiving email

Now let's now look at the other part of the problem, the receiving of the email. After bouncing through some smtp servers as described above, some email addressed to fred71@comcast.net finally lands up sitting on some computer owned by comcast in the comcast.net domain. When you, fred71, want to read your email, you want your computer to connect to comcast and suck down (ie receive) that email.

There are two common protocols for receiving email, called POP and IMAP. The two behave rather differently.

POP

POP is what most people think of as normal email. Your email goes to some central server at your ISP. Every so often you run your email client, it connects to the ISP and sucks down your mail, and now your mail is on your computer, and is no longer on the ISP's computer.

The problem with POP is that the email gets downloaded onto whatever computer you read mail on, and if you have multiple computers where you read email, your email is spread over all of them. There is a hack with POP where you set things up so that even after you download mail on one computer, the mail stays on the server for a few days, and if you download it using another computer it will also download to that second computer. But this solution still doesn't work very well. Now you have your received mail on multiple computers, but any mail you send out is only on the computer you sent it from. Any mail you delete or file away is only deleted or filed away on that computer.
The bottom line is that your mail is inconsistent across the different computers you use.

IMAP

IMAP is like webmail (and in fact most webmail uses IMAP as part of its underlying technology). The idea here is that your mail lives on a central server, not on whatever computer you are using to access it. Any changes you make to the mail, deleting it, filing it in some folder, replying to an email, all happen on the central server, so when you log in to that server from another computer you see the exact same view of your email.

Now we have consistency when you look at your mail on different computers, but we have a different problem, namely that you can't access your email if you don't have network access. This is obviously bad for people with portables who want to work on email without network access, eg on a plane.

Mail.app, however, allows you to use cached IMAP. This scheme connects to the IMAP server whenever you are on a network, but also downloads all your email to your local hard drive. When you make changes to the mail (eg delete it, or file it, or write a reply) those changes get sent to the server and also modify the local storage, so that the server storage and the local storage of your mail look identical. If you aren't connected to a network, Mail.app will change only your local storage, and when you reconnect to a network it will tell the IMAP server what changes to make to match the work you did while you were not connected to the network.

Assuming you have an IMAP server, how does Mail.app interact with it?

Well, as you would expect, first you create an account in Mail.app, which you set to be of type IMAP rather than of type POP.
Next you tell Mail.app the name of the IMAP server (which Mail.app calls "Incoming Mail Server", and the userID and password you use on that mail server.
Next Mail.app has a tab called "Special Mailboxes", where you can specify that special classes of mail (like deleted mail or junk mail) stay on the server and in sync with your local computer or not. (For example if you don't care about deleted mail or junk mail, maybe you don't want to waste time and server disk space saving these and synchronizing them.)
Finally in the Advanced tab is something called IMAP Path Prefix which can give you a whole lot of grief (and which is one of the reasons why I told you before DO NOT START MAIL.APP ON THE SERVER UNTIL WE ARE DONE WITH INSTALLATION).

Mail.app interacts with the IMAP server, as you would expect, through a specific port number, usually port 143. The IMAP server has a program running on it, usually called imapd, that listens to the IMAP port, 143, for messages and responds to those messages. So on our server we need to run imapd. Once we have imapd up and running, we again need to set up security, this time not for spammers, but so that no-one but us can login to read our mail.

Running on the email server, but on behalf of the email client

Dealing with other email accounts: fetchmail

Once we have IMAP set up, we may now want to set things up so that all our mail goes into the IMAP account. Suppose, for example, that we have a POP account with our ISP, say mjh1@comcast.net. Right now we'd have an account in Mail.app that connected to comcast.net to download this mail using POP. What we would like is for that mail automatically to land up in our IMAP mailbox (so that we see it synchronized across all our different computers) and allowing us to switch off the mjh1@comcast.net account in Mail.app.

We can do this using a program called fetchmail which also runs on our server. fetchmail connects to one or more POP servers (and any other email servers where we might have mail) every few minutes, and downloads the mail it finds into the same central location from which imapd reads incoming mail.

Dealing with spam: SpamAssassin

While Mail.app's junk mail filtering does OK, if you get a lot of spam you might want to throw more intelligence at the spam-filtering problem. You can do this using the program SpamAssassin which is installed on the server and which works nicely with postfix.

Providing a webmail view: SquirrelMail

There are sometimes situations where you want to read your mail but don't have access to your personal computer with Mail.app set up for you. You may be at a friend's office, or using a public computer at a library. In these circumstances you might want to be able to access your IMAP server through a webmail interface, something you can do with SquirrelMail which is installed on your server as part of your webserver.

The email server

Now that we have a rough idea of what a server needs to do, let's look at these tasks in more detail.

Recap: postfix, imapd, fetchmail, SpamAssassin, SquirrelMail

Mail is sent out to the world using port 25.
We run postfix to listen to port 25.
There are other programs that do this job of listening to port 25 and forwarding what comes in to the appropriate place, but postfix is built into MacOSX so we might as well use it.

A mail reader (like Mail.app) reads mail from an IMAP server which listens to the world on port 143.
Standard MacOS X Tiger does not come with an IMAP server, so we will need to install one. There are a number of alternatives for which IMAP server we use. Unfortunately the one most commonly suggested, UW-imapd, is quite awful and should be avoided. Even if you have what claims to be a special version of UW-imapd built especially for MacOS X, that doesn't change the underlying awfulness. What we will use is a simple, little-known imap server called binc-imap.

Why is UW-imapd so awful? There are at least two reasons.

The first is that it seems to have no concept of file synchronization.
UW-imapd stores all mail for a user in a single large file. It also apparently utilizes nothing to synchronize access to that file. That may not have mattered back in the year 7AD when the program was designed, but a modern IMAP client like Mail.app frequently sends multiple requests at the same time, about the same account, to the IMAP server. (For example there may be simultaneous requests asking to download the most recent mail and to synchronize filed mail on the client with the server.) In the face of these multiple requests UW-imapd doesn't actually lose data (be grateful for small mercies) but it does die, and mail processing halts for a few minutes until it is re-launched.
So if you use UW-imapd and Mail.app be prepared for frequent random delays where mail processing stops for a few minutes then starts again.

The second problem is the whole issue related to the IMAP Path Prefix, more specifically the issue of where in the server filesystem imapd stores the email it is handling on behalf of a user.
The most common UW-imapd built for MacOS X, the one that ships with Postfix Enabler, appears to be have been put together by a moron who simply doesn't understand what is going on here. It stores this file in the same place that Mail.app stores its cached version of this email account. So, you may wonder, what would happen if, perhaps deliberately, perhaps by mistake, you launched Mail.app on the server and created an IMAP account that referred to this same server? Oh the fun that will ensue at this point.

A fetcher program might be required to download mail from any previous email accounts we may have.
For this I use fetchmail which is built into Panther and Tiger.
Some people seem to obsessively hate fetchmail and prefer getmail. I personally have never had a problem with fetchmail, but if you really want to use getmail, once you've finished reading the tutorial you can download and install it using DarwinPorts.
SpamAssassin helps tag incoming mail as junk
SquirrelMail provides us with a webmail view of our mail

maildir vs mbox: Where does pending mail live on the server?

The server needs to store all the mail for each user (including all the IMAP filed mail) somewhere in the file system. The reason we need to care about this is because two programs, written by different people, the postfix smtp program (which receives mail from the outside world) and the imapd program (which delivers that mail to a mail client) need to agree on this. When new mail comes in to postfix, postfix needs to do something to make sure that imapd knows about this new mail.

Now if mail programs were being designed from scratch using modern UNIX concepts, things would be done very differently, but the fact is that mail programs have evolved over a long time, starting from before there was an internet, or heck, before there were even networks, and at each stage people did what was easiest to maintain compatability with the past.
So what we have is the following:

postfix, left to itself, stores incoming mail in /var/mail/username in a large file in so-called mbox format
binc-imap reads its data from a directory you can put anywhere you want. This directory is full of subdirectories, each corresponding to a folder of filed mail. These subdirectories (including, particularly, the inbox, which holds incoming mail) hold each separate email message in a separate file. (You should be aware that most of these directories have a name that begins with a period. Following the standard UNIX convention, neither the Finder nor ls will display these. To see these directories, as you'd expect, use ls -a.
This format is called maildir.
fetchmail simply adds a new header to each email it downloads saying "deliver this locally" and sends the mail to localhost port 25 where postfix receives it, meaning there are no files involved and we have reduced the problem to the postfix problem.

Obviously we are going to have to do something to route the mail from postfix to where binc-imap expects mail to be. Since the two programs have different ideas about how mail should be filed, it's not going to work to simply give binc-imap (which expects mail stored in a directory) the name of the file used to store mail by postfix.

Security concerns

We now need to discuss a little theory regarding security. Most people think of security in somewhat old-fashioned terms. In the old days, before networks, security mostly concerned itself with authentication (how does the computer know that a user is who he claims to be) and access control (this user isn't allowed to read those files). On networks other issues surface, including

Encryption: When our email is being transferred to our portables over a wireless network, maybe we don't want other people also on this wireless network (and receiving this wireless signal) to be able to read it.
Spoofing: Suppose that I am sending encrypted packets to some other computers. Someone may not be able to read my packets, but they could still just be destructive and throw junk packets onto the network that look like they came from me, and that confuse the computer receiving the packets.
Authenticity: If someone sends you something (eg a file or an email message), how do you know someone else didn't modify that message?
Traffic Analysis: This is something the really paranoid worry about. The idea is that even if you can't read the packets, you can still tell that over this period of time something was happening because there was a whole flurry of packets, while at this other time nothing was happening and no packets were transferred.

The current internet does not do a very good job dealing with these concepts, especially the existing email infrastructure, but we can try to identify the most realistic concerns we need to worry about and figure out a way to deal with them. (Note that I only deal below with what I consider to be the most pressing issues. If you are in a situation where people could conceivably be trying hard to tamper with your email, you should really consult a specialized book on the subject.)

Consider the usual way that people send or receive mail, using their ISP's servers, and using SMTP and POP. Both SMTP and POP (and for that matter IMAP) require that the user provide a username (so that the server knows who the mail is from/for) and a password (so that other people can't download your mail). But, and this is important, they do not encrypt the usernames, the passwords, or the email data that are sent over the network. So, particularly if you are using a wireless network, baddies can see all the wireless traffic, can extract usernames and passwords, and then cause all sorts of trouble. (In theory you can use WEP encryption to encrypt all the packets sent over the wireless network, but it is fairly easy to crack WEP encryption. Moreover if you are using public wireless access, say in a library or an airport or a coffee shop, WEP will not be used.)

Even if you don't think that you have any secrets in your email and so don't mind strangers reading it, many people use the same password and username for many different accounts, and so once baddies have your email account info, they may try to use that to get to your bank account, your eTrade account, your eBay account, and other things that can really affect your life.

What we want to do is create a situation whereby when we connect to our email server (using the standard protocols like smtp and IMAP) everything that travels over the network, including the usernames, passwords and email data, is encrypted. There are (at least) four common ways of doing this:

You can use VPN software.
This is an enterprise level solution that can handle many clients but is complicated and expensive to set up.
You can use IP-Sec.
This is a fairly new protocol. Support for it is built into MacOS X, but it is not yet used widely enough for people to have much experience with it.
I know nothing about how to set it up.
You an use a program called ssh.
We will use this. Specifically we use ssh to create an encrypted tunnel between our portable and our server.
You can use SSL/TLS.
(These are two different names, Secure Sockets Layer or Transport Layer Security, for what is basically the same thing.) SSL/TLS setup is something of a pain, but it is a widely adopted standard and so we will utilize it for some situations where ssh is not usable.

ssh

A tunnel between two computers is some magic you set up so that packets that get sent to (computer A, port p) automatically and behind the scenes get forwarded on to (computer B, port q), and likewise for replies. You think you are talking to port p on computer A, but all the time your packets are actually being handled by port q on computer B.

An encrypted tunnel is this same idea, but also makes sure that all the packets flowing between computers A and B through the tunnel are encrypted. (In fact when using ssh more than just encryption is performed. The packets are also modified in certain ways to prevent the spoofing I described above and even occasional dummy packets are sent through the connection so someone can't tell anything useful from traffic analysis.)

Suppose that I create an encrypted tunnel between port 25 on my portable and port 25 on my server. Now I tell my computer to send out all email using port 25 on my portable, ie to localhost. The packets will then flow down the encrypted tunnel to port 25 on my server, where postfix will deal with them. Because all the traffic is via the encrypted tunnel, everything that is sent to the postfix server over the public internet, including any wireless connections (including username, password and the email data) is encrypted.

A variant of the security issues above that is very important with email is that we don't want spammers to be able to send out their junk through our server so that it looks like we're the ones responsible for the spam. The same ssh tunnel technology described above will deal with this.

ssl

ssh is very nice technology for setting up a secure connection between a few machines who use is limited to you, and perhaps a few friends and family members. However it is not a complete solution.

For better or worse, ssh is not a universal technology.
Many portable devices,for example, either do not support it, or don't make that support user visible. (For example while the iPhone supports ssh, you can't get to that support through Apple-approved means.)
A second facet of non-universality is that using a web server, you may want to create some connections as secure, others not, and large 3rd party apps like web-servers aren't usually set up to do this using ssh.

SSL is sometimes a solution to both of these problems. Like I said, SSL is (IMHO) more of a hassle to set up, and, unlike ssh, it's not transparent to the underlying programs. However it is widely supported, and you will need to implement it if you want secure access your mail via an iPhone or similar, or secure mail access via SquirrelMail.

summary

So which of ssh or SSL should you use?
My recommendation is that you at least read through both sections to see what's involved in setting them up. After this reading you will probably find that

you want to set up the basic ssh infrastructure so that you can easily log in to your server via the command line
you want to set up SSL so that your SquirrelMail connection is secure
for outgoing mail,
- if you have an ISP that offers authenticated SMTP from any network, you should usually use that;
- otherwise you will probably want to use ssh to set up a secure SMTP tunnel to your server. This tunnel can also act as a backup way to send mail if you find yourself in a situation where your authenticated SMTP connection is not working.
But if you use this ssh solution, you not be able to send mail from an iPhone.

If you need to send mail from an iPhone or suchlike, AND you don't have an ISP smart enough to use authenticated SMTP, you will have to figure out how to enable postfix for SSL, something I've never done and can't help you with. But if you've read all these notes, you should probably have learned enough to figure it out.
for incoming mail, you can use the ssh solution I describe, you can use the SSL solution I describe (which will, as I've said, unlike the ssh solution, work on an iPhone), or you can use both.

← prev: TCP/IP Server Setup next: Mail postfix and IMAP Setup →

Retrieved from "http://name99.org/wiki99/index.php/Mail_Theory"

Category: Computers

Views

Personal tools

Search

Toolbox