PDA

View Full Version : The email business



Corrado Fiore
11-27-2007, 03:58 PM
Dear All,

in an effort to help WH build the best hosting environment available, I'd like to share some thoughts.

Having been a WH reseller for a couple of years, and dealing mostly with business clients, I've seen mail services become more and more the core of my hosting business - much more than web services.

Now that server space is affordable, advanced webmail services are replacing the old offline client like Thunderbird or Outlook - mostly because of ubiquity and automatic backups.

On a commercial standpoint, very few companies have gigabyte-sized websites, but almost every user has one or two gigabytes of emails -- growing every day! Emails are the new kind of data we shall sell space for.

How does WH compare to competitors? Are we ready to host our clients' gigabytes of email? Yes and no.

The bright side: Sendmail, RTBL, Procmail, SpamAssassin. All of them are rock-solid, completely configurable (a BIG plus), easy to setup. A bit of knowledge is required to reach a > 95% filtering with SA, but overall that's a state-of-art setup, and WH disk performance is quite good in my experience.

The not-so bright side: mailbox size, WU IMAP, mbox format, webmail apps.

Let's start from mailbox size. Limiting it to 50 MB is something we can't even tell to the client: "Are you saying I'm paying for having a 50 MB mailbox, when Gmail offers over 5 GB for free?".

So, why limiting? Because we are still stuck to POP3 & the ancient MBOX format ("one big file to keep them all") to store users' email. A big mbox file requires continuous parsing, i.e. an awful lot of disk activity & CPU cycles. Better keep it small (under 50 MB).

As WH launched IMAP services, I thought that finally there was a way to get over that limit. All clients' email would have been stored on server (that's what IMAP is for), clients would have purchased more space and we all have done a good business.

In brief, my experience with WU IMAPD has been a total disaster. After I converted some companies' email systems to IMAP, I was quickly recalled by angry clients who couldn't even access their email (most of them were on a ded machine with RAID disks). UW IMAPD has proven so slow as to be useless.

As far as I can tell, that was again a matter of mailbox format. UW IMAPD uses mbox, too. That's why also IMAP boxes cannot be set to hold over 50 MB of data. But 50 MB is the amount of email a heavy customer (like a graphic designer) receives in half a day... so goodbye IMAP.

Finally, webmail apps are still at an early stage at Westhost. The default webmail is Uebimiau, which is so weak I had to replace with Telaen on almost all my accounts. Uebimiau has severe performance limits when logging in and makes folder management too difficult for the average user.
The newcomer Roundcube is certainly a nice app, but can do IMAP only, and we've seen the disaster with UW IMAPD.

What is needed is a collaboration suite (email + tagging + calendar + personal web disk) where our users can store their gigabytes of data safely. Other hosting companies are providing such suites, both as services and as GPL applications.

That collab suite should interact with a high-performance IMAP server, such as Dovecot, which must keep ALL messages (also the inbox) in Maildir format. Maildir (one folder with 1 file per message) performs much better than mbox and can easily hold hundreds of megabytes. I speak from personal experience, since Telaen (and Uebimiau) store sent messages and users' folders using that format and I have had no performance problems until now.

In brief: please replace WU IMAPD with Dovecot + Maildir... and let us sell gigantic mailboxes with gigabytes of space!

Thanks for listening.

Sincerely,
Corrado Fiore

WestHost - CRussell
11-28-2007, 11:24 AM
Corrado,

Great post and I agree with most of your comments. maildirs is something on the map and does need to be addressed soon so our platform can stay competitive and meet the needs of our clients.

ai681
05-15-2008, 07:05 AM
I just want to second everything written here. Westhost is great, but these email problems are proving to be a nightmare for me. I am not a reseller, but a graphic designer who regularly sets up hosting accounts for my clients. These changes are EXACTLY what I need to keep my business here.

corvus
05-22-2008, 12:37 PM
Has anyone here looked at what the top layer would be? I know there's projects like phpgroupware/egroupware. What other players are in the space and what's everyone like?

Corrado Fiore
06-09-2008, 02:00 PM
Has anyone here looked at what the top layer would be?

That's a really good question. From a usability / performance standpoint, I would recommend Roundcube webmail. Newer versions feature a three-pane interface which resembles Outlook and Thunderbird, thus making the switch relatively pain-free even for non-geeks. Also, there's a patch which enables management of _BIG_ attachments using PHP streams, i.e. without eating up all available PHP memory.

On the other hand, if you're looking for a "all-in-one" calendaring, workgroup and messaging solution, you might want to look at some other products like Joyent Connector (which is developed using Rails). It was made free and open source a couple of months ago and seems promising to me. The email client deals with any IMAP servers, so it's basically like Roundcube + calendaring and agenda. Beware: it's still in semi-beta and installation seems like a pain. Not yet ready for prime time, IMHO, but WH techs might want to evaluate it for an inclusion among our ValueApps (hint, hint) ;)

Cheers,
Corrado Fiore

spolit
07-08-2008, 01:09 PM
I agree with the original poster. I contacted Westhost tech support and was assured that I could move my clients over to imap - no problem. Well, there are problems - plenty of them!. The imap server is almost completely useless. When you use Outlook 2007 as the client, it won't let you create subfolders and it times out randomly every time you use it. I really looked stupid to my clients because I explained to them all the cool benefits of imap. I hate looking stupid to a client especially after I checked it out with tech support. I have been using the imap service on my westhost account with Thunderbird and it worked sorta ok - I finally moved over to gmail and it works much better.

Westhost - please help us out with this one. I've heard zimbra (at zimbra.com) has a good collaboration product and a really nice ajax webmail client.

I'm going to experiment with this dovecot beast but I am a bit worried about the procmail stuff.

Lets see how it goes
Selwyn

socomusic
07-21-2008, 11:21 PM
Yes the email situation is almost unacceptable....this is 2008 right? :)

RMD
07-22-2008, 01:15 PM
...The imap server is almost completely useless. When you use Outlook 2007 as the client, it won't let you create subfolders and it times out randomly every time you use it. I really looked stupid to my clients because I explained to them all the cool benefits of imap...
I was just about to install IMAP Server for a client and this post stopped me cold. I checked out Zimba and the free version is quite loaded with features, however, with online backup/retrieval not one of them, $25/year per user is a small price to pay if you're dealing with a small firm and limited email addys. However, installing a mail server is not something I want to crunch my brain cells around. Dovecot sounds like it would do the job, but Zimbra Collaboration Suite takes a big step into the future with its ability to create mashups with 3rd party apps and support for search, tagging, sharing, RSS & VoIP. There didn't seem to be much info on scalability though.

Personally, I think it would be fabuloso if WH would consider an app that offers their clients a collaboration suite. Would users pay extra? I bet they would.

jdogg
07-26-2008, 12:20 PM
I nominate Zimbra as well.

WestHost - JFillmore
09-08-2008, 05:34 PM
In response to your feedback we will be releasing a Dovecot IMAP Server site application, which will include full maildir support. Additionally, nested folders will be available when used in conjunction with a e-mail client that supports it.

Unlike the existing IMAP server, Dovecot will require that you have a dedicated IP address for your account (whether you have a shared or dedicated IP address is shown on the top of the first page of Site Manager). I can't promise an exact date on when it will be available, but it will be within the next few weeks.

Zimbra does look like great, but unfortunately the current VPS design prevents us from releasing it as a site application.

Corrado Fiore
09-09-2008, 02:12 AM
Hey, this is great news! Thanks to WH for listening :-)

As for the top layer, I'm currently evaluating Horde / DIMP vs. Roundcube in a high traffic environment (think of business users with hundreds of emails per day). I'll share the results of my tests (speed, reliability, ease of use, features, ...) here on the forum.

Best regards,
Corrado Fiore

j103c
09-09-2008, 08:13 PM
Good news! This e-mail thing has been a pain.

One question I'd like to ask - would it be possible to POP3 the top, inbox folder in the new solution? Once upon a time I subscribed to SpamCop for my e-mail management, and they had IMAP running as well as POP3. I could use an IMAP client (like their webmail interface using Horde), manage sub-folders, etc. Then later I could POP3 the account from home and download everything in the top level folder.

I would l.o.v.e. this set up again. Having both running would allow me to easily manage the users I have on my personal domain. I'm sure I'd get a mix of those who want IMAP and those who want to stay POP3 (some of the users are remote and not technically inclined). Also, there are probably some e-mails I'd rather keep on a local machine and not stored on WestHost long-term, so it'd be nice to POP3 those off.

Corrado Fiore
09-10-2008, 07:30 AM
Dovecot does so natively. You can access the inbox via either POP3 or IMAP, and all other folders through IMAP.

In my setup, for example, all spam is delivered to a subfolder. This keeps POP boxes clean, and offers a convenient way for each user to check their spam using an IMAP webmail.

On the other hand, also POP3 will benefit from Maildir format, and let you set much higher quotas for users. No more full POP boxes during holydays :)

Cheers,
Corrado

j103c
09-10-2008, 08:31 AM
Beautiful! I'll hope WH implements it that way. :)

WestHost - JFillmore
09-11-2008, 12:25 PM
The details on how Dovecot will be implemented are not yet finalized. However, I can say that normal POP3 access will stop working properly when Dovecot is installed. Port 110 (used for POP3) is fully managed by the VPS software and it is beyond the capabilities of the software that handles site application installations to reconfigure how port 110 is used.

Fortunately, Dovecot does has native support for secure POP3, which uses port 995. Advanced users like yourselves who understand the implications of using IMAP and POP3 at the same time will be able to configure Dovecot to provide pop3s access to your inboxes if you like.

So yes, j103c, you should be able to configure Dovecot as you have described. The "unfortunate" side effect is that all POP3 access will be secure. You could also create an IP filtering rule in Site Manager to block access to port 110 to prevent confusion caused by users logging in insecurely and seeing no e-mail.

j103c
09-11-2008, 12:42 PM
Thanks Johnny! This gives me enough information to go off of to start planning for the rest of my users.

j103c
09-11-2008, 12:44 PM
One more question - do you know yet if the WH implementation of Dovecot will affect the alternate Sendmail port app? I don't think it would, but don't want to assume. Thanks!

WestHost - JFillmore
09-11-2008, 03:14 PM
Dovecot will have no affect on either Sendmail or the alternate port Sendmail site app.

Corrado Fiore
09-11-2008, 04:07 PM
Another way to keep both POP3 and IMAP access would be to assign an alternate port to the POP3 service in Dovecot, say for example 1100 instead of 110. I'm doing so for a couple of clients and it works fine (this workaround was suggested by WH support team, so credits go to them :-) ).

Note that I'm not saying that this route is by any means *better* than using secure POP3, like Johnny Fillmore correctly suggested. It's a "just in case" kind of suggestion :-), useful if you cannot for whatever reason set up a secure POP3 connection.

Cheers,
Corrado Fiore

WillieManillie
09-24-2008, 07:39 PM
Corrado,

Great post and I agree with most of your comments. maildirs is something on the map and does need to be addressed soon so our platform can stay competitive and meet the needs of our clients.

And waiting. The current platform absolutely sucks!

WestHost - JFillmore
09-25-2008, 08:30 AM
For those of you anxious for Dovecot's release it looks like we will have it available in Site Manager in approximately two weeks. As we get closer to the release date I'll update this thread with more information.

j103c
09-25-2008, 08:58 AM
Excellent - thanks!

Corrado Fiore
09-25-2008, 01:14 PM
I would like to suggest also to update the current Roundcube installation, since a couple of days ago the RC team released version 0.2 beta, which features a ton of bug fixes and enhancements.

Although it is tagged as beta, I'm using it on a couple of VPS and it seems to work reliably (actually, I've been using some nightlies from their SVN repository and they all worked flawlessly). They have a slow-but-solid development cycle IMHO.

Cheers,
Corrado Fiore

dattas
09-27-2008, 07:45 PM
WestHost generally doesn't do Betas, I've personally never seen them do it, Although I have to agree with your Roundcube 0.2B is pretty darned stable and much better than 0.1.X

WestHost - JFillmore
09-29-2008, 04:31 PM
We are ahead of schedule so you can expect to see Dovecot in Site Manager (and Server Manager for resellers) by the end of the week. You will find that there are also a few other new/updated site applications, depending on your plan.

We did take a look at including RoundCube, but decided against it for this round of updates. While it does have a history of stable alpha/beta releases, this last alpha build was (and still is, being only 8 days old) too new and untested when the list of site applications for this round of updates was finalized.

WestHost - CRussell
10-09-2008, 11:10 AM
The release of Dovecot and AtMail is fully complete.

j103c
10-11-2008, 08:23 AM
Working like a champ so far - thanks!