My ISP's response to SMTP Queueing

My ISP's response to SMTP Queueing

Post by Adam Israe » Sat, 13 Dec 1997 04:00:00



This is my ISP's response when I asked about doing SMTP, and Queuing:

This doesn't scale.
The queue is one directory.  Guess what happens when 500 domain customers
(we have well over 1000) do this, and each of them get 100 emails queued.
That's 50,000 files in a directory.  The traversal time to search and
attempt delivery from the queue on each of these will be measured in
*hours*.

What do you think?
Adam Israel

 
 
 

My ISP's response to SMTP Queueing

Post by Amiri Jone » Sat, 13 Dec 1997 04:00:00




Quote:> This is my ISP's response when I asked about doing SMTP, and Queuing:

> This doesn't scale.
> The queue is one directory.  Guess what happens when 500 domain customers
> (we have well over 1000) do this, and each of them get 100 emails queued.
> That's 50,000 files in a directory.  The traversal time to search and
> attempt delivery from the queue on each of these will be measured in
> *hours*.

        Sounds like GARBAGE to me.  What about an ISP with 5000 customers each
with 10 pieces of mail in their POP3 mailbox?  That's 50,000 pieces of mail
as well, and I'm sure there are at least a few ISPs out there in that type
of situation.  I'm no expert on SMTP mailers, but storing each e-mail in a
separate file sounds like a waste of resources to me.  Besides, has this
ISP had 500 customers come to him wanting mail relay services?  Find a new
ISP.

 
 
 

My ISP's response to SMTP Queueing

Post by Gary Hest » Sun, 14 Dec 1997 04:00:00






>>This is my ISP's response when I asked about doing SMTP, and Queuing:
>>This doesn't scale.
>>The queue is one directory.  Guess what happens when 500 domain customers
>>(we have well over 1000) do this, and each of them get 100 emails queued.
>>That's 50,000 files in a directory.  The traversal time to search and
>>attempt delivery from the queue on each of these will be measured in
>>*hours*.
>Although he is, in essence, correct in his assessment, he is ignoring
>that this is how SMTP was designed to work.

Not quite--SMTP doesn't have anything to do with how the MTA handles
files on disc. It's only the network delivery protocol. Some MTAs will
created a separate directory for each domain they're dealing with in
order to reduce this problem. Imagine how much email someone like
Uunet or AOL passes in a day....

With this kind of volume, you really need to have a full-time connection,
so that you never have 50,000 emails backed up. I have had directories
that big to deal with on old Unix boxes, and it does get incredibly slow
(and doubt that NT would be better, probably worse).

Do SMTP, but don't do queueing.

Gary

--


"Mother! I didn't expect to see you here so soon!"  "Nor I you, Princess."

 
 
 

My ISP's response to SMTP Queueing

Post by Gary Hest » Sun, 14 Dec 1997 04:00:00




>On Fri, 12 Dec 1997 14:58:31 -0600, "Amiri Jones"




  [ re: mail queueing ]

Quote:>>        Sounds like GARBAGE to me.  What about an ISP with 5000 customers each
>>with 10 pieces of mail in their POP3 mailbox?  That's 50,000 pieces of mail
>>as well, and I'm sure there are at least a few ISPs out there in that type
>>of situation.  I'm no expert on SMTP mailers, but storing each e-mail in a
>>separate file sounds like a waste of resources to me.  Besides, has this
>>ISP had 500 customers come to him wanting mail relay services?  Find a new
>>ISP.

You're right, you don't know how MTAs on Unix machines work.

A POP3 retrieval will access the mailbox associated with the users' account,
which is one file, so that's only 5,000 files, not 50,000. Most Unix MTAs
(like sendmail and smail3) will only use individual files for things in
transit. Once it's delivered, the mail is appended to the users' mailbox,
which is simply a text file owned by the user, but in a system directory.
The user interface run by the user access this mailbox, and stores the
mail locally in a directory, basically giving each user their own
information stores.

While this does make for a bigger directory, it also makes it far more
fault tolerant than a single huge database. If you lose one file, the
rest are unaffected. However, if you lose a piece out of the middle of
a large database, you may lose everything. Personally, I would much
prefer that Exchange had an individual database for each mailbox. Aside
from the fault tolerance issue, it'd make moving a user much easier.

Quote:>In thinking about it a little more, it sounds like the ISP is running
>a very inefficient mailer.

No, it'll probably outperform Exchange by a wide margin as an MTA. It
won't have any of the fancy graphics windows, or the cute bent paper
clip winking at you; it'll just deliver mail.

Gary

--


"Mother! I didn't expect to see you here so soon!"  "Nor I you, Princess."

 
 
 

My ISP's response to SMTP Queueing

Post by Greg Aske » Mon, 15 Dec 1997 04:00:00


Not really an issue; anyone with this requirement that is evaluating
Exchange can easily choose another mail package.

Most PC-based mail packages don't use this scheme anymore. GroupWise,
cc:Mail, and Exchange all have user mailboxes in multiple files and/or
databases. If you want a Unix mail system, get a Unix mail system.
It's easier than buying Exchange and wishing you had Unix mail.

Even Sun's Internet Mail Server uses a database for the message store
(although it retains support for /var/mail clients).  Their quote:

"Sun Internet Mail Server uses a committed transactions model to
prevent lost or corrupted mail messages.

"The message store represents a significant advance in reliability,
performance, and scalability among open systems message stores. The
store is built around a custom-designed database that employs a
write-once data store and two-level index to achieve excellent
performance and data integrity."

Significant advance?  Sounds like an Exchange 4.0 beta manual from
three years ago.

But the whole comparison is fundamentally invalid.  Messaging is only
a single component of a much larger system.  When you start dealing with
groupware applications, calendaring, and client-server communication
between the workstations and server(s), the whole "separate file"
scheme falls apart.  Ever hear of single-instance storage?
--
Greg Askew
Business Information Technologies, Inc.


> While this does make for a bigger directory, it also makes it far more
> fault tolerant than a single huge database. If you lose one file, the
> rest are unaffected. However, if you lose a piece out of the middle of
> a large database, you may lose everything. Personally, I would much
> prefer that Exchange had an individual database for each mailbox. Aside
> from the fault tolerance issue, it'd make moving a user much easier.

 
 
 

My ISP's response to SMTP Queueing

Post by Andy Web » Mon, 15 Dec 1997 04:00:00


I've read the other responses here, but thought an answer to the original
was most appropriate.  Comments embedded.

--
=======================================================

Simpler-Webb, Inc.                           Austin, TX
             "Mauve has more RAM" - Dilbert
(address munged to protect me from spam spewing cretins)
=======================================================


>This is my ISP's response when I asked about doing SMTP, and Queuing:

>This doesn't scale.
>The queue is one directory.

The queue is one directory by default.  You can make it as many directories
as you want (with sendmail).  If the ISP really gets this many customers
doing this, then putting a little brain power into a creative solution would
be worthwhile.  If they only have you, they should shut up - it's a non
issue.

Quote:>Guess what happens when 500 domain customers
>(we have well over 1000) do this, and each of them get 100 emails queued.

At that point you redesign the MTA to support the traffic model of your
customers.  It's not that difficult and is exactly what folks like AOL have
had to do.

This solution is really designed for the customer who only gets 5-10 emails
every 1-4 hours.  There are quite a few people like that.  Run the numbers
on randomized dial-in, weighted for business hours, etc. and the load is
VERY SMALL.

Quote:>That's 50,000 files in a directory.  The traversal time to search and
>attempt delivery from the queue on each of these will be measured in
>*hours*.

Perhaps.  But we're talking about one customer who wants to queue 5-10 email
messages.  If there are 100+ messages queued in one queue interval (a couple
hours I would guess), then the customer (you) need a dedicated connection.

>What do you think?
>Adam Israel


I think it's bunk.  Technically correct, but practically useless.  They
pulled numbers out of ??? and used them to prove their unwillingness to be
creative.  The real numbers do not support their position.  The capabilities
of the technology do not support their position.
 
 
 

My ISP's response to SMTP Queueing

Post by Anthon » Tue, 16 Dec 1997 04:00:00



>With this kind of volume, you really need to have a full-time connection,
>so that you never have 50,000 emails backed up. I have had directories
>that big to deal with on old Unix boxes, and it does get incredibly slow
>(and doubt that NT would be better, probably worse).

NT is very efficient at handling extremely large directories.  It is, after
all, thirty years younger than UNIX, and much has been learned during that
time.

--
Anthony

 
 
 

My ISP's response to SMTP Queueing

Post by Anthon » Tue, 16 Dec 1997 04:00:00


Find an ISP that wants your business.

--
Anthony


>This is my ISP's response when I asked about doing SMTP, and Queuing:

>This doesn't scale.
>The queue is one directory.  Guess what happens when 500 domain customers
>(we have well over 1000) do this, and each of them get 100 emails queued.
>That's 50,000 files in a directory.  The traversal time to search and
>attempt delivery from the queue on each of these will be measured in
>*hours*.

>What do you think?
>Adam Israel


 
 
 

My ISP's response to SMTP Queueing

Post by Anthon » Tue, 16 Dec 1997 04:00:00



>A POP3 retrieval will access the mailbox associated with the users'
account,
>which is one file, so that's only 5,000 files, not 50,000.

_Only_ 5000 files?  I think early mail products like MS Mail and cc:Mail
adequately demonstrated the overall inferiority of a messaging systemt that
uses individual, unrelated files for message storage.

Quote:>While this does make for a bigger directory, it also makes it far more
>fault tolerant than a single huge database.

No.  Individual, unrelated files have far more of a tendency to become
desynchronized than does one large database, because there is no automatic
mechanism in place to establish and enforce an application-based
relationship among them.  It is also asking for trouble to have multiple
processes accessing the same files without some sort of foolproof
synchronization among them.  And beyond that, it is very inefficient to
store data sequentially in text files.

Quote:>Personally, I would much prefer that Exchange had an
>individual database for each mailbox. Aside from the
>fault tolerance issue, it'd make moving a user much easier.

It would make the product easier to understand for users with no experience
in databases, too; however, that would not compensate for the inefficiency
of such an arrangement.

Quote:>No, it'll probably outperform Exchange by a wide margin as an MTA. It
>won't have any of the fancy graphics windows, or the cute bent paper
>clip winking at you; it'll just deliver mail.

Why not take some measurements and report back to us?

By the way, the graphics and the paperclip are part of the user agent, not
the server.

--
Anthony

 
 
 

My ISP's response to SMTP Queueing

Post by Greg Crow » Tue, 16 Dec 1997 04:00:00


Nice shooting Greg

 
 
 

My ISP's response to SMTP Queueing

Post by Andy Web » Wed, 17 Dec 1997 04:00:00


The issue isn't the storage.  It's the transport.  And Exchange puts all the
data in individual files as part of that process too.  If you had 50000
queued messages in Exchange you would have a slowdown too.

The issue is the ISP blowing smoke and being unwilling to design a system to
their usage profile.  One customer wanting to queue mail has absolutely no
statistical affect on the ISP in question.  100 customers wanting that
feature should be enough to make them build a system to handle it.  The end.

--
=======================================================
Andy Webb

Simpler-Webb, Inc.                Austin, TX     "Mauve has more RAM" -
Dilbert
=======================================================



>>A POP3 retrieval will access the mailbox associated with the users'
>account,
>>which is one file, so that's only 5,000 files, not 50,000.

>_Only_ 5000 files?  I think early mail products like MS Mail and cc:Mail
>adequately demonstrated the overall inferiority of a messaging systemt that
>uses individual, unrelated files for message storage.

>>While this does make for a bigger directory, it also makes it far more
>>fault tolerant than a single huge database.

>No.  Individual, unrelated files have far more of a tendency to become
>desynchronized than does one large database, because there is no automatic
>mechanism in place to establish and enforce an application-based
>relationship among them.  It is also asking for trouble to have multiple
>processes accessing the same files without some sort of foolproof
>synchronization among them.  And beyond that, it is very inefficient to
>store data sequentially in text files.

>>Personally, I would much prefer that Exchange had an
>>individual database for each mailbox. Aside from the
>>fault tolerance issue, it'd make moving a user much easier.

>It would make the product easier to understand for users with no experience
>in databases, too; however, that would not compensate for the inefficiency
>of such an arrangement.

>>No, it'll probably outperform Exchange by a wide margin as an MTA. It
>>won't have any of the fancy graphics windows, or the cute bent paper
>>clip winking at you; it'll just deliver mail.

>Why not take some measurements and report back to us?

>By the way, the graphics and the paperclip are part of the user agent, not
>the server.

>--
>Anthony

 
 
 

My ISP's response to SMTP Queueing

Post by Andy Web » Wed, 17 Dec 1997 04:00:00


Better is a relative term.  50000 queued messages will be a problem for any
single system.  If it has been mis-designed such that this happens, tough
luck.

--
=======================================================
Andy Webb

Simpler-Webb, Inc.                Austin, TX     "Mauve has more RAM" -
Dilbert
=======================================================



>>With this kind of volume, you really need to have a full-time connection,
>>so that you never have 50,000 emails backed up. I have had directories
>>that big to deal with on old Unix boxes, and it does get incredibly slow
>>(and doubt that NT would be better, probably worse).

>NT is very efficient at handling extremely large directories.  It is, after
>all, thirty years younger than UNIX, and much has been learned during that
>time.

>--
>Anthony

 
 
 

My ISP's response to SMTP Queueing

Post by Anthony Vea » Wed, 17 Dec 1997 04:00:00


So being 30 years younger is a good thing?

Well that would have to be the dumbest reason yet I've heard for NT
(or any operating system for that matter) being great.  You fail to
realise that all flavours of UNIX have developed over that 30 years!

Don't get me wrong, NT has it's place, but give respect to other
operating systems too.  Remember, UNIX and NetWare can both perform
quotas on the amount of data stored on a server based on the user -
NT4 is not capable of doing this straight out of the box.  So why is
this not included if "so much has been learned duing that time"?  I
would really love to have one of my users fill up my server's hard
disk just because they felt like it...

Anthony.

PS.  Your definition of "efficient" is also very subjective.  Are we
talking same hardware the UNIX and NT server?  I think not!  Throw
enough hardware at any OS and it starts to perform!




>>With this kind of volume, you really need to have a full-time connection,
>>so that you never have 50,000 emails backed up. I have had directories
>>that big to deal with on old Unix boxes, and it does get incredibly slow
>>(and doubt that NT would be better, probably worse).

>NT is very efficient at handling extremely large directories.  It is, after
>all, thirty years younger than UNIX, and much has been learned during that
>time.

>--
>Anthony

 
 
 

My ISP's response to SMTP Queueing

Post by Anthon » Thu, 18 Dec 1997 04:00:00



>So being 30 years younger is a good thing?

In the computer industry, given how much has changed in 30 years, yes, it
is.

Quote:>Well that would have to be the dumbest reason yet I've heard for NT
>(or any operating system for that matter) being great.  You fail to
>realise that all flavours of UNIX have developed over that 30 years!

Hmm... isn't one of the oft-cited great advantages of UNIX supposed to be
its portability?

Quote:>Don't get me wrong, NT has it's place, but give respect to other
>operating systems too.

UNIX is a nice operating system, but it is now past its prime.

Quote:>Remember, UNIX and NetWare can both perform
>quotas on the amount of data stored on a server based on the user -
>NT4 is not capable of doing this straight out of the box.

True for NT4.

Windows NT can handle encrypted challenge/response authentication right out
of the box; UNIX cannot.

So?

Quote:>So why is this not included if "so much has been learned
>duing that time"?

Why hasn't challenge/response been included in UNIX over a period ten times
as long?

Quote:>I would really love to have one of my users fill
>up my server's hard disk just because they felt like it...

Why?

I have administered systems in the past and had unlimited quotas for some
users, and I've never had a user fill up any device.  It's not a common
occurrence.

--
Anthony

 
 
 

My ISP's response to SMTP Queueing

Post by Anthon » Thu, 18 Dec 1997 04:00:00



>Sun srv4  runs a heck of a lot better that NT
>by a huge margin..

Based on what criteria, exactly?

--
Anthony

 
 
 

1. My ISP's response to SMTP Queueing

This is my ISP's response when I asked about doing SMTP, and Queuing:

This doesn't scale.
The queue is one directory.  Guess what happens when 500 domain customers
(we have well over 1000) do this, and each of them get 100 emails queued.
That's 50,000 files in a directory.  The traversal time to search and
attempt delivery from the queue on each of these will be measured in
*hours*.

What do you think?
Adam Israel

2. Exchange 5.5 Move Server Wizard questions

3. HELP pls. (email system w/o Internet)

4. Problems sending smtp mail's over ISP

5. CDO bug, cdoSendUsingPickup

6. Forwarding Exchange mail to ISP's SMTP

7. Delegate problem during Migration to Exchange 2000

8. queueing *some* smtp internet mail e2k

9. Outlook tries to dial ISP when starting -response

10. Slow Response when 'Save As ...'