Very Computer

by **douglass davi** » Wed, 26 Jun 2002 08:37:14

I would like to make a chat application, as well as a chat server.
The only catch is, I would like to make the chat server scale up to huge
numbers. Are there any papers, or similar distributed computing
problems that I could borrow ideas from? Any other resources?

I just need a point to start off from. Any ideas?

btw: my real address has no Zs in it.

-d

by **MSCHAEF.C** » Wed, 26 Jun 2002 08:42:55

Quote:
>I would like to make a chat application, as well as a chat server.
>The only catch is, I would like to make the chat server scale up to huge
>numbers. Are there any papers, or similar distributed computing
>problems that I could borrow ideas from? Any other resources?

>I just need a point to start off from. Any ideas?

What's wrong with any of the numerous chat applications and servers
already available? My first avenue to research this problem would be to
investigate something like IRC.

-Mike

--
http://www.mschaef.com

by **Nils O. Sel?sd** » Wed, 26 Jun 2002 16:36:05

>>I would like to make a chat application, as well as a chat server.
>>The only catch is, I would like to make the chat server scale up to huge
>>numbers. Are there any papers, or similar distributed computing
>>problems that I could borrow ideas from? Any other resources?

>>I just need a point to start off from. Any ideas?

> What's wrong with any of the numerous chat applications and servers
> already available? My first avenue to research this problem would be to
> investigate something like IRC.

To see how _not_ do do it? Despite that IRC hosts some _real_ large networks,
"everyone" know it scales badly, read the RFC, it have many good points.

by **Rainer Vo** » Wed, 26 Jun 2002 17:32:25

> I would like to make a chat application, as well as a chat server.
> The only catch is, I would like to make the chat server scale up to huge
> numbers. Are there any papers, or similar distributed computing
> problems that I could borrow ideas from? Any other resources?

> I just need a point to start off from. Any ideas?

Why not use Jabber as a start? It has the advantage that it is
available in an open-source version (www.jabber.org), which can be
modified, and a commercial version (www.jabber.com). As far as I know
it scales very well and can be extended with other services.

-----------------------------------------------------
Rainer Volz
www.vrtprj.com - virtual teams & distributed projects
www.vrtprj.de - virtuelle teams & verteilte projekte

by **T.S.Ne** » Wed, 26 Jun 2002 17:48:09

go to the

www.pscode.com

search chat program in visual basic section search.

bfn

T.S.Negi

by **MSCHAEF.C** » Wed, 26 Jun 2002 22:33:52

>>>I would like to make a chat application, as well as a chat server.
>>>The only catch is, I would like to make the chat server scale up to huge
>>>numbers.

<snip>

Quote:>> What's wrong with any of the numerous chat applications and servers
>> already available? My first avenue to research this problem would be to
>> investigate something like IRC.

>To see how _not_ do do it? Despite that IRC hosts some _real_ large
>networks, "everyone" know it scales badly, read the RFC, it have many
>good points.

Okay, that's an answer to the question I posed in my post. However,
without a definition of the word _huge_, if IRC meets his goals within his
budget, any technical problems (which I weren't aware of, obviously) seem
pretty forgiviable.

Part of the reason I mentioned IRC is that if it needs to scale to _huge_
numbers, where are the users coming from? Is this to be a competitor to an
existing system like IRC? Deployed across a company? If it's the first,
there's probably more marginal value in working in the existing
'establishment' however flawed, then trying to found a new establishment.
If it's the second, there are likely products to buy that will save lots
of time and hassle. Particularly if the chat application will
convincingly create value.

-Mike

--
http://www.mschaef.com

by JT » Wed, 26 Jun 2002 23:06:49

Or, instead of trying to complicate things more with wordless banter,
someone might just want to do it for the sake of learning how to.

> >>>I would like to make a chat application, as well as a chat server.
> >>>The only catch is, I would like to make the chat server scale up to
huge
> >>>numbers.

> <snip>

> >> What's wrong with any of the numerous chat applications and servers
> >> already available? My first avenue to research this problem would be to
> >> investigate something like IRC.

> >To see how _not_ do do it? Despite that IRC hosts some _real_ large
> >networks, "everyone" know it scales badly, read the RFC, it have many
> >good points.

> Okay, that's an answer to the question I posed in my post. However,
> without a definition of the word _huge_, if IRC meets his goals within his
> budget, any technical problems (which I weren't aware of, obviously) seem
> pretty forgiviable.

> Part of the reason I mentioned IRC is that if it needs to scale to _huge_
> numbers, where are the users coming from? Is this to be a competitor to an
> existing system like IRC? Deployed across a company? If it's the first,
> there's probably more marginal value in working in the existing
> 'establishment' however flawed, then trying to found a new establishment.
> If it's the second, there are likely products to buy that will save lots
> of time and hassle. Particularly if the chat application will
> convincingly create value.

> -Mike

> --
> http://www.mschaef.com

by **David Schwart** » Thu, 27 Jun 2002 03:59:58

> Or, instead of trying to complicate things more with wordless banter,
> someone might just want to do it for the sake of learning how to.

Then this is a bad choice. Since you're unlikely to find huge numbers
of users (huge meaning enough to break current implementations that we
all agree are bad) then you won't ever know if you've succeeded. This
means you're unlikely to learn very much.

Btw, the company I work for makes the most scaleable chat servers there
are. http://www.webmaster.com/

At least, we think they are. We've never been able to get enough warm
bodies to prove it. (And artificial load testers don't prove anything.
We have years of experience that proves that.)

DS

by **Barry Margoli** » Thu, 27 Jun 2002 04:24:57

>> Or, instead of trying to complicate things more with wordless banter,
>> someone might just want to do it for the sake of learning how to.

> Then this is a bad choice. Since you're unlikely to find huge numbers
>of users (huge meaning enough to break current implementations that we
>all agree are bad) then you won't ever know if you've succeeded. This
>means you're unlikely to learn very much.

The fact that it's a chat server seems largely irrelevant to the exercise.
The techniques needed to make a scalable server of any type are mostly
independent of the particular application. The main thing the design will
depend on probably depends on whether the application needs connections or
can make do with datagrams (the latter allows for easier scalability,
because you don't have to worry about limits on the number of sockets).

And no matter what application you implement, you're not likely to get many
users of the test system. In order to test your scalability, you'll
probably have to simulate them. Suppose if he instead said he were
implementing a scalable web server. That's clearly an application that
needs to be able to scale well, but it will probably only get enough users
to tax it when put into production use by a popular site. As an academic
exercise, the chat server is no better or worse than this.

In other words, what you're really saying is that there's no way to learn
to implement a scalable server, because it's never going to get enough use
while you're just learning. So should he just give up?

--

Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

by **David Schwart** » Thu, 27 Jun 2002 05:12:14

> And no matter what application you implement, you're not likely to get many
> users of the test system. In order to test your scalability, you'll
> probably have to simulate them.

It doesn't work. The simulated users miss many of the very critical
features of real users. These are oddly distributed packet loss and
latency, disappearing clients, and buggy TCP stacks. But the big problem
is simply that simulated clients don't do the same things real clients
do.

Quote:> Suppose if he instead said he were
> implementing a scalable web server. That's clearly an application that
> needs to be able to scale well, but it will probably only get enough users
> to tax it when put into production use by a popular site. As an academic
> exercise, the chat server is no better or worse than this.

My experience doesn't agree with this, though it may seem intuitively
correct. A web server has a much smaller command set and generally has
connection that don't continue for as long. This makes it much easier to
simulate realistic web traffic.

Quote:> In other words, what you're really saying is that there's no way to learn
> to implement a scalable server, because it's never going to get enough use
> while you're just learning. So should he just give up?

There really is no good way to learn to implement a scalable server
other than actual deployment. However, if that's not an option, you
should definitely start with something that's easy to test. A Web server
is a very good suggestion because there are at least decent load
generators for it.

DS

by **Barry Margoli** » Fri, 28 Jun 2002 08:04:48

Quote:> There really is no good way to learn to implement a scalable server
>other than actual deployment.

That's an unfortunate truth about computer systems. It's one of the things
that causes other engineering disciplines to look down on us. Could you
imagine architects saying "There's no good way to learn to design a
skyscraper other than actually building one, and see whether it stays
standing"? I wouldn't want to be working in that new architect's first
building on its first windy day. (Although architects have an impressive
track record, there have also been some spectacular disasters, such as the
Tacoma Narrows Bridge.)

The sad fact is that most server designers learned how to build scalable
servers by examining the failure modes of previous systems under real,
production use. I was in our NOC on the first New Year's Eve when there
was a firstnight.org web site, which we were hosting (I think it was
1995-1996). We had no way of predicting how much activity it would get,
and whether it could handle the load, since this was practically an
unprecedented event on the net. Our web hosting engineers were there all
night trying to keep it from dropping dead due to the overload.

--

Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

by **David Schwart** » Fri, 28 Jun 2002 08:51:48

> > There really is no good way to learn to implement a scalable server
> > other than actual deployment.
> That's an unfortunate truth about computer systems. It's one of the things
> that causes other engineering disciplines to look down on us. Could you
> imagine architects saying "There's no good way to learn to design a
> skyscraper other than actually building one, and see whether it stays
> standing"? I wouldn't want to be working in that new architect's first
> building on its first windy day. (Although architects have an impressive
> track record, there have also been some spectacular disasters, such as the
> Tacoma Narrows Bridge.)

One example that comes to mind is all the network meltdowns that Sprint
had. They were the largest network, and there was no way to simulate its
complexity and interactions with other networks. So they hit every
scalability wall and load-related failure first.

Quote:> The sad fact is that most server designers learned how to build scalable
> servers by examining the failure modes of previous systems under real,
> production use.

That's why a programmer has to stay current in dozens of ways. It's
also why you can't really learn how to program in a classroom or from a
book that only explains what things do.

Quote:> I was in our NOC on the first New Year's Eve when there
> was a firstnight.org web site, which we were hosting (I think it was
> 1995-1996). We had no way of predicting how much activity it would get,
> and whether it could handle the load, since this was practically an
> unprecedented event on the net. Our web hosting engineers were there all
> night trying to keep it from dropping dead due to the overload.

I wish I had a solution. It's not a good position when you have to say
to someone "this way works better than that way, here are a few reasons
why that should be so, but I can't prove it's so from the reasons, I
just know it to be so from extensive testing."

DS