New NNTP Server for OpenVMS

New NNTP Server for OpenVMS

Post by David Cathe » Sat, 18 Jan 1997 04:00:00



After working with ANU News for a couple of years, I've
decided that the performance just isn't there for a large feed.
I've had a DEC Alpha dedicated to the task with 6Gig of disk, and
I still have problems with backlogs, high-volume newsgroups (such as
misc.jobs.offered), massively corssposted articles duplicating
space (alt.binaries.*), and NEWSSKIM taking hours to run.  So I've
been developing a NNTP Server for OpenVMS that can provide the
performance and robustness that OpenVMS ought to be delivering.

        I'm currently ready to go "Alpha" with it, by adding some
load and seeing what happens to it.  If you are interested, please
let me know, and I'll add your site to the authorization list.  This
is limited to the first few sites that respond, but I'll take as
many I feel is feasible.  I'd like a peer site or two, and of course,
serveral readers.

The configuration is:

        Hostname: nntp.montagar.com
        Software: VNNTP V0.6
        Hardware: DEC2000-300 w/128Mb
        OperSyst: OpenVMS V6.2
        Disk:     1 RZ29 (4Gb) news spool
        Group:    around 20,000 newsgroups, generally with a 1 day
                  expiration time at present (around 150,000 articles)
        IP SW:    UCX V4.0 (NNTP software uses UCX $QIO's and
                  DECthreads)
        Connect:  T-1 via Sprintlink

The performance characteristics look promising.  I've been able to
reach inbound feed rates of over 2.5Gbytes/day.  Reading an article
via a Netscape client often has transfer rates in excess of 20K/sec.
During use (Sprintlink feed, plus a handful of users), the CPU averages
about 10% utilitization.

        I plan to go "Beta" with this around March 1st, including
tools to migrate existing ANU News hierarchies into the new NNTP
server structure.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Montagar Software Concepts           |Fone: (972)-578-5036
P. O. Box 260772, Plano, TX 75026    |http://www.montagar.com/~davidc/
Postmaster, Hostmaster, Webmaster    |http://www.montagar.com

 
 
 

New NNTP Server for OpenVMS

Post by James Kirkpatri » Wed, 22 Jan 1997 04:00:00


|>After working with ANU News for a couple of years, I've
|>decided that the performance just isn't there for a large feed.
|>I've had a DEC Alpha dedicated to the task with 6Gig of disk, and
|>I still have problems with backlogs, high-volume newsgroups (such as
|>misc.jobs.offered), massively corssposted articles duplicating
|>space (alt.binaries.*), and NEWSSKIM taking hours to run.  So I've
|>been developing a NNTP Server for OpenVMS that can provide the
|>performance and robustness that OpenVMS ought to be delivering.
|>
|>   I'm currently ready to go "Alpha" with it ...

I'd be very interested in hearing a brief description of how it is
implemented.  For example, one of the big failings of ANU is that it
stores each article as a separate file so you get into wasted space
due to cluster size, and gross inefficiencies when the directory goes
over 127 blocks.  How do you store the articles?

A second major ANU failing is the use of RMS indexed files for the NEWS.ITEMS
and history files, which become extremely awful due to random insertions (and
even worse with non-random insertions when we see a "cancel storm") and
thus need to be rebuilt frequently.  We keep articles for 3 days and a
full skim is taking over 24 hours.  How do you index the articles, and
keep track of history?

Just curious!

Jim

 
 
 

New NNTP Server for OpenVMS

Post by David Cathe » Thu, 23 Jan 1997 04:00:00



> I'd be very interested in hearing a brief description of how it is
> implemented.  For example, one of the big failings of ANU is that it
> stores each article as a separate file so you get into wasted space
> due to cluster size, and gross inefficiencies when the directory goes
> over 127 blocks.  How do you store the articles?

        I'm not going to go into*detail here, but it's done as
a series of container files.  Since lots of articles are stored in
several very large files, I can place the articles closer together than
the
allocation size would normally permit.  Since any newsgroup can
reference
the same article text from the container file, it also eliminates the
duplication that ANU does for crossposted articles.  Currently, the
granularity is in 2 blocks increments.

        This also eliminates the use of directories as a "database"
for organizing articles in a newsgroup - and the related 127 block
directory performance issue.

Quote:> A second major ANU failing is the use of RMS indexed files for the NEWS.ITEMS
> and history files, which become extremely awful due to random insertions (and
> even worse with non-random insertions when we see a "cancel storm") and
> thus need to be rebuilt frequently.  We keep articles for 3 days and a
> full skim is taking over 24 hours.  How do you index the articles, and
> keep track of history?

        No ISAM files.  Articles are hashed and internally keyed.  With
the key, it can retrieve an the full article in 2 I/O's, or 1 I/O if the
hash block is in cache.  Random/non-random inserts isn't an issue.  The
skim just walks the hash table, freeing up container file space as
needed.
Since it's only marking container space free, the I/O normally used to
open-delete-close all the articles files is eliminated.

        Currently, the server is able to "skim" over 200,000 articles in
about 10 minutes to an hour depending on server load (feed, readers,
number
of articles that need to be deleted, etc).  The "skim" runs several
times
a day, so the average article count remains more constant than the
large flucuations one gets by doing it only once a day.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Montagar Software Concepts           |Fone: (972)-578-5036
P. O. Box 260772, Plano, TX 75026    |http://www.veryComputer.com/~davidc/
Postmaster, Hostmaster, Webmaster    |http://www.veryComputer.com/

 
 
 

1. NEWSRDR 4.9-2 and/or nntp server Y2K glitch

Hi all,

As of Jan 1 newsrdr 4.9-2 won't work with our campus nntp server.
Newsrdr thinks all newsgroups are new every time it starts.
When it finishes loading the entire list of newsgroups it lists
the article ranges like [1234,0], that is, no readable articles.

My guess is that it's at the server end, but can't say for sure.

Anybody else seeing this? (In which case you're probably not reading
this message with NEWSRDR!)

Anybody know a fix?  If so, please email me directly as I'm going to be
news impaired until this is repaired.

Thanks,

David Mathog

2. Palm V, Is there a dual "different action" stylus available?

3. Announcing DNEWS 1.0c NNTP Server (NT, VMS, UNIX)

4. Where

5. NNTP Server wanted

6. info wanted: $[$] address rewriting

7. VAX/VMS NNTP SERVER/CLIENT

8. Getting the bits of a bitmap

9. nntp server for vms?

10. How can I create a VMS news server (nntp)?

11. Create directory on NT 4 server using Advanced Server on OpenVMS

12. OpenVMS Internet Features (was OpenVMS 7.0 New Features)

13. Simulatenously transferring a 1GB file from VMS to two Windows 2k servers servers servers