Very Computer

by **prasha..** » Wed, 30 Apr 2003 10:06:25

Hi,

I'm looking for information on the scalabality of the LISTEN/NOTIFY
mechanism. How well does it scale with respect to:

- hundreds of clients registered for LISTENs

I guess this translates to hundreds of the corresponding backend
processes receiving SIG_USR2 signals. The efficiency of this is
probably OS-dependent. Would anyone be in a position to give me
signal delivery benchmarks for FreeBSD on Unix?

- each client registered for thousands of LISTENs

From a look at backend/commands/async.c, it would seem that each
listening backend would get a signal for *every* LISTEN it
registered for, resulting in thousands of signals to the same
listening backend, instead of only one. Would it help if this was
optimized so that a signal was sent only once? Again, info on
relevant signal delivery benchmarks would be useful.

I'm not an expert on signals, not even a novice, so I might be totally
off base, but it seems like the Async Notification implementation does
not scale. If it does not, does anyone have a solution for the
problem of signalling a each event in a possibly very large set of
events to a large number of clients?

Thanks,

--prashanth

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

by **Tom La** » Wed, 30 Apr 2003 11:21:40

> I'm not an expert on signals, not even a novice, so I might be totally
> off base, but it seems like the Async Notification implementation does
> not scale.

Very possibly. You didn't even mention the problems that would occur if
the pg_listener table didn't get vacuumed often enough.

The pghackers archives contain some discussion about reimplementing
listen/notify using a non-table-based infrastructure. But AFAIK no one
has picked up that task yet.

regards, tom lane

---------------------------(end of broadcast)---------------------------

by **Hannu Krosi** » Wed, 30 Apr 2003 16:39:44

Quote:> Hi,

> I'm looking for information on the scalabality of the LISTEN/NOTIFY
> mechanism. How well does it scale with respect to:

> - hundreds of clients registered for LISTENs

> I guess this translates to hundreds of the corresponding backend
> processes receiving SIG_USR2 signals. The efficiency of this is
> probably OS-dependent. Would anyone be in a position to give me
> signal delivery benchmarks for FreeBSD on Unix?

> - each client registered for thousands of LISTENs

> From a look at backend/commands/async.c, it would seem that each
> listening backend would get a signal for *every* LISTEN it
> registered for, resulting in thousands of signals to the same
> listening backend, instead of only one.

But as the signals are usually generated async, you have no way to know
if a particular backend has already received a signal.

Or do you mean some mechanism that remembers "signals sent" in some
shared structure that the receiving backend can then clear when it
actually receives the signal ?

That could mean lock contention on that shared structure, unless we
decide that it is cheaper to just consult it without locking it and
accept an occasional delivery of unneeded signals.

Quote:> Would it help if this was
> optimized so that a signal was sent only once? Again, info on
> relevant signal delivery benchmarks would be useful.

I still suspect that replacing pg_listener table from the mechanism
would give gains faster. Of course we could rework the signal mechanism
as well while doing it.

Quote:> I'm not an expert on signals, not even a novice, so I might be totally
> off base, but it seems like the Async Notification implementation does
> not scale. If it does not, does anyone have a solution for the
> problem of signalling a each event in a possibly very large set of
> events to a large number of clients?

-----------------
Hannu

---------------------------(end of broadcast)---------------------------

by **prasha..** » Thu, 01 May 2003 04:38:24

> > I'm not an expert on signals, not even a novice, so I might be totally
> > off base, but it seems like the Async Notification implementation does
> > not scale.

> Very possibly. You didn't even mention the problems that would occur if
> the pg_listener table didn't get vacuumed often enough.

> The pghackers archives contain some discussion about reimplementing
> listen/notify using a non-table-based infrastructure. But AFAIK no one
> has picked up that task yet.

I found some messages in 03/2002 that also brought up the performance
issue. You had suggested the use of shared-memory, and made reference
to a "SI model". I did find see any alternative non-table-based
suggestions. What is the "SI model"?

Thanks,

--prashanth

---------------------------(end of broadcast)---------------------------

by **prasha..** » Thu, 01 May 2003 04:45:42

> > - each client registered for thousands of LISTENs

> > From a look at backend/commands/async.c, it would seem that each
> > listening backend would get a signal for *every* LISTEN it
> > registered for, resulting in thousands of signals to the same
> > listening backend, instead of only one.

> But as the signals are usually generated async, you have no way to know
> if a particular backend has already received a signal.

> Or do you mean some mechanism that remembers "signals sent" in some
> shared structure that the receiving backend can then clear when it
> actually receives the signal ?

No, I meant that a listening backend process would be sent multiple
signals from a notifying process, *in the inner loop* of
backend/commands/async.c:AtCommit_Notify().

If the listening backend had registered tens of thousands of LISTENs,
it would be sent an equivalent number of signals during a single run
of AtCommit_Notify(). I'm not sure what the cost of this is, since
I'm not sure how signal delivery works, but the tens of thousands of
system calls cannot be very cheap.

--prashanth

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate

message can get through to the mailing list cleanly

by **Tom La** » Thu, 01 May 2003 04:52:11

> I found some messages in 03/2002 that also brought up the performance
> issue. You had suggested the use of shared-memory, and made reference
> to a "SI model". I did find see any alternative non-table-based
> suggestions. What is the "SI model"?

I meant following the example of the existing shared-cache-invalidation
signaling mechanism --- see
src/backend/storage/ipc/sinvaladt.c
src/backend/storage/ipc/sinval.c
src/include/storage/sinvaladt.h
src/include/storage/sinval.h

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

by **Tom La** » Thu, 01 May 2003 07:22:22

> If the listening backend had registered tens of thousands of LISTENs,
> it would be sent an equivalent number of signals during a single run
> of AtCommit_Notify().

Not unless the notifier had notified all tens of thousands of condition
names in a single transaction.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

Very Computer

LISTEN/NOTIFY benchmarks?

LISTEN/NOTIFY benchmarks?

LISTEN/NOTIFY benchmarks?

LISTEN/NOTIFY benchmarks?

LISTEN/NOTIFY benchmarks?

LISTEN/NOTIFY benchmarks?

LISTEN/NOTIFY benchmarks?

LISTEN/NOTIFY benchmarks?