How to configure "makedbm" to handle large entry?

How to configure "makedbm" to handle large entry?

Post by Peter L. Su » Fri, 29 Oct 1999 04:00:00



Hi, there
I had a large entry in /var/yp/files/group ( about 1160 bytes) and I
can't do "make group",(error message is "Problem storing  bla...) the
problem seems to be "makedbm" cannot handle large entry, does anybody
know how to configure "makedbm" to handle large entry, or is there any
other utilities I can use to make my group file available on "YP"?

thanks,
Peter

 
 
 

How to configure "makedbm" to handle large entry?

Post by Richard L. Hamilt » Sat, 30 Oct 1999 04:00:00


Since NIS (formerly YP) and makedbm depend on the dbm or ndbm
library functions, the following from dbm_clearerr(3) (or ndbm(3)
on systems that have that page) and dbm(3b) applies:

     The sum of the sizes of a key/content pair must  not  exceed
     the  internal  block  size (currently 1024 bytes).  Moreover
     all key/content pairs that hash together must fit on a  sin-
     gle  block.   dbm_store()  will return an error in the event
     that a disk block fills with inseparable data.

There are some implementations of NIS for Linux that use gdbm
(or maybe Berkeley DB) files for storage that don't suffer from
that limitation, however precisely because they have a capability
that exceeds that of Solaris NIS servers, they aren't compatible,
i.e. a Linux master could not push out to a Solaris slave successfully.

NIS+ can go higher, at least 4KB, and up to 8KB in later versions and/or
patch levels.  However, a 4KB limit would still apply for anything being
accessed via the nsswitch.conf mechanism; that includes information
comparable to what /etc/group contains.

The usual workaround for group information is something like:

grname::2000:userid1,userid2,userid3,...useridn
grname1::2000:useridn+1,useridn+2,...

with /etc/group, that looks fairly good when doing an ls -l, because
it will consistently pick up the first entry.  But unfortunately with
NIS or NIS+, the entries are accessed in an order that depends on a
hash of the key field, so getting ls -l to display grname rather than
grname1 consistently (i.e. as other entries are changed) may not be
possible.  That won't generally break anything, as either could in
most cases be used interchangably, but it's ugly.

I've mentioned this before with reference to cases where it would have
been desirable to have groups with thousands of members, but haven't
really gotten anything out of it beyond what I've discussed above.



Quote:> Hi, there
> I had a large entry in /var/yp/files/group ( about 1160 bytes) and I
> can't do "make group",(error message is "Problem storing  bla...) the
> problem seems to be "makedbm" cannot handle large entry, does anybody
> know how to configure "makedbm" to handle large entry, or is there any
> other utilities I can use to make my group file available on "YP"?

> thanks,
> Peter

--
ftp> get |fortune
377 I/O error: smart remark generator failed

Bogonics: the primary language inside the Beltway



 
 
 

How to configure "makedbm" to handle large entry?

Post by William Mall » Sat, 30 Oct 1999 04:00:00


Good answers, but somewhat incomplete.

First it is not the size of dbm entries that makes servers that do not
use dbm incompatible with NIS servers, but rather because dbm is
actually part of the NIS replication protocol.  Thus when you change to
a new database format (like gdbm) you are NOT NIS anymore (but a variant).

Linux should not be calling it NIS (or YP) nor using the same RPC protocol
numbers or in any way confusing the issue (but I would advise against
them calling it NIS+ for obvious reaons).  Perhaps they should call it
OJT or ZQ (NIS or YP shifted one letter forward in the alphabet).  It is
certainly no longer NIS.  This is the reason the 1K limit can never be "fixed".

Second the limit in Solaris is the Name Service Switch (see nsswitch.conf(4))
which has a 4K limit.  NIS+ can handle 8K (or much more) but if you cannot
access it when you call getgrnam(3C) what is the point?  Solaris 8 will have
LDAP support, but the NSS still has the same 4K limit (and the explanation for
why it is not increased is long, but basically there is no need given the NIS
1K limit).

Use the "hack" or get another name service, NIS+ (or in the future LDAP)
or hack your own variant.  But please don't call something NIS if it will
not talk to all the NIS servers deployed on HPUX, Digitial UNIX, SGI, AIX,
BSD, Linux and Solaris 1 (SunOS 4.x)  and Solaris 2 (and who knows where else)

=wpm    William P. Malloy



>Since NIS (formerly YP) and makedbm depend on the dbm or ndbm
>library functions, the following from dbm_clearerr(3) (or ndbm(3)
>on systems that have that page) and dbm(3b) applies:

>     The sum of the sizes of a key/content pair must  not  exceed
>     the  internal  block  size (currently 1024 bytes).  Moreover
>     all key/content pairs that hash together must fit on a  sin-
>     gle  block.   dbm_store()  will return an error in the event
>     that a disk block fills with inseparable data.

>There are some implementations of NIS for Linux that use gdbm
>(or maybe Berkeley DB) files for storage that don't suffer from
>that limitation, however precisely because they have a capability
>that exceeds that of Solaris NIS servers, they aren't compatible,
>i.e. a Linux master could not push out to a Solaris slave successfully.

>NIS+ can go higher, at least 4KB, and up to 8KB in later versions and/or
>patch levels.  However, a 4KB limit would still apply for anything being
>accessed via the nsswitch.conf mechanism; that includes information
>comparable to what /etc/group contains.

>The usual workaround for group information is something like:

>grname::2000:userid1,userid2,userid3,...useridn
>grname1::2000:useridn+1,useridn+2,...

>with /etc/group, that looks fairly good when doing an ls -l, because
>it will consistently pick up the first entry.  But unfortunately with
>NIS or NIS+, the entries are accessed in an order that depends on a
>hash of the key field, so getting ls -l to display grname rather than
>grname1 consistently (i.e. as other entries are changed) may not be
>possible.  That won't generally break anything, as either could in
>most cases be used interchangably, but it's ugly.

>I've mentioned this before with reference to cases where it would have
>been desirable to have groups with thousands of members, but haven't
>really gotten anything out of it beyond what I've discussed above.



>> Hi, there
>> I had a large entry in /var/yp/files/group ( about 1160 bytes) and I
>> can't do "make group",(error message is "Problem storing  bla...) the
>> problem seems to be "makedbm" cannot handle large entry, does anybody
>> know how to configure "makedbm" to handle large entry, or is there any
>> other utilities I can use to make my group file available on "YP"?

>> thanks,
>> Peter

>--
>ftp> get |fortune
>377 I/O error: smart remark generator failed

>Bogonics: the primary language inside the Beltway



 
 
 

How to configure "makedbm" to handle large entry?

Post by Kjetil Torgrim Homm » Wed, 03 Nov 1999 04:00:00


[William Malloy]

Quote:>   First it is not the size of dbm entries that makes servers that do
>   not use dbm incompatible with NIS servers, but rather because dbm
>   is actually part of the NIS replication protocol.  Thus when you
>   change to a new database format (like gdbm) you are NOT NIS
>   anymore (but a variant).

That's a bit harsh, I think.  You are only changing the ypxfr service.
YP clients will not notice the difference.

Quote:>   Use the "hack" or get another name service, NIS+ (or in the future
>   LDAP) or hack your own variant.  But please don't call something
>   NIS if it will not talk to all the NIS servers deployed on HPUX,
>   Digitial UNIX, SGI, AIX, BSD, Linux and Solaris 1 (SunOS 4.x) and
>   Solaris 2 (and who knows where else)

The computer centre here had to switch to the Linux implementation on
all of the above platforms (and some more) when the hash algorithm in
dbm gave a netgroup.pag file larger than 4 GB...  The GDBM non-sparse
counterpart is currently 1.5 MB, a bit more palatable to Ultrix :-)

Kjetil T.

 
 
 

How to configure "makedbm" to handle large entry?

Post by Casper H.S. Dik - Network Security Engine » Wed, 03 Nov 1999 04:00:00


[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


Quote:>[William Malloy]
>>   First it is not the size of dbm entries that makes servers that do
>>   not use dbm incompatible with NIS servers, but rather because dbm
>>   is actually part of the NIS replication protocol.  Thus when you
>>   change to a new database format (like gdbm) you are NOT NIS
>>   anymore (but a variant).
>That's a bit harsh, I think.  You are only changing the ypxfr service.
>YP clients will not notice the difference.

William is not completely right but also not completely wrong.

The 1K limitation exists in both dbm and the protocol; the NIS
respones cannot be longer than 1K.

ypxfr uses a protocol that sends over DBM files; it reverts to an
entry by entry protocol if it can't use the hacked DBM protocol.

Quote:>The computer centre here had to switch to the Linux implementation on
>all of the above platforms (and some more) when the hash algorithm in
>dbm gave a netgroup.pag file larger than 4 GB...  The GDBM non-sparse
>counterpart is currently 1.5 MB, a bit more palatable to Ultrix :-)

As long as the entries themselves remain under 1K each (1K for the key,
1K for the value,) there should be no problem with interoperability.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

1. files created with "makedbm" can not used with "dbm_fetch"

I created a dbm file:

[julie:/export/home/root/C]echo a b | makedbm - db

I then wrote a C program to read it:

[julie:/export/home/root/C]more query_dbm.c
#include <ndbm.h>
#include <stdio.h>
#include <fcntl.h>

#define NAME    "a"
#define DB_NAME "db"

main()
{
DBM *db;
datum name = {NAME, sizeof (NAME)};
datum val;

db = dbm_open(DB_NAME, O_RDONLY, 0600);
val = dbm_fetch(db, name);
(void) printf("Name: %s, Value: %s\n", name.dptr, val.dptr);
dbm_close(db);
return (0);

But it cannot:

[julie:/export/home/root/C]gcc query_dbm.c
[julie:/export/home/root/C]./a.out                
Segmentation Fault(coredump)

What is the deal? Thanks.
--
Michael Wang
http://www.mindspring.com/~mwang

2. Keyboard dies during install of COL 1.3

3. " No CDDB Entry" "Error Getting CDDB Entry" ???

4. InfoMagic "March Edition" Linux 1.2.1

5. GETSERVBYNAME()????????????????????"""""""""""""

6. Can I get program name without using argv[0]?

7. """"""""My SoundBlast 16 pnp isn't up yet""""""""""""

8. Get part of content of a text file.

9. Weird :- "show-devs" OK but no" /devices" entry ??

10. how to map "/dev/root" to "/proc/partitions" entry in user prog?

11. Type "(", ")" and "{", "}" in X...

12. "Nice" or not to "nice" large jobs

13. "Standard Journaled File System" vs "Large File Enabled Journaled File System"