Why does code fail to find *exact* amouut of RAM??

Why does code fail to find *exact* amouut of RAM??

Post by Dr. David Kirkb » Tue, 02 Sep 2003 16:16:48



I've tried sending this about 5 times, and each time it seems to have
gone to /dev/null/.

Most operating systems (Linux, Solaris, AIX, Tru64, HP-UX, IRIX being
examples) support the sysconf() function to get all sorts of
information.

SYNOPSIS
     #include <unistd.h>
     long sysconf(int name);

Most OS's (IRIX, Tru64, HP-UX  being exceptions) support a method
where one
might expect to be able to determine the amount of RAM in the system
with sysconf(), but whilst it works *exactly* on some systems, it is
only *approximate* on others.

The size of memory pages (in bytes) can be found -
sysconf(_SC_PAGE_SIZE);
and the number of memory pages too - sysconf(_SC_PHYS_PAGES);

Multiplying the two together (taking care not to overflow), gives the
number of bytes of RAM the computer has - on some systems anyway.
Dividing by the number of bytes in a Mb (1048576), gives the RAM in Mb
- or so I thought. Is this not right ?????

This works exactly on Solaris 9 and exactly under AIX 5.2 too. But
under Linux it says 2017 on a system with 2 Gb, and 3886 Mb on a
system with 4 Gb.

Does anyone know if these calls are supposed to work in the way I
think ? Are they defined by  POSIX, and if so how ? It seems odd that
Solaris, Linux and AIX all support them, but Tru64, IRIX and HPUX
don't. HP-UX 11 has _SC_PAGE_SIZE, but not _SC_PHYS_PAGES.

The linux man page (for my Redhat system - 6.2 I think) says:

_SC_PAGESIZE _SC_PAGE_SIZE
    The size of a page (in bytes).

_SC_PHYS_PAGES
    The  number of pages of physical memory.  (Note that it is
possible for the product of this value and the value of _SC_PAGE_SIZE
to overflow.)

So the Linux man page is suggesting one might multiply
sysconf(_SC_PAGE_SIZE)*sysconf(_SC_PHYS_PAGES), so the man page
clearly thinks there is some use to doing this.

Any thoughts ????
--
Dr. David Kirkby,
Senior Research Fellow,
Department of Medical Physics,
University College London,
11-20 Capper St, London, WC1E 6JA.
Website: http://www.medphys.ucl.ac.uk/~davek
Author of 'atlc' http://atlc.sourceforge.net/

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by p.. » Tue, 02 Sep 2003 20:03:04




Quote:>_SC_PHYS_PAGES
>    The  number of pages of physical memory.  (Note that it is
>possible for the product of this value and the value of _SC_PAGE_SIZE
>to overflow.)

>So the Linux man page is suggesting one might multiply
>sysconf(_SC_PAGE_SIZE)*sysconf(_SC_PHYS_PAGES), so the man page
>clearly thinks there is some use to doing this.

>Any thoughts ????

Perhaps it is returned the number of physical pages available to user
space programs? Some parts of the 4Gb address space are not available
there's PCI address space mappings in there, and there's the kernel
itself of course.

So on a 4Gb box at least there will be portions of the address space
which are unavailable, therefore not all memory can be used by user
space programs.

Phil

--
http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by David W Noo » Tue, 02 Sep 2003 21:12:22






>>_SC_PHYS_PAGES
>>    The  number of pages of physical memory.  (Note that it is
>>possible for the product of this value and the value of _SC_PAGE_SIZE
>>to overflow.)

>>So the Linux man page is suggesting one might multiply
>>sysconf(_SC_PAGE_SIZE)*sysconf(_SC_PHYS_PAGES), so the man page
>>clearly thinks there is some use to doing this.

>>Any thoughts ????

> Perhaps it is returned the number of physical pages available to user
> space programs? Some parts of the 4Gb address space are not available
> there's PCI address space mappings in there, and there's the kernel
> itself of course.

Nice theory, except that's the virtual address space, not the physical
address space. AFAIAA, the PCI addresses are all on the northbridge
chipset, not the motherboard's RAM, and accessed via I/O ports 0x0CF8 and
0x0CFC.

Quote:> So on a 4Gb box at least there will be portions of the address space
> which are unavailable, therefore not all memory can be used by user
> space programs.

The parts of the physcial address space that are inaccessible to user space
will be architecture dependent. On Intel 80x86 and AMD boxes, this will be
areas such as adapter RAM, VGA aperture (maybe! or maybe not) and other
areas used for memory-mapped I/O access and I/O control.

I see no reason to exclude these from sysconf(_SC_PHYS_PAGES), as they might
be of interest even if they can't be touched. After all, there is no
guarantee to any user space code that any particular page(s) of RAM should
be accessible, merely that these pages exist. We should be aware that
sysconf(_SC_AVPHYS_PAGES) will return the number of pages of physical
memory available (whatever "available" means under Linux).

Anybody who is interested can try this code:
=============================== bigbox.c ================================
/* Program to state the size of the current machine. */

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main(void)
{
   const unsigned long page_size = sysconf(_SC_PAGESIZE);
   const unsigned long no_pages = sysconf(_SC_PHYS_PAGES);
   const unsigned long box_size = page_size*no_pages;

   printf("Page size = %lu.  No. pages = %lu.\n"
         "This machine has %lu bytes, %lu KiB or %lu MiB of RAM.\n",
         page_size, no_pages, box_size, box_size >> 10, box_size >> 20);

   return 0;

Quote:}

=========================================================================

To compile:
gcc -o bigbox -Wall -O3 bigbox.c

To run:
./bigbox

[Note that this program does *not* require root permissions.]

On my machine (an AMD K6-III) it reports a size about 8 megs too small.
--
Regards,

Dave  [RLU#314465]
======================================================

Remove spam trap to reply via e-mail.
======================================================

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by Eric Gisi » Tue, 02 Sep 2003 23:02:45




|
| Perhaps it is returned the number of physical pages available to user
| space programs? Some parts of the 4Gb address space are not available
| there's PCI address space mappings in there, and there's the kernel
| itself of course.
|
There is no such thing as physical pages available in user mode. Yes, Linux
and Win2k remove any PCI space mappings from a machine with 4GB.

| So on a 4Gb box at least there will be portions of the address space
| which are unavailable, therefore not all memory can be used by user
| space programs.
|

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by p.. » Wed, 03 Sep 2003 03:12:02






>|
>| Perhaps it is returned the number of physical pages available to user
>| space programs? Some parts of the 4Gb address space are not available
>| there's PCI address space mappings in there, and there's the kernel
>| itself of course.
>|
>There is no such thing as physical pages available in user mode. Yes, Linux
>and Win2k remove any PCI space mappings from a machine with 4GB.

Obviously the memory a user space sees is virtually addressed, however
only that which is backed by RAM can be in physical pages; the rest
has to be elsewhere.

Perhaps I should have said 'amount of virtual memory which can be
backed by RAM at any one point' instead.

Phil
--
http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by p.. » Wed, 03 Sep 2003 03:13:12





>>There is no such thing as physical pages available in user mode. Yes, Linux
>>and Win2k remove any PCI space mappings from a machine with 4GB.

>Obviously the memory a user space sees is virtually addressed, however

                            ^process

bother.

Phil

--
http://www.kantaka.co.uk/ .oOo. public key: http://www.kantaka.co.uk/gpg.txt

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by Kasper Dupon » Wed, 03 Sep 2003 05:53:57



> Perhaps it is returned the number of physical pages available to user
> space programs? Some parts of the 4Gb address space are not available
> there's PCI address space mappings in there, and there's the kernel
> itself of course.

The part about the kernel itself agree with my observations. But the
difference between the physical amount of memory and the reported size
is larger than the kernel itself.

You will find that most of this difference is the mem_map array. It
consists of one 68 byte large struct per page of memory. So you have
to substract about 1.7% from the physical memory size.

I wrote something about that in the FAQ just one week ago:
http://www.daimi.au.dk/~kasperd/comp.os.linux.development.faq.html#me...

--
Kasper Dupont -- der bruger for meget tid paa usenet.

Their business was zero and it was shrinking.

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by Dr. David Kirkb » Wed, 03 Sep 2003 11:51:40



> The size of memory pages (in bytes) can be found -
> sysconf(_SC_PAGE_SIZE);
> and the number of memory pages too - sysconf(_SC_PHYS_PAGES);

> Multiplying the two together (taking care not to overflow), gives the
> number of bytes of RAM the computer has - on some systems anyway.
> Dividing by the number of bytes in a Mb (1048576), gives the RAM in Mb
> - or so I thought. Is this not right ?????

> This works exactly on Solaris 9 and exactly under AIX 5.2 too. But
> under Linux it says 2017 on a system with 2 Gb, and 3886 Mb on a
> system with 4 Gb.

Just a quick addition to this - a very clear pattern is observed.

I have tried to obtain the RAM (and other hardware parameters) of
quite a few systems. The method used differs a little from one system
to another, but the man pages for each method claim the physical
memory will be obtained.

Commercial UNIX's - all report the *exact* physical memory.
1) Sun Ultra 80 running Solaris 9
2) HP C3000 running HP-UX 11
3) SGI Octane running IRIX 6.5.16 (or there abouts)
4) Dec Alpha running Tru64 5.1B
5) IBM RS/6000 running AIX 5.2

Commercial UNIX - so far no method found to determine the memory.
6) Cray Y-MP running UNICOS 9 - unable to determine ram.  Anyone know
how ?

Free UNIX/Linux systems, both produce an approximation to the physical
memory.

6) HP AlphaStation X1000 running FreeBSD 4.8 - inexact but very close
(2 Mb error)
7) Linux systems (various), all give inexact values.

So it is quite clear the physical memory reported by a system is
always exact in the case of the commercial operating systems, but
never so on the free ones.

PS - if you want to look the code:

The files sysdata.c, try_portable.c, try_aix.c, try_bsd.c, try_hpux.c,
try_irix.c, try solaris.c, try_tru64.c, try_unicos.c and defs.h at the
bottom of here:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/atlc/sources/tests/
have the code to get this information (ram, cpu type, #cps, number of
cpus supported on system, cache configuration etc). The binary
'sysdata' will report them. Note at the time of posting this,
Sourceforge is running on a backup server, so data is up to 24 hours
old and some of the files are not visable. There is currently no
package (.tar.gz) with these changes, so if you want to look at it,
you will need to look at the CVS tree.

--
Dr. David Kirkby,
Senior Research Fellow,
Department of Medical Physics,
University College London,
11-20 Capper St, London, WC1E 6JA.
Website: http://www.medphys.ucl.ac.uk/~davek
Author of 'atlc' http://atlc.sourceforge.net/

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by v.. » Wed, 03 Sep 2003 16:37:27




Quote:>So it is quite clear the physical memory reported by a system is
>always exact in the case of the commercial operating systems, but
>never so on the free ones.

You do realize, of course, that about the only use of such information
is sysinfo(8)?  For one thing, kernel code and data structures take
some space.  Moreover, programs do not run alone and knowing the amount
of RAM in the system is fairly pointless as far as individual programs
count.  If you need some part of address space in core (e.g. for RT-ish
stuff), you'd better use mlock(2) and it will tell you if it fails...
 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by Gavin Maltb » Wed, 03 Sep 2003 21:14:33



>>So it is quite clear the physical memory reported by a system is
>>always exact in the case of the commercial operating systems, but
>>never so on the free ones.

For the record, Solaris/SPARC (all I checked) does report the installed
physical memory as originally reported by OBP and perhaps later
modified by DR operations that introduce or remove memory.

Quote:> You do realize, of course, that about the only use of such information
> is sysinfo(8)?  For one thing, kernel code and data structures take
> some space.  Moreover, programs do not run alone and knowing the amount
> of RAM in the system is fairly pointless as far as individual programs
> count.  If you need some part of address space in core (e.g. for RT-ish
> stuff), you'd better use mlock(2) and it will tell you if it fails...

Not sure I agree.  A database, for instance, could try to size it's
SGA (for Oracle, dunno what others call it) dynamically within
physical memory.  Algorithms based on failed allocation attempts
seem messy, especially if you want them to work on system with
256MB of memory up to those with hundreds or thousands of GB.
But it would be better of starting with _SC_AVPHYS_PAGES rather
than _SC_PHYS_PAGES.  sort uses that for it's -S option.

Gavin

 
 
 

Why does code fail to find *exact* amouut of RAM??

Post by Kasper Dupon » Thu, 04 Sep 2003 01:16:52




> >>So it is quite clear the physical memory reported by a system is
> >>always exact in the case of the commercial operating systems, but
> >>never so on the free ones.

> For the record, Solaris/SPARC (all I checked) does report the installed
> physical memory as originally reported by OBP and perhaps later
> modified by DR operations that introduce or remove memory.

> > You do realize, of course, that about the only use of such information
> > is sysinfo(8)?  For one thing, kernel code and data structures take
> > some space.  Moreover, programs do not run alone and knowing the amount
> > of RAM in the system is fairly pointless as far as individual programs
> > count.  If you need some part of address space in core (e.g. for RT-ish
> > stuff), you'd better use mlock(2) and it will tell you if it fails...

> Not sure I agree.  A database, for instance, could try to size it's
> SGA (for Oracle, dunno what others call it) dynamically within
> physical memory.

In that case the value returned by Linux is more usable than the
total amount of physical memory. Of course still an application
shouldn't try to allocate that much, for performance it is better
to keep your executable, libraries, and some other stuff in
memory all the time.

Quote:> Algorithms based on failed allocation attempts
> seem messy, especially if you want them to work on system with
> 256MB of memory up to those with hundreds or thousands of GB.

It is going to be inefficient as well, because allocations will
first fail when you start to run out of swap, and your performance
will be bad because of trashing when trying to use too much
memory. And that is even the best case, in the worst case your
allocations will suceed, and eventually you get killed when the
memory actually runs out.

Quote:> But it would be better of starting with _SC_AVPHYS_PAGES rather
> than _SC_PHYS_PAGES.  sort uses that for it's -S option.

I don't know about other systems, but on Linux _SC_AVPHYS_PAGES
would be pretty useless for those purposes. It returns only the
free memory, which can be pretty small because of caching. In
some cases you want your programs to allocate enough to cause
caches to shrink and maybe even cause other programs to get
swapped out.

--
Kasper Dupont -- der bruger for meget tid paa usenet.

Their business was zero and it was shrinking.

 
 
 

1. Why oh why does DOS/Windoze work while Linux fails?

Why is it that when I have hardware problems I can use DOS
and Windoze but Linux just hangs?!

Thanks, Roger

+------------------------------------------------------------+
|Roger B. Rouse                      + # # # .               |
|                                  #           . +           |
|Arizona State University        #       + +     #           |
|Dept. Physics & Astronomy           + +   + #     +         |
|Tempe, Az, 85287-1504               +       +     #         |


|                                  +     # +   + +           |
|                                    #     + +      .        |
|                                    + .           #         |
|                                        . # + # .      Rouse|
+------------------------------------------------------------+

2. JNI problem under solaris

3. make bzImage fails on ram disk code in 2.5.5

4. unix audio programmer urgently required

5. Why is find so slow (Re: Why use find?)

6. mystery page faults

7. [RAMDISK] swapon /dev/ram fails from rc.*, succeeds manually - why?

8. Bash, get ASCII value of a character

9. How to get the exact trap code from an mdb ::findstack call frames

10. Exact meaning of C code

11. How to find out why sol 10 manifest fails?

12. copying / cloning a system disk, can't find an exact answer.