Unix process/thread RAM limits

Unix process/thread RAM limits

Post by Grant Jacob » Thu, 08 Nov 2001 13:01:31



Can anyone tell me or point me to a source indicating the process/thread
RAM limits for different Unixes?
Alternatively, I'd be happy if anyone can tell me what their systems
(max) limits are.

I am trying to get a handle of what the maximum allocatable space might
be for OSes in order to store a very large data structure in RAM
(potentially >2Gb).

Any info. will be appreciated.

Grant

 
 
 

Unix process/thread RAM limits

Post by Casper H.S. Dik - Network Security Engine » Thu, 08 Nov 2001 19:46:12


[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


>Can anyone tell me or point me to a source indicating the process/thread
>RAM limits for different Unixes?
>Alternatively, I'd be happy if anyone can tell me what their systems
>(max) limits are.

Traditionally, the limits of RAM per process are limited by two things:
        how many bits in a pointer (assuming flat address spaces)

        how much of the address space is used for other mappings
        (traditionally, Unix systems map user and kernel space in one
        address space to make user<->system switches and data copies
        cheaper)

E.g., traditionally in SunOS/Solaris for SPARC the 32-bit kernel is mapped
above the end of the user's stack; the fact that kernels of larger systems
need a bigger address space lead to the strange fact that processes on
larger systems could use less memory. (3.75GB vs 3.5 or 3.25 GB).

However, with the advent of UltraSPARC cpus, a seperate kernel context
was added so swapping back and forth remained cheap but no longer took
away virtual memory from user processes.  On UltraSPARC, 32 bit
processes can use 3.996GB of memory (4GB - 4MB) and not just 3.75GB.

Of course, systems have now progressed to the point where address
spaces are 64 bit and the typical limit is more in the order of
how much RAM you can stuff in a system (like 576GB in Sun's new F15K)

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

 
 
 

Unix process/thread RAM limits

Post by Grant Jacob » Fri, 09 Nov 2001 19:14:51


So, basically, if the limits are set to unlimited by the sysadmin,
process/thread limits are that of the system overall, not some internal OS
limit that limits the limit settings, if you can follow what I'm saying?!

Grant


> [[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]


> >Can anyone tell me or point me to a source indicating the process/thread
> >RAM limits for different Unixes?
> >Alternatively, I'd be happy if anyone can tell me what their systems
> >(max) limits are.

> Traditionally, the limits of RAM per process are limited by two things:
>         how many bits in a pointer (assuming flat address spaces)

>         how much of the address space is used for other mappings
>         (traditionally, Unix systems map user and kernel space in one
>         address space to make user<->system switches and data copies
>         cheaper)

> E.g., traditionally in SunOS/Solaris for SPARC the 32-bit kernel is mapped
> above the end of the user's stack; the fact that kernels of larger systems
> need a bigger address space lead to the strange fact that processes on
> larger systems could use less memory. (3.75GB vs 3.5 or 3.25 GB).

> However, with the advent of UltraSPARC cpus, a seperate kernel context
> was added so swapping back and forth remained cheap but no longer took
> away virtual memory from user processes.  On UltraSPARC, 32 bit
> processes can use 3.996GB of memory (4GB - 4MB) and not just 3.75GB.

> Of course, systems have now progressed to the point where address
> spaces are 64 bit and the typical limit is more in the order of
> how much RAM you can stuff in a system (like 576GB in Sun's new F15K)

> Casper
> --
> Expressed in this posting are my opinions.  They are in no way related
> to opinions held by my employer, Sun Microsystems.
> Statements on Sun products included here are not gospel and may
> be fiction rather than truth.

 
 
 

Unix process/thread RAM limits

Post by Andrew Giert » Fri, 09 Nov 2001 20:11:58


 Grant> So, basically, if the limits are set to unlimited by the
 Grant> sysadmin, process/thread limits are that of the system
 Grant> overall, not some internal OS limit that limits the limit
 Grant> settings, if you can follow what I'm saying?!

There are several limits, all of which apply:

 1. hard and soft process limits set via setrlimit() or the ulimit
    command

 2. system-wide hard limits (usually kernel parameters of some sort)
    set by the administrator which may or may not be reflected in the
    output of getrlimit(), depending on the system

 3. Architectural limits of the OS+hardware combination, which may
    vary between OS releases on the same hardware, or between
    different hardware platforms, or configurable options such as
    32 or 64 bit kernel support on the same hardware.

Obviously both 1 and 2 are up to the sysadmin, so item 3 seems to
correspond to the question you're asking.

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>

 
 
 

Unix process/thread RAM limits

Post by Grant Jacob » Sun, 11 Nov 2001 08:44:41


Thanks, this is useful, but to return to what my original question really was,
what is the *actual* RAM limits in actual OSes? I'm trying to get a feel for
what realistic values I might expect out there, given that I only use small
computers at the present, but the software I am writing could be run on larger
machines...

I guess that simplistically they are related to the RAM the CPU can access,
and that its not often that a system is going to havee threads (say)
restricted to a 32 bit space while the overall system can access a 64 bit
space, etc.Put my question another way: how often is the upper RAM limit of a
thread less that than of the process or system addressable spaces themselves?

Grant



>  Grant> So, basically, if the limits are set to unlimited by the
>  Grant> sysadmin, process/thread limits are that of the system
>  Grant> overall, not some internal OS limit that limits the limit
>  Grant> settings, if you can follow what I'm saying?!

> There are several limits, all of which apply:

>  1. hard and soft process limits set via setrlimit() or the ulimit
>     command

>  2. system-wide hard limits (usually kernel parameters of some sort)
>     set by the administrator which may or may not be reflected in the
>     output of getrlimit(), depending on the system

>  3. Architectural limits of the OS+hardware combination, which may
>     vary between OS releases on the same hardware, or between
>     different hardware platforms, or configurable options such as
>     32 or 64 bit kernel support on the same hardware.

> Obviously both 1 and 2 are up to the sysadmin, so item 3 seems to
> correspond to the question you're asking.

> --
> Andrew.

> comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
>                            or <URL: http://www.whitefang.com/unix/>

 
 
 

Unix process/thread RAM limits

Post by Andrew Giert » Sun, 11 Nov 2001 09:16:34


 Grant> I guess that simplistically they are related to the RAM the
 Grant> CPU can access, and that its not often that a system is going
 Grant> to havee threads (say) restricted to a 32 bit space while the
 Grant> overall system can access a 64 bit space, etc.Put my question
 Grant> another way: how often is the upper RAM limit of a thread less
 Grant> that than of the process or system addressable spaces
 Grant> themselves?

By definition a thread can't have a lower limit on RAM than the process
containing it, because threads don't own resources, processes do. All
threads in a process have access to the same memory space.

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>

 
 
 

Unix process/thread RAM limits

Post by Grant Jacob » Tue, 13 Nov 2001 11:23:03


Quite right and I ought to have thought of this before I asked.

Thanks anyway!,

Grant



>  Grant> I guess that simplistically they are related to the RAM the
>  Grant> CPU can access, and that its not often that a system is going
>  Grant> to havee threads (say) restricted to a 32 bit space while the
>  Grant> overall system can access a 64 bit space, etc.Put my question
>  Grant> another way: how often is the upper RAM limit of a thread less
>  Grant> that than of the process or system addressable spaces
>  Grant> themselves?

> By definition a thread can't have a lower limit on RAM than the process
> containing it, because threads don't own resources, processes do. All
> threads in a process have access to the same memory space.

> --
> Andrew.

> comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
>                            or <URL: http://www.whitefang.com/unix/>

 
 
 

Unix process/thread RAM limits

Post by David Schwart » Wed, 14 Nov 2001 04:16:42



> Thanks, this is useful, but to return to what my original question really was,
> what is the *actual* RAM limits in actual OSes? I'm trying to get a feel for
> what realistic values I might expect out there, given that I only use small
> computers at the present, but the software I am writing could be run on larger
> machines...
> I guess that simplistically they are related to the RAM the CPU can access,
> and that its not often that a system is going to havee threads (say)
> restricted to a 32 bit space while the overall system can access a 64 bit
> space, etc.Put my question another way: how often is the upper RAM limit of a
> thread less that than of the process or system addressable spaces themselves?

        32-bit operating systems (such as Linux on a P3) can support more than
2^32 bytes of RAM. However, since a process uses 32-bit pointers, it is
limited to an address space of 2^32 bytes or 4Gb. Generally only 2-3Gb
of that is usable.

        DS

 
 
 

1. Leight Weight Processing or Threads in one UNIX process

Hi

We are looking for a public domain or commercial product
that allow threads in one UNIX process.  
We want to be able to listen on a socket descriptors and do some work
after something received on the socket without blocking the whole process.
We have to implement something like a NONblocking rpc-server.
Forking another process isn't a solution in our case.
Especially we are looking for a solution for AIX 2.3.x (rs6000).

Please mail me or 'followup' to comp.unix.programmer

Thanks in advance
Stefan
--
Stefan Zimmermann       |       Tel: 0231/5599170       | Irren ist menschlich,
Dr. Materna GmbH        |       Fax: 0231/5599100       |   aber das Gefuehl
Vo"skuhle 37               |    Privat: 0231/556124        |         dabei

2. Problems with lmail temp files

3. Limiting process RAM usage?

4. kern_exec.c exploit/fix to get root

5. Limiting ram-usage of one process

6. Redhat 6.1 with NT 4.0 problems?

7. hard limit on the number of threads per process?

8. ftp problem

9. Process limit for 2.4.2 for threaded applications

10. Multi-thread process -threads in different schedulers?

11. Runaway Linux processes-Native Posix Threading Library- Old linux threads

12. Thread & Process in unix

13. Wanted: preemptive threads within a UNIX process