maximum possible memory limit ..

maximum possible memory limit ..

Post by Henti Smit » Fri, 25 Apr 2003 20:20:18



Hi all

I had a discussion with somebody watching the whole M$ server launch and mentioned then new systems supports up to a terabyte of ram.
I've tried looking for a hint at what the max momery support on linux is and cannot find it anywhere.

can somebody here enlighten me on just what the maximum amount of memory linux can deal with ?

Henti
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Valdis.Kletni.. » Fri, 25 Apr 2003 20:40:16



Quote:> I had a discussion with somebody watching the whole M$ server launch and
> mentioned then new systems supports up to a terabyte of ram.

Well.. sure.. it's easy enough to write something that supports plugging in
a terabyte.  The *tricky* part is supporting it well - you have page table
issues, you have swapping/thrashing issues (if you *do* have to page something
out, you're in trouble.. ;), you have process scheduling issues (how many
Apache processes does it take to use up a terabyte?  What's your load average
at that point?), you have multi-processor scaling issues (you're gonna want
to have 64+ processors, etc..)

Consider - the number of machines with over a terabyte of RAM is limited:

http://www.llnl.gov/asci/platforms/platforms.html

That's the sort of box that has a terabyte.  Do you *really* think that
M$ 2003 has all the stuff needed to scale to THAT size?

  application_pgp-signature_part
< 1K Download

 
 
 

maximum possible memory limit ..

Post by Carl-Daniel Hailfinge » Tue, 29 Apr 2003 15:10:14


[CC:ing lse-tech because they know better than me]


> Hi all

> I had a discussion with somebody watching the whole M$ server launch and mentioned then new systems supports up to a terabyte of ram.
> I've tried looking for a hint at what the max memory support on linux is and cannot find it anywhere.

> can somebody here enlighten me on just what the maximum amount of memory linux can deal with ?

Linux supports up to 4 GB (~2^32 bytes) of memory on 32-bit
architectures and 64 GB (~2^36 bytes) on x86 with PAE. No other
operating system can support more on 32-bit since it is a limitation of
the hardware.
On 64-bit systems, Linux supports up to 16 EB (~2^64 bytes) of memory,
which is about 16 million times more than the 1 TB limit of MS.

Current Linux 2.4 allows 32 CPUs for 32-bit arches and 64 CPUs on 64-bit
arches. However, this limit is (was?) being removed in 2.5, so you can
have up to 32767 CPUs, which should be enough for you right now.
(Note: I said _right now_, lest anybody make jokes about 640K limit)

Regards,
Carl-Daniel
--
http://www.hailfinger.org/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Andi Klee » Tue, 29 Apr 2003 16:20:12


Quote:> Linux supports up to 4 GB (~2^32 bytes) of memory on 32-bit
> architectures and 64 GB (~2^36 bytes) on x86 with PAE. No other

That's far too optimistic. 64GB will need patches, like the pgcl
patch. It is unlikely to work out of the box. Just do the math.

900MB low mem for all the kernel data structures on IA32.

44 byte struct page for each 4K page. This gives 704MB just
for the mem_map array. Leaves you 196MB left for the kernel to
manage your 64GB of memory and your processes. Unlikely to work.

2.5 uses 40 bytes for an struct page, but that doesn't help much.

Yes you can move the kernel:user split to give the kernel more memory
at the expense of the application, but that is likely to not make the
user programs happy and also helps only very limited
(even 2GB lowmem are probably not enough to make it run well with 64GB)

Realistic limit currently is ~16GB with an IA32 box.  For more you need
an 64bit architecture.

-Andi

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Carl-Daniel Hailfinge » Tue, 29 Apr 2003 17:20:16



> > Linux supports up to 4 GB (~2^32 bytes) of memory on 32-bit
> > architectures and 64 GB (~2^36 bytes) on x86 with PAE. No other

> That's far too optimistic. 64GB will need patches, like the pgcl
> patch. It is unlikely to work out of the box. Just do the math.

[explanatory math snipped]

Quote:> Realistic limit currently is ~16GB with an IA32 box.  For more you need

<marketingspeak>
This means ~16GB is available on IA32 right now with vanilla kernels,
and 64GB is available from vendors who are willing to apply the pgcl
patch. Conclusion: Linux supports 64GB on IA32
</marketingspeak>

Quote:> an 64bit architecture.
> > On 64-bit systems, Linux supports up to 16 EB (~2^64 bytes) of memory

That statement is OK?

Thanks,
Carl-Daniel
--
http://www.hailfinger.org/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Andi Klee » Tue, 29 Apr 2003 17:20:24


Quote:> and 64GB is available from vendors who are willing to apply the pgcl
> patch.

Nobody is doing that. pgcl is 2.5 only and seems to be still quite instable.
Also it's extremly intrusive.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Dave Hanse » Tue, 29 Apr 2003 18:50:14



> Realistic limit currently is ~16GB with an IA32 box.  For more you need
> an 64bit architecture.

Let's say 32GB :)  It boots just fine with 2.5.68, no additional
patches.  There's even half a gig of lowmem free.

curly:~# cat /proc/meminfo
MemTotal:     32688576 kB
MemFree:      32644196 kB
Buffers:          3632 kB
Cached:           8068 kB
SwapCached:          0 kB
Active:           9420 kB
Inactive:         4616 kB
HighTotal:    32112640 kB
HighFree:     32098240 kB
LowTotal:       575936 kB
LowFree:        545956 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:             160 kB
Writeback:           0 kB
Mapped:           4596 kB
Slab:             8316 kB
Committed_AS:     5544 kB
PageTables:        260 kB
VmallocTotal:   114680 kB
VmallocUsed:      3792 kB
VmallocChunk:   110888 kB

--
Dave Hansen

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Dave Hanse » Tue, 29 Apr 2003 19:00:26



>>and 64GB is available from vendors who are willing to apply the pgcl
>>patch.

> Nobody is doing that. pgcl is 2.5 only and seems to be still quite instable.
> Also it's extremly intrusive.

Bill will probably wake up any time now and chime in, but don't forget
all of the drivers.

# grep -r PAGE_SIZE drivers/ | wc -l
893

Each one of those needs to be audited before pgcl is acceptable to a
wide audience.  We've already seen plenty of stuff that breaks.  ext2/3
look to be all right, but I know that JFS is broken.
--
Dave Hansen

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Carl-Daniel Hailfinge » Tue, 29 Apr 2003 19:00:34




>>Realistic limit currently is ~16GB with an IA32 box.  For more you need
>>an 64bit architecture.

> Let's say 32GB :)  It boots just fine with 2.5.68, no additional
> patches.  There's even half a gig of lowmem free.

Cool. Sorry to be pestering about the 64-bit limits, but can we really
use 2^64 bytes of memory on ia64/ppc64/x86-64 etc.? (AFAIK, 64-bit
arches don't suffer from a small ZONE_LOWMEM.)

Regards,
Carl-Daniel
--
http://www.hailfinger.org/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Martin J. Blig » Tue, 29 Apr 2003 19:10:06


Quote:>>> and 64GB is available from vendors who are willing to apply the pgcl
>>> patch.

>> Nobody is doing that. pgcl is 2.5 only and seems to be still quite
>> instable. Also it's extremly intrusive.

> Bill will probably wake up any time now and chime in, but don't forget
> all of the drivers.

># grep -r PAGE_SIZE drivers/ | wc -l
> 893

> Each one of those needs to be audited before pgcl is acceptable to a
> wide audience.  We've already seen plenty of stuff that breaks.  ext2/3
> look to be all right, but I know that JFS is broken.

Well, the upside is that he's only doing s/PAGE_SIZE/MMU_PAGESIZE/
in most places, which are normally both 4K. So it will have no effect
whatsoever unless you explicitly turn it on.

M.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Andi Klee » Tue, 29 Apr 2003 19:20:15


Quote:> Cool. Sorry to be pestering about the 64-bit limits, but can we really
> use 2^64 bytes of memory on ia64/ppc64/x86-64 etc.? (AFAIK, 64-bit
> arches don't suffer from a small ZONE_LOWMEM.)

No. The hardware have far smaller physical limits.

Current AMD64 CPUs are limited to 40bit physical, 48bit virtal
(the virtual limit per process in the current Linux kernel is 39bits)

Itanium 2 afaik support a bit more 50bits (51 or 52, I forgot) physical,
probably more virtual.

Other 64bit architectures are somewhere inbetween.

The actual limit in the machines is even less. You will have a hard
time to find an affordable machine (64bit or not) with more than 8 DIMM
slots. That's 16GB Max with 2GB DIMMs.

-Andi

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Andi Klee » Tue, 29 Apr 2003 19:20:26




> > Realistic limit currently is ~16GB with an IA32 box.  For more you need
> > an 64bit architecture.

> Let's say 32GB :)  It boots just fine with 2.5.68, no additional
> patches.  There's even half a gig of lowmem free.

But what happens when you stress test it? No deadlocks?

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Dave Hanse » Tue, 29 Apr 2003 19:20:28



> Cool. Sorry to be pestering about the 64-bit limits, but can we really
> use 2^64 bytes of memory on ia64/ppc64/x86-64 etc.? (AFAIK, 64-bit
> arches don't suffer from a small ZONE_LOWMEM.)

First of all, I'm not sure any of the 64-bit arches even fully support
64-bit physical addresses.  If I remember correctly the first hammers
support 40 bits, with more to be added later.  Power4 is in close to the
same boat, but I know they go up to 256GB today (I seem to recall
something about 44-bit being the limit, though).

Don't forget that highmem starts to be needed before the 4G boundary.
The kernel has only 1GB of virtual space (look for PAGE_OFFSET, which
defines it), which means that you start needing to pull all of the
highmem trickery before you get to the actual limits.

Nobody knows how far it will go.  It's fairly safe to say that, at this
rate, Linux will keep up with whatever hardware anyone produces.
Unless, of course, someone gets even more perverse than PAE. :)

--
Dave Hansen

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Dave Hanse » Tue, 29 Apr 2003 19:30:19



>>Let's say 32GB :)  It boots just fine with 2.5.68, no additional
>>patches.  There's even half a gig of lowmem free.

> But what happens when you stress test it? No deadlocks?

Actually, it is pretty stable.  It OOM's a bit more easily, but that
isn't a surprise to anyone, especially with the smaller amount of
ZONE_NORMAL.  I routinely run kernel compiles for days on it with no
problems.

That is, of course, a NUMA-Q.  We're hoping to get a Summit box which is
just as big sometime soon.  It will be much more interesting.
--
Dave Hansen

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

maximum possible memory limit ..

Post by Carl-Daniel Hailfinge » Tue, 29 Apr 2003 19:40:17




>>Cool. Sorry to be pestering about the 64-bit limits, but can we really
>>use 2^64 bytes of memory on ia64/ppc64/x86-64 etc.? (AFAIK, 64-bit
>>arches don't suffer from a small ZONE_LOWMEM.)

> [...]
> Don't forget that highmem starts to be needed before the 4G boundary.
> The kernel has only 1GB of virtual space (look for PAGE_OFFSET, which
> defines it), which means that you start needing to pull all of the
> highmem trickery before you get to the actual limits.

It seems I misunderstood the concept of highmem. I thought highmem was
not needed on 64-bit arches. Thanks for pointing that out to me.

Quote:

> Nobody knows how far it will go.  It's fairly safe to say that, at this
> rate, Linux will keep up with whatever hardware anyone produces.

That is the answer the original poster was looking for.

Quote:> Unless, of course, someone gets even more perverse than PAE. :)

hehe ;-) Can you say PAE in userspace?

Regards,
Carl-Daniel
--
http://www.hailfinger.org/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in

More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

 
 
 

1. How to increase the Maximum limit on Shared Memory

Hi:
  Subject kind of gives an idea, what the problem is. I am running a
Client-Server model with shared memory and sockets. This runs smoothly until
certain point, i.e., with a smaller problem size. Well, when I increase size
of the problem my program dies, giving a message "Not enough memory on the
device" (errorno is 28). I would expect that the problem is exceeding the Maximum
size. We rebuild the kernel increasing the maximum limit to 3Meg. But it still
dies. Can anyone please let me what's that I am missing? Am I (ie., My Sys Admin)
building the kernel properly? Please e-mail any comments or suggestions. When
does memory swapping come into picture?
  Thanks.

P.S: I am running this on SUNOS 4.1, SUN 4/110 workstations. When I do vmstat it
shows more than 3Meg of free memory. And a fast question: How can one get BUS
alignment error (through xdbx) through malloc. My sequential
version of the above Client-Server model gives that message, but doesn't
return a NULL.
--

        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
        %                                                            %
        % Vasu Nugala                   USnail:                      %

        %                                 Logan, UT 84322.           %
        %                               Home: (801) 752-1241.        %
        %                                                            %
        %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

2. ! MASS POST Was Here! (UvPMpX)

3. Setting maximum memory limit

4. Kernel recompile lost sound

5. maximum possible virtual memory under Solaris?

6. mandrake 7.2 and sis630

7. question about process memory usage and process memory limits

8. Problems setting up Perl to run under Apache on Windows NT 4.0

9. limit memory and add memory file system

10. Make Shared Memory Block Memory-Resident Possible?

11. Limit maximum TCP connections for NAT connection

12. Linux maximum O/S limits????

13. maximum filesystem size limit