Very Computer

Hi,
is there any solution to work with files larger than 2 GB on
Intel-processor based machines? I am running RedHat Linux 5.2 with
kernel 2.0.36. I heard about patch for kernelversions 2.2.x.

Thanks in advance

    A X E L

I heard the same thing, but I can't find any information on this patch
to override the limitation.

Parick

On Mon, 28 Jun 1999 16:19:14 -0700, Patrick Letovsky

Such a change, in order to *actually prove useful,* needs to be
reflected not only within the kernel, but also in LIBC.

After all, you probably don't want to have to create a custom version of
TAR, a custom version of cp, and a custom version of grep in order to
work with those files, right?
--
Rules of the Evil Warlord #77. "I will design fortress hallways with
no alcoves or protruding structural supports which intruders could use
for cover in a firefight."

On Mon, 28 Jun 1999 16:19:14 -0700, Patrick Letovsky

Such a change, in order to *actually prove useful,* needs to be
reflected not only within the kernel, but also in LIBC.

After all, you probably don't want to have to create a custom version of
TAR, a custom version of cp, and a custom version of grep in order to
work with those files, right?
--
Rules of the Evil Warlord #77. "I will design fortress hallways with
no alcoves or protruding structural supports which intruders could use
for cover in a firefight."

I asked about it a while ago, and I got the answer that you have to use the
loopback device. I don't know how that works, but I think it moved the size
limit to 16 Gb or maybe 2 Tb, not sure.

Kent

Quote:> I heard the same thing, but I can't find any information on this patch
> to override the limitation.
> > is there any solution to work with files larger than 2 GB on
> > Intel-processor based machines? I am running RedHat Linux 5.2 with
> > kernel 2.0.36. I heard about patch for kernelversions 2.2.x.

ext2 is quite capable of handling *filesystem* sizes considerably larger
than 2GB.  (2TB rings a bell.)

But mounting across a loopback device does not change the file
descriptor in LIBC.  Growing to sizes > 2^31-1 requires an API change.
--
"Catapultam habeo!  Nisi pecuniam omnem mihi dabis, ad caput tuum saxum
immane mittam !!" (I have a catapult!  If you do not pay me the money
you owe me, I will hit you with a big rock !!)

I read your page about the 32 bits architecture limitation, actually
your page is very well documented for all kernel features.
In my case, it is just for being able to create a tar file > 2gb, so if
the only thing to do is to recompile tar under the 2.2.x with the latest
GNU C compiler, that is not that big of a deal.
But I still miss information about this patch itself, it can be in beta
or alpha, I want to give it a try. I mount a RAID5 disk array on
/dev/sda3, there, I only have backup files from other systems, and one
of this file needs to be > 2Gb. I hope the patch doesn't involved to
re-mkfs the partition.

Any information on this unobtainable patch will be very appreciate.

Patrick

On Wed, 30 Jun 1999 18:50:05 -0700, Patrick Letovsky

The *killer* part isn't a kernel patch; it's the (probably nonexistent
at this point) GLIBC patch that is needed to support big files.

SAS Institute did a summit on this, and some of the "big name" UNIX
vendors do have API extensions to allow 64 bit file accessors on 32
bit platforms.  

Note that this is anything but transparent; you really have to pick for
LIBC to either be 32-bit-oriented, or 64-bit-oriented, and *NOT BOTH
AT ONCE.*

The result of *that* is that if you want to have tar/cat/dd/...
support 64 bit access, you need to modify them as well as LIBC to use
the 64 bit API.

LIBC5 *definitely* doesn't support "big files," and I don't *think*
that GLIBC2 provides both APIs (e.g. - small&large file accessors) yet.

Net result:
Patches must be to *ALL* of:
a) Kernel,
b) LIBC,
c) Applications.

I'm certain that c) hasn't been done, which puts you Out Of Luck.

See: <http://www.sas.com/standards/large.file/x_open.20Mar96.html>
for more details on the API changes.

I do not know where the patch is.
--
Those who do not understand Unix are condemned to reinvent it, poorly.  
-- Henry Spencer          <http://www.hex.net/~cbbrowne/lsf.html>

Thanks Christopher,

        I understand the 3 parts involved into the process, so I guess I'll
need to move to a 64 bits OS. I'm gonna use an irix box to do this job.
Thanks again for the explanation.

Patrick

--
-----------------------The Web Zone, Inc.-----------------------
Your Internet & Networking Solutions Company
8100 Boone Blvd., suite 110                  Tel: (703) 744-8811
Vienna, VA 22182                             Fax: (408) 235-8881

----------------------------------------------------------------

Quote:>>>>> Christopher B Browne writes:

Christopher> The *killer* part isn't a kernel patch; it's the (probably nonexistent
Christopher> at this point) GLIBC patch that is needed to support big files.
glibc 2.1 has the full LFS interface.  It just needs (on ix86) kernel
support and some changes.  But you can already

Christopher> SAS Institute did a summit on this, and some of the "big name" UNIX
Christopher> vendors do have API extensions to allow 64 bit file accessors on 32
Christopher> bit platforms.  

Christopher> Note that this is anything but transparent; you really have to pick for
Christopher> LIBC to either be 32-bit-oriented, or 64-bit-oriented, and *NOT BOTH
Christopher> AT ONCE.*

That's not right.  libc can support both.

Christopher> The result of *that* is that if you want to have tar/cat/dd/...
Christopher> support 64 bit access, you need to modify them as well as LIBC to use
Christopher> the 64 bit API.

Or compile with -D_FILE_OFFSET_BITS=64 - check glibc 2.1.x's manual
for details.

Christopher> LIBC5 *definitely* doesn't support "big files," and I don't *think*
Christopher> that GLIBC2 provides both APIs (e.g. - small&large file accessors) yet.
libc5 doesn't but I know that glibc 2.1 does.

Christopher> Net result:
Christopher> Patches must be to *ALL* of:
Christopher> a) Kernel,
Check the linux kernel lists, AFAIK Matti Aarnio is working on them.

Christopher> b) LIBC,
Christopher> c) Applications.

Christopher> I'm certain that c) hasn't been done, which puts you Out Of Luck.

Christopher> See: <http://www.sas.com/standards/large.file/x_open.20Mar96.html>
Christopher> for more details on the API changes.

Andreas
--

On Wed, 07 Jul 1999 14:08:58 -0700, Patrick Letovsky

I would expect the issue to be resolved over the next year or perhaps
2 years at worst, particularly as 64 bit platforms become more
commonplace.  

Furthermore, it looks like the relevant code *is* getting into GLIBC
2.1, which means that it shouldn't take infinite time for it to become
usable at user level.
--
Rules of the Evil Warlord #36. "Even though I don't really care
because I plan on living forever, I will hire engineers who are able
to build me a fortress sturdy enough that, if I am slain, it won't
tumble to the ground for no good structural reason."

Quote:>>>>>> Christopher B Browne writes:
>Christopher> The *killer* part isn't a kernel patch; it's the (probably nonexistent
>Christopher> at this point) GLIBC patch that is needed to support big files.

>glibc 2.1 has the full LFS interface.  It just needs (on ix86) kernel
>support and some changes.  But you can already

I stand corrected, and am glad to see such developments in 2.1; this
nonetheless leaves the issue that applications must be coded to use
LFS.

Quote:>Christopher> SAS Institute did a summit on this, and some of the "big name" UNIX
>Christopher> vendors do have API extensions to allow 64 bit file accessors on 32
>Christopher> bit platforms.  

>Christopher> Note that this is anything but transparent; you really have to pick for
>Christopher> LIBC to either be 32-bit-oriented, or 64-bit-oriented, and *NOT BOTH
>Christopher> AT ONCE.*

>That's not right.  libc can support both.

One orientation is preferred, and the other requires coding changes of
one sort or another.
--
As of next Tuesday NCOMPLR will no longer open-code arithmetic statements.
Please update your programs.

Get a real computer :-).

From an overbearingly smug Alpha owner!

--
Richard Simpson
Farnborough, Hants, Uk                 Fax: 01252 392118

I am not aware of any views shared by myself and my employers.

->>>>>> Christopher B Browne writes:
->Christopher> The *killer* part isn't a kernel patch; it's the (probably nonexistent
->Christopher> at this point) GLIBC patch that is needed to support big files.
->
->glibc 2.1 has the full LFS interface.  It just needs (on ix86) kernel
->support and some changes.  But you can already
-
-I stand corrected, and am glad to see such developments in 2.1; this
-nonetheless leaves the issue that applications must be coded to use
-LFS.

But since it's such a small subset of applications anyway, this shouldn't be
a big deal right?

BAJ

This represents one of the "Enterprise Computing" requirements; as such,
while the incidence of need may be relatively small, those represent
situations invoving expensive/important boxes, which makes it important
out of proportion to the apparent number of boxes.

--
"The problem with the current Lisp Machine system is that nothing ever calls
anything anymore."  -- KMP