Very Computer

Hi,

we are working with large files of up to 2-4GB. The linux fs
supports only up to 1.07GB!!  Suggestions how to enlarge
the maximum file size (4x needed)  greatly appreciated!
(Other systems seems to have no limit!)

Thank's
Peter

--
______________________________________________

Peter  Mewis

Institut f. Wasserbau und Wasserwirtschaft
TU-Darmstadt
Rundeturmstr.1
64283 Darmstadt / Germany

Tel : +49-6151-163123    FAX : +49-6151-163223

______________________________________________

-Hi,
-
-we are working with large files of up to 2-4GB. The linux fs
-supports only up to 1.07GB!!  Suggestions how to enlarge
-the maximum file size (4x needed)  greatly appreciated!
-(Other systems seems to have no limit!)

Trust me, they have limits, you simply haven't encountered them yet.

From what I understand, the current limit in the ext2 filesystem is in the
2G range. The primarily is due to the use of 32 bit offsets in both the
filesystem itself and in the libraries (lseek etc) that are used to access
the files.

From what I've read on the subject so far, the process of converting these
offsets to 64 bits has begun, but that it'll be awhile before that support
will be available for public release. Please feel free to correct me if I'm
mistaken on that point.

So let's turn this issue in another direction. Let's presume that a partition
exists that will hold your data (say 6-8 GB). Is there any current mechanism
in the kernel to access all the space on that partition? If so then a small
interface library that implements the file directly on the partition may
be a possibility. Or perhaps a kernel module that provides a 64 offset
interface to a raw partition, perhaps with a ioctl to set a segment or page
number, could do the trick.

BTW this is implementable completely in user space. Create a metafile that
consists of multiple 1G file segments. Create an interface library that
maintains a 64 bit offset and maps that offset to the correct 1G file and
offset. 64 bit long long support in GCC seems to work OK. I wrote this
test program:

-----------------------------
long long testme;

main()
{
   printf("%d\n",sizeof(testme));
   testme = 1LL << 37;
   printf("%lld\n",testme);

Quote:}                      

and it delivers the expected results:
----------
8
137438953472      
------------

Note that's 137 trillion!

My initial thought on this subject is that if you're dealing with 4+ GB
files, a filesystem really isn't a necessity. The beauty of all this is
since the kernel is available to you, you can add the required functionality
right now, without having to wait for the developers to get around to a task
that affects such an extremely small part of the user population.

Hope that gives you some insight into solving your problem.

BAJ

--
Another random extraction from the mental bit stream of...
Byron A. Jeff - PhD student operating in parallel - And Using Linux!

: -Hi,
: -
: -we are working with large files of up to 2-4GB. The linux fs
: -supports only up to 1.07GB!!  Suggestions how to enlarge
: -the maximum file size (4x needed)  greatly appreciated!
: -(Other systems seems to have no limit!)

: Trust me, they have limits, you simply haven't encountered them yet.

: From what I understand, the current limit in the ext2 filesystem is in the
: 2G range. The primarily is due to the use of 32 bit offsets in both the
: filesystem itself and in the libraries (lseek etc) that are used to access
: the files.

Exactly. 32 bit _signed_ integers are used => max file length is 2G

: From what I've read on the subject so far, the process of converting these
: offsets to 64 bits has begun, but that it'll be awhile before that support
: will be available for public release. Please feel free to correct me if I'm
: mistaken on that point.

Well, _llseek() is available, but the 2G-per-file ext2 limit is still there.

: So let's turn this issue in another direction. Let's presume that a partition
: exists that will hold your data (say 6-8 GB). Is there any current mechanism
: in the kernel to access all the space on that partition?

Yep, just open the raw device would do it. Say the 2nd partition on 1st IDE
is 6MB. You can open("/dev/hda2", O_RDWR) and use it just like a normal file.
To jump at positions beyond 2GB, use _llseek() instead of lseek(). That's just
how fdisk and e2fsck do it.

: If so then a small
: interface library that implements the file directly on the partition may
: be a possibility. Or perhaps a kernel module that provides a 64 offset
: interface to a raw partition, perhaps with a ioctl to set a segment or page
: number, could do the trick.

It's already there :)

: BTW this is implementable completely in user space. Create a metafile that
: consists of multiple 1G file segments. Create an interface library that
: maintains a 64 bit offset and maps that offset to the correct 1G file and
: offset. 64 bit long long support in GCC seems to work OK. I wrote this
: test program:

That works, but using raw partitions is easier. As you also noticed:

: My initial thought on this subject is that if you're dealing with 4+ GB
: files, a filesystem really isn't a necessity.

And infact you are quite right, at least until fixed disks get much bigger than what
are now.

: The beauty of all this is
: since the kernel is available to you, you can add the required functionality
: right now, without having to wait for the developers to get around to a task
: that affects such an extremely small part of the user population.

Your picture is happy, but reality is even better: kernel support for this
is _already_ there.

                                   Massimiliano Ghilardi