BUG: S8 hsfs filesystem cannot read filesystems >4GB

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Juergen Kei » Wed, 13 Jun 2001 02:19:17



My DVD experiments uncovered an additional bug in solaris 8.

The new problem is with the HSFS (aka ISO9660) filesystem on solaris 8
x86 (I guess the same problem exists in 32-bit SPARC solaris but I
havn't verified this).  The HSFS filesystem has problems reading data
from files that are located at an address >4GB inside a partition.
DVD disks use an ISO9660 filesystem and the filesystem on such a DVD
disk is often >4GB!

Is this a known bug? If not, can anyone please file a bug report?

Again, the bug is a 32-bit 'int' or 'unsigned int' overflow, this time
somewhere in function 'hsfs_getapage()'.

Here's how the problem can be reproduced:

- Mount a DVD >4GB,  the disk should use an ISO9660 filesystem.

- Use od -x, adb, an ISO9660 low-level directory dump utitlity,
  or whatever ... to locate a file that starts below the
  4GB mark and crosses it  [*]:

% ~/src/cdrom/misc/dumpdir /cdrom/cdrom0/video_ts
...
offset of dirent        : 0x236
length of dir. record   : 54
ext. attr. record length: 0
location of extent      : 1850159               <-----------------
data length             : 1073250304            <-----------------
recording date          : 30.01.2001  8:10:27 GMT-8.00
file flags              : 0 ()
file unit size          : 0
file interleave gap size: 0
volume sequence number  : 1
filename                : (14)VTS_01_4.VOB;1
system use (6 bytes)
    00: 00 00 00 00 00 00                               ......
...

/cdrom/cdrom0/video_ts/vts_01_4.vob is such a file on the DVD I'm
using here as an example.  It starts at block 1850159 which is at
1850159*2048 = 3.7GB; it's filesize is 1GB, so the file ends somewhere
at 4.7GB.

The 4GB mark on the DVD is LBA 2^32/2048 == LBA 2097152.

That is, data beyond 2K block #246993 (2097152-1850159) in file
vts_01_4.vob is located at addresses >4GB on the DVD.

- Now we try to read some data from the problematic file (16 blocks
beyond the 4GB mark in this example):

% dd if=/cdrom/cdrom0/video_ts/vts_01_4.vob bs=2k count=1 iseek=`expr
246993 + 16`|od -c|sed 6q
1+0 records in
1+0 records out
0000000 001   C   D   0   0   1 001  \0                                
0000020                                                                
0000040                                   L   O   L   A   _   R   E  N
0000060   N   T                                                        
0000100                                  \0  \0  \0  \0  \0  \0  \0 \0
0000120 201   /   <  \0  \0   <   / 201  \0  \0  \0  \0  \0  \0  \0 \0

HSFS experts will recognize an ISO9660 primary volume discriptor (PVD)
sector, which is located at the start of an ISO9660 filesystem at LBA
16+.

Of cause the .vob file should contain MPEG encoded data here (and not
a ISO9660 PVD) ...

- Exactly the same data can be read from LBA 16 of that DVD:

% dd if=/vol/dev/aliases/cdrom0 bs=2k count=1 iseek=16|od -c|sed 6q
1+0 records in
1+0 records out
0000000 001   C   D   0   0   1 001  \0                                
0000020                                                                
0000040                                   L   O   L   A   _   R   E  N
0000060   N   T                                                        
0000100                                  \0  \0  \0  \0  \0  \0  \0 \0
0000120 201   /   <  \0  \0   <   / 201  \0  \0  \0  \0  \0  \0  \0 \0

- as a further example, a complete 2MB chunk of data in the
problematic file is bit-for-bit identical with the data found at the
start of the dvd disk:

% ksh -c "cmp <(dd if=/cdrom/cdrom0/video_ts/vts_01_4.vob bs=2k \
count=1000 iseek=246993) <(dd if=/vol/dev/aliases/cdrom0 bs=2k \
count=1000)"
1000+0 records in
1000+0 records out
1000+0 records in
1000+0 records out
%

[*] Another example for this problem is the MSDN DVD "april 2001 msdn
subscriptions library"  (yuck 8-) I know, but it could be a DVD that
others may have available for testing, too):

The error can be reproduced with the file "media/pdc2k/4-331.ASF"
found on that DVD.

% dd if=/cdrom/cdrom0/media/pdc2k/4-331.ASF bs=2k count=1 \
iseek=`expr 2097152 - 2067072 + 16` | od -c | sed 20q

dumps the contents of absolute LBA #16 (PVD).

If you play that ASF file using "mplayer" on solaris 8, playback
crashes around the 36min33sec mark (of a total of 1hr18min).

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by ken » Wed, 13 Jun 2001 04:12:39


Hi - for filesystems over 4 Gbytes you should be using UDF instead
of HSFS.

And if you replace the 'int' with a 'long' on a 32 bit machine, e.g.,
x86, it's still 32 bits.

-- Ken


> My DVD experiments uncovered an additional bug in solaris 8.

> The new problem is with the HSFS (aka ISO9660) filesystem on solaris 8
> x86 (I guess the same problem exists in 32-bit SPARC solaris but I
> havn't verified this).  The HSFS filesystem has problems reading data
> from files that are located at an address >4GB inside a partition.
> DVD disks use an ISO9660 filesystem and the filesystem on such a DVD
> disk is often >4GB!

> Is this a known bug? If not, can anyone please file a bug report?

> Again, the bug is a 32-bit 'int' or 'unsigned int' overflow, this time
> somewhere in function 'hsfs_getapage()'.

> Here's how the problem can be reproduced:

> - Mount a DVD >4GB,  the disk should use an ISO9660 filesystem.

> - Use od -x, adb, an ISO9660 low-level directory dump utitlity,
>   or whatever ... to locate a file that starts below the
>   4GB mark and crosses it  [*]:

> % ~/src/cdrom/misc/dumpdir /cdrom/cdrom0/video_ts
> ...
> offset of dirent        : 0x236
> length of dir. record   : 54
> ext. attr. record length: 0
> location of extent      : 1850159               <-----------------
> data length             : 1073250304            <-----------------
> recording date          : 30.01.2001  8:10:27 GMT-8.00
> file flags              : 0 ()
> file unit size          : 0
> file interleave gap size: 0
> volume sequence number  : 1
> filename                : (14)VTS_01_4.VOB;1
> system use (6 bytes)
>     00: 00 00 00 00 00 00                               ......
> ...

> /cdrom/cdrom0/video_ts/vts_01_4.vob is such a file on the DVD I'm
> using here as an example.  It starts at block 1850159 which is at
> 1850159*2048 = 3.7GB; it's filesize is 1GB, so the file ends somewhere
> at 4.7GB.

> The 4GB mark on the DVD is LBA 2^32/2048 == LBA 2097152.

> That is, data beyond 2K block #246993 (2097152-1850159) in file
> vts_01_4.vob is located at addresses >4GB on the DVD.

> - Now we try to read some data from the problematic file (16 blocks
> beyond the 4GB mark in this example):

> % dd if=/cdrom/cdrom0/video_ts/vts_01_4.vob bs=2k count=1 iseek=`expr
> 246993 + 16`|od -c|sed 6q
> 1+0 records in
> 1+0 records out
> 0000000 001   C   D   0   0   1 001  \0
> 0000020
> 0000040                                   L   O   L   A   _   R   E  N
> 0000060   N   T
> 0000100                                  \0  \0  \0  \0  \0  \0  \0 \0
> 0000120 201   /   <  \0  \0   <   / 201  \0  \0  \0  \0  \0  \0  \0 \0

> HSFS experts will recognize an ISO9660 primary volume discriptor (PVD)
> sector, which is located at the start of an ISO9660 filesystem at LBA
> 16+.

> Of cause the .vob file should contain MPEG encoded data here (and not
> a ISO9660 PVD) ...

> - Exactly the same data can be read from LBA 16 of that DVD:

> % dd if=/vol/dev/aliases/cdrom0 bs=2k count=1 iseek=16|od -c|sed 6q
> 1+0 records in
> 1+0 records out
> 0000000 001   C   D   0   0   1 001  \0
> 0000020
> 0000040                                   L   O   L   A   _   R   E  N
> 0000060   N   T
> 0000100                                  \0  \0  \0  \0  \0  \0  \0 \0
> 0000120 201   /   <  \0  \0   <   / 201  \0  \0  \0  \0  \0  \0  \0 \0

> - as a further example, a complete 2MB chunk of data in the
> problematic file is bit-for-bit identical with the data found at the
> start of the dvd disk:

> % ksh -c "cmp <(dd if=/cdrom/cdrom0/video_ts/vts_01_4.vob bs=2k \
> count=1000 iseek=246993) <(dd if=/vol/dev/aliases/cdrom0 bs=2k \
> count=1000)"
> 1000+0 records in
> 1000+0 records out
> 1000+0 records in
> 1000+0 records out
> %

> [*] Another example for this problem is the MSDN DVD "april 2001 msdn
> subscriptions library"  (yuck 8-) I know, but it could be a DVD that
> others may have available for testing, too):

> The error can be reproduced with the file "media/pdc2k/4-331.ASF"
> found on that DVD.

> % dd if=/cdrom/cdrom0/media/pdc2k/4-331.ASF bs=2k count=1 \
> iseek=`expr 2097152 - 2067072 + 16` | od -c | sed 20q

> dumps the contents of absolute LBA #16 (PVD).

> If you play that ASF file using "mplayer" on solaris 8, playback
> crashes around the 36min33sec mark (of a total of 1hr18min).

--

========================================================================
Ken Simpson                              Well Connected Computing, Inc.

Voice: +1.415.332.5018                   Sausalito, CA 94965
FAX:   +1.415.332.9197                   USA, Earth
========================================================================

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Juergen Kei » Wed, 13 Jun 2001 17:59:42



> Hi - for filesystems over 4 Gbytes you should be using UDF instead
> of HSFS.

Well, I'm forced to use whatever the creator of the DVD decided to use
(note dvd == read only media).  And they are using ISO9660
filesystems!  At least all the dvd videos and most of the data dvds
I've seen upto now use ISO9660. So, if we want to use these dvds
under solaris, sun better fixes the broken hsfs filesystem (or enhance
the hsfs fs to allow use of partition / mediasizes >4GB).

Quote:> And if you replace the 'int' with a 'long' on a 32 bit machine, e.g.,
> x86, it's still 32 bits.

Sure, that's why there are types like int64_t and uint64_t in the sys
header files.  It would be a good idea to use them in the
hsfs_getapage() subroutine.
 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Joerg Schilli » Wed, 13 Jun 2001 19:28:51




Quote:

>Hi - for filesystems over 4 Gbytes you should be using UDF instead
>of HSFS.

>And if you replace the 'int' with a 'long' on a 32 bit machine, e.g.,
>x86, it's still 32 bits.

The ISO-9660 filesystem supports filesystems > 4 GB. It does _not_
support a single file being > 2 GB inside a ISO-9660 oartition.

... Of course you need to use a decent ISO-9660 formatter (like mkisofs).

The question is:

Is this a bug in the Solaris HSFS implementation or is it a bug inside
the ISO-9660 structure on the disk.

Background information:

ESO (European Southern Observatory the largest supertelescope in the world)
see www.eso.org uses mkisofs and cdrecord to archive ~ 40 GB per day.
They write 4.7 GB DVD-R media.

I know that they got a patch for the hsfs filesystem from Sun and now have no
problems, but I am not shure about the reason for the patch...

--



URL:  http://www.fokus.gmd.de/usr/schilling    ftp://ftp.fokus.gmd.de/pub/unix

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Kjetil Torgrim Homm » Wed, 13 Jun 2001 19:34:06


[Juergen Keil]

Quote:>   Well, I'm forced to use whatever the creator of the DVD decided to
>   use (note dvd == read only media).  And they are using ISO9660
>   filesystems!  At least all the dvd videos and most of the data
>   dvds I've seen upto now use ISO9660.

Strange.  I've only seen UDF so far.  I haven't looked at that many,
though.

Quote:>   So, if we want to use these dvds under solaris, sun better fixes
>   the broken hsfs filesystem (or enhance the hsfs fs to allow use of
>   partition / mediasizes >4GB).

I think you mean, Solaris should handle out-of-spec HSFS filesystems.
You should complain to the DVD author, not Sun.

Kjetil T.

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Juergen Kei » Wed, 13 Jun 2001 20:13:59



Quote:> >   So, if we want to use these dvds under solaris, sun better fixes
> >   the broken hsfs filesystem (or enhance the hsfs fs to allow use of
> >   partition / mediasizes >4GB).

> I think you mean, Solaris should handle out-of-spec HSFS filesystems.
> You should complain to the DVD author, not Sun.

Is that really out-of-spec ISO9660?  Chapter and verse from the
ISO9660:1988 standard please...

All I see is:

8.4.8 Volume Space Size

This field shall specify as a 32-bit number the number of the Logical
Blocks in which the Volume Space of the volume is recorded.

9.1.3 Location of Extent

This field shall specify as a 32-bit number the Logical Block Nummber
of the first Logical Block allocated to the Extent.

6.2.2 Logical Block

...
Each Logical Block shall consist of 2^(n+9) bytes, where n equals 0
or a positive integer.
...

I've found no limitations for the allowable range of the 32-bit
logical block numbers, so I assume 0 .. 2^32-1 is allowed here.  Which
gives a maximum ISO9660 filesystem size of 2TB, using 512 bytes
Logical Blocks (more using bigger Logical Blocks).

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Juergen Kei » Wed, 13 Jun 2001 20:29:49



> Is this a bug in the Solaris HSFS implementation or is it a bug inside
> the ISO-9660 structure on the disk.

The ISO-9660 structure on the disk is OK.  It's a bug in the Solaris
HSFS filesystem code.

That is, using a solaris 8 kernel hotpatch to fix the buggy
hsfs_getapage() routine [+ the sd driver bug, see my other usenet
article] allows you to read the whole file without problems!

Quote:> I know that they got a patch for the hsfs filesystem from Sun and now have no
> problems, but I am not shure about the reason for the patch...

Do you know if the patch they've got frmo sun is an offical solaris 8
patch?

There is 109765-02, but that does not fix (apparently) anything about
huge filesystems (it's part of the MU4 update and I have that
installed, anyway):

Patch-ID# 109765-02
Synopsis: SunOS 5.8_x86: /kernel/fs/hsfs patch
BugId's fixed with this patch: 4305026 4328133
Changes incorporated in this version: 4305026
Date: Nov/07/00

...

Problem Description:

4305026 df can't resolve devices mounted by vold

(from 109765-01)
4328133 assertion failed: io_off == io_off_tmp, file:
         ../../common/fs/hsfs/hsfs_vnops.c

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Juergen Kei » Wed, 13 Jun 2001 21:09:22



> Is this a bug in the Solaris HSFS implementation or is it a bug inside
> the ISO-9660 structure on the disk.

The following test might show the bug more easily (without the need
for a special program to access binary data in such a problematic
file beyond the 4GB mark):

                                -----

For those, who have the MSDN "april 2001 msdn subscriptions library"
DVD accessable for testing, check the text (html) file
samples/vs98/readme.htm on that dvd on a solaris 8 box (it's file data
is located at LBA 2580118, ~4.9GB):

% more /cdrom/cdrom0/samples/vs98/readme.htm
jkashd $kl/sjh
              asjkdhj ashdas&%$iu
[... more binary junk omitted ...]

The expected contents of that file with a correct iso9660 filesystem
implementation should be: (standard html text)

% more /cdrom/cdrom0/samples/vs98/readme.htm
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" Content="text-html; charset=Windows-1252">
<title>Island Hopper News Sample</title>
<META NAME="DESCRIPTION" CONTENT=" ">
<META NAME="MS.LOCALE" CONTENT="EN-US">
<META NAME="MS.HAID" CONTENT="islandhopperc">
<LINK REL=stylesheet TYPE="text/css" HREF="msdn.css">
</HEAD>
<BODY>
<FONT FACE="Verdana, arial">
<h1><a name="veovrislandhoppernewssample"></a>Island Hopper News Sample</h1>
</FONT>
<FONT FACE="Verdana, arial" SIZE=2>
<p>
[ ... more html code omitted ... ]

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Joerg Schilli » Thu, 14 Jun 2001 03:12:10




>> Hi - for filesystems over 4 Gbytes you should be using UDF instead
>> of HSFS.

>Well, I'm forced to use whatever the creator of the DVD decided to use
>(note dvd == read only media).  And they are using ISO9660
>filesystems!  At least all the dvd videos and most of the data dvds
>I've seen upto now use ISO9660. So, if we want to use these dvds
>under solaris, sun better fixes the broken hsfs filesystem (or enhance
>the hsfs fs to allow use of partition / mediasizes >4GB).

>> And if you replace the 'int' with a 'long' on a 32 bit machine, e.g.,
>> x86, it's still 32 bits.

>Sure, that's why there are types like int64_t and uint64_t in the sys
>header files.  It would be a good idea to use them in the
>hsfs_getapage() subroutine.

I just asked the guys from ESO....

The bug is known by Sun for a long time. There is a non public patch
but this patch is not a general use solution, it does not work with all
drives.

So I would recommend to send an urgen bug report to show Sun that it is a
real problem that needs to be worked on.

--



URL:  http://www.fokus.gmd.de/usr/schilling    ftp://ftp.fokus.gmd.de/pub/unix

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Bruce Adle » Thu, 14 Jun 2001 08:01:31



> ...
> I just asked the guys from ESO....

> The bug is known by Sun for a long time. There is a non public patch
> but this patch is not a general use solution, it does not work with all
> drives.

> So I would recommend to send an urgen bug report to show Sun that it is a
> real problem that needs to be worked on.

I don't think Sun is going to worry too much that a MSDN DVD isn't
readable on Solaris (just as MS isn't too concerned that the UFS
filesystem and Rockridge extensions on the Solaris installation CDs
aren't readable by Windows). And the only DVDs that are widely
available are movie DVDs and Sun doesn't yet support playing DVD
movies on the x86 platform. If you suggest that using a HSFS
filesystem on a hard disk (rather than a DVD drive) might be useful
for interchanging data between platforms, Sun is likely to tell
you to use UDF instead (hopefully, UDF doesn't have the same 32-bit
error).

If you just want a patch for yourself, you do realize that you
can probably use Sun's "per incident" support method to get a fix
for around $200? But I suspect you're looking for a Public Patch
which would be available to everyone. I don't think one customer
escalating a bug like this is likely to get you a generic Public
Patch (or even getting a fix in the next MU). It just doesn't have a
wide enough impact for Sun to do anything other than give whoever
reports the bug the same (or another) private patch. You need to
directly contact some key Sun manager (like Greg Baker for example)
and persuade (or bribe or blackmail or embarrass) him/her that a Public/Recommended Patch would be a good thing.

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Joerg Schilli » Sun, 24 Jun 2001 02:09:16




Quote:>[Juergen Keil]

>>   So, if we want to use these dvds under solaris, sun better fixes
>>   the broken hsfs filesystem (or enhance the hsfs fs to allow use of
>>   partition / mediasizes >4GB).

>I think you mean, Solaris should handle out-of-spec HSFS filesystems.
>You should complain to the DVD author, not Sun.

These filesystems are completely within specs.

--



URL:  http://www.fokus.gmd.de/usr/schilling    ftp://ftp.fokus.gmd.de/pub/unix

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Joerg Schilli » Sun, 24 Jun 2001 02:12:21





>> ...
>> I just asked the guys from ESO....

>> The bug is known by Sun for a long time. There is a non public patch
>> but this patch is not a general use solution, it does not work with all
>> drives.

>> So I would recommend to send an urgen bug report to show Sun that it is a
>> real problem that needs to be worked on.

>I don't think Sun is going to worry too much that a MSDN DVD isn't
>readable on Solaris (just as MS isn't too concerned that the UFS
>filesystem and Rockridge extensions on the Solaris installation CDs
>aren't readable by Windows). And the only DVDs that are widely
>available are movie DVDs and Sun doesn't yet support playing DVD
>movies on the x86 platform. If you suggest that using a HSFS
>filesystem on a hard disk (rather than a DVD drive) might be useful
>for interchanging data between platforms, Sun is likely to tell
>you to use UDF instead (hopefully, UDF doesn't have the same 32-bit
>error).

Sun should worry because these filesystems are completely within specs.

The problems is that there are many versions of the UDF standard.

--



URL:  http://www.fokus.gmd.de/usr/schilling    ftp://ftp.fokus.gmd.de/pub/unix

 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Bruce Adle » Sun, 24 Jun 2001 05:44:05






> >> ...
> >> I just asked the guys from ESO....

> >> The bug is known by Sun for a long time. There is a non public patch
> >> but this patch is not a general use solution, it does not work with all
> >> drives.

> >> So I would recommend to send an urgen bug report to show Sun that it is a
> >> real problem that needs to be worked on.

> >I don't think Sun is going to worry too much that a MSDN DVD isn't
> >readable on Solaris (just as MS isn't too concerned that the UFS
> >filesystem and Rockridge extensions on the Solaris installation CDs
> >aren't readable by Windows). And the only DVDs that are widely
> >available are movie DVDs and Sun doesn't yet support playing DVD
> >movies on the x86 platform. If you suggest that using a HSFS
> >filesystem on a hard disk (rather than a DVD drive) might be useful
> >for interchanging data between platforms, Sun is likely to tell
> >you to use UDF instead (hopefully, UDF doesn't have the same 32-bit
> >error).

> Sun should worry because these filesystems are completely within specs.

Yes, but DVD media (which is the only media which doesn't support
512-byte sectors and which has a large enough capacity to trigger
this bug) isn't yet officially supported on Solaris x86. The fact
that just looking at the code and the spec trivially shows that
the code doesn't follow the spec merely classifies the problem as
a bug (rather than a (mis)feature). You still have to convince
someone within Sun that it's a bug that's worthy of being fixed
and that the fix needs to be made available to everyone. Sun has
limited resources and therefore can't possibly fix every single bug
they've created (no matter how out of spec it leaves the product).
Sun managers don't get a bigger bonuses for avoiding or eliminating
spec violations if that makes their projects miss deadlines. And,
not every programmer within Sun goes out of their way to fix spec
violation bugs that only show up in unsupported environments.

I'm merely trying to make you aware of the fact that even escalating
the bug to P1 doesn't guarantee you'll ever get Sun to issue a public
generic patch since you're asking them to fix a feature they
officially don't support.

Quote:> The problems is that there are many versions of the UDF standard.

Which is only a problem if there isn't a common implementation on
whatever specific systems you're interchanging media between. I've
never actually tried to use UDF to interchange data between
different platforms (and I've no idea what platforms you're working
with), but I imagine that if didn't work due to Sun applying the
"wrong" standard on Solaris x86, and if it was an important enough
interchange (for example, between Solaris and a Cobalt server), you
might have better luck getting Sun to fix such a problem rather than
trying to get them to fix a Solaris x86 DVD bug.
 
 
 

BUG: S8 hsfs filesystem cannot read filesystems >4GB

Post by Juergen Kei » Tue, 26 Jun 2001 03:42:31



> I don't see why this should be related to 512 bytes sectors.

> SCSI curently supports 1 TB of disk space if you use 512 byte sectors.

> DVD-R is currently limited to 4.7 GB and DVD-ROM to 17 GB. This is
> far away from the 1 TB limit. If Sun has problems with disks > 4 GB, then
> there really must be a * bug in Sun's code.

> Maybe the problem is caused by the way Sun computes faked inode numbers.
> AFAIK they use the byte offset of the dir entry on the disk.

> The ISO-9660 filesystem itself does not require to compute things based on
> the byte offset (except offsets inside a file). As a single file on ISO-9660
> cannot be > 2 GB, there should be not problem.

To repeat the two issues:

There are two separate problems:

1.

On Solaris 8 x86, the 'sd' driver re-blocks disc accesses from the
kernel's internal DEV_BSIZE (512 Byte) block numbers to the device's
logical block size, which is typicalls 2K for a CDROM/DVD device.

It uses something like the following piece of code:

    unsigned offset;
    unsigned device_blkno;

    offset = (buf->b_blkno + un->un_offset[part]) * DEV_BSIZE;
    device_blkno = offset / un->un_lbasize;

The conversion is via byte offsets and that conversion overflows for a
device access >4GB.  This bug is independant from the filesystem used.
It happens for HSFS, UDFS, UFS, whatever...  It even happens when you
access the raw 'sd' device.  Note: The buggy conversion code is only
used for devices with a logical block size != DEV_BSIZE.

The Solaris 8 SPARC version of the 'sd' driver does not contain the
subroutine with the buggy block size conversion
(sd_nodevbsize_blksize()).  Probably because you're supposed to attach
Sun CDROM/DVD devices to your Sun machines, and these Sun SCSI devices
use 512 byte logical blocks anyway.  (I'm unsure if that is the case
for Sun machines with ATAPI CDROM/DVD devices, too?)

2.

On 32-bit Solaris 8 the hsfs filesystem cannot read files > the 4GB
mark on the device (x86 for sure; probably SPARC, too).

Again, the code converts from iso9660 logical block numbers to byte
offsets and does that using 32-bit variables/arithmetic.

On DVD media you might be able to workaround this bug by using the
UDFS filesystem, in case the DVD disc vendor uses dual filesystems on
the media (HSFS/UDFS; quite common on DVD video discs).  Of cause on a
solaris 8 Intel box this workaround is useless, because of problem #1.

 
 
 

1. BUG: S8 x86 sd driver cannot read beyond 4GB on DVD

While playing with mplayer (http://www.tools.de/solaris/mplayer) and
some DVDs on solaris 8 x86 (MU4, ATAPI DVD-COM), I wondered why the
mpeg movies are (apparently) stored *multiple* times on such a DVD
disk.

Well, it turned out that the data is *not* duplicated multiple times
on the media; the effect I've observed is a bug in the solaris sd
driver.  On devices with a logical block size != 512 bytes, there are
problems reading data beyond the 4GB mark.  Due to a 32bit overflow
somewhere in the subroutines sd_nodevbsize_blksize() and
sd_shadow_iodone()[*], all dvd accesses are modulo 2^32!  You can't
read data beyond the 4GB mark!

Quick test:

Insert a DVD (must contain more than 4GB of data, and it's probably
best if you find an unencrypted DVD - otherwise you've to perform the
css authentication protocol using something like libcss-0.1.0):

  % df -k /cdrom/cdrom0/.
  Filesystem            kbytes    used avail capacity Mounted on
  /vol/dev/dsk/c0t1d0/lola_rennt
                       7888642 7888642     0   100%   /cdrom/lola_rennt

7.8GB, fine.

  % ksh -c "cmp <(dd if=/vol/dev/aliases/cdrom0 bs=2k count=1000 \
  iseek=0) <(dd if=/vol/dev/aliases/cdrom0 bs=2k count=1000 \
  iseek=2097152)"
  1000+0 records in
  1000+0 records out
  1000+0 records in
  1000+0 records out
  %

The above command compares 2 MByte of data starting at device offset 0
with a 2 MByte block of data starting at device offset 4GB.  The two
blocks are bit for bit identical!

Is this a known bug?  In case this is a new bug, can someone please
file a bug report?

[*] it seems, as if sd_nodevbsize_blksize() and sd_shadow_iodone()
contain something like

  unsigned tmp = buf->b_blkno * DEV_BSIZE;
  device_block = tmp / device->logical_block_size;

to convert from DEV_BSIZE (512byte) blocks to device block numbers
(2048byte on DVD/CDROM).  'tmp' is the byte offet into the device; and
we have an overflow when accessing data >4GB.

2. Pgp for Linux

3. hsfs and ufs filesystem migration question

4. newfs question....

5. BUG: mountd / share_nfs cannot export NFS filesystem to subnet

6. Kernel malloc prob. with NFS

7. remote mounting, "cannot stat <host>:/filesystem"

8. Hostname resolution problems on subnetwork

9. msdos (fat) filesystem bug 1.3.62->

10. Noutmp entry : ---> read-only filesystem ????

11. >Filesystem mounted (created(?)) read only

12. How to edit a read only filesystem -> lilo.conf and fstab

13. 4GB SCSI disks and filesystems ?