Log-structured file system?

Log-structured file system?

Post by Brad Fig » Wed, 23 Oct 1996 04:00:00



Any pointers to a Linux log-structured file system are
appreciated.

Thanks,
Brad Figg

 
 
 

Log-structured file system?

Post by Christopher B. Brow » Wed, 23 Oct 1996 04:00:00



Quote:>Any pointers to a Linux log-structured file system are
>appreciated.

http://lucien.blight.com/~c-cook/prof/lfs/

Apparently research and preliminary implementation work is going on now;
"production code" isn't expected 'til well into 1997.
--

Web: http://www.conline.com/~cbbrowne/ SAP Basis Consultant, UNIX Guy
Windows NT - How to make a 100 MIPS Linux workstation perform like an 8 MHz 286

 
 
 

Log-structured file system?

Post by Maciej Stachowi » Fri, 25 Oct 1996 04:00:00






> >>Any pointers to a Linux log-structured file system are
> >>appreciated.

> >http://lucien.blight.com/~c-cook/prof/lfs/

> >Apparently research and preliminary implementation work is going on now;
> >"production code" isn't expected 'til well into 1997.

>    That project has a lot of ambition, but, as Christoper
> mentioned, a long way to go.  For example, their plan begins with
> "Learn about standard filesystems."  I wish them luck though, and I am
> glad somebody set up a mailing list and a good web page on the
> subject.

Well, we're well past that step now. We have mklfs working (and verified
correct by mounting under NetBSD) and kernel code that can successfully mount the
filesystem (but can't do much more). The project will take some time to complete,
but by being based on the BSD-LFS code will get  the benefits of all the research
done on BSD-LFS, and be compatible with FreeBSD, NetBSD, etc. I haven't been
working much on it myself lately, since I've been pretty busy with class, but I
should have time to do some work in the near future.

Quote:>    If you want to see actual code (albiet not quite working
>yet), look at the /usr/src/linux/fs/log directory in the last two
>editions of Yggdrasil Plug & Play Linux.

>    You can find an updated (2.0.12) kernel source tree that includes
>this not yet working filesystem as a loadable module on ftp.yggdrasil.com
>in ~ftp/private/adam/linux-2.0.12.ygg.tar.gz, along with a matching mklogfs
>program as mklogfs.c in the same directory.  The code is actually for a
>compressed log filesystem.

I'd be interested in seeing this code, especially what modifications, if any
had to be made to the buffer cache to support a log-structured fs. Off to the
ftp site...

- Maciej Stachowiak

 
 
 

Log-structured file system?

Post by Adam J. Richt » Fri, 25 Oct 1996 04:00:00





>>Any pointers to a Linux log-structured file system are
>>appreciated.

>http://lucien.blight.com/~c-cook/prof/lfs/

>Apparently research and preliminary implementation work is going on now;
>"production code" isn't expected 'til well into 1997.

        That project has a lot of ambition, but, as Christoper
mentioned, a long way to go.  For example, their plan begins with
"Learn about standard filesystems."  I wish them luck though, and I am
glad somebody set up a mailing list and a good web page on the
subject.

        If you want to see actual code (albiet not quite working
yet), look at the /usr/src/linux/fs/log directory in the last two
editions of Yggdrasil Plug & Play Linux.

        You can find an updated (2.0.12) kernel source tree that includes
this not yet working filesystem as a loadable module on ftp.yggdrasil.com
in ~ftp/private/adam/linux-2.0.12.ygg.tar.gz, along with a matching mklogfs
program as mklogfs.c in the same directory.  The code is actually for a
compressed log filesystem.

        Be warned, however, that when I get back to working on this
filesystem (if nobody else picks it up), I intend to make some
incompatible changes to it.  In particular, I plan to change to
directory format to support something faster than a linear search for
finding, adding and removing a file name in a directory.

--
Adam J. Richter     __     ______________       4880 Stevens Creek Blvd

+1 408 261-6630         | g g d r a s i l       San Jose, CA 95129
fax +1 408 261-6631      "Free Software For The Rest Of Us."

 
 
 

Log-structured file system?

Post by Larry McV » Wed, 13 Nov 1996 04:00:00


: but by being based on the BSD-LFS code will get  the benefits of all the research

Before you invest a lot of time in LFS, consider the following:
LFS places blocks pretty much where the disk head happens to be.
The allocator is naive in the extreme and subsequent research has shown
that the shor term win of writing stuff all clumped together is more
than offset by the long term pain of reorganizing the data.

LFS missed the point that all file systems are basically block allocators.
Their job is to place data such that it can be quickly read back.
LFS fails this taks for anything other than a single *process* operating
system (note, not single user, single process).  To test this, try the
following test on LFS

        $ for i in 1 2 3 4 5 6 7 8 8
        do lmdd of=XXX.$i bs=1m count=20 &
        done

        $ lmdd if=XXX.1 bs=1m

Then try the same test on FFS, EXT2, SGI's XFS, etc, etc.  You'll find
that real file systems will unscramble the stream of requests into N
streams, and put each of the N contiguous on disk.

If you want to look at good file system technology, check out network
appliance's file system.  It's the best I've ever seen and I have
fairly deep understanding of UFS, XFS, and EXT2 (not mention that
thing called LFS).

--lm

P.S.  For those who don't know - LFS sucks because the file system is
the log.  That is a far cry from a logging file system.  Ask yourself
why a database isn't just the database log.  The answer to that is the
answer to why LFS is stupid.

 
 
 

Log-structured file system?

Post by Terje Malmeda » Wed, 13 Nov 1996 04:00:00



> Before you invest a lot of time in LFS, consider the following:
> LFS places blocks pretty much where the disk head happens to be.
> The allocator is naive in the extreme and subsequent research has shown
> that the shor term win of writing stuff all clumped together is more
> than offset by the long term pain of reorganizing the data.

[deleted]

Quote:> If you want to look at good file system technology, check out network
> appliance's file system.  It's the best I've ever seen and I have
> fairly deep understanding of UFS, XFS, and EXT2 (not mention that
> thing called LFS).

> --lm

> P.S.  For those who don't know - LFS sucks because the file system is
> the log.  That is a far cry from a logging file system.  Ask yourself
> why a database isn't just the database log.  The answer to that is the
> answer to why LFS is stupid.

Uhm. We had a visit from a NetApp salesperson, and from his
description their filesystem sound like a variant of LFS. What exactly
is different/better about NetApp's WAFL compared to LFS? (Apart from
the very clever RAID4 stuff).

--
 - Terje

 
 
 

Log-structured file system?

Post by Stephen Tweedi » Fri, 15 Nov 1996 04:00:00


Hi all,

Larry, you may be quite correct with your criticisms of BSD's LFS.
However, I'd take issue with some of the points raised about log
structured filesystems in general.


> Before you invest a lot of time in LFS, consider the following: LFS
> places blocks pretty much where the disk head happens to be.  The
> allocator is naive in the extreme and subsequent research has shown
> that the shor term win of writing stuff all clumped together is more
> than offset by the long term pain of reorganizing the data.

Maybe, maybe not --- it depends enormously on the workload.  If you
are running an LFS volume very close to capacity with a lot of
activity on it, then yes, the background cleaner activity is
substantial.  However, one of the things we have learnt with our
experience on OpenVMS's log-structured filesystem, Spiralog, is that
cleaning is a lot less necessary than you initially expect.

We have a Spiralog home disk here which is typically running at about
80% full.  We run it without the cleaner active for much of the time,
and it works just fine.  The cleaner is enabled briefly prior to doing
an online backup, and that is enough to keep the filesystem
fragmentation down.  Cleaning cost is essentially negligible in this
environment.  

Quote:> LFS missed the point that all file systems are basically block allocators.
> Their job is to place data such that it can be quickly read back.

This misses a fundamental property of LFSes.  The whole point about
log structured filesystem technology was to design a system
specifically suited to running on server-class machines where large
caches eliminate a lot of the user reads.  A great deal of profiling
was done on filesystem activity in the early stages of LFS work, and
we know for a fact that under many typical loads, writes predominate
enormously over reads at the disk IO level.  The fact that LFS
prioritises disk write performance over disk read performance is a
deliberate design decision, not a fault of the system.

Of course, this means that on a typical desktop machine, performance
with a LFS is unlikely to be nearly as good as using a traditional
update-in-place filesystem.  That's OK --- LFS was never intended for
that sort of environment.

Quote:> LFS fails this taks for anything other than a single *process* operating
> system (note, not single user, single process).

Perhaps this is true for BSD LFS, but it is certainly not true for
Spiralog.  The trick is that although the filesystem does not have
freedom to place blocks anywhere it wants, it _does_ have freedom to
decide which blocks to write, and when.  Spiralog operates a
write-back cache policy default, and the cache management layers do a
good job of coalescing logically adjacent (or overlapping) writes
together so that they eventually get shipped to disk as a single,
sequential write.  The filesystem cannot do clustering of data at the
disk level, but write-back caching permits very effective clustering
at the cache level.

Quote:> P.S.  For those who don't know - LFS sucks because the file system is
> the log.

That is not a sufficient condition for a filesystem to suck, although
it is certainly true that it is easy for a naive LFS implementation to
have poor performance under very common load conditions.

Cheers,
 Stephen.
---

OFST: OpenVMS Filesystem Technologies