Very Computer

I understand that RAW is, like, TIFF, a lossless format; that it, however,
produces smaller files.

Accordingly, RAW would seem a valid format for archival purposes.  However,
I read that there is no standard for the data format, and wonder whether
this might be a problem.  Does anyone know whether any standards are in the
pipeline and, if so, what impact they are likely to have upon current
implementations.

NikonScan software does not include RAW, but it does have 'NEF': 'The
formats include NEF, (Nikon Electronic image Format) which allows files to
be saved, and then reopened and modified without harming the original
scanned information. NEF is not legible to Adobe Photoshop or any other
imaging applications, these files can only be opened using Nikon Scan
software.'  What is the format; and should it be avoided (sems like it to
me, but my understanding has me question why they'd have this at all)?

So far, my conclusion is to use TIFF.  However, any advice which would
suggest that this is inappropriate would be appreciated.

The only challenge is that I want slides to be available on my PC, rather
than stored on DVD, for example (I'm moving around all the time and carting
hundreds of DVDs around is a lot harder than throwing a PC into a box); I
could use DVD as offline archival fof the shots that really matter, though.
The problem is, with TIFF, I'm looking at 130MB per shot; ultimately 10,000+
shots (and growing); add to that RAID 1 and a need for regular back-ups and
I need an almost unmanageable amount of disk space.

Incidentally, does anyone know how RAW achieves smaller files than TIFF, and
approximately how much smaller they are?

Thank you in advance.

Very often, RAW files are actually TIFF variants, as TIFF (by definition) is
tagged image file format. Different applications can produce TIFFs including
application-specific data. TIFF can be losslessly compressed, as is often
the case with RAW files, hence the smaller files (30-40% smaller, depends on
image).

The reason for not using TIFF as a basis for RAW files normally implies that
the device/application requires very peculiar data to be stored (or, in a
peculiar way). This is why I don't think there would be any standard for RAW
files. (Do you really wish there to exist yet another file format??? ;-)

I'm dealing with the same capacity problem as you are. Without removable
media (like DVD), there's almost no way to do it. 130MB * 10000 figures out
1300GB... Expensive and unpractical, I would say. If you don't agree, see
LaCie (http://www.lacie.com/), they have big disks (up to 500GB).

    Jouko

Quote:

> I understand that RAW is, like, TIFF, a lossless format; that it, however,
> produces smaller files.

> Accordingly, RAW would seem a valid format for archival purposes.
However,
> I read that there is no standard for the data format, and wonder whether
> this might be a problem.  Does anyone know whether any standards are in
the
> pipeline and, if so, what impact they are likely to have upon current
> implementations.

> NikonScan software does not include RAW, but it does have 'NEF': 'The
> formats include NEF, (Nikon Electronic image Format) which allows files to
> be saved, and then reopened and modified without harming the original
> scanned information. NEF is not legible to Adobe Photoshop or any other
> imaging applications, these files can only be opened using Nikon Scan
> software.'  What is the format; and should it be avoided (sems like it to
> me, but my understanding has me question why they'd have this at all)?

> So far, my conclusion is to use TIFF.  However, any advice which would
> suggest that this is inappropriate would be appreciated.

> The only challenge is that I want slides to be available on my PC, rather
> than stored on DVD, for example (I'm moving around all the time and
carting
> hundreds of DVDs around is a lot harder than throwing a PC into a box); I
> could use DVD as offline archival fof the shots that really matter,
though.
> The problem is, with TIFF, I'm looking at 130MB per shot; ultimately
10,000+
> shots (and growing); add to that RAID 1 and a need for regular back-ups
and
> I need an almost unmanageable amount of disk space.

> Incidentally, does anyone know how RAW achieves smaller files than TIFF,
and
> approximately how much smaller they are?

> Thank you in advance.

Quote:

> The reason for not using TIFF as a basis for RAW files normally implies
that
> the device/application requires very peculiar data to be stored (or, in a
> peculiar way). This is why I don't think there would be any standard for
RAW
> files. (Do you really wish there to exist yet another file format??? ;-)

Quote:> I'm dealing with the same capacity problem as you are. Without removable
> media (like DVD), there's almost no way to do it. 130MB * 10000 figures
out
> 1300GB... Expensive and unpractical, I would say. If you don't agree, see
> LaCie (http://www.lacie.com/), they have big disks (up to 500GB).

I was thinking that Firewire drives might be an option - 250GB, portable and
it is easy to add quite a few to the system.

If you don't want to have to reprocess the images, use TIFF and compress it.

Quote:> The only challenge is that I want slides to be available on my PC, rather
> than stored on DVD, for example (I'm moving around all the time and carting
> hundreds of DVDs around is a lot harder than throwing a PC into a box); I
> could use DVD as offline archival fof the shots that really matter, though.
> The problem is, with TIFF, I'm looking at 130MB per shot; ultimately 10,000+
> shots (and growing); add to that RAID 1 and a need for regular back-ups and
> I need an almost unmanageable amount of disk space.

The other issue of archiving is for how long.  If you are talking about
a few years, then any solution is fine.  However if you are talking about
10, 20, 30 years, one must always think about whether you will have the
software to read the particular file format you archive, not to mention
whether the media will be readable.

Quote:>Accordingly, RAW would seem a valid format for archival purposes.

Quote:>So far, my conclusion is to use TIFF.  However, any advice which would
>suggest that this is inappropriate would be appreciated.

Quote:>The problem is, with TIFF, I'm looking at 130MB per shot

Quote:>Incidentally, does anyone know how RAW achieves smaller files than TIFF ...

Quote:> ... and approximately how much smaller they are?

Bill

TIFF is a tagged image format, well specified such that many
applications, both existing now and in the future, will be able to read
it. Because of that, it is the sensible archive choice regardless of
size.

Of course, there are other good bets for archive format too. For
instance, I don't expect that Adobe Systems' PhotoShop file format will
be lost anytime soon as I believe it has been licensed for development
of Photoshop compatible utility software and add-ons. Even if it
weren't, Photoshop has such a large footprint in the image processing
market that it would be worth the effort for someone to backwards
engineer it if Adobe Systems went out of business, all current products
that knew the format suddenly disappeared, and all the existing source
code that implemented it somehow magically was permanently erased
overnight. Probabilities of all that happening at once are quite low...
;-) I archive my original scan as a TIFF and subsequently modified and
developed images as .PSD to preserve all layers and other work, along
with a web display resolution sized JPEG for quick perusal. The last is
for convenience' sake since I can always generate that from the PSD
file easily. It solves one of your problems: keep the JPEGs on your PC,
keep the full resolution archive files on off-line media.

The disk space problem is taxing for the number of images you're
interested in storing, not to mention an image retrieval system that
would allow you to find what you want. I don't think it's totally
unmanageable, however ... although it isn't particularly inexpensive.
Neither CDR nor DVD have enough capacity to be considered. Let's see,
10,000 130MB image files is about 1,300 GBytes of data ... that would
fit handily on three 500 Gbyte drives. You could buy them each, one at
a time, as you need them and take advantage of the price drop as they
become cheaper over time. Larger storage devices will become available
at increasing lower prices as well.

My data archive volume might get up into that range eventually too, so
I've started with a pair of 100GB drives ($90 apiece for the bare
drive, plus the cost of an enclosure, backs up the other) with FireWire
interfaces so I can plug and unplug them anytime I need. The
performance is far far better than CDR and DVDR, and given the large
MTBF ratings of good quality drives, they're probably more reliable
too. Plus, if operating system or file system parameters change over
time, they can be updated easily to keep step.

That's my solution. I'll tell you how well it works in another 20
years... ;-)

Godfrey

Quote:> A RAW format is exactly what it states, raw data in whatever arbitrary
> format whomever designs a device and its software delivers it.

I shall continue to save in TIFF.

And I'll archive onto hard drives (La Cie ha s 500GB; WDC has 250GB, but
faster).

I'll use 4,000dpi for the images I like, and (most likely) 2000dpi for the
detritus.  That should keep things manageable for the time being.

Now I'll see how much time I can waste deliberating - for 10,000 slides -
whether an image is a 4,000dpi scan or only a 2,000dpi one.

Again, thank you to everyone.

Graham Millington

You might want to check PNG and JPEG2000 as storage formats. Both are known
published standards (PNG http://www.w3c.org/Graphics/PNG/ and JPEG2000
http://www.jpeg.org/JPEG2000.html), both support also 16 bit and both are
lossless. Photoshop has already a jpeg2000 plugin from Adobe ($99), there
are also freeware ones around. I think Photoshop will have jpeg2000 "native"
in version 8. Size gets down from 115MB tiff to around 85MB with jpeg2000 16
bit lossless compression. Might help a little... Downside is of course that
the conversion takes quite a bit of processing power and time. On my PC
opening/saving a 115MB tiff in PS takes about 3-4 sec, jpeg2000 with
freeware plugin takes around half a minute.

I have not examined PNGs any closer, but it should be on the same level. An
example from my Canon EOS 10D camera raw (converted with BreezeBrowser)

raw    5.9M
8 bit tiff   18.4M
16 bit tiff  36.9M
jpeg    3.0 M (lossy, best quality)
8 bit jpeg2000  5.4M (lossless)
16 bit jpeg2000  12.3M (lossless)
8 bit png  6.6M
16 bit png 11.0M

HTH
Ilkka

Greg.