canonicalizing tar

canonicalizing tar

Post by phil-news-nos.. » Mon, 13 May 2002 18:28:27



Anyone know of a good way to make tar archive files in canonical
(sorted) order, as opposed to the order they come from readdir()?

When I try to feed it a list of files pre-sorted I run into these
problems:

1.  If directory names are included, it backs up that directory
    tree at that point.  Those are then duplicates of the files
    separately specified, and are not sorted, either.

2.  If directory names are not included, directory metadata is
    not included, either.  While tar does create the directories
    at extract time, it doesn't apply the metadata.

3.  The -T option can get around command line argument limits, but
    it doesn't help the above 2 problems.

Here is what happens when I try some methods:

=============================================================================




drwx------ phil/phil         0 2002-05-12 04:16 testdir/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/z/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/1
drwx------ phil/phil         0 2002-05-12 04:16 testdir/y/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/1
drwx------ phil/phil         0 2002-05-12 04:16 testdir/x/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/1
drwx------ phil/phil         0 2002-05-12 04:16 testdir/c/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/b/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/a/

-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/2

drwx------ phil/phil         0 2002-05-12 04:16 testdir/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/z/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/1
drwx------ phil/phil         0 2002-05-12 04:16 testdir/y/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/1
drwx------ phil/phil         0 2002-05-12 04:16 testdir/x/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/1
drwx------ phil/phil         0 2002-05-12 04:16 testdir/c/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/b/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/a/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/a/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/b/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/c/
drwx------ phil/phil         0 2002-05-12 04:16 testdir/x/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/x/2
drwx------ phil/phil         0 2002-05-12 04:16 testdir/y/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/y/2
drwx------ phil/phil         0 2002-05-12 04:16 testdir/z/
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/2
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/1
-rw------- phil/phil         0 2002-05-12 04:16 testdir/z/2

=============================================================================

What I ultimately want is for each object to be archived exactly once
and in sorted (by name) order.  BTW, even running find through sort
gets it wrong for filenames with characters lower in collation than
the "/" character is in ASCII.  I can get around this by translating
the "/" character to something much lower before sort and translating
it back afterwards.  But it isn't solving the above problems which is
explained from the man page as:

  The use of a directory name always implies that the subdirectories
  below should be included in the archive.

If only there was an option to turn that feature off.

--
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |

-----------------------------------------------------------------

 
 
 

canonicalizing tar

Post by Andrew Giert » Mon, 13 May 2002 22:53:26


 phil> Anyone know of a good way to make tar archive files in
 phil> canonical (sorted) order, as opposed to the order they come
 phil> from readdir()?

pax -wd -x ustar

(feed it the filenames on stdin in your chosen order...)

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>

 
 
 

canonicalizing tar

Post by Terran Melconi » Tue, 14 May 2002 13:48:34



>Anyone know of a good way to make tar archive files in canonical
>(sorted) order, as opposed to the order they come from readdir()?

Any chance using cpio (or equivalent) is an option?  It behaves more
like the way you want.
 
 
 

canonicalizing tar

Post by phil-news-nos.. » Tue, 14 May 2002 20:21:40




|
|  phil> Anyone know of a good way to make tar archive files in
|  phil> canonical (sorted) order, as opposed to the order they come
|  phil> from readdir()?
|
| pax -wd -x ustar
|
| (feed it the filenames on stdin in your chosen order...)

I don't find this at all in Slackware.  There's no project for it in
freshmeat.net.  How common is it found?  Got a FTP/URL for source?

--
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |

-----------------------------------------------------------------

 
 
 

canonicalizing tar

Post by Victor Wagne » Tue, 14 May 2002 21:42:20




> |  phil> canonical (sorted) order, as opposed to the order they come
> |  phil> from readdir()?
> |
> | pax -wd -x ustar
> |
> | (feed it the filenames on stdin in your chosen order...)
> I don't find this at all in Slackware.  There's no project for it in

Use better distro. It is in Debian (and you can get site tarball
from ftp.debian.org)

Quote:> freshmeat.net.  How common is it found?  Got a FTP/URL for source?

It predates freshmeat. It was in BSDi long ago before freshmeat was
launched. For old programs look to ftp.ibiblio.org

--
http://www.communiware.ru                     http://www.ice.ru/~vitus

 
 
 

canonicalizing tar

Post by Andrew Giert » Tue, 14 May 2002 21:52:20


 > | pax -wd -x ustar
 > |
 > | (feed it the filenames on stdin in your chosen order...)

 phil> I don't find this at all in Slackware.  There's no project for
 phil> it in freshmeat.net.  How common is it found?

everywhere except Linux, apparently :-)

It's a POSIX.2 / SUS requirement, and it's available on *BSD and on
every commercial Unix I've used.

 phil> Got a FTP/URL for source?

you could try src/bin/pax in the FreeBSD distributions; it also needs
the FTS library, but doesn't appear to rely on much else that's
BSD-specific (though for the reasons given above I've never tried
compiling it on any other system :-)

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>

 
 
 

canonicalizing tar

Post by Joe Halpi » Tue, 14 May 2002 22:19:44




>  > | pax -wd -x ustar
>  > |
>  > | (feed it the filenames on stdin in your chosen order...)

>  phil> I don't find this at all in Slackware.  There's no project
>  phil> for it in freshmeat.net.  How common is it found?

> everywhere except Linux, apparently :-)

Slackware != Linux. It's certainly in the RedHat distribution.

The OP can get the source RPM from (where else?) rpmfind:

http://rpmfind.net/linux/RPM/redhat/7.1/i386/pax-1.5-3.i386.html

Joe

 
 
 

canonicalizing tar

Post by Lew Pitch » Tue, 14 May 2002 22:51:12


On 13 May 2002 08:19:44 -0500, Joe Halpin




>>  > | pax -wd -x ustar
>>  > |
>>  > | (feed it the filenames on stdin in your chosen order...)

>>  phil> I don't find this at all in Slackware.  There's no project
>>  phil> for it in freshmeat.net.  How common is it found?

>> everywhere except Linux, apparently :-)

>Slackware != Linux. It's certainly in the RedHat distribution.

>The OP can get the source RPM from (where else?) rpmfind:

>http://rpmfind.net/linux/RPM/redhat/7.1/i386/pax-1.5-3.i386.html

OTOH, Slackware isn't restricted to RPMs (although you _can_ install from
RPM).

To the OP: Take a look at
http://ibiblio.org/pub/Linux/utils/compress/pax-2.1.tar.gz
which should be installable in Slackware (or any other Linux distro)

Remember, Slackers aren't afraid to get their hands dirty (I'm a Slacker,
too  ;-) )

Lew Pitcher, Information Technology Consultant, Toronto Dominion Bank Financial Group

(Opinions expressed are my own, not my employer's.)

 
 
 

canonicalizing tar

Post by Julie » Wed, 15 May 2002 11:36:15


 ha scritto:

Quote:>| pax -wd -x ustar
[...]
> I don't find this at all in Slackware.  There's no project for it
> in freshmeat.net.  How common is it found?

How common? Well, it's the only standard POSIX utility found in Windows
NT. That should give you an idea

--
 < http://www.reactos.com/ >  Open source clone of Windows NT. Current
     Don't stand, REACT!      version 0.0.19. C, C++ and ASM developers
                              and beta testers are welcome!

 
 
 

canonicalizing tar

Post by phil-news-nos.. » Mon, 20 May 2002 09:27:37




|> |  phil> canonical (sorted) order, as opposed to the order they come
|> |  phil> from readdir()?
|> |
|> | pax -wd -x ustar
|> |
|> | (feed it the filenames on stdin in your chosen order...)
|
|> I don't find this at all in Slackware.  There's no project for it in
|
| Use better distro. It is in Debian (and you can get site tarball
| from ftp.debian.org)

I evetually found it at ftp.gnu.org.

Let me know when Debian comes out with a version that uses BSD style
bootscripts instead of SysV style.

| It predates freshmeat. It was in BSDi long ago before freshmeat was
| launched. For old programs look to ftp.ibiblio.org

I found it.  Took a while to port it to Linux, but I finally got it
compiled.  Strange that it did use autoconf, but it wasn't set up in
autoconf correctly to make it portable.

--
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |

-----------------------------------------------------------------

 
 
 

canonicalizing tar

Post by phil-news-nos.. » Mon, 20 May 2002 09:32:40




|  > | pax -wd -x ustar
|  > |
|  > | (feed it the filenames on stdin in your chosen order...)
|
|  phil> I don't find this at all in Slackware.  There's no project for
|  phil> it in freshmeat.net.  How common is it found?
|
| everywhere except Linux, apparently :-)
|
| It's a POSIX.2 / SUS requirement, and it's available on *BSD and on
| every commercial Unix I've used.
|
|  phil> Got a FTP/URL for source?

I found it at ftp://ftp.gnu.org/gnu/cpio/cpio-2.4.2.tar.gz
(md5: e651ca1e1ac53aaebfa7ad256b0fe4fc)

It didn't compile.  Looked like the problem with an assumption about
the implementation of strdup().  Even though glibc for Linux implements
strdup(), this GNU version of cpio didn't know how to handle that, even
though it used autoconf to configure things.  Looks like they simply
didn't put "strdup" into the autoconf setup, so it just assumed that
strdup() wasn't there and compiled its own (in conflict).  Patching a
few things got it going.

--
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |

-----------------------------------------------------------------

 
 
 

canonicalizing tar

Post by phil-news-nos.. » Mon, 20 May 2002 09:34:05





|>
|>  > | pax -wd -x ustar
|>  > |
|>  > | (feed it the filenames on stdin in your chosen order...)
|>
|>  phil> I don't find this at all in Slackware.  There's no project
|>  phil> for it in freshmeat.net.  How common is it found?
|>
|> everywhere except Linux, apparently :-)
|
| Slackware != Linux. It's certainly in the RedHat distribution.

RedHat != Linux, either.  Linux is just a kernel.  Looks like cpio
is in the latest version of Slackware.  I eventually did find it on
a newer machine I had installed a newer Slackware on (but not the
one I needed it on).

--
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |

-----------------------------------------------------------------

 
 
 

canonicalizing tar

Post by phil-news-nos.. » Mon, 20 May 2002 09:39:14



| OTOH, Slackware isn't restricted to RPMs (although you _can_ install from
| RPM).
|
| To the OP: Take a look at
| http://ibiblio.org/pub/Linux/utils/compress/pax-2.1.tar.gz
| which should be installable in Slackware (or any other Linux distro)
|
| Remember, Slackers aren't afraid to get their hands dirty (I'm a Slacker,
| too  ;-) )

True.  I just wasn't originally trying to use cpio at all (hoping to
use plain tar).  I had used cpio in the past, and encountered the fact
that it didn't handle devices, pipes, and sockets, even in tar mode.
Then when someone suggested it, I was just making quick checks for it.

Interestingly enough, among LFS (Linux From Scratch) people, Slackware
is the most popular starting system, enve though LFS can work find from
just about any Linux distribution.  I guess it's in the personality.

--
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |

-----------------------------------------------------------------

 
 
 

canonicalizing tar

Post by Joe Halpi » Mon, 20 May 2002 23:39:01






> |>
> |>  > | pax -wd -x ustar
> |>  > |
> |>  > | (feed it the filenames on stdin in your chosen order...)
> |>
> |>  phil> I don't find this at all in Slackware.  There's no project
> |>  phil> for it in freshmeat.net.  How common is it found?
> |>
> |> everywhere except Linux, apparently :-)
> |
> | Slackware != Linux. It's certainly in the RedHat distribution.

> RedHat != Linux, either.

I didn't say it was (that's why I spoke of the RedHat *distribution*),
I just pointed him at a place he could get the source from. I don't
know where Slackware keeps its stuff.

Joe

 
 
 

canonicalizing tar

Post by Andrew Giert » Mon, 20 May 2002 14:13:59


 phil> True.  I just wasn't originally trying to use cpio at all
 phil> (hoping to use plain tar).

pax is not cpio.

--
Andrew.

comp.unix.programmer FAQ: see <URL: http://www.erlenstar.demon.co.uk/unix/>
                           or <URL: http://www.whitefang.com/unix/>

 
 
 

1. How to canonicalize a filename - or f$parse for unix

I would like to canonicalize a file or directory name (relative to the
current directory if it's a relative path).

Sort of the reverse of basename.

In VMS I would use f$parse, I wonder if there is any standard Unix
equivalent, or shell idiom to do this.

E.g. given   PWD = /usr/stuff   then

canonicalize hello      => /usr/stuff/hello
canonicalize /hello     => /hello
canonicalize ../hello   => /usr/hello

I could write a script to do this pretty easily, but I don't see how to do
it in just one or two lines, and I suspect there is already something that
exists, I just haven't found it.

Malcolm

2. Q: Serial ports: COM1-4 and Interrupts

3. cannot canonicalize

4. Solaris 2.5.1 & PPP Server

5. tar -xvf mozilla.tar = tar: directory checksum error

6. NATd and IPFW

7. Wanted: rcmd host tar x_?_vqf file.tar file1 ... fileN | tar xvf -

8. O(1) J9 scheduler: set_cpus_allowed

9. tar won't tar!

10. backup multiple tar-files on a tape using tar and mt - command

11. Extracting files with tar and uncompress fails with tar: Archive - EOF not on block boundary

12. "rm -rf /usr/ports" before "tar -xvzf ports.tar.gz"???

13. tar & mt command. Getting Tar:Unexpected EOF