A question about appending with the write system call

A question about appending with the write system call

Post by Chris Sherm » Sat, 04 Jan 1992 07:05:09



A unix question:

Two (or more) processes 'open' (not fopen) the same file with
O_APPEND.  Then both processes 'write', say, 80 byte blocks to the
file.

Do the two (or more) processes stomp on each other's blocks of data?
And by stomp, I mean will characters or blocks be interleaved or over-
written.  And what happens at block boundaries?

And what happens with nfs and afs in the equation?

The big question is how do we get a whole bunch of processes to open a
single file (a log file, in fact) and write to the end of it at the same
time.

We are thinking HP/UX here, but a general answer would be useful too.

Thanx,
--
     ____/     /     /     __  /    _  _/    ____/
    /         /     /     /   /      /     /          Chris Sherman
   /         ___   /        _/      /          /

 
 
 

A question about appending with the write system call

Post by Matthew Farwe » Sun, 05 Jan 1992 01:17:59



>A unix question:
>Two (or more) processes 'open' (not fopen) the same file with
>O_APPEND.  Then both processes 'write', say, 80 byte blocks to the
>file.

>Do the two (or more) processes stomp on each other's blocks of data?
>And by stomp, I mean will characters or blocks be interleaved or over-
>written.

I asked this very same question about a year ago.  (Apologies if the
answers aren't quite correct - my old mail is buried in a pile of tapes
and can't be gotten at easily)

The answer is, generally, the writes will be atomic and the data won't
be overwritten or interleaved.  Apparently there is a race condition in
4.2 BSD, so any systems derived from said may or may not have this
problem.

Quote:>          And what happens at block boundaries?

I believe some Unices split the writes at block boundaries, so the write
is no longer atomic. The best way to make sure of avoiding this is to
have your data block size as a divisor of your disk block size (ie pad
it to 128 or something like that). This way you avoid the problem.

Quote:>And what happens with nfs and afs in the equation?

NFS?  Oh, sorry, I thought we were talking about Unix filesystem
semantics.  With NFS, all bets are off.

Quote:>The big question is how do we get a whole bunch of processes to open a
>single file (a log file, in fact) and write to the end of it at the same
>time.

Just using O_APPEND should be sufficient.  However, what you may want to
do is use your own logging daemon.  The bunch of processes just connect
to the daemon and give the data to it.  Then you only have one process
writing, and you avoid all the above problems.

Dylan.
--
This is the ordinary signature

 
 
 

A question about appending with the write system call

Post by Al Cla » Sun, 05 Jan 1992 06:22:48


To mingle appends, suggest you consider fopen(filename, "a");
You can use setbuf() to force writes on line or block boundaries,
and fflush() to for writes when you choose.  You also then have
available fputs(), fprintf(), fwrite(), and putc(), and fputc() as
well.

--

 
 
 

A question about appending with the write system call

Post by Guy Harr » Sun, 05 Jan 1992 10:10:46


Quote:>To mingle appends, suggest you consider fopen(filename, "a");

Which, in *SOME* versions of the standard I/O library, but *NOT* all,
has the effect of turning O_APPEND on.  (Older BSD versions do not turn
O_APPEND on; at some point, they will probably change the standard I/O
library to do so, as ANSI C says that the "a" flag forces all subsequent
writes "to be forced to the then current end-of-file".)
 
 
 

A question about appending with the write system call

Post by John Chambe » Sun, 05 Jan 1992 13:31:34



> A unix question:

> Two (or more) processes 'open' (not fopen) the same file with
> O_APPEND.  Then both processes 'write', say, 80 byte blocks to the
> file.

> Do the two (or more) processes stomp on each other's blocks of data?
> And by stomp, I mean will characters or blocks be interleaved or over-
> written.  And what happens at block boundaries?

Not unless some turkey has broken something internally.  The point  of
O_APPEND is to solve exactly this problem. Each write() is atomic, and
with O_APPEND, they will all append to the file. Caveat: The atomicity
of  write()  calls  is  guaranteed  only up to BUFSIZ bytes.  But with
O_APPEND, longer writes won't stomp  on  each  other;  they  might  be
interlaced.

Quote:> And what happens with nfs and afs in the equation?

No idea.  I wouldn't trust them unless I could get a definitive answer
from someone who I personally believed was knowledgeable.

Quote:> The big question is how do we get a whole bunch of processes to open a
> single file (a log file, in fact) and write to the end of it at the same
> time.

Open()  with  O_APPEND,  and  use  write().   Don't  use  fopen()  and
fprintf();  it  is free to chop messages up in interesting and amusing
ways,  and  they'll  come  out  interlaced.   If  you  want   printf's
formatting,  use  sprintf()  to  write to a buffer (which is no bigger
than BUFSIZ), and write it with a single write() call.  This is what I
do  in  my  personal  C  debugging  module,  and  the result is highly
portable and quite reliable.  Don't use any of the stdio  routines  to
write  it  out;  even if you call fflush(), you aren't guaranteed that
the data will make it into the kernel much less to the disk within any
finite time limit.  (Sad but true of too many commercial systems.) You
might also consider calling fsync() if you're worried  about  loss  of
messages  during  a  crash,  and  of  course  that's  when you're most
interested in not losing the messages.

Quote:> We are thinking HP/UX here, but a general answer would be useful too.

It isn't very well documented, but anything that calls  itself  "Unix"
will guarantee atomicity of read() and write() up to BUFSIZ bytes, and
O_APPEND will work to prevent lost output to audit trails.  (You  then
have only the minor problem of the files filling up your disk.  ;-)

--
All opinions Copyright (c) 1991 by John Chambers.  Inquire for licensing at:
Home: 1-617-484-6393 ...!{bu.edu,harvard.edu,ima.com,eddie.mit.edu,ora.com}!minya!jc
Work: 1-508-486-5475 {sppip7.lkg.dec.com!jc,ub40::jc}

 
 
 

A question about appending with the write system call

Post by John F Ca » Mon, 06 Jan 1992 05:33:06


[context: O_APPEND and network filesystems]

|> > And what happens with nfs and afs in the equation?
|>
|> No idea.  I wouldn't trust them unless I could get a definitive answer
|> from someone who I personally believed was knowledgeable.

If you open an NFS file with O_APPEND, the client kernel checks the file
size before each write request and writes where it is told the end of the
file is.  There is a race condition if two clients are writing at the same
time, but appending will often work if the interval between calls to write()
is longer than the NFS server response time (i.e. for important data, don't
rely on this feature).

O_APPEND does not work right in AFS, because clients cache files and the
last client to store back to the server will replace the old contents.
(General advice: always assume that AFS does the wrong thing.  You will rarely
be disappointed and sometimes you will be pleasantly surprised when it works.)

--

 
 
 

A question about appending with the write system call

Post by Boyd Rober » Sun, 05 Jan 1992 20:44:06



> Caveat: The atomicity of  write()  calls  is  guaranteed  only up
> to BUFSIZ bytes.

Ok, this is about the third time I've read this type of statement, so
just _who_ has broken write so the whole write is no longer atomic?

The only case where a write is not atomic is on a pipe when the size
of the write is greater than the _maximum_ size of the pipe (I'm talking
about V7 pipe semantics here; not socket pipes, stream pipes or any other conceivable form of pipe).

So just who is guilty?  Or are you people confused?


``When the going gets wierd, the weird turn pro...''

 
 
 

A question about appending with the write system call

Post by Lyle_Sea.. » Wed, 08 Jan 1992 04:53:25


Excerpts from netnews.comp.unix.internals: 4-Jan-92 Re: A question about

Quote:> If you open an NFS file with O_APPEND, the client kernel checks the file
> size before each write request and writes where it is told the end of the
> file is.  There is a race condition if two clients are writing at the same
> time, but appending will often work if the interval between calls to write()
> is longer than the NFS server response time (i.e. for important data, don't
> rely on this feature).
> O_APPEND does not work right in AFS, because clients cache files and the
> last client to store back to the server will replace the old contents.
> (General advice: always assume that AFS does the wrong thing.  You will
> rarely be disappointed and sometimes you will be pleasantly surprised
> when it works.)

John's statements here are pretty much on the mark, except for that last one...

The AFS-3 file semantics are that all clients will see the state of the
file as of the last time it was close() - ed.  So, for processes A and B
on one client workstation, and process C on another, if process A is
writing the file, process B will see all the writes, but process C
won't.  The way to guarantee that process C sees the writes is to
write(),close(),open(), and continue, or write(),fsync(), continue, but
there is still a race condition there.  

AFS-4  (DCE DFS, available sometime from a vendor near you, nag one)
will support fine-grain read-write sharing, so any two processes writing
a file with O_APPEND will behave essentially identically regardless of
whether they are on the same host or different ones, albeit at a small
performance penalty (due to the added communications involved with
repeatedly notifying each client where the end of the file is, and what
the contents of the file are).

AFS-3 is designed for general usage, (which is read-mostly,
write-rarely, and most of the writes are complete rewrites of a file,
which is usually re-read immediately) and
not for append-always, read-almost-never applications.  NFS would be
suitable for the cited use.

Ob Tip:  If you know you're going to delete a file which you currently
have open (eg, a temp file), unlink() before the close() instead of
vice-versa: it's faster, especially with log-based filesystems, or AFS.  

Lyle            Transarc                707 Grant Street
412 338 4474    The Gulf Tower  Pittsburgh 15219
Ich bin ein Virus. Mach' mit und kopiere mich in Deine .signature.

 
 
 

1. Question on writing c program implement cp function using UNIX system calls - read, write, etc.

I need to write a c program that will perform like the cp command in UNIX
and copy one file to another. I need to use the UNIX system calls read(),
write(), open(), close(). I know how to open the source file for read access
and destination file for write access with open(), but I am unsure of how to
use read() and write().

I know read takes my file descriptor (fd) of the source file I opened as its
1st arguement. For the second, I need to put those bytes in some sort of
buffer right? Could I just write them directly to the destination file
instead of storing into a buffer? Well if I have to make a buffer for input
I read in I do this:

c = (char *) calloc(100, sizeof(char));

Is that right? If I do this will it work for all files whether they are
binary or ascii files?

The 3rd arg of read() takes in how many  bytes to read, how do I tell it to
read in the entire source file?

So for read, this is what I have so far: read(fd, c, ????) where fd is my
source file descriptor and c as define above.

As for write, I have: write(fd2, ???, ???) where fd2 is the file descriptor
of my destination file. How do I specify it to write what I read from the
input source file and write all of those bytes inputted? Thanks for any
help.

2. Way to discard a dirty buffer_head

3. Need help writing C program using UNIX system calls (read, write, etc) that copies files

4. X Freezing

5. How I could add a new system call to linux or modify a system calls

6. Help with ioctl call !!!

7. How to use open system call in a new system call

8. Resources for Linux X terminal ?

9. "Interrupted system call" at "low level" - system calls

10. How to write my own system call ?

11. Read & Write MSR system calls

12. System() Call & Writing to Spawned Process' STDIN

13. writing system calls