mbx mailbox corruption and quotas?

mbx mailbox corruption and quotas?

Post by Marc » Sat, 08 Feb 2003 02:42:45



Back in Dec. I posted some messages about getting dmail and procmail
to work together to deliver to MBX mailboxes in ~/INBOX. I've since
gotten that working but have recently notice that I've been getting
some mailbox corruption in user's mailboxes. My question is, is dmail
quota safe when delivering to mbx mailboxes?

On the imap client end, users who get corrupt mailboxes are greeted
with this message when trying to login:

   IMAP protocol error: command unrecognized: "SEARCH ERRFLG = 2"
   Unknown on line 0

   Command unrecognized:  "SEARCH ERRFLG = 2"
   Unknown on line 0

On the server side, running pine as the user results in an error
message about bad UID's in the mailbox. Pine tries to repair the
mailbox, but is unable. Examining ~/INBOX directly reveals a ^M at the
end of each line.

The common thread for these users is that all are at or beyond their
soft quota limit (100M). This makes me suspect that our problem is
quota related. Should dmail be able to correctly handle quotas? If a
message is being written to a mbx mailbox and the hard quota is
reached in mid-message write, will the the integrity of the mailbox be
preserved? Are there non-quota scneraios in which mailbox corruption
can occur? (Perhaps I have something misconfigured.)

I did find a post to this newsgroup from 2001-09-26 with a perl script
that can fix mbx corruption (Subject: Re: Corrupt mbx mailbox: any
fix?). It worked, but I'd like to find out what's causing this problem
rather than picking up the pieces after it occurs.

Relevant info: Red Hat 7.3, ext3 filesystem, dmail v2002.
--

Marc Wallman
North Dakota State University

 
 
 

mbx mailbox corruption and quotas?

Post by Mark Crispi » Sat, 08 Feb 2003 05:53:22



> Back in Dec. I posted some messages about getting dmail and procmail
> to work together to deliver to MBX mailboxes in ~/INBOX. I've since
> gotten that working but have recently notice that I've been getting
> some mailbox corruption in user's mailboxes. My question is, is dmail
> quota safe when delivering to mbx mailboxes?

Yes, dmail (as with all other c-client applications) is supposed to be
quota-safe.  If it is unable to write to the mailbox, it will back out of
the append and restore the mailbox to the previous state.

Note that this will happen on a hard quota limit (a write failure), not a
soft quota limit.

Make sure that you have the latest version.  There were issues on SVR4 and
TRU64 UNIX in older versions.  In particular, a bug in one of the TRU64
filesystem types got tickled by the sequence of system calls used in the
older version, resulting in nulls being written instead of the actual
data.

Since you say that you're running Linux, neither of these issues should
apply.  On the other hand, I don't have much experience with the Linux
implementation of quotas; most people just throw enough disk on Linux
systems.

Quote:> On the imap client end, users who get corrupt mailboxes are greeted
> with this message when trying to login:
>    IMAP protocol error: command unrecognized: "SEARCH ERRFLG = 2"
>    Unknown on line 0
>    Command unrecognized:  "SEARCH ERRFLG = 2"
>    Unknown on line 0

I haven't a clue as to what this means.  "SEARCH ERRFLG = 2" is certainly
not a valid IMAP protocol command, nor is this anything that would be
babbled by UW imapd.

Quote:> On the server side, running pine as the user results in an error
> message about bad UID's in the mailbox. Pine tries to repair the
> mailbox, but is unable.

What precisely is the message, and more importantly, what is the final
message issued by Pine before it is "unable".  The bad UID messages are
generally just warnings; it's the last message that says what really
happened.

Quote:> Examining ~/INBOX directly reveals a ^M at the
> end of each line.

This is correct.  mbx format uses Internet newlines, not UNIX newlines.
This means that the server doesn't need to do newline conversion nor do
count calculation.

Quote:> Should dmail be able to correctly handle quotas? If a
> message is being written to a mbx mailbox and the hard quota is
> reached in mid-message write, will the the integrity of the mailbox be
> preserved?

Yes, and yes.  If not, and you have the latest version, then you have
found a bug and I am very interested in the details.

One problem with dealing with quotas is that there are multiple
implementations, each with different characteristics.  One *
implementation that I once encountered set a bit on an open file
descriptor when a quota exceeded error was hit, and once set it refused to
allow any other operations including file-truncate!

However, it seems to me that you're talking about soft quota, not hard
quota.  As far as the c-client library is concerned, hard quota is the
only thing that exists; the presumption is that the quota system will
produce an appropriate snarl.  The UW computer center has a local patch to
UW imapd that, in conjunction with the local quota and accounting system,
presents the quota snarl as an IMAP alert.  That patch isn't in the
distributed version since it's highly dependent upon our local accounting
system and (ugh!) AIX.

Quote:> Are there non-quota scneraios in which mailbox corruption
> can occur? (Perhaps I have something misconfigured.)

Access via NFS or other network filesystem is the most significant
problem.  Besides the locking issues, mbx format assumes that the inode
and data are updated atomically and instantaneously.  This happens with
local files, but over NFS there is a timelag and more importantly the
inode and data are updated independently.  Modern versions of NFS have
reduced this problem, but it has not gone away entirely.

Quote:> I'd like to find out what's causing this problem
> rather than picking up the pieces after it occurs.

As would I.

-- Mark --

http://www.veryComputer.com/
Science does not emerge from voting, party politics, or public debate.

 
 
 

mbx mailbox corruption and quotas?

Post by Marc » Sun, 09 Feb 2003 05:18:29



> > On the imap client end, users who get corrupt mailboxes are greeted
> > with this message when trying to login:
> >    IMAP protocol error: command unrecognized: "SEARCH ERRFLG = 2"
> >    Unknown on line 0
> >    Command unrecognized:  "SEARCH ERRFLG = 2"
> >    Unknown on line 0

> I haven't a clue as to what this means.  "SEARCH ERRFLG = 2" is certainly
> not a valid IMAP protocol command, nor is this anything that would be
> babbled by UW imapd.

This could just be a funny error reported by the mail client. I
recieved another report today (as I was just finishing composing this
post) with the following error message which is the same as one I got
during the test described below. The client in this case was
squirrelmail 1.2.10.

      ERROR : Could not complete request.
      Query:EXAMINE "INBOX"
      Reason Given: EXAMINE failed: Unable to find CRLF at 20022401 in
64
      bytes, text: From root Fri Feb 7 00:02:09 CST 2003

      ERROR : Could not complete request.
      Query:SELECT "INBOX"
      Reason Given: SELECT failed: Unable to find CRLF at 20022401 in
64
      bytes, text: From root Fri Feb 7 00:02:09 CST 2003

Quote:> > On the server side, running pine as the user results in an error
> > message about bad UID's in the mailbox. Pine tries to repair the
> > mailbox, but is unable.

> What precisely is the message, and more importantly, what is the final
> message issued by Pine before it is "unable".  The bad UID messages are
> generally just warnings; it's the last message that says what really
> happened.

I didn't have any corrupt mailboxes on hand, so I restored a bad one
from tape backup and got pretty much the same results. (This, however,
raises another question for me. Will mbx mailboxes that are backed up
to tape maintain their intregrety even if they are being modified as
they are being backed up?)

Running pine gave a message like this...

   Unable to find CRLF at some number in ... text: From

...then said that no mailbox was open. The message passed by before I
could copy it to the clipboard. This is a reconstruction from memory.

After that, re-running pine starts off with messages like this:

     Invalid UID 00000008 in message 8, rebuilding UIDs]

After a whole bunch of these, the final message is:

          [No folder is currently open]
--

Marc Wallman
North Dakota State University

 
 
 

mbx mailbox corruption and quotas?

Post by Mark Crispi » Sun, 09 Feb 2003 06:35:03



>       Reason Given: EXAMINE failed: Unable to find CRLF at 20022401 in 64
>       bytes, text: From root Fri Feb 7 00:02:09 CST 2003

OK, this is what happened.

Some program, probably *NOT* anything based upon the c-client library,
appended a traditional UNIX format message to an mbx-format INBOX.  This
program ran as root.

I suggest that you read the UW IMAP Toolkit FAQ; specifically, that you
split the corrupted mail file into two pieces.  The first piece should be
20022401 bytes long, and can be returned immediately to the user as
recovered (and uncorrupted) mailbox data.

The second piece, starting at byte 20022401 of the file, is the corrupted
data, and needs manual examination.  The very first thing to do is to
remove that traditional UNIX format message from it.  If there are any
mbx-format messages after that, you might want to try appending it to a
copy of the first piece -- making sure that you do not do anything to
remove the CRs from the CRLF newlines!! -- and see if the result is valid.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

 
 
 

mbx mailbox corruption and quotas?

Post by Marc » Sat, 15 Feb 2003 04:43:09




> >       Reason Given: EXAMINE failed: Unable to find CRLF at 20022401 in 64
> >       bytes, text: From root Fri Feb 7 00:02:09 CST 2003

> OK, this is what happened.

> Some program, probably *NOT* anything based upon the c-client library,
> appended a traditional UNIX format message to an mbx-format INBOX.  This
> program ran as root.

Thank's for the help on this. You identified exactly what happened.
Each night we have (had) a script that notifies users who are over
quota that they are using too much storage. What I didn't know is that
this script directly modifies the user's mailbox and it assumes that
it's in mbox format. Thanks again!!!
--

Marc Wallman
North Dakota State University

 
 
 

mbx mailbox corruption and quotas?

Post by Mark Crispi » Sat, 15 Feb 2003 05:31:52



> Thank's for the help on this. You identified exactly what happened.
> Each night we have (had) a script that notifies users who are over
> quota that they are using too much storage. What I didn't know is that
> this script directly modifies the user's mailbox and it assumes that
> it's in mbox format. Thanks again!!!

You're welcome.  If you can persuade that script to pipe its email to a
program, you can have it pipe to tmail to deliver it and all should be
well.

How did that script know to write into ~/INBOX?  Did you change things so
that mail is delivered to the /var/spool/mail directory?

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

 
 
 

mbx mailbox corruption and quotas?

Post by Marc » Sat, 15 Feb 2003 23:48:38




> > Thank's for the help on this. You identified exactly what happened.
> > Each night we have (had) a script that notifies users who are over
> > quota that they are using too much storage. What I didn't know is that
> > this script directly modifies the user's mailbox and it assumes that
> > it's in mbox format. Thanks again!!!

> You're welcome.  If you can persuade that script to pipe its email to a
> program, you can have it pipe to tmail to deliver it and all should be
> well.

I'm now constructing a warning in ~/INBOX.quotawarn and using mailutil
to append it to ~/INBOX (i.e.  su username -c "mailutil appenddelete
INBOX.quotawarn INBOX"). I will investigate doing this with tmail
instead.

Quote:> How did that script know to write into ~/INBOX?  Did you change things so
> that mail is delivered to the /var/spool/mail directory?

All mail is being delivered to ~/INBOX. We do this so that user's
files do not end up getting spread across filesystems. It is easier to
manage quotas that way.

The 'over quota notification' was a shell script that built a quota
notification message with a subroutine, Emit_Mail, and ran it like so:

       Emit_Mail >> ~username/INBOX

Since it was being run as root, it had the advantage of being able to
append to the mailbox even for users over their hard quota.
--

Marc Wallman
North Dakota State University

 
 
 

mbx mailbox corruption and quotas?

Post by Mark Crispi » Sun, 16 Feb 2003 03:28:26



> The 'over quota notification' was a shell script that built a quota
> notification message with a subroutine, Emit_Mail, and ran it like so:

>        Emit_Mail >> ~username/INBOX

tmail respects quotas, so you wouldn't be able to append to users over
hard quota (this will also be a problem with using mailutil), but
something like
        Emit_Mail | tmail username
should work.  You probably want to fix Emit_Mail not to write a "From "
line since that is not used in mbx format.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.

 
 
 

1. Mailbox corruption

Trying to track down an odd bug....

I am using Pine (4.33) to connect to Exchange (5.5.2653.13) through
IMAP.  I generally save my outgoing messages to a folder, and also
often take advantage of the pOstpone feature.

I've noticed, however, that outgoing messages (either saved to a
folder when sent, or saved to the postponed-msgs folder) are often
getting corrupted.  The nature of the corruption is that the end
of the message will appear before the beginning (including headers).

For example, this happened today:
-------------------------------- cut here ---------------------------
#=-
-=#| 1412C DCL, Workstation Services Group, CCSO Ofc:(217)244-3862 |#=-

Date: Tue, 18 Dec 2001 14:00:37 -0600 (CST)




Subject: Re: SGI surplus

Message-ID:

MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII

It is *very* common.  I currently have four dead Indy power supplies on
my shelf.  I'm hoping to figure out how to fix them some day....

Thanks,

Damian Menscher
--

-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |
-------------------------------- cut here ---------------------------
(space removed from after the sigdashes above to prevent issues
with people's newsreaders truncating the rest of my message)

I think it's somewhat suspicious that the first 1000 characters of
the message (including headers) are stored together (though moved
to the end.  I'm told that Exchange uses a database to store mail
files... perhaps this is related?

And yes, the messages are corrupt on the server.  Viewing them
using LookOut or the web interface shows the same corruption.

I'd be very interested to hear if anyone has experienced anything
similar.  Right now, I don't know whether the problem is with Pine
or with Exchange.  But it's something we've been seeing off-and-on
for about a year, so it would be really nice to track down!

Damian Menscher
--

-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 1412C DCL, Workstation Services Group, CCSO Ofc:(217)244-3862 |#=-

2. |||||FREE ,Your own webserver, using your cable-modem||||||

3. Eudora: continuous mailbox corruption

4. Not selecting cells in a DataTable

5. Mailbox Corruption

6. Can't update Subordinate Reference

7. Mailbox corruption

8. Problems With Constant Sign-Up Screen

9. Sendmail 8.6.8 mailbox corruption

10. inbox.mbx and deleted.mbx keep growing

11. MSN V2 (UK) ands Internet Mail

12. If I convert mailboxes -> mbx, is there any going back?