Very Computer

I am seeing an odd problem on a customer's machine and am stumped as
to what is going on.

The software I support uses rename() to rotate a data log file for
processing. When this particular problem crops up the rotation fails
with error '2', which is defined:
--------------------
  #define ENOENT  2       /* No such file or directory            */
--------------------

This is the log file rotation code:
--------------------------
 if (rename(mFileName.data(), toName.data())) {
        ourErrLog::Log(ourErrLog::anError, otherLogRotateFailed,
errno);
        ++mRotateErrors;
        return otherLogRotateFailed;

ourMessages.msg:otherLogRotateFailed  Log rotation failed with error
%d
--------------------------

The man page for "rename()" says this for ENOENT:

     ENOENT
           The link named by old does not exist,  or  either  old
           or new points to an empty string.

Since both the 'old' and the 'new' file names show up in the error
message log, that seems to indicate the old and new arguments are not
empty, which begs the question: why does rename think the 'old' name
does not exist?

The customer is running (uname -a output):
--------------------------
SunOS <hostname> 5.8 Generic_108528-15 sun4u sparc SUNW,UltraAX-MP
--------------------------

Also, when this occurs I have been able to take the value of 'old'
filename and 'vi' it, saving the data off to another file name.
However, if I 'ls' the directory where 'old' resides just before I 'vi
old', 'old' does not show up in the list. So 'vi' sees 'old' but 'ls'
does not.

This customer is the only customer of many running this product and
code on Solaris who sees this error. The customer has a similar
machine running this product also (same software version, same OS,
down to the patch level) and that machine does not see this problem.

Is anyone aware of any bugs in Solaris that have to do with this?
Also, if you have any ideas about what may be going on, please say so.

thanks,

Phil

Quote:> Also, when this occurs I have been able to take the value of 'old'
> filename and 'vi' it, saving the data off to another file name.
> However, if I 'ls' the directory where 'old' resides just before I 'vi
> old', 'old' does not show up in the list. So 'vi' sees 'old' but 'ls'
> does not.

> This customer is the only customer of many running this product and
> code on Solaris who sees this error. The customer has a similar
> machine running this product also (same software version, same OS,
> down to the patch level) and that machine does not see this problem.

> Is anyone aware of any bugs in Solaris that have to do with this?
> Also, if you have any ideas about what may be going on, please say so.

Any chance the box has been compromised, and a dodgy 'ls' placed on there?

Does 'echo *' show the old filename ?

And what about find . -print ?

Quote:> I am seeing an odd problem on a customer's machine and am stumped as
> to what is going on.

> The software I support uses rename() to rotate a data log file for
> processing. When this particular problem crops up the rotation fails
> with error '2', which is defined:
> --------------------
>   #define ENOENT  2       /* No such file or directory            */
> --------------------

> This is the log file rotation code:
> --------------------------
>  if (rename(mFileName.data(), toName.data())) {
>         ourErrLog::Log(ourErrLog::anError, otherLogRotateFailed,
> errno);
>         ++mRotateErrors;
>         return otherLogRotateFailed;

Windoze background, eh?  (Hungarian notation, bleurgh!)

Quote:> Since both the 'old' and the 'new' file names show up in the error
> message log, that seems to indicate the old and new arguments are not
> empty, which begs the question: why does rename think the 'old' name
> does not exist?

Does the file contain 01 in its name?  If so, congratualtions.
Your customer has been cracked.  They need to reinstall their
machine(s) from scratch, apply all the latest patches, and lock
them down - before putting them back on the network.

--
Rich Teer

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

Quote:>Does the file contain 01 in its name?  If so, congratualtions.
>Your customer has been cracked.  They need to reinstall their
>machine(s) from scratch, apply all the latest patches, and lock
>them down - before putting them back on the network.

This is what I would have answered, based on just the Subject line.  The
rootkit that hacks "ls" is well known.

But does this rootkit also hack libc so that rename() doesn't see these
files?  If not, why is his application also failing?

--

Genuity Managed Services, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.

Good question.  That did occur to me too, but I thought I'd
err on the side of caution, and assume the worst.

--
Rich Teer

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.rite-online.net

<snip>

Quote:>The customer is running (uname -a output):
>--------------------------
>SunOS <hostname> 5.8 Generic_108528-15 sun4u sparc SUNW,UltraAX-MP
>--------------------------

>Also, when this occurs I have been able to take the value of 'old'
>filename and 'vi' it, saving the data off to another file name.
>However, if I 'ls' the directory where 'old' resides just before I 'vi
>old', 'old' does not show up in the list. So 'vi' sees 'old' but 'ls'
>does not.

>This customer is the only customer of many running this product and
>code on Solaris who sees this error. The customer has a similar
>machine running this product also (same software version, same OS,
>down to the patch level) and that machine does not see this problem.

>Is anyone aware of any bugs in Solaris that have to do with this?
>Also, if you have any ideas about what may be going on, please say so.

One of our admins here saw something very similar.  The file would
list if the full path was given, but not if just the directory
was listed.  The listed file showed it had 0 (zero) links to it.
It was possible to cat the file as well.  I haven't followed up
with him since then (sorry, so I don't have any further info).
The uname output was:
SunOS hostname 5.8 Generic_108528-16 sun4u sparc SUNW,Sun-Fire-280R

Yeah.  Sounds like either 1. NFS problems (if the filesystem is mounted
over NFS), 2. a corrupt filesystem, or 3. an exploit has compromised the
machine.

--

 __ San Jose, CA, USA / 37 20 N 121 53 W / &tSftDotIotE
/  \ That it will never come again is what makes life so sweet.
\__/ Emily*inson
    WebVal / http://www.veryComputer.com/
 URL scanner, maintainer, and validator in Python.

Joe Bloggs sez:
... So 'vi' sees 'old' but 'ls'

Quote:>> does not.
...

> Does 'echo *' show the old filename ?

> And what about find . -print ?

And what about /usr/bin/sparcv9/ls ?

Dima
--
The speed at which a mistyped command executes is directly proportional
to the amount of damage done.                                       -- Joe Zeff

my tuppence worth and it me seem trival, but the file does not begin with a .
does it?  If so `ls -a` will show it, but `ls` won't.

Others mentioned a root kit infection.  You *should* check for that.

Directory permissions can explain this issue.

Keep on execute on the directory.  Turn off read on it.  The result is a
directory where "ls" will fail with permission denied because it takes read
permission to scan the directory for the names in it, but "vi" of a file
with a specific name will succeed because execute of a directory allows you
to open files by specific name.  This is true if you are running as a
non-root user or if you are in an NFS mounted directory that maps root to
nobody.  If you're root on a local system, directory permissions don't
matter and this doesn't apply.  Fix with "chmod 755 ."

But check for a root kit anyways.

I very much appreciate all your replies and wanted to say that the
file name did have '01' in it's name for one of three occurrences of
this problem. The other two it did not.

As for the other tests suggested, since we have to reboot the machine
to clear the problem and the problem occurs sporadically, I can only
save them for the next time it comes up.

Last, I hadn't thought of the "this machine's been compromised"
scenario and will have my customer look into it.

thanks again,

Phil

------------------------------------

Just as a follow up to my previous post in this thread.

The problem we saw here is SUN BugID: 4395913 File found with link count of zero.
It was suppose to be patched in the latest 5.8 kernel patch.  We are
still checking to see (it looked like it was suppose to be fixed in -14 and
we were running -16 at the time of the problem, so we are still checking).

Hope this helps,

Matt S