Very Computer

After a patch bundle update on my SPARC Solaris 5.8 (which didn't go
smoothly), /var/logs was empty and no matter what I try, syslogd
doesn't log messages to /var/adm/messages anymore.  Short of taking the
server down for the day and reapplying all the patches, I was hoping
someone might have a suggestion or spot something I missed??

I have tried with no success:
- copying over syslogd from a nearly identical server to make sure I
wasn't trojaned, no luck
- reapplying the most recent syslogd related patch 110945-08, no
difference
-  /etc/init.d/syslog stop
/etc/init.d/syslog start
which does start and stop syslogd, but doesn't make it log
- changing /etc/syslog.conf and making sure all the spaces are tabs
- /usr/ucb/logger -p mail.emerg "Test message" but it doesn't come
through
- I ran syslogd in debug mode with /usr/sbin/syslogd -d -t and
successive logger attempts show nothing in the debug (why nothing at
all?)
- I can see that syslogd does pick up changes to /etc/syslog.conf in
the debug, but there's no indication of any errors
- I've tried truss'ing it and comparing the output to the near
identical server but I can't see any errors, the logger messages just
don't seem to have any effect
- I've tried a restart but there's no difference...
- A lot of google posts/articles I read concluded with the author
saying "It just fixed itself" - I've tried this approach too but it's
not working!! :)

I can post any config files, truss output (or the blank log file :) )
if needed.

I'm guessing a library syslogd uses might be corrupt?

If anyone has any other ideas on what I might have missed or ways of
finding out exactly what the problem is I'd really appreciate it -
Thanks,
    John

http://www.sunfreeware.com/programlistsparc8.html#lsof

It will tell you the files that your syslogd has open, which might be
useful if you're trying to find a missing or corrupted library file.

Here's the output from my Solaris 2.8 system (Generic_108528-29) with a
working syslogd:

# ps -ef | grep syslogd
root  1279     1  0   Sep 09 ?        9:52 /usr/sbin/syslogd
root 27895 26197  0 11:27:15 pts/7    0:00 grep syslogd
# lsof -p 1279
COMMAND  PID USER   FD   TYPE        DEVICE SIZE/OFF   NODE NAME
syslogd 1279 root  cwd   VDIR          32,1     1536      2 /
syslogd 1279 root  txt   VREG          32,3    79152 209842 /usr
(/dev/dsk/c0t0d0s3)
syslogd 1279 root  txt   VREG          32,3    13184  42160 /usr
(/dev/dsk/c0t0d0s3)
syslogd 1279 root  txt   VREG          32,3    44836  42697
/usr/lib/nss_files.so.1
syslogd 1279 root  txt   VREG          32,3    25608 184710
/usr/lib/locale/en_US.ISO8859-15/en_US.ISO8859-15.so.2
syslogd 1279 root  txt   VREG          32,3   183496  42690
/usr/lib/libthread.so.1
syslogd 1279 root  txt   VREG          32,3    17096  92252
/usr/platform/sun4u/lib/libc_psr.so.1
syslogd 1279 root  txt   VREG          32,3    24968  42116
/usr/lib/libmp.so.2
syslogd 1279 root  txt   VREG          32,3  1158072  42536
/usr/lib/libc.so.1
syslogd 1279 root  txt   VREG          32,3    12708  42546
/usr/lib/libdoor.so.1
syslogd 1279 root  txt   VREG          32,3   911328  42640
/usr/lib/libnsl.so.1
syslogd 1279 root  txt   VREG          32,3     4872  42083
/usr/lib/libdl.so.1
syslogd 1279 root  txt   VREG          32,3    38904  42664
/usr/lib/libpthread.so.1
syslogd 1279 root  txt   VREG          32,3    24628  42086
/usr/lib/libcmd.so.1
syslogd 1279 root  txt   VREG          32,3   255488  42107
/usr/lib/ld.so.1
syslogd 1279 root    0r  VDIR          32,1     1536      2 /
syslogd 1279 root    1r  VDIR          32,1     1536      2 /
syslogd 1279 root    2r  VDIR          32,1     1536      2 /
syslogd 1279 root    3u  IPv6 0x3000255e6c0      0t0    UDP *:syslog
(Idle)
syslogd 1279 root    4u  IPv4 0x3000255ed00      0t0    UDP *:syslog
(Idle)
syslogd 1279 root    5r  DOOR         308,0      0t0  10993
/etc/.name_service_door (door to nscd[1299])
syslogd 1279 root    6w  VCHR          97,0    0t217 218027

syslogd 1279 root    7w  VREG          32,5       29  16347
/var/adm/messages
syslogd 1279 root    8w  VREG          32,5 12587893  16582 /var
(/dev/dsk/c0t0d0s5)
syslogd 1279 root    9r  VCHR          21,6      0t0 218016

syslogd 1279 root   10u  DOOR          0,27      0t0     27 (this PID's
door)
syslogd 1279 root   11w  VREG          32,5    29389  73287 /var
(/dev/dsk/c0t0d0s5)
#

A couple of quick things you can check:
-  Look at your /var filesystem and make sure it is available and
writable.
- If you have a remore log server available, try sending syslog from
this server to the remote log server.
BTG
--
Boston Technology Group
http://www.bostontechgroup.com

You can also run the syslog program in interactive, debug mode. Type man
syslogd for more information. I believe the -d is for debug option.

Anonymous

Thanks for the reply, I tried out lsof and checked it against the other
server I maintain that has a working syslogd and your output and all
the file sizes of the libs seem to be the same.

One thing I forgot to mention in my post is that on a restart, a "going
down" message is logged to /var/adm/messages like this:

cat /var/adm/messages
Jan  5 21:00:56 tserver syslogd: going down on signal 15

So I know it can write to /var/adm/messages, but it never logs anything
from logger, sshd etc.

Any other advice/tools I can try out would be very helpful,  thanks
again,
   John

Here's my syslog.conf (it's the same as the near identical server I
also maintain):

auth.err        /dev/console
*.emerg         *

local5.info             /var/log/htsearch.log

I've made sure it's all tabbed and no spaces.

I forgot to mention in my original post that on a restart of syslogd,
"going down" messages are logged like this:
Jan  5 21:00:56 tserver syslogd: going down on signal 15
dmesg just reports these messages, no others like ones I can
arbitrarily send from logger

/var has loads of space and is writable:
/dev/dsk/c0t0d0s7       15200475     7328339     7720132    49%    /var
drwxr-xr-x  55 root     sys         1024 Jul 22 16:09 /var

How do you set up the remote log server to take another servers log
messages?  That'd be a good test to narrow down where the problem might
be.

Thanks a million,
    John

I tried the logger program, sending different messages of different
priorities etc., but still nothing in my /var/adm/messages

When I ran syslogd in debug mode and used logger to send messages to
it, but nothing happens.  Nothing is reported as an error in the
previous debug and the last line just says that it's running and
waiting:

main(1): Started at time Wed Jan  5 21:37:43 2005
hnc_init(1): hostname cache configured 128 entry ttl:600
getnets(1): network is being turned off
amiloghost(1): testing 136.201.1.124.2.2
conf_init(1): I am loghost
cfline(1): (auth.err    /dev/console)
cfline(1): (*.emerg             *)
cfline(1): (*.info;local5.none          /var/adm/messages)
cfline(1): (local5.info         /var/log/htsearch.log)

syslogd: version 1.92
Started: Wed Jan  5 21:37:43 2005
Input message count: system 0, network 0
# Outputs: 4

------------------------ priority = [file, facility]
------------------------

0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4
--------------------------------------------------
X X X X 3 X X X X X X X X X X X X X X X X X X X X CONSOLE: /dev/console
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 X WALL:
6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 X 6 6 X FILE:
/var/adm/messages
X X X X X X X X X X X X X X X X X X X X X 6 X X X FILE:
/var/log/htsearch.log

Facilities:
[00] kern:   0
[01] user:   8
[02] mail:  16
[03] daemon:  24
[04] auth:  32
[05] security:  32
[06] mark: 192
[07] syslog:  40
[08] lpr:  48

[10] uucp:  64
[11] cron: 120
[12] local0: 128
[13] local1: 136
[14] local2: 144
[15] local3: 152
[16] local4: 160
[17] local5: 168
[18] local6: 176
[19] local7: 184

Priorities:
[00] panic:   0
[01] emerg:   0
[02] alert:   1
[03] crit:   2
[04] err:   3
[05] error:   3
[06] warn:   4
[07] warning:   4
[08] notice:   5
[09] info:   6
[10] debug:   7
[11] none:  16

Per File Statistics
File                            Tot     Dups    Nofwd   Errs
----                            ---     ----    -----   ----
/dev/console                    0       0       0       0
WALL                            0       0       0       0
/var/adm/messages               0       0       0       0
/var/log/htsearch.log           0       0       0       0

logmsg(8): msg dispatcher started
sys_poll(9): sys_thread started
init(1): accepting messages from local system
init(1): syslogd: started
main(1): off & running....

Occasionally it'll get an alarm signal, but this is normal.

Can you perhaps advise me if this debug shows anything wrong with my
setup?  Or any ideas as to why in debug, nothing happens when I log a
message?

Thanks for your help,
    John

If nothing's listening on udp/514 when syslogd is down, then start
syslogd and you should see syslogd connected to it using the same
command above.

# lsof -i udp:514
COMMAND  PID USER   FD   TYPE        DEVICE SIZE/OFF NODE NAME
syslogd 1279 root    3u  IPv6 0x3000255e6c0      0t0  UDP *:syslog
(Idle)
syslogd 1279 root    4u  IPv4 0x3000255ed00      0t0  UDP *:syslog
(Idle)
#

Also, you might double-check the permissions on your /var /var/log and
/var/adm subdirs, and the syslog and messages files.

# ls -ld /var /var/adm /var/log
drwxr-xr-x  37 root     sys         1024 Sep 10  2003 /var/
drwxrwxr-x  11 root     sys          512 Jan  2 03:10 /var/adm/
drwxr-xr-x   2 root     sys          512 Jan  2 03:10 /var/log/
# ls -l /var/adm/messages /var/log/syslog
-rw-r--r--   1 root     root         113 Jan  4 14:47 /var/adm/messages
-rw-r--r--   1 root     sys        52201 Jan  6 09:46 /var/log/syslog
#

---cut---

*.notice                      /var/log/notice
mail.info                     /var/log/notice
*.crit                        /var/log/critical
kern,mark.debug               /dev/console

*.emerg                       *
*.alert                       root,operator
*.alert;auth.warning          /var/log/auth

---cut---

Thanks for the suggestions, apologies for the late reply but I was sick
with flu.

I've tried checking the UDP port and it's there alright, but I only
have one (presumably you have more than one network card?).
# lsof -i udp:514
COMMAND  PID USER   FD   TYPE        DEVICE SIZE/OFF NODE NAME
syslogd 8267 root    3u  IPv4 0x30004142e58      0t0  UDP *:syslog
(Idle)

And my file permissions are the same as yours:
# ls -ld /var /var/adm /var/log
drwxr-xr-x  55 root     sys         1024 Jul 22 16:09 /var
drwxrwxr-x   9 root     sys          512 Jan  5 23:58 /var/adm
drwxr-xr-x   5 root     sys          512 Dec 19 20:26 /var/log
# ls -l /var/adm/messages /var/log/syslog
-rw-r--r--   1 root     root         442 Jan 17 20:44 /var/adm/messages
-rw-r--r--   1 root     sys            0 Dec 16 20:49 /var/log/syslog

This problem totally stumps me, it seems to have nothing stopping it
from logging properly

Any other suggestions are gladly welcome!

Thanks,
John

I've tried that too and I still can't log messages to it.  I created
just the /var/log/notice file and I issued a

# logger -p mail.info "Just a test"

so it would log this to /var/log/notice but all I get is:
Jan 17 20:57:54 tau syslogd: /var/log/critical: No such file or
directory
Jan 17 20:57:54 tau syslogd: /var/log/auth: No such file or directory

I have no idea how syslog works, but perhaps it is working fine but
whatever (if there is one) process that sends messages to syslog or
flags/queues messages for logging is not working.  So in this case,
let's say that logger isn't working, it would appear that syslogd isn't
logging, but it may not be getting any messages to log.  So what does
logger and say sshd use to pass messages to syslogd?  Perhaps I can
test/replace that?

Thanks,
    John

Quote:> so it would log this to /var/log/notice but all I get is:
> Jan 17 20:57:54 tau syslogd: /var/log/critical: No such file or
> directory
> Jan 17 20:57:54 tau syslogd: /var/log/auth: No such file or directory

If logging still doesn't work, make sure you can at least do the
following:

# echo hello > /var/log/auth
# echo hello > /var/log/critical
# echo hello > /var/log/notice
# echo hello > /var/adm/messages

If that worked, then look at syslogd:

# sum /usr/sbin/syslogd
29510 155 /usr/sbin/syslogd
# what /usr/sbin/syslogd
/usr/sbin/syslogd:
SunOS 5.8 Generic 110945-08 Apr 2003
#

Let me know how that goes.

Did you edit the /etc/syslog.conf. You need to be very carefull to edit this
config file.

Please read how to edit syslog.conf

cheers

Quote:> > so it would log this to /var/log/notice but all I get is:
> > Jan 17 20:57:54 tau syslogd: /var/log/critical: No such file or
> > directory
> > Jan 17 20:57:54 tau syslogd: /var/log/auth: No such file or directory

> Make sure /var/log/critical and /var/log/auth exist too and are
> writable by root.  Then bounce syslogd again.

> If logging still doesn't work, make sure you can at least do the
> following:

> # echo hello > /var/log/auth
> # echo hello > /var/log/critical
> # echo hello > /var/log/notice
> # echo hello > /var/adm/messages

> If that worked, then look at syslogd:

> # sum /usr/sbin/syslogd
> 29510 155 /usr/sbin/syslogd
> # what /usr/sbin/syslogd
> /usr/sbin/syslogd:
> SunOS 5.8 Generic 110945-08 Apr 2003
> #

> Let me know how that goes.

Recently I was involved in an incident whereby syslog would not log
properly even on localhost (but the problem was noticed because certain
hosts weren't sending their syslog output to the central loghost).  The
problem ended up being a difference in the way that syslog was
configured by default.  More specifically, Solaris 9 changed the way
that you disable TCP/IP connections to syslog. In Solaris 8 a -t flag
was added to syslogd when it was started. Solaris 9 now uses a config
file to determine where the connections are allowed
(/etc/default/syslogd must have LOG_FROM_REMOTE set to YES (which is
the default in Sol9) ).

Anyway, I realize you're running Sol8, but I'm thinking maybe your
patches might have changed things to work like Sol9?  It's worth
looking into at any rate.

Buenos suerte,

Patrick

I created those files, stopped, started syslogd, sent a logger message
but still nothing!

The files all exist and are writable, it's just that the messages are
not getting through to syslogd

# echo hello > /var/log/auth
# echo hello > /var/log/critical
# echo hello > /var/log/notice
# echo hello > /var/adm/messages
# cat /var/log/auth /var/log/critical /var/log/notice /var/adm/messages
hello
hello
hello
hello

My syslogd seems to be the same as yours:
# sum /usr/sbin/syslogd
29510 155 /usr/sbin/syslogd
# /usr/ccs/bin/what /usr/sbin/syslogd
/usr/sbin/syslogd:
SunOS 5.8 Generic 110945-08 Apr 2003

Is it possible that syslogd is fine, but whatever gives the messages to
syslogd isn't working?  Any idea what it might be and how I check it
out?

Thanks for all your help so far,
John