Kernel panic due to NF_IP_LOCAL_OUT handler calling itself again

Kernel panic due to NF_IP_LOCAL_OUT handler calling itself again

Post by Morfea » Wed, 02 Mar 2005 22:50:33



Hi,
There is an update.

The protocol I am writing is a source routing based protocol called
Meghadoot similar to the wireless networks DSR, as written by Alex Song
(http://piconet.sf.net). I am using netfilter, I am using linux 2.6.5
(supplied along with FC2), without the SMP and PREEMPTIBLE options.
I have a seperate header called MeghHeader, which carries the source
route and it comes b/w the iph and the tcph. I am using an isolated
machine called A from which I send out ping packets. Here is what is
happening:
In my local_out_handler, I cruft the source routing header, route the
packet using ip_route_output (which causes an arp request to be send).
And then call the output routine passed to me by netfilter (okfn).
As the arp request obviously will fail, and hence
skb->dst->neighbour->output points to neigh_resolve_output, which sends
the arp request, and queues the skb in the neigh->arp_queue, the
neigh->state is NUD_INCOMPLETE.
When neigh_timer_handle fires, it walks thru the arp_queue and sends an
icmp dest unreach for each of the queued skb's thru the
ipv4_link_failure function.
Please note that my local_out_handler is serialized by having a
spinlock at its entry.
Hence my debugging output looks something like this:
        Entering local out with icmp packet type 8, code 0 : The normal ping
packet, in pings context.
        ... Some more of the above
        Entering local out with icmp packet type 3, code 1 : The icmp dest
unreach, given off in the TIMER_SOFTIRQ context, when
neigh_timer_handler is fired.
        ...And then some more of the two...

        Now this is something, that has totally freaked me out ....

        I sometimes get an icmp skb in my local out handler with some invalid
type say like 234 etc, As a hack in my local_out_handler, I return
NF_DROP ('cos I noticed that icmp_send, assumes that the type is
correct before indexing into the icmp table). As soon as my
local_out_handler exits with this screwed up skb,(Again, my
local_out_handler is not reentrant, it is locked by a spinlock) I get
either a bad eip or a null eip somewhere in the neigh_timer_handler.
The stack dump looks like this. (The full debugging output, including
the oops message is available online at
http://www.cs.iitm.ernet.in/~dvagr/helpme):
I am trying to ping the machine 10.6.6.2, whose route is hardcoded into
10.6.6.5.
<------------ My debugging output begins ---------------->
Meghadoot: ****************** Entering local out *******************
function addr 224c483, dst output 224a842<2>
Meghadoot: okfn points to :Address: 224c483, name: dst_output, offset:
0,size: 1c, module: Kernel<2>,
Meghadoot: skb->dst->output points to : Address: 224a842, name:
ip_output, offset: 0,size: 5c, module: Kernel<2>
Meghadoot: local out handler, dumping call trace
 [<22926360>] local_out_handler+0x4e/0x296 [megh_linux]
 [<0224c483>] dst_output+0x0/0x1c
 [<0224a842>] ip_output+0x0/0x5c
 [<0223c787>] nf_iterate+0x40/0x89
 [<0224c483>] dst_output+0x0/0x1c
 [<0223ca91>] nf_hook_slow+0x90/0x101
 [<0224c483>] dst_output+0x0/0x1c
 [<0224c0dc>] ip_push_pending_frames+0x2cf/0x37c
 [<0224c483>] dst_output+0x0/0x1c
 [<0226686f>] icmp_send+0x35e/0x397
 [<02107352>] do_IRQ+0x134/0x169
 [<02239dc9>] neigh_timer_handler+0x0/0x112
 [<0211de03>] run_timer_softirq+0x10b/0x12a
 [<0211af6d>] __do_softirq+0x35/0x73
 [<021078f5>] do_softirq+0x46/0x4d
 =======================
 [<0210737b>] do_IRQ+0x15d/0x169
 [<0210403b>] default_idle+0x23/0x26
 [<0210408c>] cpu_idle+0x1f/0x34
 [<02318612>] start_kernel+0x174/0x176

Meghadoot: In context: in_softirq:1024,in_irq:0,in_interrupt:1024<2>
Meghadoot: local out called in context of swapper<2>
Meghadoot: finished dumping stack ....<2>
Meghadoot: Entered local Out handler,printing skb info<2>
Meghadoot: Local out handler , icmp type is 0, code is 0<2>s:127.0.0.1
d:127.0.0.1 and protocol is 1<2> (Valid icmph->type, I return NF_DROP
if icmph->type > NR_ICMP_TYPES )
Meghadoot: Got Locally destined packet (ie iph->daddr = MYIPADDR )<2>,
returned NF_ACCEPT.
Meghadoot: ****************** Exiting local out
*******************<1>Unable to handle kernel NULL pointer dereference
at virtual address 00000000
 printing eip:
00000000
*pde = 00000000
Oops: 0000 [#1]
CPU:    0
EIP:    0060:[<00000000>]    Not tainted
EFLAGS: 00010206   (2.6.5-1.358)
EIP is at 0x0
eax: 00000000   ebx: 00000000   ecx: 0000b404   edx: 00000007
esi: 00000000   edi: 00000000   ebp: 00000000   esp: 02344fb0
ds: 007b   es: 007b   ss: 0068
Process swapper (pid: 0, threadinfo=02344000 task=022c5aa0)
Stack: 00020000 00000000 00594400 1c8e4900 00000056 02239dc9 1c8e495c
0211de03
       02344fd0 02344fd0 0234007b 00000001 02369428 0000000a 02316000
0211af6d
       02317f94 00000046 00000000 021078f5
Call Trace:
 [<02239dc9>] neigh_timer_handler+0x0/0x112
 [<0211de03>] run_timer_softirq+0x10b/0x12a
 [<0211af6d>] __do_softirq+0x35/0x73
 [<021078f5>] do_softirq+0x46/0x4d
 =======================
 [<0210737b>] do_IRQ+0x15d/0x169
 [<0210403b>] default_idle+0x23/0x26
 [<0210408c>] cpu_idle+0x1f/0x34
 [<02318612>] start_kernel+0x174/0x176

Code:  Bad EIP value.
 <0>Kernel panic: Fatal exception in interrupt
In interrupt handler - not syncing

<---------------------- End of debugging dump --------------------->

I will be really greateful, if any one of you could help me out with
this. Please.

Expecting your reply eagerly.
Regards,
Vinay

 
 
 

Kernel panic due to NF_IP_LOCAL_OUT handler calling itself again

Post by jimx » Fri, 04 Mar 2005 17:16:17


When you receive the icmp dest unreach message, do you drop it or inform the
kernel?
and can you reset the atomic_t variable?
try to stop local_out_handler when receive icmp dest unreach message by set
a flag.

--
jimxu
*NO PAINS, NO GAINS*

Quote:> Hi,
> I am writing an implementation of a source routing protocol as a
> loadable module. I am using netfilter, I am not using the IP SSR
> option, I am using kernel 2.6.5, without smp and preemption support. My
> design is based on the DSR protocol. I have a header after the IP
> header, describing the source route and the route error. If 's' is the
> src, 'd' is the dst, and x1,x2 ... are the hops, the source route is :
> x1-x2...d. However unlike SSR I don't change the dst field of the ip
> header, which is set to d. Also every src routed packet carries with it
> an ack request for the next hop, to which the next hop is supposed to
> reply to.
> The jist of my code is as follows:

> pre_route_handler:
> For all source routed packets:
> ackReply to previous hop
> nxtHop = getNextHop(packet);
> if(ip_route_input(myaddr,nxtHop)!=0){
> drop packet;
> }
> awaitAck(nxtHop);
> return NF_ACCEPT.

> In local_out_handler:
> create a packet with the source route. route it to the first hop,
> using ip_route_output_key(flowi,skb). I then do an awaitAck(firstHop).

> I intend to use this protocol in wireless networks. And hence I have
> implemented an ack based design, where every node is responsible for
> ensuring that the packet makes it to the next hop. For this the
> function awaitAck does the following:

> awaitAck(skb):
> add an ackRequest header.
> add a timer(a pointer to it) in the skb->cb.
> put this skb to a queue.

> When this timer fires:
> if rtxCount for the skb less than a MAX_RXMT, retransmit, else send a
> route error on the reverse route to the source.

> This scheme is working fine when there are no errors. I am testing it
> across an ethernet lan by pinging. However when there are link
> failures, weird things happen.
> The topology I am using is A - - - B, where B is some fictional node
> not connected to A, but in the same subnet (the routing table so
> configured to put such packets to eth0). In the local out handler, I do
> a ip_route_output to B, which succeeds. I call the output function
> okfn, given to me by netfilter directly and return NF_STOLEN. The next
> packet to come to me in local_out is an icmp dest unreach. It is
> destined to me, so I accept it. The next packet is again an icmp dest
> unreach, and after that somehow my local_out_handler is called again,
> while the first call of it hasn't finished. (My kernel is not smp and
> not preemptible). At times this happens over and over. My kernel then
> panics, either due to a stack overflow, or some bad eip value, or
> something else (with eip value not decoded, and nowhere in the
> /proc/kallsyms). I detect this double calling of my local_out_handler
> by using an atomic_t variable. The same effect is seen if I return an
> NF_DROP or NF_ACCEPT on the skb in the reentrant call. I have also used
> spinlocks but the kernel always crashes.

> After some poking around the kernel code, I found that if an arp entry
> to a node is not present, an entry is created in the arptable with
> status set to pending. All packets waiting for this arp request to be
> resolved are queued up. When the arp request ultimately fails(times
> out), the packets in this queue are freed and an icmp destination
> unreachable error is sent back to the source of these packets.

> But I are still unable to figure out as to why EIP is getting
> corrupted(or the reason for stack overflow).

> I am quite helpless, I have posted on several mailing lists(linux-net,
> kernelnewbies, linux-kernel), but haven't received any replies, Also I
> couldn't find any decent material on the web related to my problem.
> I would be obliged if someone could please help me with this. What am I
> doing wrong, where can I find information on this?
> My code is completely based on Alex Song's DSR implementation for linux
> 2.4, available online at http://piconet.sf.net.

> Regards,
> Vinay Reddy


 
 
 

1. kernel panic "killing interrupt handler" and kernel BUG at sched.c:468

Hi everyone,

And then after approximately 1.5 hours on 2.4.19-xfs (XFS CVS snapshot
20020930 patched only with Randy Dunlap's sys-magic 20020314, built
using gcc 2.95.4 running Debian GNU/Linux).

Now in the process running syslogd. Attached kernel-panic-3.out is the
oops.

I did this, as well, and hit a kernel BUG at sched.c:566. My sched.c is
now exactly the same as the one in the XFS tree (which I think is the
same as the one in vanilla 2.4.19). Oops is attached as
kernel-bug-2.out.

System has been up for two hours so far and is thankfully alive. I hope
someone can point me to what the probably cause of this problem is. I
did an xfs_check -- which is from the latest 2.3.1 package -- and all my
XFS partitions are okay. I will be looking for a memory scanning tool
shortly to make sure I don't suddenly have bad RAM, although the problem
doesn't seem to be that.

Thanks again to everyone for your time.

 --> Jijo

--
Federico Sevilla III   :  http://jijo.free.net.ph
Network Administrator  :  The Leather Collection, Inc.
GnuPG Key ID           :  0x93B746BE

  kernel-panic-3.out
2K Download

  kernel-bug-2.out
3K Download

2. JAZZ drive or SyJet drives... Do they work with Linux?

3. Threads performance - allow signal handler to not call handler

4. T2000 running Clearcase

5. kernel panic due to swap problem or broken SCSI driver or ???

6. SuSE 7.3 installation hosed, reinstall failed; please help

7. aiee, killed interrupt handler - kernel panic

8. How do I change ip address on NIS+ server

9. kernel panic, killing interrupt handler

10. Kernel panic: Aiee, killing interrupt handler!

11. Kernel panic:Aiee killing interrupt handler

12. 2.4.5 Kernel panic: Aiee, killing interrupt handler!

13. RH7.1 Install -->Kernel panic: Aiee, killing interrupt handler!