Very Computer

by **Bigdaki** » Tue, 27 Jan 2004 08:12:14

Hi folks,

I was wondering how NFS failover works in practice. I have two servers for
each shared file system, with the shared files being identical on the servers.

The automount maps also reflect that there are two servers.

So if one server fails, how quickly will the nfs clients respond, and mount
from the alternative server? And should things run smoothly after that, or is
their anything else that needs to be done?

Yours,

Stuart
Dr. Stuart A. Weinstein
Ewa Beach Institute of Tectonics
"To err is human, but to really foul things up
requires a creationist"

by **Shivakanth Mundr** » Tue, 27 Jan 2004 15:17:23

> Hi folks,

> I was wondering how NFS failover works in practice. I have two servers for
> each shared file system, with the shared files being identical on the servers.

> The automount maps also reflect that there are two servers.

> So if one server fails, how quickly will the nfs clients respond, and mount
> from the alternative server? And should things run smoothly after that, or is
> their anything else that needs to be done?

> Yours,

> Stuart
> Dr. Stuart A. Weinstein
> Ewa Beach Institute of Tectonics
> "To err is human, but to really foul things up
> requires a creationist"

Can you try something like this

mount -o ro host1,host2,host3:/usr/share/man /usr/share/man

and stop nfs on host1. you should still be able to access them via
host2/3 if the failover works properly.

I would be curious to know as well.

Shivakanth.

by **Bigdaki** » Tue, 27 Jan 2004 19:17:53

>Subject: Re: NFS failover info..

>Date: 1/25/04 8:17 PM Hawaiian Standard Time

>> Hi folks,

>> I was wondering how NFS failover works in practice. I have two servers
>for
>> each shared file system, with the shared files being identical on the
>servers.

>> The automount maps also reflect that there are two servers.

>> So if one server fails, how quickly will the nfs clients respond, and mount
>> from the alternative server? And should things run smoothly after that, or
>is
>> their anything else that needs to be done?

>> Yours,

>> Stuart
>> Dr. Stuart A. Weinstein
>> Ewa Beach Institute of Tectonics
>> "To err is human, but to really foul things up
>> requires a creationist"

>Can you try something like this

>mount -o ro host1,host2,host3:/usr/share/man /usr/share/man

>and stop nfs on host1. you should still be able to access them via
>host2/3 if the failover works properly.

>I would be curious to know as well.

Hmmm.. I've never tried that syntax before with the mount command.

Might be worth a try.. if its legal.

Stuart
Dr. Stuart A. Weinstein
Ewa Beach Institute of Tectonics
"To err is human, but to really foul things up
requires a creationist"

by **Darren Dunha** » Wed, 28 Jan 2004 02:28:09

> Hi folks,
> I was wondering how NFS failover works in practice. I have two servers for
> each shared file system, with the shared files being identical on the servers.
> The automount maps also reflect that there are two servers.
> So if one server fails, how quickly will the nfs clients respond, and mount
> from the alternative server? And should things run smoothly after that, or is
> their anything else that needs to be done?

This only works with read-only mounts.

And even so, it's not very popular. I've seen a few posts from admins
asking about it and not being able to see the failover occur.

--

Unix System Administrator Taos - The SysAdmin Company
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

by **Bigdaki** » Wed, 28 Jan 2004 05:41:28

>Subject: Re: NFS failover info..

>Date: 1/26/04 7:28 AM Hawaiian Standard Time

>> Hi folks,

>> I was wondering how NFS failover works in practice. I have two servers
>for
>> each shared file system, with the shared files being identical on the
>servers.

>> The automount maps also reflect that there are two servers.

>> So if one server fails, how quickly will the nfs clients respond, and mount
>> from the alternative server? And should things run smoothly after that, or
>is
>> their anything else that needs to be done?

>This only works with read-only mounts.

>And even so, it's not very popular. I've seen a few posts from admins
>asking about it and not being able to see the failover occur.

Yeah thats what I was wondering. I had my suspicions about that working well in
an actual operating environment.

Stuart
Dr. Stuart A. Weinstein
Ewa Beach Institute of Tectonics
"To err is human, but to really foul things up
requires a creationist"

by **Bigdaki** » Wed, 28 Jan 2004 06:33:40

>Subject: Re: NFS failover info..

>Date: 1/26/04 10:41 AM Hawaiian Standard Time

>>Subject: Re: NFS failover info..

>>Date: 1/26/04 7:28 AM Hawaiian Standard Time

>>> Hi folks,

>>> I was wondering how NFS failover works in practice. I have two servers
>>for
>>> each shared file system, with the shared files being identical on the
>>servers.

>>> The automount maps also reflect that there are two servers.

>>> So if one server fails, how quickly will the nfs clients respond, and
>mount
>>> from the alternative server? And should things run smoothly after that, or
>>is
>>> their anything else that needs to be done?

>>This only works with read-only mounts.

>>And even so, it's not very popular. I've seen a few posts from admins
>>asking about it and not being able to see the failover occur.

>Yeah thats what I was wondering. I had my suspicions about that working well
>in
>an actual operating environment.

Ok, I did the following basic experiment which was suggested earlier.

I made an old version of blt I had laying around exportable read only from two
machines. The directory /export/blt2.4 is absolutely identical on both
machines.

I then mounted that partition on a client:

mount -o ro host1,host2:/export/blt2.4 /blt

host1 is running 2.7. host2 running 2.9 and the client running 2.8. Not ideal,
I know, but if it works like that, its at least somewhat robust. Now I'm pretty
share failover capability was introduced in 2.6. If it was introduced after
2.7, then obviously, this is a bad experiment.

After issuing the mount command I cd'ed into /blt on the client. I then
determined it was mounting from host1. So I unshared /export/blt2.4 on host1.
When trying to do an ls on the client, it responded "Stale NFS Handle". Cd'ing
in and out didn't help.

THere was no failover as I understand "failover" to mean.

SO much for that, I guess.

Stuart

Dr. Stuart A. Weinstein
Ewa Beach Institute of Tectonics
"To err is human, but to really foul things up
requires a creationist"

by **Dragan Cvetkovi** » Wed, 28 Jan 2004 06:38:23

> THere was no failover as I understand "failover" to mean.

I think you need Sun Cluster for that.

Bye, Dragan

--
Dragan Cvetkovic,

To be or not to be is true. G. Boole No it isn't. L. E. J. Brouwer

!!! Sender/From address is bogus. Use reply-to one !!!

by **Darren Dunha** » Wed, 28 Jan 2004 06:40:56

> I then mounted that partition on a client:
> mount -o ro host1,host2:/export/blt2.4 /blt
> host1 is running 2.7. host2 running 2.9 and the client running 2.8. Not ideal,
> I know, but if it works like that, its at least somewhat robust. Now I'm pretty
> share failover capability was introduced in 2.6. If it was introduced after
> 2.7, then obviously, this is a bad experiment.
> After issuing the mount command I cd'ed into /blt on the client. I then
> determined it was mounting from host1. So I unshared /export/blt2.4 on host1.
> When trying to do an ls on the client, it responded "Stale NFS Handle". Cd'ing
> in and out didn't help.
> THere was no failover as I understand "failover" to mean.

Hmm. To me, that doesn't test failover. The client sends a valid
request and gets a valid response ("Stale NFS Handle") back. I see no
reason that it should try another server.

The man page says "and the first server in the list is down..". Your
test does not match that situation.

--

Unix System Administrator Taos - The SysAdmin Company
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

by **Shivakanth Mundr** » Wed, 28 Jan 2004 07:55:01

>>I then mounted that partition on a client:

>>mount -o ro host1,host2:/export/blt2.4 /blt

>>host1 is running 2.7. host2 running 2.9 and the client running 2.8. Not ideal,
>>I know, but if it works like that, its at least somewhat robust. Now I'm pretty
>>share failover capability was introduced in 2.6. If it was introduced after
>>2.7, then obviously, this is a bad experiment.

>>After issuing the mount command I cd'ed into /blt on the client. I then
>>determined it was mounting from host1. So I unshared /export/blt2.4 on host1.
>>When trying to do an ls on the client, it responded "Stale NFS Handle". Cd'ing
>>in and out didn't help.

>>THere was no failover as I understand "failover" to mean.

> Hmm. To me, that doesn't test failover. The client sends a valid
> request and gets a valid response ("Stale NFS Handle") back. I see no
> reason that it should try another server.

> The man page says "and the first server in the list is down..". Your
> test does not match that situation.

Darren,

May be I didnt read the Original Post properly. I some how thought that
failover was being asked in "ro" scenario and quoted that example.

Yeah! I just looked at the man page and what it says. Does that mean the
server itself has to be down and not reachable on the network? I thought
it would access the second host listed when the NFS share on the
first host is unavailable. I dont clearly understand what difference
does it make from a NFS client point of view whether just NFS share is
unavailable or the NFS server itself is not reachable. In either case
the client is not getting the required service from the server and
should go to the next box in the listed order.

Shivakanth.

by **Darren Dunha** » Wed, 28 Jan 2004 09:02:23

>> Hmm. To me, that doesn't test failover. The client sends a valid
>> request and gets a valid response ("Stale NFS Handle") back. I see no
>> reason that it should try another server.

>> The man page says "and the first server in the list is down..". Your
>> test does not match that situation.

> Darren,
> May be I didnt read the Original Post properly. I some how thought that
> failover was being asked in "ro" scenario and quoted that example.

Yup. I think that's correct.

Quote:> Yeah! I just looked at the man page and what it says. Does that mean the
> server itself has to be down and not reachable on the network?

I haven't tested it myself in recent memory, but that is exactly how I
interpret it.

Quote:> I thought
> it would access the second host listed when the NFS share on the
> first host is unavailable. I dont clearly understand what difference
> does it make from a NFS client point of view whether just NFS share is
> unavailable or the NFS server itself is not reachable. In either case
> the client is not getting the required service from the server and
> should go to the next box in the listed order.

Having an NFS share unavailable could be an administrative decision.
Maybe you *meant* to turn it off. That's not a good enough reason to go
ask another server for the data.

I see it like DNS nameserver responses. Additional nameservers aren't
there because the first one doesn't have the address for the name,
they're there in case the first one doesn't answer at all.

You're not allowed to keep asking servers until you get the answer you
want, you have to believe the first answer, even if it's "the data you
want doesn't exist".

--

Unix System Administrator Taos - The SysAdmin Company
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >

by **Bigdaki** » Wed, 28 Jan 2004 09:02:53

>Subject: Re: NFS failover info..

>Date: 1/26/04 11:40 AM Hawaiian Standard Time

>> I then mounted that partition on a client:

>> mount -o ro host1,host2:/export/blt2.4 /blt

>> host1 is running 2.7. host2 running 2.9 and the client running 2.8. Not
>ideal,
>> I know, but if it works like that, its at least somewhat robust. Now I'm
>pretty
>> share failover capability was introduced in 2.6. If it was introduced after
>> 2.7, then obviously, this is a bad experiment.

>> After issuing the mount command I cd'ed into /blt on the client. I then
>> determined it was mounting from host1. So I unshared /export/blt2.4 on
>host1.
>> When trying to do an ls on the client, it responded "Stale NFS Handle".
>Cd'ing
>> in and out didn't help.

>> THere was no failover as I understand "failover" to mean.

>Hmm. To me, that doesn't test failover. The client sends a valid
>request and gets a valid response ("Stale NFS Handle") back. I see no
>reason that it should try another server.

>The man page says "and the first server in the list is down..". Your
>test does not match that situation.

What this is saying is that the server must go completely down for there to be
a problem?

It seems to me that if the NFS link is busted, it should try another server
after a timeout period. If NFS is *smart* enough to figure out the handle is
stale, it should be smart enough to use this as a clue and see if the other
designated servers are available.

I guess SUN and I disagree on what constitutes *failure* in this case.

Stuart
Dr. Stuart A. Weinstein
Ewa Beach Institute of Tectonics
"To err is human, but to really foul things up
requires a creationist"

by **Shivakanth Mundr** » Wed, 28 Jan 2004 10:06:09

>>Subject: Re: NFS failover info..

>>Date: 1/26/04 11:40 AM Hawaiian Standard Time

>>>I then mounted that partition on a client:

>>>mount -o ro host1,host2:/export/blt2.4 /blt

>>>host1 is running 2.7. host2 running 2.9 and the client running 2.8. Not

>>ideal,

>>>I know, but if it works like that, its at least somewhat robust. Now I'm

>>pretty

>>>share failover capability was introduced in 2.6. If it was introduced after
>>>2.7, then obviously, this is a bad experiment.

>>>After issuing the mount command I cd'ed into /blt on the client. I then
>>>determined it was mounting from host1. So I unshared /export/blt2.4 on

>>host1.

>>>When trying to do an ls on the client, it responded "Stale NFS Handle".

>>Cd'ing

>>>in and out didn't help.

>>>THere was no failover as I understand "failover" to mean.

>>Hmm. To me, that doesn't test failover. The client sends a valid
>>request and gets a valid response ("Stale NFS Handle") back. I see no
>>reason that it should try another server.

>>The man page says "and the first server in the list is down..". Your
>>test does not match that situation.

> What this is saying is that the server must go completely down for there to be
> a problem?

> It seems to me that if the NFS link is busted, it should try another server
> after a timeout period. If NFS is *smart* enough to figure out the handle is
> stale, it should be smart enough to use this as a clue and see if the other
> designated servers are available.
> I guess SUN and I disagree on what constitutes *failure* in this case.

Darren,

Thanks for shedding more light as usual.

Yeah, I thought it would go to the second server because it knows who
else holds the resource from the mount command irrespective of what kind
of error it gets from the first one.

May be If you did a (# /etc/init.d/nfs.server stop) on host1 instead of
just unsharing the resource and still have NFS on host1 responding to
requests it might have gone to the next host listed in the mount option
because it doesnt hear anything from host1.(even then it would try host1
and get a reply something like NFS server not responding). Any ways I
too agree with Dr Stuart that its not a 100% fail over (as I expected).

Shivakanth

by **Bigdaki** » Wed, 28 Jan 2004 11:52:56

>Subject: Re: NFS failover info..

>Date: 1/26/04 3:06 PM Hawaiian Standard Time

>>>Subject: Re: NFS failover info..

>>>Date: 1/26/04 11:40 AM Hawaiian Standard Time

>>>>I then mounted that partition on a client:

>>>>mount -o ro host1,host2:/export/blt2.4 /blt

>>>>host1 is running 2.7. host2 running 2.9 and the client running 2.8. Not

>>>ideal,

>>>>I know, but if it works like that, its at least somewhat robust. Now I'm

>>>pretty

>>>>share failover capability was introduced in 2.6. If it was introduced
>after
>>>>2.7, then obviously, this is a bad experiment.

>>>>After issuing the mount command I cd'ed into /blt on the client. I then
>>>>determined it was mounting from host1. So I unshared /export/blt2.4 on

>>>host1.

>>>>When trying to do an ls on the client, it responded "Stale NFS Handle".

>>>Cd'ing

>>>>in and out didn't help.

>>>>THere was no failover as I understand "failover" to mean.

>>>Hmm. To me, that doesn't test failover. The client sends a valid
>>>request and gets a valid response ("Stale NFS Handle") back. I see no
>>>reason that it should try another server.

>>>The man page says "and the first server in the list is down..". Your
>>>test does not match that situation.

>> What this is saying is that the server must go completely down for there
>to be
>> a problem?

>> It seems to me that if the NFS link is busted, it should try another server
>> after a timeout period. If NFS is *smart* enough to figure out the handle
>is
>> stale, it should be smart enough to use this as a clue and see if the other
>> designated servers are available.

>> I guess SUN and I disagree on what constitutes *failure* in this case.

>Darren,

>Thanks for shedding more light as usual.

>Yeah, I thought it would go to the second server because it knows who
>else holds the resource from the mount command irrespective of what kind
>of error it gets from the first one.

>May be If you did a (# /etc/init.d/nfs.server stop) on host1 instead of
>just unsharing the resource and still have NFS on host1 responding to
>requests it might have gone to the next host listed in the mount option
>because it doesnt hear anything from host1.(even then it would try host1
>and get a reply something like NFS server not responding).

Thats not a bad idea, unfortunately host1 serves a boatload of other stuff, and
I don't have that many computers to mess around with.

Stuart
Dr. Stuart A. Weinstein
Ewa Beach Institute of Tectonics
"To err is human, but to really foul things up
requires a creationist"

by **Thomas Na** » Wed, 28 Jan 2004 18:07:31

|
| >
| > THere was no failover as I understand "failover" to mean.
| >
|
| I think you need Sun Cluster for that.

No, you can make it 'easily' work without SunCluster. Here are the
incredience we use for several years:

- externale disk(s) (RAID box prefered) attached to _BOTH_ hosts
- Solaris Volume Manager (SVM) formerly OnlineDisksuite
- virtual interface
- some code for monitoring and initiating failover

The disk array should have 2 independend host channels. If this is not
supported you _MUST_ change the SCSI initiator ID for one of the boxes
using NVRAM commands during bootup. BTW, the setup works with old-style
SCSI as well as fiber channel.

2nd important thing is that the device path to the drives must be
identical on _BOTH_ boxes therefore having similar hardware helps but
it's not required. In the later case you may have to rename/rearrange
the paths a little bit. As they are used to create the mount handle it's
vital to have them identical. If not you will see a 'stale NFS handle'
error message.

Next you really wanna setup a metaset holding the external disk(s). Doing
so the Solaris kernel will enforce that only the machine you reserved the
disks for usage will be allowed to access them. Other machines will be
paniced by their SVM code.

I also recommend using virtual interfaces to provide services to clients.
Seperating them makes it much easier to move a service to a different
server independend from the other jobs the box is used for.

All the piece of code has to do is to

- take the metaset
- check if log replaying or fsck is required
- mount the filesystems
- export them
- bring up the NFS-service interface

The clients can now access the NFS server using one single IP independend
of the actual box. Failover time is about 1-2 minutes but this highly
depends on your service check interval and how many times you repeat the
check.

Hope that helps,
Thomas

-----------------------------------------------------------------
PGP fingerprint: B1 EE D2 39 2C 82 26 DA A5 4D E0 50 35 75 9E ED

by **Dragan Cvetkovi** » Wed, 28 Jan 2004 22:25:59

> |
> | >
> | > THere was no failover as I understand "failover" to mean.
> | >
> |
> | I think you need Sun Cluster for that.

> No, you can make it 'easily' work without SunCluster. Here are the
> incredience we use for several years:

> - externale disk(s) (RAID box prefered) attached to _BOTH_ hosts
> - Solaris Volume Manager (SVM) formerly OnlineDisksuite
> - virtual interface
> - some code for monitoring and initiating failover

[snip]

Quote:
> All the piece of code has to do is to

> - take the metaset
> - check if log replaying or fsck is required
> - mount the filesystems
> - export them
> - bring up the NFS-service interface

Well, what you have basically described is a function of Sun Cluster and
its monitoring agents. Your other requirements (that are snipped) are also
the ones that Sun Cluster provides...

So basically, you are right, you don't need SC, but you need to emulate its
functionality.

Bye, Dragan

--
Dragan Cvetkovic,

To be or not to be is true. G. Boole No it isn't. L. E. J. Brouwer

!!! Sender/From address is bogus. Use reply-to one !!!