During the course of experimentation with 11gR2, I was given some space on a Sun Storage 7410 NAS. In the past NAS meant using NFS with obscure mount options that seemed to vary from platform to platform. So, at first I went scrambling for the “best practices” to use with Oracle NAS on Solaris.
There is a nice Metalink article Note:359515.1 with the latest information for all platforms. This Metalink note does include the “tcp” option which is not necessary on Solaris. So it boiled down to the following mount options for using Oracle data files on NAS devices with Solaris.
rw,bg,hard,nointr,rsize=32768,
wsize=32768,noac,
forcedirectio, vers=3,suid
But wait, what about the new 11g feature to use direct NFS “dNFS”? More searching…
configuring dNFS on Solaris
This is a fairly simple process. Although Oracle dNFS configuration is fairly well documented for Linux, I will post my interpretation and commentary to help other Solaris users that might want to configure dNFS.
First, create mount the NFS share just as you would have in the past. Oracle still needs to see the file system from the OS point of view. You don’t have to use the mount options as in the past, but you might want them anyway for OS tools may access the mount. You would most likely place these options in the “/etc/vfstab” file, but I will just show the mount command.
mount -o rw,bg,hard,nointr,rsize=32768,\
wsize=32768,noac,forcedirectio,vers=3,suid \
toromondo.west:/export/glennf /ar1
Second, you have to link the direct NFS libraries in place of ODM. This is a little clunky, but not terrible.
cd $ORACLE_HOME/lib
cp libodm11.so libodm11.so_stub
ln -s libnfsodm11.so libodm11.so
Third, create the “$ORACLE_HOME/dbs/oranfstab” file. This file defines the various details Oracle needs to directly access the NFS share. You can configure multiple paths, so that Oracle can multiplex access to the NFS share. This is for redundancy and load balancing. There is another Metalink article ID:822481.1 that details how to configure dNFS with multiple paths across the same subnet and force the OS to not route packets. This is a great feature, which I will try once I get some more network plumbing. For now, I just did the most simple configuration as shown below.
cat $ORACLE_HOME/dbs/oranfstab
server: toromondo.west
path: toromondo.west
export: /export/glennf mount:/ar1
Finally, you will be able to see if this takes effect by looking at the “alert.log” file. When Oracle starts up it places debug information in the alert.log file so we can see if Oracle is using Direct NFS or not.
grep NFS alert_*.log
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0
Direct NFS: attempting to mount /export/glennf on filer toromondo.west defined in oranfstab
Direct NFS: channel config is:
Direct NFS: mount complete dir /export/glennf on toromondo.west mntport 38844 nfsport 2049
Direct NFS: channel id [0] path [toromondo.west] to filer [toromondo.west] via local [] is UP
Direct NFS: channel id [1] path [toromondo.west] to filer [toromondo.west] via local [] is UP
That’s all there is to it. Hopefully, you will find this useful.
Interesting – I’d be curious about the performance against a 7410. Can you share any results?
I am just getting started with my testing, so no real results at the moment. Also, most of this is for internal projects, but I will share what I can.
I am working on a followup post a the moment that shows how to monitor this environment with Solaris. So, stay tuned.
Hi Glenn,
If you have time, I’d be interested in a comparison in terms of IO throughput, latency, and CPU utilitsation, between dNFS and traditional OS based NFS. Would also be interested in how reliable dNFS is, any packet drops/retransmits, any RAC issues you are running/ran across?
That is kind of my plan to do a comparison now I have gone down this path
So far, using 10gigE I have not seen any issues with drops/retransmits, but I would guess that could be environment and application dependent. Also, this is a single-instance environment. I do understand the allure of using this kind of setup for RAC. It certainly simplifies things.
Thanks for your comments.
btw, if possible it would be nice to see some performance measurements done with 1 through 4 active 1GE links instead of a single 10Gb/s.
fyi – the “server:” stanza is a descriptive field in oranfstab.
You really, really want to make sure you have the latest patches before you attempt to use DNFS as this technology was very fragile up until three months ago or so.
If you are using ethernet to connect to your storage device be sure your host, storage device, and switches support and are using jumbo frames, and be sure you have tuned your TCP stack on the host(s) and storage devices to have a large receive window to account for the jumbo frames coming in.
-Adam
I am using 11gR2 that was just released (11.2.0.1) for Solaris SPARC and things have been stable. I agree that you definitely need to use Jumbo frames to get decent bandwidth. I tuned the TCP stack a little, but the defaults for Solaris are pretty good as well. I went with the following…
ndd -set /dev/tcp tcp_recv_hiwat 400000
ndd -set /dev/tcp tcp_xmit_hiwat 400000
ndd -set /dev/tcp tcp_max_buf 2097152
ndd -set /dev/tcp tcp_cwnd_max 2097152
Generally I do not learn article on blogs, however I would like to say that this write-up very compelled me to
check out and do so! Your writing taste has been amazed
me. Thanks, quite nice post.
Thanks. I have been not able to post much at all lately due to a change in jobs, but hopefully I will be doing some more soon. I have recently been re-acquainted with ZFS inside of the SPARC SuperCluster. The analytics are quite amazing to me still.