Direct NFS access to Sun Storage 7410 with Oracle 11g and Solaris… configuration and verifcation

During the course of experimentation with 11gR2, I was given some space on a Sun Storage 7410 NAS.   In the past NAS meant using NFS with obscure mount options that seemed to vary from platform to platform.  So, at first I went scrambling for the “best practices” to use with Oracle NAS on Solaris.

There is a nice Metalink article Note:359515.1 with the latest information for all platforms.  This Metalink note does include the “tcp” option which is not necessary on Solaris.  So it boiled down to the following mount options for using Oracle data files on NAS devices with Solaris.

rw,bg,hard,nointr,rsize=32768,
wsize=32768,noac,
forcedirectio, vers=3,suid

But wait, what about the new 11g feature to use direct NFS “dNFS”?    More searching…

configuring dNFS on Solaris

This is a fairly simple process.  Although Oracle dNFS configuration is fairly well documented for Linux, I will post my interpretation and commentary to help other Solaris users that might want to configure dNFS.

First, create mount the NFS share just as you would have in the past.  Oracle still needs to see the file system from the OS point of view.  You don’t have to use the mount options as in the past, but you might want them anyway for OS tools may access the mount.  You would most likely place these options in the “/etc/vfstab” file, but I will just show the mount command.

mount -o rw,bg,hard,nointr,rsize=32768,\
wsize=32768,noac,forcedirectio,vers=3,suid \

toromondo.west:/export/glennf /ar1

Second, you have to link the direct NFS libraries in place of ODM.  This is a little clunky, but not terrible.

cd $ORACLE_HOME/lib
cp libodm11.so libodm11.so_stub
ln -s libnfsodm11.so libodm11.so

Third, create the “$ORACLE_HOME/dbs/oranfstab” file.  This file defines the various details Oracle needs to directly access the NFS share.  You can configure multiple paths, so that Oracle can multiplex access to the NFS share.  This is for redundancy and load balancing.  There is another Metalink article ID:822481.1 that details how to configure dNFS with multiple paths across the same subnet and force the OS to not route packets.  This is a great feature, which I will try once I get some more network plumbing.  For now, I just did the most simple configuration as shown below.

cat $ORACLE_HOME/dbs/oranfstab
server: toromondo.west
path: toromondo.west
export: /export/glennf mount:/ar1

Finally, you will be able to see if this takes effect by looking at the “alert.log” file.  When Oracle starts up it places debug information in the alert.log file so we can see if Oracle is using Direct NFS or not.

grep NFS alert_*.log
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0
Direct NFS: attempting to mount /export/glennf on filer toromondo.west defined in oranfstab
Direct NFS: channel config is:
Direct NFS: mount complete dir /export/glennf on toromondo.west mntport 38844 nfsport 2049
Direct NFS: channel id [0] path [toromondo.west] to filer [toromondo.west] via local [] is UP
Direct NFS: channel id [1] path [toromondo.west] to filer [toromondo.west] via local [] is UP

That’s all there is to it.  Hopefully, you will find this useful.

About these ads

15 Responses to “Direct NFS access to Sun Storage 7410 with Oracle 11g and Solaris… configuration and verifcation”


  1. 1 fmozza November 19, 2009 at 5:19 pm

    Interesting – I’d be curious about the performance against a 7410. Can you share any results?

  2. 2 glennfawcett November 20, 2009 at 7:58 am

    I am just getting started with my testing, so no real results at the moment. Also, most of this is for internal projects, but I will share what I can.

    I am working on a followup post a the moment that shows how to monitor this environment with Solaris. So, stay tuned.

  3. 3 konrad November 23, 2009 at 2:57 am

    Hi Glenn,

    If you have time, I’d be interested in a comparison in terms of IO throughput, latency, and CPU utilitsation, between dNFS and traditional OS based NFS. Would also be interested in how reliable dNFS is, any packet drops/retransmits, any RAC issues you are running/ran across?

    • 4 glennfawcett November 23, 2009 at 10:17 am

      That is kind of my plan to do a comparison now I have gone down this path :) So far, using 10gigE I have not seen any issues with drops/retransmits, but I would guess that could be environment and application dependent. Also, this is a single-instance environment. I do understand the allure of using this kind of setup for RAC. It certainly simplifies things.

      Thanks for your comments.

      • 5 Jason Buchanan November 24, 2009 at 6:14 am

        btw, if possible it would be nice to see some performance measurements done with 1 through 4 active 1GE links instead of a single 10Gb/s.

  4. 6 Jason Buchanan November 23, 2009 at 6:24 pm

    fyi – the “server:” stanza is a descriptive field in oranfstab.

  5. 7 Adam Fletcher November 30, 2009 at 7:27 am

    You really, really want to make sure you have the latest patches before you attempt to use DNFS as this technology was very fragile up until three months ago or so.

    If you are using ethernet to connect to your storage device be sure your host, storage device, and switches support and are using jumbo frames, and be sure you have tuned your TCP stack on the host(s) and storage devices to have a large receive window to account for the jumbo frames coming in.

    -Adam

    • 8 glennfawcett November 30, 2009 at 9:17 am

      I am using 11gR2 that was just released (11.2.0.1) for Solaris SPARC and things have been stable. I agree that you definitely need to use Jumbo frames to get decent bandwidth. I tuned the TCP stack a little, but the defaults for Solaris are pretty good as well. I went with the following…

      ndd -set /dev/tcp tcp_recv_hiwat 400000
      ndd -set /dev/tcp tcp_xmit_hiwat 400000
      ndd -set /dev/tcp tcp_max_buf 2097152
      ndd -set /dev/tcp tcp_cwnd_max 2097152

  6. 9 industry February 28, 2013 at 11:41 am

    Generally I do not learn article on blogs, however I would like to say that this write-up very compelled me to
    check out and do so! Your writing taste has been amazed
    me. Thanks, quite nice post.

    • 10 glennfawcett March 13, 2013 at 1:32 pm

      Thanks. I have been not able to post much at all lately due to a change in jobs, but hopefully I will be doing some more soon. I have recently been re-acquainted with ZFS inside of the SPARC SuperCluster. The analytics are quite amazing to me still.


  1. 1 Words About Oracle Direct NFS On Sun Storage 7410 And Chip Multithreading Technology (CMT) For Oracle Database 11g Release 2. « Kevin Closson’s Oracle Blog: Platform, Storage & Clustering Topics Related to Oracle Databases Trackback on November 19, 2009 at 9:27 pm
  2. 2 c0t0d0s0.org Trackback on November 20, 2009 at 11:00 am
  3. 3 Blogroll Report 13/11/2009-20/11/2009 « Coskan’s Approach to Oracle Trackback on November 22, 2009 at 1:56 pm
  4. 4 Direct NFS in Solaris with Oracle 11g: Benchmarks « UNIX Administratosphere Trackback on December 15, 2009 at 2:01 am
  5. 5 More Words About Oracle Direct NFS On Sun Storage 7410 And Chip Multithreading Technology (CMT) For Oracle Database 11g Release 2. « Kevin Closson’s Oracle Blog: Platform, Storage & Clustering Topics Related to Oracle Databases Trackback on December 15, 2009 at 9:06 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





Follow

Get every new post delivered to your Inbox.

Join 338 other followers

%d bloggers like this: