Direct NFS access to Sun Storage 7410 with Oracle 11g and Solaris… configuration and verifcation

During the course of experimentation with 11gR2, I was given some space on a Sun Storage 7410 NAS.   In the past NAS meant using NFS with obscure mount options that seemed to vary from platform to platform.  So, at first I went scrambling for the “best practices” to use with Oracle NAS on Solaris.

There is a nice Metalink article Note:359515.1 with the latest information for all platforms.  This Metalink note does include the “tcp” option which is not necessary on Solaris.  So it boiled down to the following mount options for using Oracle data files on NAS devices with Solaris.

rw,bg,hard,nointr,rsize=32768,
wsize=32768,noac,
forcedirectio, vers=3,suid

But wait, what about the new 11g feature to use direct NFS “dNFS”?    More searching…

configuring dNFS on Solaris

This is a fairly simple process.  Although Oracle dNFS configuration is fairly well documented for Linux, I will post my interpretation and commentary to help other Solaris users that might want to configure dNFS.

First, create mount the NFS share just as you would have in the past.  Oracle still needs to see the file system from the OS point of view.  You don’t have to use the mount options as in the past, but you might want them anyway for OS tools may access the mount.  You would most likely place these options in the “/etc/vfstab” file, but I will just show the mount command.

mount -o rw,bg,hard,nointr,rsize=32768,\
wsize=32768,noac,forcedirectio,vers=3,suid \
toromondo.west:/export/glennf /ar1

Second, you have to link the direct NFS libraries in place of ODM.  This is a little clunky, but not terrible.

cd $ORACLE_HOME/lib
cp libodm11.so libodm11.so_stub
ln -s libnfsodm11.so libodm11.so

Third, create the “$ORACLE_HOME/dbs/oranfstab” file.  This file defines the various details Oracle needs to directly access the NFS share.  You can configure multiple paths, so that Oracle can multiplex access to the NFS share.  This is for redundancy and load balancing.  There is another Metalink article ID:822481.1 that details how to configure dNFS with multiple paths across the same subnet and force the OS to not route packets.  This is a great feature, which I will try once I get some more network plumbing.  For now, I just did the most simple configuration as shown below.

cat $ORACLE_HOME/dbs/oranfstab
server: toromondo.west
path: toromondo.west
export: /export/glennf mount:/ar1


Finally, you will be able to see if this takes effect by looking at the “alert.log” file.  When Oracle starts up it places debug information in the alert.log file so we can see if Oracle is using Direct NFS or not.

grep NFS alert_*.log
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0
Direct NFS: attempting to mount /export/glennf on filer toromondo.west defined in oranfstab
Direct NFS: channel config is:
Direct NFS: mount complete dir /export/glennf on toromondo.west mntport 38844 nfsport 2049
Direct NFS: channel id [0] path [toromondo.west] to filer [toromondo.west] via local [] is UP
Direct NFS: channel id [1] path [toromondo.west] to filer [toromondo.west] via local [] is UP

That’s all there is to it.  Hopefully, you will find this useful.

Exadata V2 IOPS are massive… but predictable latency will keep your applications humming and help you sleep at night.

After talking with customers over the years, I find that performance is necessary… but predictability is what really matters.  Today’s data centers are complex.  Applications  are very dynamic changing often on a daily basis and new applications are being added to the mix as well.  The constant application morphing is enough to scare most DBA’s.  But, the scariest thing of all is, the resource requirements are often not well understood… chaos indeed.

the dreaded 1:00AM call

How many DBAs and system admins have got the dreaded 1:00AM call only to track down the problem to be “application” update that was applied just prior?  So, the patch added some “new” functionality or added some additional batch processing.  This typically amounts to an increase in IOPS and unless you are well provisioned,  application response time will typically increase as well.  How do you provide predictable user response time in this environment?

response time with latency-sensitive applications

I was just reading through Kevin’s most recent post on the Sun Oracle database machine.  He does a great job of describing the subtleties of a consolidated environment, and the benefit that Sun FlashFire technology provides.  I particularly like the following quote:

What if you have a database that doesn’t require extreme IOPS but requires very low latency I/O served from a data set of, say, 1 TB. Imagine further that this latency-sensitive database isn’t the only database you have.

In typical ERP/CRM environments, end-user response time is often directly tied to IO response time.  If you look at the recent Sun/Oracle TPC-C announcements with FlashFire, you will get an idea of what I mean.  This Sun/Oracle TPC-C result showed a 10x drop in response time over the next highest result.  TPC-C is very sensitive to IO latency,  so it is no wonder that Flash was a win in this environment.

For another example, consider this AWR excerpt from a customer application…

                                 Reads               CPU   Elapsed
Physical Reads  Executions    per Exec   %Total Time (s)  Time (s)    SQL Id
-------------- ----------- ------------- ------ -------- --------- -------------
667,627             16,020          41.7   45.3  2456.29  20703.06 ggyyzqn066f38
Module: java@myhost (TNS V1-V3)
BEGIN PKGCUST.SHOW_CUST (:1, :2); END;

The above query averages 41.7 reads with a query response time of 1.3 seconds.  The above example has an average IO latency of 7ms, so 41.7 reads would take about 292ms.  But these are 15,000 rpm drives and at 7ms response time, these disks are getting pretty hot.  This environment experienced a 4x increase in IOPS due to new functionality and some rogue applications.  This increase caused the query response time to jump from 1.3 -> 2.2 seconds.

Now, if this query was running on Exadata V2, things would look different.  This sudden spike in IOPS would not affect the IO response time.  With Exadata V2, the 41.7 reads would happen in about 21ms and this would not change with an 4x increase in IOPS.  We have measured the response time and it remains flat at about 0.5 ms whether you are doing 100 IOPS or a Million.

sleep well at night with FlashCache

Exadata V2 solves the problem of predictable IO latency by providing more IOPS than applications can actually use.  If a workload increases or new functionality is added, you don’t have to worry about IO latency.    I like the way the Kevin put’s it in his blog post:

No, I’m not as excited about the IOPS capability of the Database Machine as much as the FLASH storage-provisioning model it offers. In spite of all the hype, the Database Machine IOPS story is every bit as much a story of elegance as it is brute force.

FlashCache puts the IOPS where they are needed most.  If you run multiple instances in a dynamic environment, Exadata V2 with FlashCache might just help you avoid the dreaded 1:00AM call.

Glenn Fawcett’s transplanted blog… Discussing Oracle database performance with Sun servers

With the acquisition of Sun quickly approaching, I decided it was time to get a personal blog to discuss Oracle performance on Sun. I have maintained a blog at Sun for the past 3 years, so I am not new to the powers of blogging.  I want to continue to post material during the transition and not have to come up to speed on another blogging platform and the required logistics.  Blogging helps me discuss current trends regarding database performance in real-time as well as preview and refine material that eventually becomes a presentation or white paper.

Why did I choose WordPress?

The are multiple reasons.  The functionality is one of the best in the business.  The interface is smooth and intuitive which allows you quickly express yourself.  Also, for the past 5 months I have been working closely with an old friend Kevin Closson on Exadata V2 performance.  Kevin is  an avid blogger on WordPress as well… So all-in-all, WordPress seemed a natural fit.

take care,
Glenn