Posts Tagged 'sar'

Solaris Eye for the Linux Guy… Part II (oprofile, Dtrace, and Oracle Event Trace)

Proper tool for the job

My grandfather used to say to me: “Use the proper tool for the job”.  This is important to keep in mind when faced with performance issues.  When I am faced with performance problems in Oracle, I typically start at a high level with AWR reports or Enterprise Manager to get a high level understanding of the workload.   To drill down further, the next step is to use Oracle “10046 event” tracing.  Cary Millsap created a methodology around event tracing called “Method-R” which shows how to focus in on the source of a performance problem by analyzing the components that contribute to response time.   These are all fine places to start to analyze performance problems from the “user” or “application” point of view.  But what happens if the OS is in peril?

If you are experiencing high system time or strange application behavior, it is likely time to drill deeper with OS based tools.  I mentioned in my last post , “prstat” is the “top” equivalent for Solaris.  “prstat” is the best place to start to see how processes are running on Solaris, but at some point you may need to drill down deeper to gain a better understanding of the problem.

With Linux “oprofile” allows you to sample the kernel and user code to build a profile of how the system and applications are behaving.  This is an incredibly useful tool, but it doesn’t exist on Solaris.  Luckily, there is something that is arguably better – Dtrace.

Solaris Dtrace(1m) / Linux “oprofile”

Dtrace was developed for the release of Solaris 10 by kernel engineers as a way to better debug and monitor Solaris.  Unlike “oprofile”, Dtrace is an really an environment that involves writing code in “D” to make use of the numerous amounts of probe data that exist.  Dtrace is really powerful, but it does require some heavy lifting to get started.  This is where the “Dtrace Toolkit” comes in handy.

The “Dtrace Toolkit” is a set of scripts that server as a starting point for those interested in using Dtrace.  Also included in the “Dtrace Toolkit” are some real clever utilities.  My two favorite utilities for Dtrace are the “hotkernel” and “hotuser” scripts.  These scripts analyze either the kernel or a user “PID” to show which routines are most used.  This can be extremely useful when diagnosing performance problems that extend beyond the V$ tables or Oracle “10046 event trace” data.

To illustrate the use of these utilities, I have included output from a benchmark that shows how these might be used.


root@apl5-1> ./hotkernel
Sampling... Hit Ctrl-C to end.
FUNCTION                                                COUNT   PCNT
nxge`nxge_check_10g_link                                    1   0.0%
genunix`ioctl                                               1   0.0%
genunix`fop_read                                         5730   2.1%
genunix`kstrgetmsg                                       6091   2.2%
unix`utl0                                                7058   2.6%
FJSV,SPARC64-VII`cpu_halt_cpu                            7220   2.6%
FJSV,SPARC64-VII`copyout                                 9340   3.4%
ip`tcp_fuse_output                                      12637   4.6%
unix`_resume_from_idle                                  12922   4.7%
unix`disp_getwork                                       18864   6.8%
unix`mutex_enter                                        34033  12.3%


root@apl5-1> ./hotuser -p 12626
Sampling... Hit Ctrl-C to end.
FUNCTION                                                COUNT   PCNT
oracle`kxsInitExecutionHeap                                 1   0.0%
oracle`0x10b319ad0                                          1   0.0%
oracle`kews_pls_jvm_event_resume_i                          1   0.0%
oracle`0x10b319ac8                                          1   0.0%
oracle`kghfrh                                               1   0.0%
oracle`opiptc                                               1   0.0%
oracle`qertbFetchByRowID                                   91   1.0%
oracle`kghalf                                              94   1.1%`memcpy                                      102   1.2%
oracle`opikndf2                                           105   1.2%
oracle`kpofcr                                             113   1.3%
oracle`opiodr                                             120   1.4%
oracle`kslwtectx                                          120   1.4%
oracle`kslwt_update_stats_int                             126   1.4%
oracle`opitsk                                             126   1.4%
oracle`ksupucg                                            151   1.7%
oracle`nsbasic_brc                                        153   1.7%
oracle`kdxbrs1                                            187   2.1%
oracle`kdxlrs2                                            192   2.2%
oracle`kews_sqlcol_end                                    194   2.2%
oracle`opifch2                                            212   2.4%
oracle`opiexe                                             250   2.8%
oracle`skgslcas                                           265   3.0%`memset                                      416   4.7%
oracle`kcbgtcr                                            826   9.4%

You can begin to see how Dtrace can be useful to see the effect of the workload on Solaris and profile the user application – in this case an Oracle shadow process.  But this is just the beginning.  If you are so inclined, Dtrace can be used to correlate all sorts of performance data both inside the kernel and application.

Solaris Eye for the Linux guy… or how I learned to stop worrying about Linux and Love Solaris (Part 1)

This entry goes out to my Oracle techie friends that have been in the Linux camp for sometime now and are suddenly finding themselves needing to know more about Solaris… hmmmm… I wonder if this has anything to do with Solaris now being an available option with Exadata?  Or maybe the recent announcement that the SPARC T3 multiplier for T3-x servers is now 0.25.  Judging by my inbox recently, I suspect a renewed interest in Solaris to continue.

I have focused on Oracle database performance on Solaris for 14 years now.  In the last few years, I began to work on Exadata and found myself  needing to learn the “Linux” way of performance analysis.   I will cover some basic tools needed for Oracle performance analysis with Solaris as well as some special performance topics.  I am not a deep dive kernel type of guy, so don’t expect esoteric Dtrace scripts to impress your friends.  I am not going to cover how to patch and maintain Solaris – this is out of scope. With this in mind, lets get started.

prstat(1M) … top on steriods!

Probably the first tool that the typical Linux performance guy reaches for is top.  This is part of every Linux distribution that I know of but is sadly missing from Solaris… But Solaris has something much better “prstat(1m)“.  I know the name is boring but it simply means “process status”.  This is the first place to get an idea of how processes are performing on a system and quite likely the most useful tool in general.

# prstat
 5623 oracle     32G   32G cpu63    0    0   0:02:23 1.0% oracle/1
 5625 oracle     32G   32G cpu60    0    0   0:02:22 1.0% oracle/1
 5627 oracle     32G   32G sleep    0    0   0:02:18 1.0% oracle/1
 5629 oracle     32G   32G cpu38    0    0   0:02:16 1.0% oracle/1
 5609 oracle     43M   38M sleep    0    4   0:01:21 0.6% rwdoit/1
 5605 oracle     43M   38M sleep    0    4   0:01:18 0.6% rwdoit/1
 5607 oracle     43M   38M sleep    0    4   0:01:18 0.6% rwdoit/1
 5601 oracle     43M   38M sleep    0    4   0:01:17 0.6% rwdoit/1
Total: 106 processes, 447 lwps, load averages: 5.03, 7.11, 19.48

Top would show you....

# top
load averages:  5.06,  6.84, 18.81                                                                             14:50:13
109 processes: 100 sleeping, 1 stopped, 8 on cpu
CPU states: 92.7% idle,  4.0% user,  3.3% kernel,  0.0% iowait,  0.0% swap
Memory: 128G real, 60G free, 50G swap in use, 135G swap free

 5623 oracle     1   0    0    0K    0K cpu57   3:08 66.41% oracle
 5625 oracle     1   0    0    0K    0K cpu33   3:06 66.02% oracle
 5627 oracle     1   0    0    0K    0K cpu14   3:02 64.45% oracle
 5629 oracle     1   0    0    0K    0K cpu41   2:59 63.28% oracle
 5609 oracle     1   0    4    0K    0K cpu31   1:46 37.70% rwdoit
 5605 oracle     1   0    4    0K    0K cpu55   1:43 36.33% rwdoit
 5607 oracle     1   0    4    0K    0K sleep   1:43 36.33% rwdoit
 5601 oracle     1   0    4    0K    0K cpu32   1:42 35.94% rwdoit

What is happening?  “top” shows details from a from process point of view where as by default prstat(1M) shows the system aggregates.  To make prstat(1M) look more like top, you have to enable micro-state accounting with the “-m” option.  With the “-m” option, prstat(1M) shows CPU utilization of various processes like top but with a LOT more detail.  You have access to details regarding CPU time broken out by user and system.  You can find out the percentage of time spent in traps, sleep, and time spent waiting for CPU “LAT”.  Finally, you can see the number of voluntary and involuntary context switches along with the number of threads per process.

# prstat -m
 5623 oracle    48  18 0.0 0.0 0.0 0.0  29 4.3 38K   8 .3M   0 oracle/1
 5625 oracle    48  18 0.0 0.0 0.0 0.0  30 4.3 38K   9 .3M   0 oracle/1
 5627 oracle    43  21 0.0 0.0 0.0 0.0  30 5.5 32K   7 .3M   0 oracle/1
 5629 oracle    42  21 0.0 0.0 0.0 0.0  31 5.6 33K   6 .3M   0 oracle/1
 5609 oracle    17  21 0.0 0.0 0.0 0.0  57 5.7 33K   5 77K   0 rwdoit/1
 5605 oracle    20  17 0.0 0.0 0.0 0.0  59 4.5 38K   5 89K   0 rwdoit/1
 5607 oracle    17  19 0.0 0.0 0.0 0.0  58 5.4 32K   4 75K   0 rwdoit/1
 5601 oracle    20  16 0.0 0.0 0.0 0.0  59 4.5 38K   5 90K   0 rwdoit/1

There are a lot of options available to prstat(1M) so do take a look at the prstat(1M) man page. Also look at the “scalingbits” blog for an excellent discussion on prstat(1M) monitoring.  Stephan goes into much more detail about the utility and how to monitor by “zones” or “projects”… very useful stuff.

lsof = pfiles(1M)… the proc(1) commands

pfiles(1M) mirrors what “lsof” does but there is much more information available on a per-process basis available.  A man of “proc(1)” shows the available process related commands: “pstack(1M)”, “pldd(1M)”, and “pflags(1M) to name a few.  These utilities referred to as process introspection commands are detailed nicely on the website.

vmstat(1M), mpstat(1M), iostat(1M), sar(1M)… basically the same!

It is good to know that somethings are not all that different.  The above tools have minor differences, but generally look the same.  Some options are different and expanded.   Take for example iostat.

“iostat(1M) for Solaris has as a “z” option that takes out the devices that are not doing any IO and have “Zeros”.  The biggest issue with most of these tools come into play when there are missing options or the formatting is different.  This messes up scripts that have been developed to help aid analysis.  This is not too hard to fix… just something that is going to have to be worked out.


The best references for Solaris performance and analysis would be the “Solaris Internals” and the “Solaris Performance and Tools” books.  These books describe the architecture of Solaris and show how to analyze and diagnosis performance issues… and you can get them on the kindle 🙂  The books also have an accompanying website “” to continue the discussion.

That is all for now…