Analyzing IO at the Exadata Cell level… a simple tool for IOPS.

Lately I have been drawn into to a fare number of discussions about IO characteristics while helping customers run benchmarks.  I have been working with a mix of developers, DBAs, sysadmin, and storage admins.  As I have learned, every group has there own perspective – certainly when it comes to IO and performance.

  • Most DBA’s want to see data from the DB point of view so AWR’s or EM works just fine.
  • Most System Admin’s look at storage from the Filesystem or ASM disk level.
  • Storage Admins want to see what is going on within the array.
  • Performance geeks like myself, like to see all up and down the stack :)

As part of pulling back the covers, I came up with a simple little tool for show IOPS at the cell level.

Mining IO statistics from cellcli

The cellsrv process collects data about various events and performance metrics in an Exadata storage cell.  I certainly am a huge fan of the table and index usage data gathered using the  “pythian_cell_cache_extract.pl” written by Christo Kutrovsky.  It is really provides a great look inside the Exadata Smart Flash Cache.  So, this got me to thinking.  What about IOPS data?

With the introduction of the Write Back Flash cache in X3, there is much more analysis about what is going to flash vs disk – and how what is written to flash is flushed to disk.

To look at all the current metrics gathered from the storage cells in your Exadata or SuperCluster you can run “cellcli -e list metriccurrent” on all the storage cells.  The “metriccurrent” parameters are updated every minute by cellsrv to store performance data.  There are a few convient parameters that can be used to sum up all the IOPS.

  • CD_IO_RQ_R_LG_SEC + CD_IO_RQ_R_SM_SEC
  • CD_IO_RQ_W_LG_SEC + CD_IO_RQ_W_SM_SEC

These parameters shore the number of IO/sec for reads and writes.  By mining this data and breaking it down by “FD” vs “CD” you can see hit ratios for reads from an overall cell point of view, but now you can also see how many writes are going to FLASH vs DISK.

The “ciops-all.sh” script will look at all the cells and sum up all the IOPS and report the findings.  This is very useful to get a quick look at the IO profile in the cells.

[oracle@exa6db01 WB]$ ./ciops-all.sh
FLASH_READ_IOPS: 6305
DISK_READ_IOPS: 213
FLASH_WRITE_IOPS: 488203
DISK_WRITE_IOPS: 6814
TOTAL_NUMBER_OF_DRIVES: 84
WRITE_PCT_to_FLASH: 98
READ_PCT_from_FLASH: 96
IOPS_PER_DISK: 83

This can be very helpful when trying to figure out if you need to go with high performance or high capacity disks.  This case shows most IO going to the flash and only 83 IOPS are spilled to each disk.  So, with this case HC disks would be a fine choice.  With a simple modification, I made the “ciops-mon.sh” script to print out the throughput every few minutes to graph the results over time.

ciops_data_x3-2

This has been helpful as I have been investigating and explaining the inter-workings of the Exadata smart flash cache.  Hopefully, you will find this useful when trying to analyze and understand Exadata Cell level IO with your workload.

About these ads

6 Responses to “Analyzing IO at the Exadata Cell level… a simple tool for IOPS.”


  1. 1 Rich Headrick June 18, 2013 at 11:00 am

    Nice simple write up. As an FYI, the cell_group file can be located on most compute nodes in /opt/oracle.Support Tools/onecommand/cell_group. You’ll need to modify the script to account for that.

    • 2 glennfawcett June 18, 2013 at 11:27 am

      Thanks Rich for the feedback.

      The /opt/oracle.SupportTools/onecommand/cell_group location works great if you have root. I only had access to the “oracle” user on the DB nodes and the “celladmin” user on the storage cells so I had to create a new file. We should make the “cell_group” readable but that is not the default. I will add that the Pythian scripts parse the “/etc/oracle/cell/network-config/cellip.ora” file for the ip addresses. This file is readable by the oracle user by default.

      • 3 Rich Headrick June 18, 2013 at 11:35 am

        One more little mod for formatting: I aligned all your variable at the $ so they line up nicely when prting. It doesn’t format well on this comment page however.

  2. 4 kevinclosson June 18, 2013 at 12:46 pm

    500K IOPS from a half rack…right on datasheet spec :-)

    Must be SLOB? :-)

    Can you grep ‘db file sequential read’ from the AWR report for us?
    :-)

    Nice to see you blogging Glenn… We need to get together soon.

    • 5 glennfawcett June 19, 2013 at 5:52 am

      It wasn’t actually SLOB, but that might be interesting.

      I used a mod of my blkhammer populate script to populate a bunch of tables OLTP style to show how WriteBack is used. As expected, Exadata is real good on “db file sequential read”… in the sub picosecond range if I am not mistaken :)


  1. 1 Analyzing IO at the Cell level with cellcli… a new and improved script | Glenn Fawcett's Oracle blog Trackback on January 21, 2014 at 8:46 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





Follow

Get every new post delivered to your Inbox.

Join 340 other followers

%d bloggers like this: