Introduction

This article illustrates the use the open source software for Storage Array Performance Monitoring. There are tools for performance monitoring but they are either expensive ( EMC's ECC ), hard to maintain ( HDS Tuning Manager )  or both ( HP's Storage Essentials and Veritas's Command Central ). Most of these software tools are bloated i.e they offer too many metrics out of which the SAN Administrator probably requires less than 5%. The author has implemented a Cacti based replication monitoring solution for a large IT company in San Jose, CA. This solution monitors key HUR ( Hitachi Universal Replicator ) metrics such as Journal Usage, RPO, throughput and also FCIP metrics from Brocade 7500's.

About Cacti

Cacti is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices.

Installing and configuring Cacti on AIX 5.3

Installing Apache2

Download and extract apache2 source from http://httpd.apache.org/download.cgi

From within the extracted directory run the commands below

Installing MySQL

Download the version of MySQL for AIX 5.3 64 bit from http://dev.mysql.com/downloads/mysql/5.1.html#aix and extract to /opt/mysql64

[root@trencher]/opt/mysql64
# ls
COPYING            README             data               lib                scripts            support-files
EXCEPTIONS-CLIENT  bin                docs               man                share              tests
INSTALL-BINARY     configure          include            mysql-test         sql-bench
[root@trencher]/opt/mysql64

Follow the steps below to configure and startup mysql

shell> groupadd mysql
shell> useradd -g mysql mysql
shell> cd /usr/local
shell> gunzip < /path/to/mysql-VERSION-OS.tar.gz | tar xvf -
shell> ln -s full-path-to-mysql-VERSION-OS mysql
shell> cd mysql
shell> chown -R mysql .
shell> chgrp -R mysql .
shell> scripts/mysql_install_db --user=mysql
shell> chown -R root .
shell> chown -R mysql data
shell> bin/mysqld_safe --user=mysql &

Installing PHP

Download PHP from http://www.php.net/downloads.php. Extract to a directory and run the configure command below.

'./configure' '--prefix=/usr/local' '--enable-mbstring' '--with-apxs2=/usr/local/apache2/bin/apxs' '--with-mysql=shared' '--with-config-file-path=/usr/local/apache2/conf' '--with-gd' '--with-pear' '--with-gettext' '--with-mime-magic' '--with-zlib-dir=/opt/freeware/lib' '--enable-shared' '--disable-static' '--with-zlib' '--with-bz2' '--with-jpeg-dir=/opt/freeware/lib' '--with-png-dir=/opt/freeware/lib' '--with-xpm-dir=/opt/freeware/lib' '--with-iconv'

Finally gmake install from the same directory.

Installing RRDTool

Download RRDTool from http://oss.oetiker.ch/rrdtool/download.en.html.

Follow the build instructions at http://oss.oetiker.ch/rrdtool/doc/rrdbuild.en.html

Installing Performance Export Tool

The key to this solution is the performance export tool which is not really open source and cannot be posted here owing to export restrictions on the encryption algorithms used. Essentially it is a java program which uses RMI to connect to the SVP on the storage array. Since it uses a digitial certificate to connect to the SVP it is governed by export regulations. However if your company owns a HDS storage array i.e USP/USPV/NSC55/USPVM you can find it on the microcode CD in the tools directory. If you own a 9900V then you have to ask your HDS account team to order it for you.

Extract the archive to a directory and edit rununix.sh to use JRE 1.4

#! /bin/sh
/usr/HDVM/agent/JRE1.4/jre/bin/java -classpath "./lib/JSanExport.jar:./lib/JSanRmiServerEx.jar" -Xmx268435456 -Dmd.command=command.txt -Dmd.logpath=log sanproject.getmondat.RJMdMain

The metrics that you get off the array are specified in command.txt that is in the same directory as rununix.sh. See below for a sample command.txt. Ensure that you create a read only account on the array since the userid and password are in plain text.

[root@trapeze]"/usr/local/apache2/htdocs/scripts"# more command.txt
svpip 10.2.125.12
login user "user"
show

group Port
group PG
group LDEV
group PhyPG
group PhyProc
group PhyCSW
group RemoteCopy
group RCLU
group RCLDEV

shortrange -0001:
longrange -000015:
outpath "/usr/local/apache2/htdocs/cacti/scripts/out"
option compress
apply

Now copy the following scripts to the /usr/local/apache2/htdocs/cacti/scripts directory

[root@trapeze]"/usr/local/apache2/htdocs/scripts"# more getdata.ksh
#!/usr/bin/ksh
perl /usr/local/apache2/htdocs/cacti/scripts/getNSC55data.pl

[root@trapeze]"/usr/local/apache2/htdocs/scripts"# more getNSC55data.pl
#!/usr/bin/perl
system ("touch /usr/local/apache2/htdocs/cacti/scripts/.cronlock");
system ("rm -rf /usr/local/apache2/htdocs/cacti/scripts/out");
system ("rm -rf /usr/local/apache2/htdocs/cacti/scripts/PhyPG_dat");
system ("rm -rf /usr/local/apache2/htdocs/cacti/scripts/PhyCSW_dat");
system ("rm -rf /usr/local/apache2/htdocs/cacti/scripts/PhyProc_dat");
system ("rm -rf /usr/local/apache2/htdocs/cacti/scripts/RemoteCopy_dat");
system ("/usr/local/apache2/htdocs/cacti/scripts/runUnix.bat ");
system ("unzip -o /usr/local/apache2/htdocs/cacti/scripts/out/PhyPG_dat.ZIP -d /usr/local/apache2/htdocs/scripts/");
system ("unzip -o /usr/local/apache2/htdocs/cacti/scripts/out/PhyProc_dat.ZIP -d /usr/local/apache2/htdocs/scripts/");
system ("unzip -o /usr/local/apache2/htdocs/cacti/scripts/out/PhyCSW_dat.ZIP -d /usr/local/apache2/htdocs/scripts/");
system ("unzip -o /usr/local/apache2/htdocs/cacti/scripts/out/RemoteCopy_dat.ZIP -d /usr/local/apache2/htdocs/scripts/");
system ("rm -rf /usr/local/apache2/htdocs/cacti/scripts/.cronlock");
[root@trapeze]"/usr/local/apache2/htdocs/scripts"#

Edit your crontab as below

[root@trapeze]"/usr/local/apache2/htdocs/scripts"# crontab -l
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /usr/local/apache2/htdocs/cacti/scripts/getdata.ksh

Installing the Data Gathering Scripts

Copy the two scripts below to the cacti/scripts directory

 [root@trapeze]"/usr/local/apache2/htdocs/scripts"# more getrecord.ksh
#!/usr/bin/ksh

perl /usr/local/apache2/htdocs/scripts/getNSC55record.pl $1 $2
[root@trapeze]"/usr/local/apache2/htdocs/scripts"# more getNSC55record.pl
#!/usr/bin/perl

my $group = $ARGV[0];
my $resource = $ARGV[1];

my $BASEDIR = "/usr/local/apache2/htdocs/scripts/".$group."_dat/";
my $file = undef;
my $FILENAME = undef;

while ( -e "/usr/local/apache2/htdocs/scripts/.cronlock") {
        sleep 5;
}

if (($group =~ m/PhyCSW/g ) && ($resource =~ m/CLPR/g )) { $file = "PHY_Short_Write_Pending_Rate.csv"; $FILENAME = $BASEDIR.$file; }
if ($group =~ m/PhyPG/g ) { $file = "PHY_Short_PG.csv"; $FILENAME = $BASEDIR.$file; }
if (($group =~ m/PhyProc/g ) && ($resource =~ m/CHP/g )) { $file = "PHY_Short_CHP.csv"; $FILENAME = $BASEDIR.$file; }
if ($group =~ m/RemoteCopy/g ) { $file = "RemoteCopy.csv"; $FILENAME = $BASEDIR.$file; }

my $lineheader = `cat $FILENAME | grep $resource`;
my $line = `tail -1 $FILENAME`;
my @linearray = split(",", $line);
my @lineheaderarray = split(",", $lineheader);
my $counter = 0;
foreach (@lineheaderarray) {
        if ( $_ =~ m/$resource/g ) { print $linearray[$counter]; goto endprog; }
        else { $counter = $counter + 1 ; }
}
endprog:
[root@trapeze]"/usr/local/apache2/htdocs/scripts"#

Import the Data and Graph templates

Download and import the Data and Graph Templates from here

Click the link below to see some examples