[Orca-users] Re: Orcallator CPU Questions

[Adam Levin]

On Wed, 14 Aug 2002, Gary M.Blumenstein wrote:
> I'm trying to better understand how Orcallator and SEtoolkit determines
> CPU utilization on a multiprocessor Sparc system.  First here's a little
> background...

Don't forget that CPU utilization is only one factor in the system
performance area, and in fact just because a system shows a lot of CPU
util. does not necessarily mean that something bad is going on.  Load
average is generally a better indicator of how busy a system is, but in
your specific case, I can see where CPU util. would be important.

Since you say that watching mpstat shows close to 100% utilization when
the image processor is running, it's possible that the system would
benefit from more CPUs for the guy to get his images finished faster.
More likely, though, based on what you're saying, *faster* CPUs, and not
necessarily more of them, would help.

As for Orca, I believe what you see on the graphs is a five minute
*average*.  It's also important to note that RRD is not able to show
*every* data point, so some points are dropped.  The graphs represent
average trends, not exact calculations.

The first thing you should do is look at the five minute numbers in the
orca data files themselves.  They may show occasional peaks where the
graphs don't, simply because of the way RRD handles the data.

You may also want to lower your sampling to one minute intervals for a day
and see what that shows you.  I think that RRD is still going to average
out the data for the graphs, though.

The graphs provide a good overview of what the system is doing, but you
have to look at the numbers to get the hard details.  In a 16 CPU system,
it wouldn't surprise me to see a lot of idle time if there is only one CPU
intensive application running (even if that app is multi-threaded or
multi-processor enabled).

> Based on mpstat, the programmer thinks we're running our Sparc E6500
> system at full-bore during image processing and we would see that if we
> decreased Orcallator's sample interval.  In the past he has made the case
> to management that his application is very CPU intensive and thus requires
> massive amounts of hardware to run.

I think he would probably benefit from upgrading to UltraSPARC III
processors, especially for image processing.  He'd probably get about
three-four times the performance out of a 750MHz or higher USIII than you
get currently out of your USII (I'm guessing you've got the 400 or
450MHz).  Nevertheless, if the rest of the system is idle for a good
amount of time in between image processes, then it seems like he shouldn't
really be complaining.  2-6 mins isn't too bad, unless you're doing
thousands of images.

> My theory is that we're looking at two related but seperate things.  The
> near real-time output from mpstat does indeed show instances where all 16
> CPUs sustain high peak loads.  However, the results from Orcallator shows
> the actual workload for the past 5 minutes.  We're not "missing" data as
> the developer is suggesting but rather Orcallator's histogram is based on
> the total capacity of the machine and that includes all available CPU
> cycles for the entire sampling period.  OTOH, mpstat shows the
> instantaneous load and like the Heisenberg uncertainty principle, the CPUs
> may be in a different state while you're watching it's output.

I agree, for the most part.  Orca is indeed showing an overall view of
system resources, while mpstat is an instantaneous picture of the system
at that moment.  I don't think the measurement is really affecting the
CPUs, though -- mpstat goes right to the kernel metrics, and shouldn't be
affecting performance all that much, just like Orca and the SE Toolkit.

It's hard to gauge utilization on a big MP system when you're only running
one application.  That app is going to monopolize the system whenever it
can, but if it isn't running constantly (like, say, a database), then can
you really justify more hardware?  I'm surprised the image processor app
is using all 16 CPUs at once.  I'd be more inclined to buy a two- or
four-way machine with bigger processors and more memory rather than using
a 6500 for this, but I don't know what your application is capable of.

-Adam

Adam Levin, Senior Unix Systems Administrator | http://www.audible.com/
Audible, Inc.     
Wayne, NJ, 07470        Bigjobs!
973-837-2797