[Orca-users] Interpreting kernel contention results

[Sean O'Neill]

Lester, Mike wrote:
> I am trying to drill down into what kernel contention is and how I 
> improve the situation.  Orca is monitoring an Oracle database server 
> running Solaris 9 with over 50 databases on it and I consistently see 
> Kernel contention on the System Overview Chart above 1.  The databases 
> are in a development environment and the vast majority are idle.  They 
> are accessed on an ad-hoc basis and are not all in use at one time. 
>  
> First of all, is this a problem or just the way a database server 
> looks?  If I understand the description of kernel contention, this graph 
> indicates multiple processors are trying to access the same data.  More 
> processors would seem to make the problem worse.  Does anyone have any 
> words of wisdom for this behavior? 
>  

 From http://www.sun.com/950901/columns/adrian/column1.html:

The Kernel Contention Rule
The kernel uses mutual exclusion locks to synchronize its operation and 
keep multiple CPUs from concurrently accessing critical code and data 
regions. Each time a CPU has to wait for a lock to become free it wastes 
CPU power and this event is counted.

Table 8 Mutex Rule

CPU Rule			Level	Action
------------------------------------------------------
mpstat30.smtx < 200		Green	No Problem
200 <= mpstat30.smtx < 400	Amber	1. Mutex Stall
400 <= mpstat30.smtx		Red	1. Mutex Stall
------------------------------------------------------

    1. Mutex Stall
       If the number of stalls per CPU per second exceeds the limit, 
mutex contention is happening in the kernel, which wastes CPU time and 
degrades multiprocessor scaling.

Asking a question like this usually doesn't get much of a response 
without more information.

What is your CPU doing ?
Memory Problem ?
What type of server ?
How many CPUs ?
How much memory ?
I/O Configuration ?
   - JBOD ?
   - SAN ?
   - NAS / DAS ?
   - # of controllers and type (type of SCSI or type of SAN [1Gb / 2Gb])
   - Is backend storage intelligent e.g. Hitachi, IBM Shark, etc ?
Storage Configuration ?
   - VxVM / SVM (using Soft partitions in SVM ??) ?
   - Raw or Cooked FSs ?
   - Is your volume strip consistent with your backend storage ?  (If 
appropriate that is)
   - If cooked, is your file system maxcontig tuned to be consistent 
with your volume/backend strip config ?
Oracle Configuration ?
   - How big is SGA ?