[Orca-users] Re: High Sleeps on mutex rate during heavy oracle activity

[Adam Levin]

On Thu, 8 Aug 2002, Dave Lowenstein wrote:
> I'm having a hard time figuring out exactly what the sleeps on mutex rating
> corresponds to and how to look at it/tune it. just using se toolkit's
> zoom.se I don't see any warnings, but on this particular oracle server orca
> is reporting 80-95 sleeps on mutex pretty much steadily throughout a big
> import.
> Am I overreacting?

Probably.

How many CPUs do you have?  Is in 80-95 systemwide, or per CPU?  Either
way, until you're up in the hundreds *per CPU*, you have nothing to worry
about.

You can use lockstat to check on more lock details, and mpstat can be
somewhat useful too.

Here's some detail from Cockroft's book.  Typos are all mine:
---BEGIN EXCERPT---
One key capability in shared memory, multiprocessor systems is the ability
to perform interprocessor synchronization by means of atomic load/store or
swap instructions.  All SPARC chips have an instruction called LDSTUB,
which means load-store-unsigned-byte.  The instruction reads a byte from
memory into a register, then writes 0xFF into memory in a single,
indivisible operation.  The value in the register can then be examined to
see if it was already 0xFF, which means that another processor got there
first, or if it was 0x00, which means that this processor is in charge.
This instruction is used to make mutual exclusion locks (known as mutexes)
that make sure only one processor at a time can hold the lock.  The lock
is acquired through LDSTUB and cleared by storing 0x00 back to the memory.

If a processor does not get the lock, then it may decide to spin by
sitting in a loop and testing the lock until it becomes available.  By
checking with a normal load instruction in a loop before issuing an
LDSTUB, the processor performs the spin within the cache, and the bus
snooping logic watches for the lock being cleared.  In this way, spinning
causes no bus traffic, so processors that are waiting do not slow down
those that are working.  A spin lock is appropriate when the wait is
expected to be short.  If a long wait is expected, the process should
sleep for a while so that a different job can be scheduled onto the CPU.
The extra SPARC V9 instructions introduced with UltraSPARC include an
atomic compare-and-swap operation, which is used by the
UltraSPARC-specific version of the kernel to make locking more efficient.

...

In Solaris 2.6 [the book does not cover Solaris 8, but this is useful
anyway], the lockstat(1) command offers a kernel lock monitoring
capability that is very powerful and easy to use.  The kernel can
dynamically change its locks to start collecting information and return to
the normarl, highly optimized lock code when the collection is complete.
There is some extra overhead while the locks are instrumented, but useful
measurements can be taken in a few seconds without disruption of the work
on production machines.  Data is presented clearly with several options
for how it is summarized, but you still must have a very good
understanding of Unix kernel architecture and Solaris 2, in particular, to
really understand what is happening.

You should read the lockstat(1) manual page to see all its options, but in
normal operation, it is run by the root user for the duration of a process
for which you want to monitor the effects.  If you want to run it for an
interval, use the sleep command.  I had trouble finding an example that
would cause lock contention in Solaris 2.6.  Eventually, I found that
putting a C shell into an infinite loop, as shown below, generates 120,000
or more system calls per second duplicating and shutting file descriptors.
When two shells are run at the same time on a dual CPU system, mpstat
shows that hundreds of mutex spins/second occur on each CPU.  This test is
not intended to be serious!
---END EXCERPT---

-Adam

Adam Levin, Senior Unix Systems Administrator | http://www.audible.com/
Audible, Inc.
Wayne, NJ, 07470             Those who are willing to give up essential
973-837-2797                    liberties for security deserve neither.