[Orca-users] Possible memory leak in orcallator?

[Stoltzfus, Mark (HQP)]

I recently upgraded many of our systems to orcallator.se r404, running a
combination of SE Toolkit versions 3.4, 3.3.1, and 3.2.1, depending on
Solaris version (we've got 2.6 through 9 here).  I noticed a 2GB
orcallator process on one of our 6800s running Solaris 2.8 and SE 3.4
this morning:

$ ps -eofname,vsz | grep se.sparc
se.sparc 2061840
$

This has been running about a month.  A pmap reveals a bulging heap:

$ pmap 11018
11018:  /opt/RICHPse/bin/se.sparcv9 -I/opt/RHIorca/lib/SE/3.4 -DWATCH_OS
/opt/
0000000100000000    296K read/exec         /opt/RICHPse/bin/se.sparcv9
0000000100148000     24K read/write/exec   /opt/RICHPse/bin/se.sparcv9
000000010014E000 2059280K read/write/exec     [ heap ]
FFFFFFFF7DA00000     32K read/exec         /usr/lib/sparcv9/libaio.so.1
FFFFFFFF7DB08000      8K read/write/exec   /usr/lib/sparcv9/libaio.so.1
FFFFFFFF7DC00000     24K read/exec         /usr/lib/sparcv9/librt.so.1
FFFFFFFF7DD06000      8K read/write/exec   /usr/lib/sparcv9/librt.so.1
FFFFFFFF7DE00000      8K read/write/exec     [ anon ]
FFFFFFFF7DF00000      8K read/write/exec     [ anon ]
FFFFFFFF7E000000     16K read/exec         /usr/lib/sparcv9/libmp.so.2
FFFFFFFF7E104000      8K read/write/exec   /usr/lib/sparcv9/libmp.so.2
FFFFFFFF7E200000    712K read/exec         /usr/lib/sparcv9/libc.so.1
FFFFFFFF7E3B2000     56K read/write/exec   /usr/lib/sparcv9/libc.so.1
FFFFFFFF7E3C0000      8K read/write/exec   /usr/lib/sparcv9/libc.so.1
FFFFFFFF7E400000      8K read/write/exec     [ anon ]
FFFFFFFF7E500000    664K read/exec         /usr/lib/sparcv9/libnsl.so.1
FFFFFFFF7E6A6000     56K read/write/exec   /usr/lib/sparcv9/libnsl.so.1
FFFFFFFF7E6B4000     40K read/write/exec   /usr/lib/sparcv9/libnsl.so.1
FFFFFFFF7E700000     56K read/exec
/usr/lib/sparcv9/libsocket.so.1
FFFFFFFF7E80E000     16K read/write/exec
/usr/lib/sparcv9/libsocket.so.1
FFFFFFFF7E900000     88K read/exec         /usr/lib/sparcv9/libm.so.1
FFFFFFFF7EA16000      8K read/write/exec   /usr/lib/sparcv9/libm.so.1
FFFFFFFF7EB00000     32K read/exec         /usr/lib/sparcv9/libgen.so.1
FFFFFFFF7EC08000      8K read/write/exec   /usr/lib/sparcv9/libgen.so.1
FFFFFFFF7ED00000    112K read/exec         /usr/lib/sparcv9/libelf.so.1
FFFFFFFF7EE1C000      8K read/write/exec   /usr/lib/sparcv9/libelf.so.1
FFFFFFFF7EF00000      8K read/exec
/usr/platform/sun4u-us3/lib/sparcv9/libc_psr.so.1
FFFFFFFF7F000000      8K read/write/exec     [ anon ]
FFFFFFFF7F100000      8K read/exec
/usr/lib/sparcv9/libkstat.so.1
FFFFFFFF7F202000      8K read/write/exec
/usr/lib/sparcv9/libkstat.so.1
FFFFFFFF7F300000     16K read/exec         /usr/lib/sparcv9/libkvm.so.1
FFFFFFFF7F404000      8K read/write/exec   /usr/lib/sparcv9/libkvm.so.1
FFFFFFFF7F500000      8K read/exec         /usr/lib/sparcv9/libdl.so.1
FFFFFFFF7F600000    152K read/exec         /usr/lib/sparcv9/ld.so.1
FFFFFFFF7F724000     16K read/write/exec   /usr/lib/sparcv9/ld.so.1
FFFFFFFF7F728000      8K read/write/exec   /usr/lib/sparcv9/ld.so.1
FFFFFFFF7FFEC000     80K read/write          [ stack ]
         total  2061904K
$

This doesn't seem to be unique to SE 3.4, however.  I've also got an
orcallator process on a Solaris 7 host running SE Toolkit 3.3.1 that's
climbed to a 219MB, and one on a Solaris 2.6 host running SE Toolkit
3.2.1 that's up over 50MB (which is big for that box, it's an Ultra 2).
Restarting orcallator trims the size substantially (for example, on the
6800, it went down to 148MB.  It appears to grow slowly, i.e. not with
every sample iteration.  However, whatever issue that may exist is
probably exacerbated by the fact that I've set the SAMPLE_INTERVAL to
60.

Has anyone else seen this/know what's going on?

Thanks,

Mark