[Orca-users] ongoing problem with Orca master abend after filling memory

[Glen Gunselman]

(Note: The server is:
                     uname -a
                     SunOS fozzie 5.10 Generic_118833-17 sun4u sparc SUNW,Sun-Fire-V490)

I seem to have an ongoing problem with Orca master filling memory and then abending.

(Note: I deleted the Orca graph showing memory usage as the post was rejected because it was larger than 40K.)


This has been happening for almost a year about every 2 weeks:

grep 'dumped core' /var/svc/log/network-csworca:default.log
[ Jul 11 10:05:40 Stopping because process dumped core. ]
[ Oct 27 05:15:42 Stopping because process dumped core. ]
[ Nov  9 08:57:33 Stopping because process dumped core. ]
[ Nov 22 07:05:39 Stopping because process dumped core. ]
[ Dec  5 08:00:42 Stopping because process dumped core. ]
[ Dec 18 12:10:39 Stopping because process dumped core. ]
[ Dec 31 16:15:38 Stopping because process dumped core. ]
[ Jan 17 11:00:43 Stopping because process dumped core. ]
[ Jan 31 13:45:32 Stopping because process dumped core. ]
[ Feb 13 12:40:36 Stopping because process dumped core. ]
[ Feb 26 08:40:40 Stopping because process dumped core. ]
[ Mar 11 17:42:37 Stopping because process dumped core. ]
[ Apr  4 10:14:12 Stopping because process dumped core. ]
[ Apr 17 13:10:42 Stopping because process dumped core. ]
[ Apr 30 11:25:37 Stopping because process dumped core. ]
[ May 13 07:10:40 Stopping because process dumped core. ]
[ May 26 06:25:38 Stopping because process dumped core. ]
[ Jun  8 18:22:26 Stopping because process dumped core. ]

Here is a complete set of log messages for the last two failures:

[ May 26 06:25:38 Stopping because process dumped core. ]
[ May 26 06:25:39 Executing stop method ("/var/opt/csw/svc/method/svc-csworca stop") ]
/var/opt/csw/svc/method/svc-csworca: kill: no such process
[ May 26 06:25:39 Method "stop" exited with status 0 ]
[ May 26 06:25:41 Executing start method ("/var/opt/csw/svc/method/svc-csworca start") ]
[ May 26 06:25:41 Method "start" exited with status 0 ]
Version string '1.05 ' contains invalid data; ignoring: ' ' at /opt/csw/bin/orca line 66.
[ Jun  8 18:22:26 Stopping because process dumped core. ]
[ Jun  8 18:22:27 Executing stop method ("/var/opt/csw/svc/method/svc-csworca stop") ]
/var/opt/csw/svc/method/svc-csworca: kill: no such process
[ Jun  8 18:22:27 Method "stop" exited with status 0 ]
[ Jun  8 18:22:29 Executing start method ("/var/opt/csw/svc/method/svc-csworca start") ]
[ Jun  8 18:22:29 Method "start" exited with status 0 ]
Version string '1.05 ' contains invalid data; ignoring: ' ' at /opt/csw/bin/orca line 66.

I believe the problem started after I installed CSWhobbit and CSWhobbitc:


pkginfo -l CSWhobbit CSWhobbitc
   PKGINST:  CSWhobbit
      NAME:  hobbit - Hobbit Monitoring Server
  CATEGORY:  application
      ARCH:  sparc
   VERSION:  4.2.0,REV=2007.04.12
   BASEDIR:  /
    VENDOR:  http://hobbitmon.sf.net/ packaged for CSW by Mike Arnold
      DESC:  Hobbit is a system for monitoring servers and networks. (Server)
    PSTAMP:  solaris8-sparc64.razorsedge.org20070412151442
  INSTDATE:  Jun 23 2010 14:34
   HOTLINE:  http://www.opencsw.org/bugtrack/
     EMAIL:  mike at opencsw.org
    STATUS:  completely installed
     FILES:      346 installed pathnames
                  17 shared pathnames
                  50 directories
                  78 executables
                   1 setuid/setgid executables
               13858 blocks used (approx)

   PKGINST:  CSWhobbitc
      NAME:  hobbit_client - Hobbit Monitoring Client
  CATEGORY:  application
      ARCH:  sparc
   VERSION:  4.2.0,REV=2007.04.12
   BASEDIR:  /
    VENDOR:  http://hobbitmon.sf.net/ packaged for CSW by Mike Arnold
      DESC:  Hobbit is a system for monitoring servers and networks. (Client)
    PSTAMP:  solaris8-sparc64.razorsedge.org20070412151516
  INSTDATE:  Jun 23 2010 14:41
   HOTLINE:  http://www.opencsw.org/bugtrack/
     EMAIL:  mike at opencsw.org
    STATUS:  completely installed
     FILES:       54 installed pathnames
                  10 shared pathnames
                  15 directories
                  24 executables
                1355 blocks used (approx)

pkgutil -c    shows Orca and Hobbit are current:
...
CSWhobbit                 4.2.0,REV=2007.04.12      SAME
CSWhobbitc                4.2.0,REV=2007.04.12      SAME
...
CSWorca                   snapshot_r535,REV=2010.08.04 SAME
CSWorcaweb                snapshot_r535,REV=2010.08.04 SAME



Thanks for any clues,
GlenG
(Note: Cross posted to users at lists.opencsw.org on 6/14/2011 but no help as of 6/20/2011.)