[Orca-users] Threads

[Hudes, Dana]

The current behavior is that we end up with many bunzip2 processes.  One for each bz2 file that needs processing. The result is that on a 16GB v490 with 8 1.8Mhz cores I have about 190 bunzip2s running. More than somewhere around 30-35 monitored hosts and orca server process quits, loggin "out of memory!". That there is 64GB swap untouched doesn't matter. Limits don't apply its running as root

If the orca server host falls behind or is rebooted one gets this flood of bunzip2s running on files previously processed. 
This seems counterintuitive for cases where a period of time was fully processed (no back-filling orcallator data). 



----- Original Message -----
From: Blair Zajac <blair at orcaware.com>
To: Hudes, Dana
Cc: orca-users at orcaware.com <orca-users at orcaware.com>
Sent: Sun Aug 30 22:05:13 2009
Subject: Re: [Orca-users] Threads

I don't think we need threads for this.  The main loop in Orca is  
single threaded, so when it goes to read from a data file, it'll use  
use the Compress::Bzip2 instance for that file.

Plus, we probably don't want as many threads as there are hosts being  
monitored, that would easily kill a box.

Regards,
Blair

On Aug 30, 2009, at 6:55 PM, Hudes, Dana wrote:

> Blair: I will work on that this week
> Note that threads all share common stack so you need a lot more  
> stack space than default. I'll put in some code to check for / deal  
> with that
>
>
>
> ----- Original Message -----
> From: Blair Zajac <blair at orcaware.com>
> To: Hudes, Dana
> Cc: orca-users at orcaware.com <orca-users at orcaware.com>
> Sent: Sun Aug 30 21:28:18 2009
> Subject: Re: [Orca-users] Threads
>
>
> On Aug 30, 2009, at 6:14 PM, Hudes, Dana wrote:
>
> > Has anyone tried replacing the spawning of bunzip2 processes with
> > threads using Compress::Bzip2?
> > If so any feedback?
> >
> > Blair: you probably chose not to do this because the solaris-
> > distributed Perl is not threaded. Is that correct?
> >
>
> IIRC, at the time there wasn't a Compress::Bzip2 module.  Having patch
> to use it would be nice instead of forking bzunip2's.
>
> Blair
>
>
> _______________________________________________
> Orca-users mailing list
> Orca-users at orcaware.com
> http://www.orcaware.com/mailman/listinfo/orca-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/orca-users/attachments/20090830/f6bfcb03/attachment.html>