[Orca-users] Re: orca speed continued

[bias16]

I didn't get any responses to my latest posting on orca speed, but did
come up with the following from reading past posting on the issue and
direct responses on previous queries I have made.  

I have about 200 servers that were taking few hours to get all the
maps updated and am now down to a matter of few minutes depending on
when orca updates recieved.

I still get a ton a errors with files being outta date and and other
hoopla, but the graphs looks wonderful.  As I update all clients to
orcallator.se 1.32 I expect many errors to go away.

1) All clients rsync orcallator data to central server via rsyncd
every 5 minutes.  Most clients are still running old version of
orcallator.se (1.23?), but I'm gradually updating them to 1.32.

2) Central server is (T1400, 4CPU, 4Gig memory).  Since I have enough
memory, I create a directory /tmp/orca/orcallator and /tmp/orca/rrd to
store data files.  Web pages (graphs) are stored on striped veritas
volume.

3) I adjusted the orcallator.cfg script to only print the graphs for
things that we care about.   For example, we don't use nfs or web
stats, so I take those out.  Also - with all the volumes we have, a
large number of disk use percentage graphs are printed, so I take
those out since not very helpful to us.

4) When initially bringing up orca on systems, I place all historic
data in orcallator directory and run global orca script on everything
to create graphs, html, and initial rrd files.  This take a while...
8-10 hours.  We really like the comparism graphs for all systems, so
there is no way to "split" this up until I take a look at orca perl
script and rewrite to just generate html (no rrd, no png).  I may also
look at running this "full" orca periodically to keep html up to date
as new systems automatically added.

5) After single run of orca for all files, I break it up in multiple
instances based on what logically makes since for our environment.  In
our case, 5 processes.   To do this, I create a separate .cfg for each
instance changing that orca.state name to orca-1.state, etc. and
find_files reverence to something like following:

/tmp/orca/orcallator/(a.*|b.*)/(?:(?:orcallator)|(?:percol))-\d{4}-\d{2}-\d{2}(?:\.(?:Z|gz))?

I change the .* in search to a.*|b.* and removed the |bz2 from file
ending search.  As long as upto date, I don't need it stat'ing all
those bz2 files.

6) I run the individual instances of orca in -daemon mode with the
-no-html flag.  I want them to continuous check for updates and I
don't want to mess up html charts... just the .png graphs.

I'm still working on seemless way to handle rare orca sever reboots
and, even though bzip2 orcallator historic data is small, I need to
handle automatic moving of that from /tmp to disk to preserve space.

I've also noticed that updating the orca server to 0.27b and
orcallator.se 1.32 has caused quarterly and yearly graphs to only be
generated once a day.  I don't know if I did this or it was automatic,
but it is a nice change.

This is AWESOME product and has proven extremely useful for
montitoring system.  Thanks Blair!

Thanks,
Liston