[Orca-dev] Orcallator.se changes and additions

[Blair Zajac]

Dmitry Berezin wrote:

> I would like to suggest a couple of changes to the orcallator.se script 
> as well as make some additions to it.

Dmitry,

That would be great.  I think you should send me patches to fix and make 
the additions you want to.

> 1. “Fix” timestamp.
> 
>             Timestamps that orcallator.se uses for each data entry do 
> not perfectly lineup with the data gathering interval and could be off 
> by a couple of seconds.

Where do you see this happening.  On slow systems?  On large systems 
with many things to measure?

 > In case when the data represents a whole
> quantities (such as number of http processes or number of database 
> connections) this creates erroneous results (30.998 database connections 
> for instance). This problem could be avoided by fixing the timestamp to 
> the interval boundary:   
> 
>     now    = time(0);
> 
>     now = (now/interval)*interval;
> 
> I agree that “fixing” any data is not a very good idea, but since the 
> change in the value of each timestamp will be small, I do not think that 
> this will have any significant effect on data averaging.

Recent versions of orcallator.se try to avoid this problem to get now to 
  be exactly an integer multiple of the interval.  What version are you 
working with?

Also, are you using the web log processing code?  I could see this 
having an impact.

> 2. Problem with compressing data files.
> 
>             There are two places in orcallator.se where it calls 
> “compress” program defined in the environment. Both times it is executed 
> on the background. If the header information changes and log file needs 
> to be switched, orcallator.se first closes that log file and calls 
> “compress”, then goes to search for the new available log file name, 
> compressing all uncompressed files that it finds. The problem is that 
> the log file that was just closed is still being compressed on the 
> background at this time, so orcallator.se calls another “compress” for 
> the same file. Obviously, one of these calls fails. I suggest changing 
> orcallator.se, so that when it closes a log file, it compresses it on 
> the foreground.

Hmmm, I'd rather keep that still in the background.

How about adding a second test, where orcallator.se will check to see if 
  the compressed fileame exists, and if it does, it'll assume that the 
compression program is still running.

Patch welcome for this.

> 3. Problem with proc_next().
> 
>             It appears that first_proc() / next_proc() combination 
> doesn’t always traverse through all of the processes on the system. 
> Every 10^th -20^th time it stops somewhere in the middle. I have ran 
> some tests on Solaris 8 with SE 3.3.1 that consistently confirmed that 
> behavior. Here are the two loops that go through all processes – the 
> second one seems to be consistent (orcallator.se uses the first one).
> 
>  
> 
> *******************************************************
> 
>   for (p=first_proc(); p.pr_pid != -1; p=next_proc()) {
> 
>   }
> 
> *******************************************************
> 
>   proc_class_t pp;
> 
>   while (pp.index$ != -1) {
> 
>     refresh$(pp);
> 
>   }
> 
> *******************************************************

Do you get the pr_fname member function on the proc_class_t?  As it is 
used here:

   for (p=first_proc(); p.pr_pid != -1; p=next_proc()) {
     for (i=0; i<number_regexs; ++i) {
       if (p.pr_fname =~ regexs[i]) {
         ++count_procs_results[i];
       }
     }
   }


> These three are minor changes and I have already implemented and tested 
> them on my server. Blair, I can send the diffs for all/any of these if 
> you would like to merge them into the official release.

Yes, send them along.  They would be great additions to orcallator.se.


> 4. The next topic is Disk Statistics.
> 
>             The RAWDISK code appears to be the biggest problem for the 
> new installations of Orca, especially on the newer hardware.

Agreed.

 > It also
> appears that it DOES NOT bring any advantages, since it was written for 
> a very specific situation, and the original code handles everything well 
> enough. I would like to know if anybody is still running orcallator.se 
> on the servers where RAWDISK code is necessary, and if not, it would 
> probably be wise to remove it (or at least disable it by default). I am 
> also going to write a patch for tape (st) statistics collection that 
> will not depend on RAWDISK code being enabled and post it here after 
> testing it.

Let's wait for some feedback here, especially from Alan, who wrote the code.

> 5. Process count.
> 
>             Orcallator.se provides the ability for counting http and 
> https processes, but in a lot of cases there might be a need to count 
> some other types of processes (oracle database connections, for 
> example). It would probably be better to have a generic function that 
> would count “any” processes based on some initialization variables.
> 
> Blair, is this of any interest. If yes, I could work on this… However, 
> see the next topic

Yes, a general way of doing this would be good.

> (I have also written a patch that counts Oracle database connections for 
> all Oracle databases, dynamically finding all instances; but now I am 
> not sure if this is of any value to anybody.)

There have been requests for Oracle data before, so I would say yes.

> 6. Workload statistics.
> 
>             I have written a new class based on the workload class that 
> now comes with SE Toolkit. The new class is more “orca friendly” and 
> allows for easier initialization (IMHO). I have also written a patch for 
> orcallator.se to use this new class to collect workload statistics. 
> Coincidentally, the new class provides the ability to get process counts 
> (based on all/any of the following: program name, program arguments, 
> user name, processor set binding) as well, so it could be used to count 
> the number of http processes, oracle processes, etc.
> 
> Blair, I can send this for review. Also, I need some help with graphing 
> the data.

Be glad to help.

After reviewing several of the patches, let's get you Subversion access 
so you can get these changes into Orca/orcallator.se.

Best,
Blair