[Orca-dev] Procallator Serial Number overwriting

[C. L. Villani]

One issue we have had is that while we're testing, we
tend to stop and start procallator a lot.  Normally, this
isn't a problem -- procallator just looks in
   /var/log/orcaware/procallator/machinename.caltech.edu/
and iterates through the files until it finds an empty
-XXX location, at which point it assigns a new serial
number.

This is the code as it exists:
    # on first execution check file existence
    if ( !$rate_ok ) {
        while ( -f $out_filename[$r] ) {
            $num++;
            $out_filename[$r] = sprintf "%s/proccol-%04d-%02d-%02d-%03d",
              $DEST_DIR, $year, $mon, $mday, $num;
        }
    }

Which works fine if the files are:
-rw-rw-rw-   1 sams sams 172334 Aug  3 14:50 proccol-2005-08-03-000
-rw-rw-rw-   1 sams sams   7221 Aug  3 15:05 proccol-2005-08-03-001
-rw-rw-rw-   1 sams sams  39025 Aug  3 18:05 proccol-2005-08-03-002

However, occasionally, procallator would compress files, and then we
would stop and start, and we'd have:
-rw-rw-rw-   1 sams sams 172334 Aug  3 14:50 proccol-2005-08-03-000.bz2
-rw-rw-rw-   1 sams sams   7221 Aug  3 15:05 proccol-2005-08-03-001.bz2
-rw-rw-rw-   1 sams sams  39025 Aug  3 18:05 proccol-2005-08-03-002

At which point, procallator would then create a new "proccol-2005-08-03-001"
file.  At midnight, or when the number of columns changed, proccolator
would then try to compress the file, and complain about the existence
of proccol-2005-08-03-001.bz2 already.  It would then not compress the
file.  Sort of annoying, but no data lost -- yet.  Until it happened a
third time in the same day, at which point we'd start losing data.

Right now, I have the following kludge which prevents this from 
happening:
    # on first execution check file existence
    if ( !$rate_ok ) {
        while ( -f $out_filename[$r] || -f "$out_filename[$r].bz2") {
            $num++;
            $out_filename[$r] = sprintf "%s/proccol-%04d-%02d-%02d-%03d",
              $DEST_DIR, $year, $mon, $mday, $num;
        }
    }

But I thought it would be nice for folks -- and actually allow
me to make a useful contribution to the software -- if I could 
solve the general case (and without doing something ugly like 
making an index or having people specify their compressor extension 
on startup).

I tried this:
    # on first execution check file existence
    if ( !$rate_ok ) {
        while ( glob($out_filename[$r] ) {
            $num++;
            $out_filename[$r] = sprintf "%s/proccol-%04d-%02d-%02d-%03d",
              $DEST_DIR, $year, $mon, $mday, $num;
        }
    }

Which I thought would work, but for some reason that's not picking
up the extensions.

Any suggestions?

Thanks,

Connie-Lynne
clynne at ofb.net