[Orca-users] RE: Multiple hosts (fwd)

[Blair Zajac]

Forwarding email discussions regarding orca data client and server for
public discussions.

Best,
Blair

-------- Original Message --------
Subject: RE: [orca-discuss] Multiple hosts (fwd)
Date: Tue, 16 Jul 2002 14:19:56 -0700 (PDT)
From: Aaron Kushner <akushner at bitmover.com>
To: "Berezin, Dmitry" <dberezin at surfside.rutgers.edu>
CC: Blair Zajac <blair at orcaware.com>

Hello Dmitry,

All very good points and I appreciate the time you spent on your
writeup. I've thought about many of these issues and will comment
on them below.

On Tue, 16 Jul 2002, Berezin, Dmitry wrote:

> 1. It would be good to add some "protocol" to conversation between client
> and server:
>       They should exchange with some greetings to identify themselves.

Was considering this approach, but didn't think the overhead (small as
it is) was necessary. I was considering making the server look like a
webserver and send the data as a post. In fact, now that I think of it,
the server side could really be a webserver running a CGI that handles
the data. However, I was initially planning on running the server in
a chrooted environment and just wanted something simple, but then I
found that I wouldn't have root access on the server machine anyway...

(Speaking of which, I've created an Apache solaris package that runs
in a chrooted environment if you are interested)

>
>       They should exchange their versions, in case in the future, there
> will be different versions and some of them might be incompatible.
>
>       The client should have a chance to announce its name, since it might
> be coming through some firewalls with NAT, etc., and it could be impossible
> to resolve its name.

Good point. By using the ip name, I was trying not to trust arbitrary
names that might be part of the protocol exchange. Things like:

      host../../etc/stomp

came to mind (hence the chroot'ed environment).

>       The client should have a chance to announce the file name to use.

> Notice, that orcallator.se will change the file to <file-name>.001,
> <file-name>.002, etc, if the header changes or orcallator.se is restarted.
> In which case, unless you address filename issues, you might end up with
> incorrect data on the server side.

I was willing to deal with a days worth of corrupted data.

>
> 2. It would be good to handle the case when server daemon is not available.
> Client should then store all data in some file (queue it up), and send it to
> the server when it becomes available.

Thought of this, but this has lots of problems that would need
significant time to work out. Say that server is down for two
days. When the server comes back up, the client would have to then
send the saved data and the server would then have to place it in the
correct file names (so the exchange would have to include extra info
or the server would have to parse the data and figure out how to name
the file.) Obviously, not insurmountable, but how valuable is this?

I was considering just saving the data to disk and manually merging it
into the server. After doing that once, might do the work of making the
server smarter... I'm beginning to like the idea of using wget/curl
on the client side and apache on the server to do the transport. Can
you see the problem with that approach? However, Blair is not crazy
about execing programs within orcallator.se.

OK, just reread this post -- I can see why using apache is not a good
idea -- CGIs are fairly CPU intensive and having 100 machines executing
the CGI at the same time and invoking Perl or Python wouldn't be real
smart. I know there is mod_perl, but I am trying to limit dependcies,
not create them.

>
> 3. Server should handle file compression similar to orcallator.se. For
> instance, it could keep the list of all current filenames for each known
> client and compress them, when filename for a client changes.

Was planning on having a cronjob that ran after midnight that
compressed the previous days files. Why make the server more
complicated...

>
> 4. Default port, 1500, is not a very good choice since it is likely to be
> used by some other applications. Oracle uses 1521 as its default listener
> port, but DBAs often change it to something else in the 15xx range. I'd use
> something higher, in 10xxx for example.

Good point.

>
> 5. If I run orcaReceive -l <two other options>, for example ./orcaReceive -l
> -d -v, I'd get "Segmentation Fault (core dumped)".

Will fix.

>
> I don't know how much of these you have already addressed in your new
> version, but I thought I'd share my thoughts with you. Unfortunately, I
> don't have a lot of time to do all this myself, since I've just started a
> new job, but I'd be glad to help if I can. If this is too much, I mean it
> falls out of the scope of your needs, just ignore my comments. In any case,
> as I said it is a good start, and something like this would be very useful
> for orca implementations.
>
> Thank you.
>
>  -Dmitry.
>

Once again, thanks for the feedback. As it stands now, is this program
useful for you?

Regards,
Aaron

> -----Original Message-----
> From: Aaron Kushner [mailto:akushner at bitmover.com]
> Sent: Monday, July 15, 2002 1:49 PM
> To: Berezin, Dmitry
> Subject: RE: [orca-discuss] Multiple hosts (fwd)
>
>
> By the way, I created a Solaris package that only puts the necessary
> client files on the machines that need to be monitored.
>
> If you want to make your own package, install epm
> (http://www.easysw.com/epm/)
> and then update your orca bitkeeper repository. From the repo, type
> 'bk pull' to get the epm config files to make the client solaris package.
> In the orcaT/src dir, the makefile has a 'make pkg' rule. Use that to
> create the package.
>
>
> (I haven't uploaded my latest changeset to orca.bkbits.net -- will let you
> know when that is done)
>
> pkgadd -d ./Orca_Client.pkg
>
>
> ## Installing part 1 of 1.
> /etc/init.d/orcallator
> /etc/rc2.d/S99orcallator <symbolic link>
> /opt/orca/bin/orcaSend
> /opt/orca/bin/restart_orcallator
> /opt/orca/bin/start_orcallator
> /opt/orca/bin/stop_orcallator
> /opt/orca/lib/orcallator.se
> [ verifying class <none> ]
>
>
>
>
> ---------- Forwarded message ----------
> Date: Mon, 15 Jul 2002 09:14:19 -0700 (PDT)
> From: Aaron Kushner <akushner at bitmover.com>
> To: "Berezin, Dmitry" <dberezin at surfside.rutgers.edu>
> Subject: RE: [orca-discuss] Multiple hosts
>
> Hello Dmitry,
>
> Here is the deal. I've created a simple client server app that sends
> the orcallator data to a central machine. The orca network server
> daemon saves the sent data to an orcallator-date file in a directory
> named for the host sending the data.
>
> I modified the orcallator.se and start_orcallator scripts to use my
> orcaSend program. In the start_orcallator file, if the ORCA_REPOS is
> set to some host, the orcallator.se file will do something like:
>
>       echo "orcallator data" | orcaSend orca_server_hostname
>
>
> On the orca_server_hostname, there is orcaReceive running that saves
> the data into the same hierarchy that the orcallator.se program would
> have done. So I didn't need to modify the orca perl program at all.
>
> Does this sound like something you can use? I've been using it for
> a couple of weeks internally and it seems to work. I'm not a real
> competent C programmer so I'm sure that the program can be improved.
>
> I've used bitkeeper source management for this program, so if you
> install bitkeeper, you can just do a:
>
>       bk clone http://orca.bkbits.net/orca orca
>
> to get the latest version of the project. My stuff is in the
> contrib/orcaT directory. I'm in the middle of writing docs and
> such.
>
> Regards,
> Aaron
>
>
> On Mon, 15 Jul 2002, Berezin, Dmitry wrote:
>
> >  Aaron,
> > 
> > Blair Zajac had mentioned it to me that you are working on setting up a
> > daemon for Orca to collect Orcallator data from other hosts. I wonder if
> you
> > could share some details of this project.
> > 
> > Thank you.
> > 
> > Dmitry Berezin.
> > 
> >  -----Original Message-----
> > From: Blair Zajac [mailto:blair at orcaware.com]
> > Sent: Friday, July 12, 2002 2:13 PM
> > To: orca-discuss at yahoogroups.com
> > Cc: akushner at bitmover.com
> > Subject: Re: [orca-discuss] Multiple hosts
> >
> >
> > dmitryb17 wrote:
> > >
> > > What is the correct way to send data that orcallator collects from
> > > multiple hosts to a centralized location? Just use NFS?
> >
> > There is no one correct way.  It depends mostly on your security
> > issues.  I used NFS and had no problems with it, but if you can't
> > use NFS, then you end up using rsync over ssh to get the files to
> > the Orca host.
> >
> > >
> > > Is there a built-in functionality to send new data to the host that
> > > creates graphs via some sort of daemon that runs on it and listens
> > > for incoming connections with new data? (BTW, is this a topic for the
> > > orca-developer group?)
> >
> > Yes, this is on topic.
> >
> > In fact Aaron Kushner is working on this right now.  You can probably
> > ask him for a copy of the code.  At some point, I'd like to see this
> > code incorporated into the Orca distribution.
> >
> > Best,
> > Blair
> >
> >
>