From mixal at paconet.us Sat Aug 8 01:56:21 2009 From: mixal at paconet.us (Allen Eastwood) Date: Sat, 8 Aug 2009 03:56:21 -0500 Subject: [Orca-users] Orca dying on the vine? Message-ID: So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. If dead, what are folks using out there for replacement? Thanks! -A -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Sat Aug 8 19:23:45 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Sat, 8 Aug 2009 22:23:45 -0400 Subject: [Orca-users] Orca dying on the vine? Message-ID: I still use orca. I need some work to rebuild and make it work under sol 10 in container and memory is a problem with bunzip1 but I have that source and will recompile for native code and autopar with studio 12 After that config to get zfs and other sol 10. Kstats. ________________________________ From: orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com To: orca-users at orcaware.com Sent: Sat Aug 08 04:56:21 2009 Subject: [Orca-users] Orca dying on the vine? So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. If dead, what are folks using out there for replacement? Thanks! -A -------------- next part -------------- An HTML attachment was scrubbed... URL: From mixal at paconet.us Mon Aug 10 15:28:52 2009 From: mixal at paconet.us (Allen Eastwood) Date: Mon, 10 Aug 2009 17:28:52 -0500 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: <4A8053D4.CC54.005D.3@emporia.edu> References: <4A8053D4.CC54.005D.3@emporia.edu> Message-ID: I've run across some issues with older revs of Orca, and those seem to be resolved with the latest svn snapshot (R535). If you notice the se process crashing, then that's teh way to go. I've got a tar/install script that installs orca, SE, and the packages needed for support (I use Sunfreeware). It's not pretty, but it seems to be working. If anyone wants to try it out: http://www.mediafire.com/?wzka51zwjjg It's also got my older installation for Solaris 8 and 9, I'm not bothering to update those. It does have support for sparc and x86. Pretty much, as root: untar the file, run the install script in it. I have the web server stuff turned off in the orcallator.cfg as I use this for general purpose monitoring. Also, I place 2 cron entries in root's crontab - one to clean up files over 1 year old, and the other runs orca hourly to update the html. The se data collection runs all the time, but I don't like clogging up my servers with the perl processing running continuously. If you want to put the html files on a central host, I used to just do a quick shell script to copy /opt/orca/html over to :// and another script to create a quick & dirty index.html where each hostname directory is a hyperlink. Sunday I messed around with trying to compile the latest SE from svn, but it kept crashing...and I'm not up to figuring that out after working my way through getting the new Orca going. Blair...if you're still around and doing any development, and need access to a sparc or CMT box, let me know. I should be able to hook you up. Regards, -A On Mon, Aug 10, 2009 at 17:07, Glen Gunselman wrote: > Allen, > > I'm still hoping to use ORCA with Solaris 10. I have it installed using > OpenCSW. > > My thought is that system admins today are not as interested in what's > going in their servers. > > > > > > > Glen Gunselman > Systems Software Specialist > TCS > Emporia State University > > >>> Allen Eastwood 08/08/09 3:56 AM >>> > > So, with the rather low level of activity both on the mailing lists and > development for Orca and SE, have we gotten to the point where Orca/SE are > pretty much dead? They've been great tools over the years, and I haven't > seen anything that gives the same level of insight. > > If dead, what are folks using out there for replacement? > > Thanks! > > -A > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ggunselm at emporia.edu Mon Aug 10 15:07:32 2009 From: ggunselm at emporia.edu (Glen Gunselman) Date: Mon, 10 Aug 2009 17:07:32 -0500 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: References: Message-ID: <4A8053D4.CC54.005D.3@emporia.edu> Allen, I'm still hoping to use ORCA with Solaris 10. I have it installed using OpenCSW. My thought is that system admins today are not as interested in what's going in their servers. Glen Gunselman Systems Software Specialist TCS Emporia State University >>> Allen Eastwood 08/08/09 3:56 AM >>> So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. If dead, what are folks using out there for replacement? Thanks! -A -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Mon Aug 10 16:16:29 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Mon, 10 Aug 2009 19:16:29 -0400 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: <4A8053D4.CC54.005D.3@emporia.edu> References: <4A8053D4.CC54.005D.3@emporia.edu> Message-ID: Interested yes obsessed no. Throw enough hardware at the problem and it goes away: 2 core PIV server slow? Throw 4x quad core Xeon at it. 4GB not enough? Try 16 GB. this isn't new. I've consolidated numerous systems which were using 10-20% of an E10K domain (4GB, 4x400mhz Ultra II cpu) into a single 5240 (128 x 1.4Mhz sun4v more-or-less ultraIII+ cores, 32 GB RAM). ________________________________ From: orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com [mailto:orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com] On Behalf Of Glen Gunselman Sent: Monday, August 10, 2009 6:08 PM To: orca-users at orcaware.com; Allen Eastwood Subject: Re: [Orca-users] Orca dying on the vine? Allen, I'm still hoping to use ORCA with Solaris 10. I have it installed using OpenCSW. My thought is that system admins today are not as interested in what's going in their servers. Glen Gunselman Systems Software Specialist TCS Emporia State University >>> Allen Eastwood 08/08/09 3:56 AM >>> So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. If dead, what are folks using out there for replacement? Thanks! -A -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.stoltzfus at pmigroup.com Mon Aug 10 16:51:33 2009 From: mark.stoltzfus at pmigroup.com (Mark Stoltzfus) Date: Mon, 10 Aug 2009 16:51:33 -0700 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: <4A8053D4.CC54.005D.3@emporia.edu> References: <4A8053D4.CC54.005D.3@emporia.edu> Message-ID: <77CAAFB7C7CE3843A83557DB8857A3AC02F36515@PMI-HMBX1.us.corp.pmigroup.com> We have ~100 Solaris 10 (06/06) systems here running Orca 0.28b523 with RICHPse 3.4.1 and it works fine. Granted, it's mostly older hardware (USIII) so I can't speak to compatibility with newer hardware releases, but it works. My $.02, Mark ________________________________ From: orca-users-bounces+mark.stoltzfus=pmigroup.com at orcaware.com [mailto:orca-users-bounces+mark.stoltzfus=pmigroup.com at orcaware.com] On Behalf Of Glen Gunselman Sent: Monday, August 10, 2009 3:08 PM To: orca-users at orcaware.com; Allen Eastwood Subject: Re: [Orca-users] Orca dying on the vine? Allen, I'm still hoping to use ORCA with Solaris 10. I have it installed using OpenCSW. My thought is that system admins today are not as interested in what's going in their servers. Glen Gunselman Systems Software Specialist TCS Emporia State University >>> Allen Eastwood 08/08/09 3:56 AM >>> So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. If dead, what are folks using out there for replacement? Thanks! -A -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Mon Aug 10 18:27:03 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Mon, 10 Aug 2009 21:27:03 -0400 Subject: [Orca-users] Orca dying on the vine? Message-ID: I am using orcallator on a variety of systems. All RICHPse 3.5. Orca-server is in a container on a v490 under sol 10 5/09 patched to latest. Built with sun studio 12 and perl 5.8.8 (compile with studio 12 from coolstack release) The bunzip24 go wild. After 150 of them or so all 8gb of RAM is gone. My thinking is recompile bunzip2 from source use 64!it native (not sparcv8 as shipped) and tuen on autopar. I looked at pbunzip but its docs sayit won't help decompress perf with bzips compressed with the regular. The hope is to run each bunzip faster so that we don't reach critical mass. Host doing it has 2 4gbit FC ports to Hitachi 9990 6+1 raid 15l rpm disks. Of couse those drives have other apps on them but still the Hitachi has more capacity to deliver than a mere 8 gbit can pull ________________________________ From: orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com To: Glen Gunselman ; orca-users at orcaware.com ; Allen Eastwood Sent: Mon Aug 10 19:51:33 2009 Subject: Re: [Orca-users] Orca dying on the vine? We have ~100 Solaris 10 (06/06) systems here running Orca 0.28b523 with RICHPse 3.4.1 and it works fine. Granted, it?s mostly older hardware (USIII) so I can?t speak to compatibility with newer hardware releases, but it works. My $.02, Mark ________________________________ From: orca-users-bounces+mark.stoltzfus=pmigroup.com at orcaware.com [mailto:orca-users-bounces+mark.stoltzfus=pmigroup.com at orcaware.com] On Behalf Of Glen Gunselman Sent: Monday, August 10, 2009 3:08 PM To: orca-users at orcaware.com; Allen Eastwood Subject: Re: [Orca-users] Orca dying on the vine? Allen, I'm still hoping to use ORCA with Solaris 10. I have it installed using OpenCSW. My thought is that system admins today are not as interested in what's going in their servers. Glen Gunselman Systems Software Specialist TCS Emporia State University >>> Allen Eastwood 08/08/09 3:56 AM >>> So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. If dead, what are folks using out there for replacement? Thanks! -A -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mick.Sheppard at cpp.co.uk Tue Aug 11 01:04:06 2009 From: Mick.Sheppard at cpp.co.uk (Mick Sheppard) Date: Tue, 11 Aug 2009 09:04:06 +0100 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: References: Message-ID: <6999E2BC89CF664781139A8F9A289A720DE98DB7@MERCURY.cpp-group.com> Hi, I've just been through installing and configuring Orca at my latest place of work. Whilst there is no work on SE, though the source is available should anyone want to, and Orcallator is pretty stable, I'm writing a lot of custom collectors to monitor things that aren't otherwise easily visible. Mick ________________________________ From: orca-users-bounces+mick.sheppard=cpp.co.uk at orcaware.com [mailto:orca-users-bounces+mick.sheppard=cpp.co.uk at orcaware.com] On Behalf Of Allen Eastwood Sent: 08 August 2009 09:56 To: orca-users at orcaware.com Subject: [Orca-users] Orca dying on the vine? So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. If dead, what are folks using out there for replacement? Thanks! -A This is an email from the CPP Group Plc, Holgate Park, York, YO26 4GA; telephone +44 (0)1904 544500. This message may contain information that is confidential. If you are not the intended recipient, you may not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify the sender immediately by email, facsimile or telephone and either return or destroy the original message. The CPP Group Plc accepts no responsibility for any changes made to this message after it has been sent by the original author. This email has been scanned for all viruses by the MessageLabs Email Security System. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mixal at paconet.us Tue Aug 11 12:08:43 2009 From: mixal at paconet.us (Allen Eastwood) Date: Tue, 11 Aug 2009 14:08:43 -0500 Subject: [Orca-users] Orca-users Digest, Vol 78, Issue 4 In-Reply-To: References: Message-ID: Mick, Any chance you'd be willing to share what you've been adding? Regards, -A From: "Mick Sheppard" > To: > Date: Tue, 11 Aug 2009 09:04:06 +0100 > Subject: Re: [Orca-users] Orca dying on the vine? > > Hi, > > > > I?ve just been through installing and configuring Orca at my latest place > of work. Whilst there is no work on SE, though the source is available > should anyone want to, and Orcallator is pretty stable, I?m writing a lot of > custom collectors to monitor things that aren?t otherwise easily visible. > > > > Mick > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mick.Sheppard at cpp.co.uk Wed Aug 12 01:04:02 2009 From: Mick.Sheppard at cpp.co.uk (Mick Sheppard) Date: Wed, 12 Aug 2009 09:04:02 +0100 Subject: [Orca-users] Orca-users Digest, Vol 78, Issue 4 In-Reply-To: References: Message-ID: <6999E2BC89CF664781139A8F9A289A720DE993B9@MERCURY.cpp-group.com> Allen, If I write anything that's generic I'll be sure to share them. However the current ones are specific to custom application metric we have at our organisation. As such they wouldn't be relevant to others. Mick ________________________________ From: orca-users-bounces+mick.sheppard=cpp.co.uk at orcaware.com [mailto:orca-users-bounces+mick.sheppard=cpp.co.uk at orcaware.com] On Behalf Of Allen Eastwood Sent: 11 August 2009 20:09 To: orca-users at orcaware.com Subject: Re: [Orca-users] Orca-users Digest, Vol 78, Issue 4 Mick, Any chance you'd be willing to share what you've been adding? Regards, -A From: "Mick Sheppard" To: Date: Tue, 11 Aug 2009 09:04:06 +0100 Subject: Re: [Orca-users] Orca dying on the vine? Hi, I've just been through installing and configuring Orca at my latest place of work. Whilst there is no work on SE, though the source is available should anyone want to, and Orcallator is pretty stable, I'm writing a lot of custom collectors to monitor things that aren't otherwise easily visible. Mick This is an email from the CPP Group Plc, Holgate Park, York, YO26 4GA; telephone +44 (0)1904 544500. This message may contain information that is confidential. If you are not the intended recipient, you may not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify the sender immediately by email, facsimile or telephone and either return or destroy the original message. The CPP Group Plc accepts no responsibility for any changes made to this message after it has been sent by the original author. This email has been scanned for all viruses by the MessageLabs Email Security System. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Fri Aug 14 09:00:21 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Fri, 14 Aug 2009 12:00:21 -0400 Subject: [Orca-users] Interface speed assumptions Message-ID: I've got the newer version of Orca (can't seem to find the version number in any of the files). In the orcallator.cfg used for the server, I see in the network interface computations the assumption that a gigabit-capable interface is in fact running at gigabit speed. I have systems where we don't have enough copper gigabit switch ports and therefore the interface is running 100 FDX -- and in fact I have one now, and this maddingly happens because of our network group, which has autonegotiate mismatched and therefore is at 100HDX (I have had other, 100Mbit interfaces such as hme where they screw up autonegotiate and I end up with 10HDX). How do I work around this assumption? We DO have ce interfaces, both old Sun gigaswift fiber cards and newer copper gigabit ports, which are running at gigabit speed. Also missing from the list seems to be nxge, which are the ports on all our T5240s. Do I just add that to the regex below or do I have to also update the orcallator.cfg used by RICHPse? # Interface bits per second for 1 Gbit interfaces. plot { title %g Interface Bits Per Second: $1 source orcallator data 1024 * 8 * ((?:(?:bge)|(?:ce)|(?:fjg[ei])|(?:v?ge)|(?:skge)|(?:e1000g)|(?:ipge)|(?: bnx))\d+)InKB/s data 1024 * 8 * $1OuKB/s line_type area line_type line1 legend Input legend Output y_legend Bits/s data_min 0 data_max 1000000000 href http://www.orcaware.com/orca/docs/orcallator.html#interface_bits_per_sec ond } -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Fri Aug 14 09:03:39 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Fri, 14 Aug 2009 12:03:39 -0400 Subject: [Orca-users] zfs Message-ID: How do I add collection and plotting for zfs information? Starting with the various kstat data such as the ARC. Collecting and plotting ZFS filesystem compression ratios is desirable as well. That comes from zfs list command output. I need to double-check I suspect the raw data for zpool iostat is available in kstat but not sure, there are a lot of kstats.... -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragon at raytheon.com Fri Aug 21 09:45:56 2009 From: dragon at raytheon.com (David Michaels) Date: Fri, 21 Aug 2009 10:45:56 -0600 Subject: [Orca-users] Interface speed assumptions In-Reply-To: References: Message-ID: <4A8ECF44.8070207@raytheon.com> On 8/14/2009 10:00 AM, Hudes, Dana wrote: > I've got the newer version of Orca (can't seem to find the version > number in any of the files). In the orcallator.cfg used for the > server, I see in the network interface computations the assumption > that a gigabit-capable interface is in fact running at gigabit speed. > I have systems where we don't have enough copper gigabit switch ports > and therefore the interface is running 100 FDX -- and in fact I have > one now, and this maddingly happens because of our network group, > which has autonegotiate mismatched and therefore is at 100HDX (I have > had other, 100Mbit interfaces such as hme where they screw up > autonegotiate and I end up with 10HDX). > How do I work around this assumption? We DO have ce interfaces, both > old Sun gigaswift fiber cards and newer copper gigabit ports, which > are running at gigabit speed. That's a tricky one. I suppose you could take out the data_max line in those graphs, and they'll be computed based on whatever the highest value is for that bracket of time, but that's not useful for cross-the-board comparisons. I would say that rather than change how your data is presented, you should fix your autonegotiation problems. You paid for 1gb, you should be getting 1gb. ;) We had that problem in our Sun shop also -- Cisco and Sun don't seem to want to autonegotiate ports correctly sometimes, especially when dealing with fiber (I recall some issue with pause frames). Another option is to create a special group and plot stanza for those hosts with the problem, and set data_max to 100mb. That can all be done in orcallator.cfg, but it results in something rather messy. A third option is to leave data_max alone, but fake out the bit rate of afflicted machines by multiplying it by 10. That puts them all at the same scale as the other working servers. > Also missing from the list seems to be nxge, which are the ports on > all our T5240s. Do I just add that to the regex below or do I have to > also update the orcallator.cfg used by RICHPse? Yes, just add a regex. Be careful to follow the same pattern as the other regex's -- parens and |'s and whatnot. I think in your case it would look like this: data 1024 * 8 * ((?:*(?:nxge)*|(?:bge)|(?:ce)|(?:fjg[ei])|(?:v?ge)|(?:skge)|(?:e1000g)|(?:ipge)|(?:bnx))\d+)InKB/s I'd modify both files. The Orca server needs to know how to present the data, and the RICHPse running on the T5240s needs to know to collect the data. Though I didn't think RICHPse even used orcallator.cfg... I thought it only used orcallator.se. O.o > # Interface bits per second for 1 Gbit interfaces. > plot { > title %g Interface Bits Per Second: $1 > source orcallator > data 1024 * 8 * > ((?:(?:bge)|(?:ce)|(?:fjg[ei])|(?:v?ge)|(?:skge)|(?:e1000g)|(?:ipge)|(?:bnx))\d+)InKB/s > data 1024 * 8 * $1OuKB/s > line_type area > line_type line1 > legend Input > legend Output > y_legend Bits/s > data_min 0 > data_max 1000000000 > href > http://www.orcaware.com/orca/docs/orcallator.html#interface_bits_per_second > } > -- --Dragon "draco nemo omnibus horis sapit" -------------- next part -------------- An HTML attachment was scrubbed... URL: From dragon at raytheon.com Fri Aug 21 10:00:46 2009 From: dragon at raytheon.com (David Michaels) Date: Fri, 21 Aug 2009 11:00:46 -0600 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: References: <4A8053D4.CC54.005D.3@emporia.edu> Message-ID: <4A8ED2BE.3010306@raytheon.com> On 8/10/2009 5:16 PM, Hudes, Dana wrote: > Interested yes obsessed no. > Throw enough hardware at the problem and it goes away: 2 core PIV > server slow? Throw 4x quad core Xeon at it. > 4GB not enough? Try 16 GB. Obviously not all places have money to throw. But in any case, it's not what I use Orca for, or at least, not exclusively. I'm not the obsessive type to try to squeeze out the next 1% of performance boost by over-analyzing my data, but I will use Orca to help debug strange problems. It may be because we're not in an ops/service environment but rather a development environment, and our servers are frequently hammered in strange ways by overzealous programmers. Being able to point at a graph and say, "your code was opening a bazillion TCP ports" is fantastic. Being able to point at a graph and say, "we just need more memory, not more CPU" is also good (though less common), because it avoids the problem you cited of "2 core PIV server slow? throw 4x quad core Xeon at it" -- that solution wouldn't be helpful if the slowness is actually caused by memory rather than CPU. While there are ways to determine such resource shortages without Orca, Orca makes it very easy -- so easy that even a manager can see it. ;) --Dragon > this isn't new. I've consolidated numerous systems which were using > 10-20% of an E10K domain (4GB, 4x400mhz Ultra II cpu) into a single > 5240 (128 x 1.4Mhz sun4v more-or-less ultraIII+ cores, 32 GB RAM). > From blair at orcaware.com Sat Aug 29 22:33:12 2009 From: blair at orcaware.com (Blair Zajac) Date: Sat, 29 Aug 2009 22:33:12 -0700 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: References: Message-ID: <34362727-6F17-4F55-989F-6F3029CBC860@orcaware.com> On Aug 8, 2009, at 1:56 AM, Allen Eastwood wrote: > So, with the rather low level of activity both on the mailing lists > and development for Orca and SE, have we gotten to the point where > Orca/SE are pretty much dead? They've been great tools over the > years, and I haven't seen anything that gives the same level of > insight. I've always felt that Orca targets sysadmin people more than developers so doesn't see as many contributions as compared to an open- source package that targets developers. Orca is GPL and I do take patches for it. So there's nothing stopping people from contributing. It could be a good idea to move it to github so that people can make use of github's forking model that makes it easier for people to work on their copy of the code and I can pull those changes into my version of Orca for official releases. I'm always looking for people who want to get involved and contribute code. I've given commit access to a few people to make it better. So the more the merrier. Regards, Blair From hudesd at hra.nyc.gov Sun Aug 30 03:20:46 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Sun, 30 Aug 2009 06:20:46 -0400 Subject: [Orca-users] Orca dying on the vine? Message-ID: If you would move to using mercurial instead of git I think you might like it. Hosting of project at btibucket.org Mercurial has a very active community with many add-on tools Thus far my modifications are to orcallator.cfg and orcallator.se I was considering trying my hand at modifying to use use threads and the Compres'Bzip2 library instead of forking off bunzip2 child processes in the hope that might make orca scale better. If you've tried that and found it unsatisfactory, I'd like to know. ----- Original Message ----- From: orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com To: Allen Eastwood Cc: orca-users at orcaware.com Sent: Sun Aug 30 01:33:12 2009 Subject: Re: [Orca-users] Orca dying on the vine? On Aug 8, 2009, at 1:56 AM, Allen Eastwood wrote: > So, with the rather low level of activity both on the mailing lists > and development for Orca and SE, have we gotten to the point where > Orca/SE are pretty much dead? They've been great tools over the > years, and I haven't seen anything that gives the same level of > insight. I've always felt that Orca targets sysadmin people more than developers so doesn't see as many contributions as compared to an open- source package that targets developers. Orca is GPL and I do take patches for it. So there's nothing stopping people from contributing. It could be a good idea to move it to github so that people can make use of github's forking model that makes it easier for people to work on their copy of the code and I can pull those changes into my version of Orca for official releases. I'm always looking for people who want to get involved and contribute code. I've given commit access to a few people to make it better. So the more the merrier. Regards, Blair _______________________________________________ Orca-users mailing list Orca-users at orcaware.com http://www.orcaware.com/mailman/listinfo/orca-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From blair at orcaware.com Sun Aug 30 13:51:45 2009 From: blair at orcaware.com (Blair Zajac) Date: Sun, 30 Aug 2009 13:51:45 -0700 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: References: Message-ID: <4A9AE661.9000101@orcaware.com> I have forked projects at github: http://github.com/blair and a Mercurial one at http://code.google.com/p/scala-migrations/ So I don't think I would have another one at btibucket.org. I do like the stacked git (stg) interface for dealing with patches better than Mercurial Queues, which I have been unable to see a way to reorder patches. Blair Hudes, Dana wrote: > If you would move to using mercurial instead of git I think you might > like it. Hosting of project at btibucket.org > > Mercurial has a very active community with many add-on tools > > Thus far my modifications are to orcallator.cfg and orcallator.se > > I was considering trying my hand at modifying to use use threads and the > Compres'Bzip2 library instead of forking off bunzip2 child processes in > the hope that might make orca scale better. If you've tried that and > found it unsatisfactory, I'd like to know. > > > ----- Original Message ----- > From: orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com > > To: Allen Eastwood > Cc: orca-users at orcaware.com > Sent: Sun Aug 30 01:33:12 2009 > Subject: Re: [Orca-users] Orca dying on the vine? > > > On Aug 8, 2009, at 1:56 AM, Allen Eastwood wrote: > > > So, with the rather low level of activity both on the mailing lists > > and development for Orca and SE, have we gotten to the point where > > Orca/SE are pretty much dead? They've been great tools over the > > years, and I haven't seen anything that gives the same level of > > insight. > > I've always felt that Orca targets sysadmin people more than > developers so doesn't see as many contributions as compared to an open- > source package that targets developers. > > Orca is GPL and I do take patches for it. So there's nothing stopping > people from contributing. > > It could be a good idea to move it to github so that people can make > use of github's forking model that makes it easier for people to work > on their copy of the code and I can pull those changes into my version > of Orca for official releases. > > I'm always looking for people who want to get involved and contribute > code. I've given commit access to a few people to make it better. So > the more the merrier. > > Regards, > Blair From mixal at paconet.us Sun Aug 30 14:55:09 2009 From: mixal at paconet.us (Allen Eastwood) Date: Sun, 30 Aug 2009 16:55:09 -0500 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: <34362727-6F17-4F55-989F-6F3029CBC860@orcaware.com> References: <34362727-6F17-4F55-989F-6F3029CBC860@orcaware.com> Message-ID: Hey Blair, I'm glad to see your response. Being a sysadmin type myself, I agree totally with what you are saying on the targeting side. Unfortunately, that also means that I'm not able to contribute much on the writing code side of things. However, I can provide some solaris hosts for a few people, both x86 (virtual) and sun4u. I can do testing also on sun4v. At some point, I'm thinking of getting a IBM workstation and when I do, then AIX becomes a possibility also. I would like to figure out some better packaging for Orca and the dependent packages, and I'm willing to contribute any packages I create. I've used git, mercurial and svn, and I have no issues either way, whatever works best for the developer types. ;-p Regards, -A On Sun, Aug 30, 2009 at 00:33, Blair Zajac wrote: > > On Aug 8, 2009, at 1:56 AM, Allen Eastwood wrote: > > So, with the rather low level of activity both on the mailing lists and >> development for Orca and SE, have we gotten to the point where Orca/SE are >> pretty much dead? They've been great tools over the years, and I haven't >> seen anything that gives the same level of insight. >> > > I've always felt that Orca targets sysadmin people more than developers so > doesn't see as many contributions as compared to an open-source package that > targets developers. > > Orca is GPL and I do take patches for it. So there's nothing stopping > people from contributing. > > It could be a good idea to move it to github so that people can make use of > github's forking model that makes it easier for people to work on their copy > of the code and I can pull those changes into my version of Orca for > official releases. > > I'm always looking for people who want to get involved and contribute code. > I've given commit access to a few people to make it better. So the more > the merrier. > > Regards, > Blair > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mixal at paconet.us Sun Aug 30 15:37:35 2009 From: mixal at paconet.us (Allen Eastwood) Date: Sun, 30 Aug 2009 17:37:35 -0500 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: <957C12E0-8063-4C8D-B48B-C14E841FF9E8@baltic-online.de> References: <34362727-6F17-4F55-989F-6F3029CBC860@orcaware.com> <957C12E0-8063-4C8D-B48B-C14E841FF9E8@baltic-online.de> Message-ID: I'm a sunfreeware user myself. But the CSW packages are also good. It's been a while since looked at CSW, so it maybe time to re-evaluate. The reason I prefer sunfreeware is mostly habit and I prefer to install straight packages via a jumpstart install, and I prefer the /usr/local install location. That being said, there's nothing wrong with the other as well. One reason I'd like to see more packaging is that often I need performance data from a client's servers and it's much easier if I can get them a fairly contained package and an install script. As much as I hate the bloat that comes from installing multiple versions of some packages, a self-contained packaging that doesn't change what else is on the server is sometimes needed. The other thing is that usually, I'm not dealing with a lot of web servers, so I tend to turn those graphs off. In the long run, it would be good to have that more easily configurable with some kind of post install scripting. Maybe some kind of pre-defined server classes (web server, DB server, etc?). Any reason you haven't done the SMF integration? I figure if I can at least get the se collecting data, then I can always grab that data and analyze it later. Typically, I put in a cron job to run orca to generate the rrd's and html hourly. In the past, keeping orca doing that continuously had a larger impact on the systems. -A On Sun, Aug 30, 2009 at 17:16, Dagobert Michelsen wrote: > Hi Allen, > > Am 30.08.2009 um 23:55 schrieb Allen Eastwood: > >> I would like to figure out some better packaging for Orca and the >> dependent packages, and I'm willing to contribute any packages I create. >> > > I have a preliminary version of Orca available at > > which currently lacks SMF/RC integration but should be > usable otherwise. There is also an official SE Toolkit > available from OpenCSW which I also maintain, feedback > is very welcome on these packages! > > > Best regards > > -- Dago > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Sun Aug 30 15:54:03 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Sun, 30 Aug 2009 18:54:03 -0400 Subject: [Orca-users] Orca dying on the vine? Message-ID: Sure -- the mercurial one at google should work; bitbicket was a suggestion made ibecause I did not. Know of the other one . I will give a go tomorrow at work at setting up a local repository from the google so I have one to work from First I need to build perl 5.10.1 and see if that improves performance over 5.8.8 ----- Original Message ----- From: Blair Zajac To: Hudes, Dana Cc: orca-users at orcaware.com Sent: Sun Aug 30 16:51:45 2009 Subject: Re: [Orca-users] Orca dying on the vine? I have forked projects at github: http://github.com/blair and a Mercurial one at http://code.google.com/p/scala-migrations/ So I don't think I would have another one at btibucket.org. I do like the stacked git (stg) interface for dealing with patches better than Mercurial Queues, which I have been unable to see a way to reorder patches. Blair Hudes, Dana wrote: > If you would move to using mercurial instead of git I think you might > like it. Hosting of project at btibucket.org > > Mercurial has a very active community with many add-on tools > > Thus far my modifications are to orcallator.cfg and orcallator.se > > I was considering trying my hand at modifying to use use threads and the > Compres'Bzip2 library instead of forking off bunzip2 child processes in > the hope that might make orca scale better. If you've tried that and > found it unsatisfactory, I'd like to know. > > > ----- Original Message ----- > From: orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com > > To: Allen Eastwood > Cc: orca-users at orcaware.com > Sent: Sun Aug 30 01:33:12 2009 > Subject: Re: [Orca-users] Orca dying on the vine? > > > On Aug 8, 2009, at 1:56 AM, Allen Eastwood wrote: > > > So, with the rather low level of activity both on the mailing lists > > and development for Orca and SE, have we gotten to the point where > > Orca/SE are pretty much dead? They've been great tools over the > > years, and I haven't seen anything that gives the same level of > > insight. > > I've always felt that Orca targets sysadmin people more than > developers so doesn't see as many contributions as compared to an open- > source package that targets developers. > > Orca is GPL and I do take patches for it. So there's nothing stopping > people from contributing. > > It could be a good idea to move it to github so that people can make > use of github's forking model that makes it easier for people to work > on their copy of the code and I can pull those changes into my version > of Orca for official releases. > > I'm always looking for people who want to get involved and contribute > code. I've given commit access to a few people to make it better. So > the more the merrier. > > Regards, > Blair -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Sun Aug 30 15:58:48 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Sun, 30 Aug 2009 18:58:48 -0400 Subject: [Orca-users] Orca dying on the vine? Message-ID: I have run orca server on sun4v (T5240) and sun4u (v490, v490). Since the orca server is single-threaded and spawns many bunzip2 children which are i/o bound the coolthreads many cpus doesn't help. A single Ultra IV+ chip suffices @ 1.8GHz (and if could get 2GHz I would) The T2 is 1.4GHz. . ________________________________ From: orca-users-bounces+hudesd=hra.nyc.gov at orcaware.com To: orca-users at orcaware.com Sent: Sun Aug 30 17:55:09 2009 Subject: Re: [Orca-users] Orca dying on the vine? Hey Blair, I'm glad to see your response. Being a sysadmin type myself, I agree totally with what you are saying on the targeting side. Unfortunately, that also means that I'm not able to contribute much on the writing code side of things. However, I can provide some solaris hosts for a few people, both x86 (virtual) and sun4u. I can do testing also on sun4v. At some point, I'm thinking of getting a IBM workstation and when I do, then AIX becomes a possibility also. I would like to figure out some better packaging for Orca and the dependent packages, and I'm willing to contribute any packages I create. I've used git, mercurial and svn, and I have no issues either way, whatever works best for the developer types. ;-p Regards, -A On Sun, Aug 30, 2009 at 00:33, Blair Zajac wrote: On Aug 8, 2009, at 1:56 AM, Allen Eastwood wrote: So, with the rather low level of activity both on the mailing lists and development for Orca and SE, have we gotten to the point where Orca/SE are pretty much dead? They've been great tools over the years, and I haven't seen anything that gives the same level of insight. I've always felt that Orca targets sysadmin people more than developers so doesn't see as many contributions as compared to an open-source package that targets developers. Orca is GPL and I do take patches for it. So there's nothing stopping people from contributing. It could be a good idea to move it to github so that people can make use of github's forking model that makes it easier for people to work on their copy of the code and I can pull those changes into my version of Orca for official releases. I'm always looking for people who want to get involved and contribute code. I've given commit access to a few people to make it better. So the more the merrier. Regards, Blair -------------- next part -------------- An HTML attachment was scrubbed... URL: From dam at baltic-online.de Sun Aug 30 15:16:12 2009 From: dam at baltic-online.de (Dagobert Michelsen) Date: Mon, 31 Aug 2009 00:16:12 +0200 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: References: <34362727-6F17-4F55-989F-6F3029CBC860@orcaware.com> Message-ID: <957C12E0-8063-4C8D-B48B-C14E841FF9E8@baltic-online.de> Hi Allen, Am 30.08.2009 um 23:55 schrieb Allen Eastwood: > I would like to figure out some better packaging for Orca and the > dependent packages, and I'm willing to contribute any packages I > create. I have a preliminary version of Orca available at which currently lacks SMF/RC integration but should be usable otherwise. There is also an official SE Toolkit available from OpenCSW which I also maintain, feedback is very welcome on these packages! Best regards -- Dago From cnk at caltech.edu Sun Aug 30 16:41:35 2009 From: cnk at caltech.edu (Cynthia Kiser) Date: Sun, 30 Aug 2009 16:41:35 -0700 Subject: [Orca-users] Orca dying on the vine? In-Reply-To: <957C12E0-8063-4C8D-B48B-C14E841FF9E8@baltic-online.de> References: <34362727-6F17-4F55-989F-6F3029CBC860@orcaware.com> <957C12E0-8063-4C8D-B48B-C14E841FF9E8@baltic-online.de> Message-ID: <20090830234135.GA13472@clyde.caltech.edu> Blair, thanks for creating the git and mercurial repositories. I haven't done any extending of Orcaware for a couple of years now but having the code available in version control ecosystems that expect private and public forks will make it much easier to keep track of things should I do so in the future. I am sorry to say I don't have anything to contribute back to the project. The 'extensions' I made were useful to me - but uninteresting to anyone other than the handful of people still running AOLserver. -- Cynthia Kiser cnk at caltech.edu From hudesd at hra.nyc.gov Sun Aug 30 18:14:43 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Sun, 30 Aug 2009 21:14:43 -0400 Subject: [Orca-users] Threads Message-ID: Has anyone tried replacing the spawning of bunzip2 processes with threads using Compress::Bzip2? If so any feedback? Blair: you probably chose not to do this because the solaris-distributed Perl is not threaded. Is that correct? -------------- next part -------------- An HTML attachment was scrubbed... URL: From blair at orcaware.com Sun Aug 30 18:28:18 2009 From: blair at orcaware.com (Blair Zajac) Date: Sun, 30 Aug 2009 18:28:18 -0700 Subject: [Orca-users] Threads In-Reply-To: References: Message-ID: On Aug 30, 2009, at 6:14 PM, Hudes, Dana wrote: > Has anyone tried replacing the spawning of bunzip2 processes with > threads using Compress::Bzip2? > If so any feedback? > > Blair: you probably chose not to do this because the solaris- > distributed Perl is not threaded. Is that correct? > IIRC, at the time there wasn't a Compress::Bzip2 module. Having patch to use it would be nice instead of forking bzunip2's. Blair From hudesd at hra.nyc.gov Sun Aug 30 18:55:43 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Sun, 30 Aug 2009 21:55:43 -0400 Subject: [Orca-users] Threads Message-ID: Blair: I will work on that this week Note that threads all share common stack so you need a lot more stack space than default. I'll put in some code to check for / deal with that ----- Original Message ----- From: Blair Zajac To: Hudes, Dana Cc: orca-users at orcaware.com Sent: Sun Aug 30 21:28:18 2009 Subject: Re: [Orca-users] Threads On Aug 30, 2009, at 6:14 PM, Hudes, Dana wrote: > Has anyone tried replacing the spawning of bunzip2 processes with > threads using Compress::Bzip2? > If so any feedback? > > Blair: you probably chose not to do this because the solaris- > distributed Perl is not threaded. Is that correct? > IIRC, at the time there wasn't a Compress::Bzip2 module. Having patch to use it would be nice instead of forking bzunip2's. Blair -------------- next part -------------- An HTML attachment was scrubbed... URL: From blair at orcaware.com Sun Aug 30 19:05:13 2009 From: blair at orcaware.com (Blair Zajac) Date: Sun, 30 Aug 2009 19:05:13 -0700 Subject: [Orca-users] Threads In-Reply-To: References: Message-ID: <60565503-BF07-4118-AFCF-E6E9366638D8@orcaware.com> I don't think we need threads for this. The main loop in Orca is single threaded, so when it goes to read from a data file, it'll use use the Compress::Bzip2 instance for that file. Plus, we probably don't want as many threads as there are hosts being monitored, that would easily kill a box. Regards, Blair On Aug 30, 2009, at 6:55 PM, Hudes, Dana wrote: > Blair: I will work on that this week > Note that threads all share common stack so you need a lot more > stack space than default. I'll put in some code to check for / deal > with that > > > > ----- Original Message ----- > From: Blair Zajac > To: Hudes, Dana > Cc: orca-users at orcaware.com > Sent: Sun Aug 30 21:28:18 2009 > Subject: Re: [Orca-users] Threads > > > On Aug 30, 2009, at 6:14 PM, Hudes, Dana wrote: > > > Has anyone tried replacing the spawning of bunzip2 processes with > > threads using Compress::Bzip2? > > If so any feedback? > > > > Blair: you probably chose not to do this because the solaris- > > distributed Perl is not threaded. Is that correct? > > > > IIRC, at the time there wasn't a Compress::Bzip2 module. Having patch > to use it would be nice instead of forking bzunip2's. > > Blair > > > _______________________________________________ > Orca-users mailing list > Orca-users at orcaware.com > http://www.orcaware.com/mailman/listinfo/orca-users From hudesd at hra.nyc.gov Sun Aug 30 19:15:02 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Sun, 30 Aug 2009 22:15:02 -0400 Subject: [Orca-users] Threads Message-ID: The current behavior is that we end up with many bunzip2 processes. One for each bz2 file that needs processing. The result is that on a 16GB v490 with 8 1.8Mhz cores I have about 190 bunzip2s running. More than somewhere around 30-35 monitored hosts and orca server process quits, loggin "out of memory!". That there is 64GB swap untouched doesn't matter. Limits don't apply its running as root If the orca server host falls behind or is rebooted one gets this flood of bunzip2s running on files previously processed. This seems counterintuitive for cases where a period of time was fully processed (no back-filling orcallator data). ----- Original Message ----- From: Blair Zajac To: Hudes, Dana Cc: orca-users at orcaware.com Sent: Sun Aug 30 22:05:13 2009 Subject: Re: [Orca-users] Threads I don't think we need threads for this. The main loop in Orca is single threaded, so when it goes to read from a data file, it'll use use the Compress::Bzip2 instance for that file. Plus, we probably don't want as many threads as there are hosts being monitored, that would easily kill a box. Regards, Blair On Aug 30, 2009, at 6:55 PM, Hudes, Dana wrote: > Blair: I will work on that this week > Note that threads all share common stack so you need a lot more > stack space than default. I'll put in some code to check for / deal > with that > > > > ----- Original Message ----- > From: Blair Zajac > To: Hudes, Dana > Cc: orca-users at orcaware.com > Sent: Sun Aug 30 21:28:18 2009 > Subject: Re: [Orca-users] Threads > > > On Aug 30, 2009, at 6:14 PM, Hudes, Dana wrote: > > > Has anyone tried replacing the spawning of bunzip2 processes with > > threads using Compress::Bzip2? > > If so any feedback? > > > > Blair: you probably chose not to do this because the solaris- > > distributed Perl is not threaded. Is that correct? > > > > IIRC, at the time there wasn't a Compress::Bzip2 module. Having patch > to use it would be nice instead of forking bzunip2's. > > Blair > > > _______________________________________________ > Orca-users mailing list > Orca-users at orcaware.com > http://www.orcaware.com/mailman/listinfo/orca-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From hudesd at hra.nyc.gov Mon Aug 31 07:45:14 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Mon, 31 Aug 2009 10:45:14 -0400 Subject: [Orca-users] Threads In-Reply-To: <60565503-BF07-4118-AFCF-E6E9366638D8@orcaware.com> References: <60565503-BF07-4118-AFCF-E6E9366638D8@orcaware.com> Message-ID: >Blair Zajac [mailto:blair at orcaware.com] >I don't think we need threads for this. The main loop in Orca is single threaded, so when it goes to read from a data file, it'll use >use the Compress::Bzip2 instance for that file. >Plus, we probably don't want as many threads as there are hosts being monitored, that would easily kill a box. If the main loop is single-threaded 1) that would imply one bunzip2 process launched at a time and wait for the result. That is not in fact the case. If you look at the code it just keeps spawning bunzip2 processes until the spawn fails. 2) Single-threaded main loop misses an easy obvious parallelism. The optimal approach is not to launch a thread per host but to put work into a queue which is read by a pool of worker threads. The number of worker threads is configurable. This is how, for example, Apache2 worker thread model works (of course most folks run apache2 in prefork mode ). From blair at orcaware.com Mon Aug 31 10:50:30 2009 From: blair at orcaware.com (Blair Zajac) Date: Mon, 31 Aug 2009 10:50:30 -0700 Subject: [Orca-users] Threads In-Reply-To: References: <60565503-BF07-4118-AFCF-E6E9366638D8@orcaware.com> Message-ID: <4A9C0D66.1070409@orcaware.com> Hudes, Dana wrote: > >> Blair Zajac [mailto:blair at orcaware.com] >> I don't think we need threads for this. The main loop in Orca is > single threaded, so when it goes to read from a data file, it'll use >> use the Compress::Bzip2 instance for that file. > >> Plus, we probably don't want as many threads as there are hosts being > monitored, that would easily kill a box. > > If the main loop is single-threaded > 1) that would imply one bunzip2 process launched at a time and wait for > the result. That is not in fact the case. > If you look at the code it just keeps spawning bunzip2 processes until > the spawn fails. Right, but the bunzip2 processes block until the main thread reads them. > 2) Single-threaded main loop misses an easy obvious parallelism. > > The optimal approach is not to launch a thread per host but to put work > into a queue which is read by a pool of worker threads. The number of > worker threads is configurable. This is how, for example, Apache2 > worker thread model works (of course most folks run apache2 in prefork > mode ). I suggest to break the work into two steps, the first is just switching to Compress::Bzip2 using the main loop to read the files with a second step is to have a configurable thread pool. Each of these would be separate commits. Regards, Blair From hudesd at hra.nyc.gov Mon Aug 31 11:06:21 2009 From: hudesd at hra.nyc.gov (Hudes, Dana) Date: Mon, 31 Aug 2009 14:06:21 -0400 Subject: [Orca-users] Threads In-Reply-To: <4A9C0D66.1070409@orcaware.com> References: <60565503-BF07-4118-AFCF-E6E9366638D8@orcaware.com> <4A9C0D66.1070409@orcaware.com> Message-ID: -----Original Message----- From: Blair Zajac [mailto:blair at orcaware.com] Sent: Monday, August 31, 2009 1:51 PM To: Hudes, Dana Cc: orca-users at orcaware.com Subject: Re: [Orca-users] Threads Hudes, Dana wrote: > >> Blair Zajac [mailto:blair at orcaware.com] monitored, that would easily kill a box. > >Right, but the bunzip2 processes block until the main thread reads them. Amd they consume system resources sitting there sleeping. Sure, not cpu resources but process table slots and memory. As to why swap never gets touched before spawning runs out of memory I don't know. >I suggest to break the work into two steps, the first is just switching to >ompress::Bzip2 using the main loop to read the files with a second step is to have a configurable thread pool. Each of these would be >separate commits. I agree completely.