[Orca-users] RE: Script to pull data to central Orca server in parallel

[Cohen, Joel]

My bad. Here are the scripts inline rather than as attachments.
#!/usr/local/bin/perl -w
# 
# netgroup.pl -- print out members of a netgroup (including sub-netgroups)
#
use strict;
use Getopt::Std;

our($opt_n);

getopts('n') or die "Could not read options: $!";

sub usage {
	print "USAGE: netgroup <netgroup name>\n";
	exit 1;
}

sub printgroup {
	my @group = @_;
	my $separator = " ";
	$opt_n && ($separator = "\n");
	foreach my $tmp (@group) {
		print $tmp, $separator;
	}
	if (! $opt_n) { (print "\n"); }
}

sub get_members {
	my $args = shift @_;
	my @pre_netgroup = ();
	my @post_netgroup = ();

	@pre_netgroup = split /\s+/, `/bin/ypmatch $args netgroup`;
	foreach my $member (@pre_netgroup) {
		if ( $member =~ /(\w+\-\w*(\-\w+)*)/ ) {
			my @tmp_netgroup = &get_members($1);
			push @post_netgroup, @tmp_netgroup;
		}
		elsif ( $member =~ /(\(\,)*(\w+)(\,\))*/ ) {
			push @post_netgroup, $2;
		}
	}
	return @post_netgroup;
}

if ( @ARGV == 0 ) { &usage; }

my $arg = shift;

my @netgroup = &get_members($arg);
printgroup (sort @netgroup);


#!/usr/local/bin/perl -w
#
# parallel_pull.pl script

use strict;
use Getopt::Std;
use Parallel::ForkManager;

my $MAX_FORKS = 10; # Number of hosts to do in parallel
my $date = `/bin/date +%Y'-'%m'-'%d`; chomp $date;
my $rcp = "/usr/bin/rcp";
my $srcdir = "/var/orcallator";
my $netgroup = "/usr/local/bin/netgroup -n";
my @hostlist = ();

our($opt_e);
getopts('e:') or die "Couldn't read options: $!";

if (! $opt_e) {
	print "You must specify -e <prod|dev>";
	exit 1;
}

my $environment = $opt_e;

my $dstdir = "/orca/var/$environment/orcallator";
open(HOSTS, "$netgroup orca-$environment |") or die "Can't open netgroup
map: $!";
while (<HOSTS>) {
        push @hostlist, $_;
}
close HOSTS;

my $pm = new Parallel::ForkManager($MAX_FORKS);

$pm->run_on_finish(
	sub {
		my ($pid, $exit_code, $ident) = @_;
		print "** host $ident PID $pid exit code: $exit_code\n";
	}
);

# Actual program code here
foreach my $host (@hostlist) {
	chomp $host;
	$pm->start($host) and next;
	my $rc = system($rcp, "$host.acml.com:$srcdir/orcallator-$date*",
"$dstdir/$host/");
	$pm->finish($rc);
}

print "Waiting for children to finish...\n";
$pm->wait_all_children;
print "ALL DONE!\n";


-----Original Message-----
From: Cohen, Joel 
Sent: Monday, November 24, 2003 9:46 AM
To: 'orca-users at orcaware.com'
Subject: Script to pull data to central Orca server in parallel


Hi all,

I wrote this small perl script to do the pulls from all your hosts that run
the SE toolkit in *parallel*. I'm running an environment with 60+ servers
using this and it takes less than 30 seconds to do all the copies.

Requirements:
Parallel::ForkManager module from CPAN
NIS - I'm using a netgroup that contains all my SE hosts to pull from. This
makes it easy. My netgroup lister script (unless you have one already)

You can drop me a line you have any question about this script.

Happy Orcaing! (is that even a word?)

Joel Cohen
Alliance Capital Management
CSSA - Sr. UNIX Administrator
New York, NY
-----------------------------------------
The information contained in this transmission may contain privileged and confidential information and is intended only for the use of the person(s) named above. If you are not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender immediately by reply e-mail and destroy all copies of the original message. Please note that we do not accept account orders and/or instructions by e-mail, and therefore will not be responsible for carrying out such orders and/or instructions.