Parallel::Depend Parallel-dependent dispatch of Perl or shell code
Download

Parallel::Depend Ranking & Summary

Parallel::Depend Tags

perl functions methods Perl functions Perl methods parallel-dependent dispatch

Parallel::Depend Description

Parallel-dependent dispatch of Perl or shell code Parallel::Depend is a Perl module that does parallel, dependent dispatch of Perl methods, perl functions, inline code blocks, or external shell commands. The schedule syntax is derived from Make but does not require that all jobs be wrapped in shell code to execute and also supports sub-schedules ("groups" ) that are dispatched as a unit.Execution history, including stdout and stderr of each job, is kept in per-job files for simpler access after the fact.SYNOPSIS package Mine; use base qw( Parallel::Depend Whatever::Else ); my $manager = Mine->constructify( @whatever ); my @argz = ( # assign values to attributes restart => '', # restart in-process queue force => '', # ignore/overwrite previous execution verbose => 1, # quiet (0), progress (1), detail (2). debug => 0, # DB::single set before first parse, dispatch nofork => '', # single-stream, useful for testing. maxjobs => 8, # 0 => unlimited, x < 0 => nofork. fork_ttys => '', # used for $DB::fork_TTY with perl debugger autoload => '', # choose between autoload and shell logdir => "$Bin/../var/log", # stderr, stdout files rundir => "$Bin/../var/run", # job status files sched => ( 'bar' ) bim = { your code here } # $anon_sub->( 'bim' ) bam = /path/to/shell # system( '/path/to/shell', 'bam' ) this = ./blah -a -b # system( './blah -a -b', 'this' ) # example of reusing an alias: zip a # maxjobs files in parallel. /tmp/bakcup/bigdump.aa = squish /tmp/bakcup/bigdump.ab = squish /tmp/bakcup/bigdump.ac = squish /tmp/bakcup/bigdump.ad = squish /tmp/bakcup/bigdump.ae = squish /tmp/bakcup/bigdump.af = squish /tmp/bakcup/bigdump.aa : /tmp/bakcup/bigdump.ab : /tmp/bakcup/bigdump.ac : /tmp/bakcup/bigdump.ad : /tmp/bakcup/bigdump.ae : /tmp/bakcup/bigdump.af : # groups are sub-schedules that have their # own namespace for jobs, are skpped entirely # on restart if the group completes successfully, # and can set their own attributes. pass2 < maxjob % 4 > # throttle heavy-duty jobs. pass2 < fee fie foe fum : > # all these can run in pass2 < this that other : > # parallel, no harm splitting them up pass2 < this = squish > # locally defined aliases: these are pass2 < that = squash > # *not* the same as the jobs above. pass2 < other = frobnicate > # attributes can be set, apply to all nested # levels. helpful within groups. # # for example, extract the contents of some # files, process them, then cleanup the # results. each group uses its own alias # to handle the files. maxjob % 0 # unlimited forks after < maxjob % 2 > # two-way only for after group prior < alias % extract > # default alias within the group after < alias % cleanup > during < alias % process > prior < file1 : > # $mgr->extract( 'file1' ) prior < file2 : > # $mgr->extract( 'file2' ) during < file1 : > # $mgr->process( 'file1' ) during < file2 : > # $mgr->process( 'file2' ) after < file1 : > # $mgr->cleanup( 'file1' ) after < file2 : > # $mgr->cleanup( 'file2' ) # assign job-specific attributes -- mainly to control # verbosity or flag jobs as installing ad-hoc schedules. this ~ ad_hoc # default for all attributes is 1 this ~ verbose 0 # or add your own value that ~ verbose 2 # as you might have guessed by now, text after # an un-escaped hash sign is treated as a comment.END ); my $outcome = eval { $manager->prepare ( @argz ); # parase, validate the queue. $manager->validate; # check for deadlocks $manager->execute; # do the deed "The End" } or die $@; or just: $manager->prepare( @argz )->validate->execute; # if you want to derive a new object from a # new one and use it to execute the que or add # ad_hoc jobs (e.g., factory class) then you # must share the queue for the new object. sub derive_new_object { my $mgr = shift; my $derived = $mgr->new( @_ ); $mgr->share_queue( $derived ); $derived } # at this point the derived object uses # the same queue as the original $manager # (not a clone, the same one). executing # with $derived will have the same effect # on the queue as $manager. $derived->execute; Requirements: · Perl

Parallel::Depend Related Software