Log::Parallel Cluster computing framework
Download

Log::Parallel Ranking & Summary

Log::Parallel Tags

log framework Perl module cluster cluster computing framework batch log processing

Log::Parallel Description

Cluster computing framework Log::Parallel is the main driver module at the heart of a cluster computing framework used for batch log processing. It sets things up, figures out what jobs can run and in what order, and queues them up to run.Everything it does is driven from the configuration data, probably parsed by Config::YAMLMacros and validated by Log::Parallel::ConfigCheck.Only one program, process_logs, is expected to use this module. As such, documentation for the API for this module in particular is not particularly relevant, but this is as good a place as any to document the overall system.process_logs is the driver script for processing data logs through a series of jobs specified in a configuration file.SYNOPSIS % bin/process_logs -c config_file use Log::Parallel; use Log::Parallel::ConfigCheck qw(validate_config); use Proc::JobQueue::DependencyQueue; opitons(); run($opts); validate_config($config); add_recnums($config); $files_by_recnum{$_->{recnum}} = get_files_by_srec($_, $config->{hostsinfo}) for @{$config->{sources}}; my $dependency_graph = make_dependency_graph(make_task_list($opts, $config, %files_by_recnum)) my $job_queue = new Proc::JobQueue::DependencyQueue(dependency_graph => $dependency_graph, hosts => [], hold_all => 1); setup_slave_hosts($config, $job_queue); Requirements: · Perl

Log::Parallel Related Software