log2csv.pl - grep data from any logfile format

Some people here were using LiveGraph, it looks a little grotty but works very very well. I had a quick look at using other things, custom graph generation in perl or perl + gnuplot. But LiveGraph gives a very neat solution with:
* GUI and immediate display of data by just pointing at .dat or .csv file
* immediate and easy tweaking of graph
* LIVE graph plotting
* Export graph to image using GUI
* Very nice generic automatic figure-format-out LiveGraph .csv/.dat file format

And hopefully, I think I did it before: * multiple sources of data on one plot

This script, log2csv.pl allows any textual log file with data in it to be fed to LiveGraph. So maybe I never have to write another data parsing + plotting script ever again! :)
log2csv takes:
* output .csv file name
* input log file name
* pairs of data name and regexp to grep out data value from log

#e.g.
log2csv.pl testdata.csv some.log \
D0 'WSS.*D00,PORT.*=\s*(\d+)\b' D1 'WSS.*D01,PORT.*=\s*(\d+)\b' \
PD_A 'PD:pd_a power:([-]*\d+[\.]*\d*)\b' \
PD_O 'PD:pd_o1a power:([-]*\d+[\.]*\d*)\b' \
PD_D1 'PD_D1:([-]*\d+[\.]*\d*)\b' \
PD_D2 'PD_D2:([-]*\d+[\.]*\d*)\b'

Run it on windows using cygwin (tail -f needed).

Here is the script:





#!/usr/bin/perl -w

=head1 NAME

log2csv.pl - process test log files, generate .csv file for LiveGraph

=head1 USAGE

=cut

my $usage = <
log2csv.pl [ ..]

e.g.
log2csv.pl testdata_22Feb_xx.csv /cygdrive/m/TclTestManager/Logs/TestRun_22Feb_*/*.log \\
D0 'WSS.*D00,PORT.*=\\s*(\\d+)\\b' D1 'WSS.*D01,PORT.*=\\s*(\\d+)\\b' \\
PD_A 'PD:pd_a power:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_O 'PD:pd_o1a power:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_D1 'PD_D1:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_D2 'PD_D2:([-]*\\d+[\\.]*\\d*)\\b'
END

=head1 SYNOPSIS

e.g. parse this:

$ bash ./monitor_log.sh

log::logarray:aWSSinfo0(D00,ATTN) = 7.0
log::logarray:aWSSinfo0(D00,PORT) = 0
log::logarray:aWSSinfo0(D01,ATTN) = 7.0
log::logarray:aWSSinfo0(D01,PORT) = 1
MONITOR: WSS configured and check okay. D00 att:7.0 port:0 D01 att:7.0 port:1
MONITOR loop state:lock PD:pd_a power:-15.99 target:-16.00
TESTSTEP: FAIL fPDVal:-1.26 check PD_D1:-1.26 is about 7dBm, thresh:3 target:7 delta:8.26 thresh:3
TESTSTEP: FAIL fPDVal:-1.26 check PD_D2:-1.90 is about 7dBm, thresh:3 target:7 delta:8.26 thresh:3
MONITOR: OFSM processes check PASS
MONITOR PING test TX:1 RX:1 result:PASS


TODO: parse this:
TESTSTEP: FAIL waitForOlcControlState bResult=FAIL iTestTime=213 match(1,2):(0,0) check1:state:\s+lock\y
MONITOR: WSS init try:1

TODO: get time at every monitor loop, plot that, then can see when disconnected/waiting.
Test Start Thu Feb 18 13:17:32 GMT Standard Time 2010

=head1 DESCRIPTION

=head1 DESIGN/NOTES

Started with testreport.pl

tail /cygdrive/m/TclTestManager/Demo/DemoFeb2010/ofso_demo.log
tail /cygdrive/m/TclTestManager/Demo/DemoFeb2010/monitor_log.sh


=cut


sub openCSV {
my $outfile=shift;
my $refDetails=shift;

# open file, create and write header
# overwrite or append to outfile? overwrite.
( open( OUTFILE, ">$outfile" )) || die "file err: Can't open $outfile for write: $!";

# write header
my $bWriteComma = 0;
for my $ds ( @$refDetails ) {
#print "write var $ds->{var} val $ds->{val} regexp $ds->{regexp}\n";
($bWriteComma == 1) && print OUTFILE ", ";
if (defined($ds->{var})) { print OUTFILE "$ds->{var}"; }
$bWriteComma = 1;
}
print OUTFILE "\n";
close OUTFILE;
}

sub writeCSV {
my $outfile=shift;
my $refDetails=shift;

# open file, append
( open( OUTFILE, ">>$outfile" )) || die "file err: Can't open $outfile for append: $!";

# write data
my $bWriteComma = 0;
for my $ds ( @$refDetails ) {
#print "write var $ds->{var} val $ds->{val} regexp $ds->{regexp}\n";
($bWriteComma == 1) && print OUTFILE ", ";
if (defined($ds->{val})) { print OUTFILE "$ds->{val}"; }
$bWriteComma = 1;
}
print OUTFILE "\n";
close OUTFILE;
}


# TODO user passes in filename [optional outfile] [optional regexp/list of famille to take]
my $c = $#ARGV + 1;
print "ARGC=$c\n";
foreach my $i (0 .. $#ARGV) {
print "arg ARGV[$i]=$ARGV[$i]\n";
}

(defined $ARGV[0]) || die "args err: no outfile specified. \n$usage";
my $outfile = shift;
(defined $ARGV[0]) || die "args err: no infile specified. \n$usage";
my @infiles;
my $infilecount = 0;
while (defined($ARGV[0]) && -e $ARGV[0]) {
$infiles[$infilecount++] = shift;
print "args infile $infilecount is $infiles[$infilecount-1]\n";
}
($infilecount>0) || die "args err: no infile found. \n$usage";

my @data_spec;
my $data_spec_count = 0;
while (defined $ARGV[0]) {
# TODO: if match ^- option parse

# pair of data and regexp
(defined $ARGV[1]) || die "err: pair of data and regexp expected. \n$usage";
my $ds = {};
$ds->{var} = shift;
$ds->{regexp} = shift;
# mark first regexp as the one to trigger a write of values (e.g. 1st regexp is timestamp)
if ($data_spec_count == 0) { $ds->{write} = 1; }
push @data_spec, $ds;
$data_spec_count++;
print "data spec $data_spec_count is $data_spec[$data_spec_count-1]->{var} and $data_spec[$data_spec_count-1]->{regexp}\n";
#print "data spec 0 is $data_spec[0]->{var} and $data_spec[0]->{regexp}\n";
}

&openCSV ($outfile,\@data_spec);

#use File::Tail;
#my $file=File::Tail->new($infiles[0]);
#while (defined(my $line=$file->read)) {
# print "$line";
#}
open(INFILE, '-|', "tail -f $infiles[0]") or die "file err: Can't open $infiles[0]: $!";
my $lastline = "";
while () {
my $line=$_;
while (defined($_) && $line ne $lastline) {
#print "line: $_";
$lastline = $line;
for my $ds ( @data_spec ) {
#print "grep var $ds->{var} regexp $ds->{regexp}\n";
$_ = $line;
if (my ($val) = ($line =~ m/$ds->{regexp}/ ) ) {
print "GROPPED var $ds->{var} regexp $ds->{regexp} val $val\n";
# store value
$ds->{val} = $val;
# write out values when we get match on first regexp ...
if (defined($ds->{write})) {
print "GRO write\n";
&writeCSV ($outfile,\@data_spec);
}
last; # next; #break; continue;
}
}
}
}

return;