Daily_UpdateBacklog.au3

Task reminders in morning or evening don't really work. Especially when quite busy last thing and opening explorer, locking a file, opening excel takes a few minutes. So. Automate the open folder, get svn lock and open file part and pop all up as a reminder. Hudson daily task and AutoIt script.

Prompt user, svn update, get lock, open .xls file, commit and release lock (or timeout and release lock) handle lock fail or user not present run from Hudson or other regular cron tool.



;http://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-automation.html
;http://www.autoitscript.com/autoit3/docs/libfunctions/_IECreate.htm
;http://stackoverflow.com/questions/9667957/automation-of-right-clicking-of-file-in-windows-explorer


#include <GUIListView.au3>
#include  <Excel.au3>

$Title = "UPDATE BACKLOG"
$Path = "C:\SVN\DocsPlanning\XXX\"
$File = "C:\SVN\DocsPlanning\XXX\Sprint36.xls"

$TortoisePath = "c:\Program Files\TortoiseSVN\bin\TortoiseProc.exe"
;http://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-automation.html
; TODO: change all to /closeonend:1 when all are working
;/closeonend:1 auto close if no errors

Local $iSvnPid = Run($TortoisePath & " /closeonend:1 /command:update /path:" & $Path)
ProcessWaitClose($iSvnPid)
$x = MsgBox(0,$Title & ": svn update done","svnpid:"&$iSvnPid&"\nsvn update done",5)

;http://www.autoitscript.com/autoit3/docs/libfunctions/_IECreate.htm
;#include
;Local $oIE = _IECreate("www.autoitscript.com")

$x = MsgBox(33,$Title & ": Try and Lock Backlog?",$File&"\nTry and Lock Backlog?",1800)
;$x = MsgBox(0,$Title & ": Debug","Debug answer = " & $x,900)
; $x = 1 okay $x = 2 cancel

If $x = 1 Then
  ;Hmpfh. the lock command doesn't take logmsg
  Local $iSvnPid = Run($TortoisePath & " /closeonend:0 /command:lock /logmsg:""update jco integration tasks"" /path:" & $File)
  ProcessWaitClose($iSvnPid)
  $x1 = MsgBox(33,$Title & ": Locked?",$File&"\nDid you lock it?")

  ;http://stackoverflow.com/questions/9667957/automation-of-right-clicking-of-file-in-windows-explorer
  ;Local $hList = ControlGetHandle("[CLASS:CabinetWClass]", "", "[CLASS:SysListView32; INSTANCE:1]")
  ;Local $aClient = WinGetPos($hList)
  ;Local $aPos = _GUICtrlListView_GetItemPosition($hList, _GUICtrlListView_GetSelectedIndices($hList))
  ;MouseClick("Right", $aClient[0] + $aPos[0] + 4, $aClient[1] + $aPos[1] + 4)
  
  ;MouseClick("Right", "Optical_SW_Sprint_36.xls")

  If $x1 = 1 Then

    ;pull up explorer window for commit or if there is trouble
    Local $iPid = Run("C:\WINDOWS\EXPLORER.EXE /n,/e," & $Path)
    ProcessWait($iPid)
    MsgBox(0,$Title & ": EXPLORER","pid:"&$iPid,5)
    Sleep(3000)
  
    $excel_file = _ExcelBookOpen($File)

    $x2 = MsgBox(33,$Title & ": Commit and unlock Backlog","Now edit backlog.\nWhen finished commit and unlock Backlog.\nFinished?\n")

    If $x2 = 1 Then
      ; commit and release lock
      Local $iSvnPid = Run($TortoisePath & " /closeonend:0 /command:commit /logmsg:""update jco integration tasks"" /path:" & $File)
      ProcessWait($iSvnPid)
    EndIf
   
  EndIf

  ; unlock in case the commit didn't unlock
  Sleep(30000)
  Local $iSvnPid = Run($TortoisePath & " /closeonend:1 /command:unlock /path:" & $File)
  ProcessWait($iSvnPid)

  ; close explorer
  ProcessClose($iPid)

EndIf

Exit

log2csv.pl - grep data from any logfile format

Some people here were using LiveGraph, it looks a little grotty but works very very well. I had a quick look at using other things, custom graph generation in perl or perl + gnuplot. But LiveGraph gives a very neat solution with:
* GUI and immediate display of data by just pointing at .dat or .csv file
* immediate and easy tweaking of graph
* LIVE graph plotting
* Export graph to image using GUI
* Very nice generic automatic figure-format-out LiveGraph .csv/.dat file format

And hopefully, I think I did it before: * multiple sources of data on one plot

This script, log2csv.pl allows any textual log file with data in it to be fed to LiveGraph. So maybe I never have to write another data parsing + plotting script ever again! :)
log2csv takes:
* output .csv file name
* input log file name
* pairs of data name and regexp to grep out data value from log

#e.g.
log2csv.pl testdata.csv some.log \
D0 'WSS.*D00,PORT.*=\s*(\d+)\b' D1 'WSS.*D01,PORT.*=\s*(\d+)\b' \
PD_A 'PD:pd_a power:([-]*\d+[\.]*\d*)\b' \
PD_O 'PD:pd_o1a power:([-]*\d+[\.]*\d*)\b' \
PD_D1 'PD_D1:([-]*\d+[\.]*\d*)\b' \
PD_D2 'PD_D2:([-]*\d+[\.]*\d*)\b'

Run it on windows using cygwin (tail -f needed).

Here is the script:





#!/usr/bin/perl -w

=head1 NAME

log2csv.pl - process test log files, generate .csv file for LiveGraph

=head1 USAGE

=cut

my $usage = <
log2csv.pl [ ..]

e.g.
log2csv.pl testdata_22Feb_xx.csv /cygdrive/m/TclTestManager/Logs/TestRun_22Feb_*/*.log \\
D0 'WSS.*D00,PORT.*=\\s*(\\d+)\\b' D1 'WSS.*D01,PORT.*=\\s*(\\d+)\\b' \\
PD_A 'PD:pd_a power:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_O 'PD:pd_o1a power:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_D1 'PD_D1:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_D2 'PD_D2:([-]*\\d+[\\.]*\\d*)\\b'
END

=head1 SYNOPSIS

e.g. parse this:

$ bash ./monitor_log.sh

log::logarray:aWSSinfo0(D00,ATTN) = 7.0
log::logarray:aWSSinfo0(D00,PORT) = 0
log::logarray:aWSSinfo0(D01,ATTN) = 7.0
log::logarray:aWSSinfo0(D01,PORT) = 1
MONITOR: WSS configured and check okay. D00 att:7.0 port:0 D01 att:7.0 port:1
MONITOR loop state:lock PD:pd_a power:-15.99 target:-16.00
TESTSTEP: FAIL fPDVal:-1.26 check PD_D1:-1.26 is about 7dBm, thresh:3 target:7 delta:8.26 thresh:3
TESTSTEP: FAIL fPDVal:-1.26 check PD_D2:-1.90 is about 7dBm, thresh:3 target:7 delta:8.26 thresh:3
MONITOR: OFSM processes check PASS
MONITOR PING test TX:1 RX:1 result:PASS


TODO: parse this:
TESTSTEP: FAIL waitForOlcControlState bResult=FAIL iTestTime=213 match(1,2):(0,0) check1:state:\s+lock\y
MONITOR: WSS init try:1

TODO: get time at every monitor loop, plot that, then can see when disconnected/waiting.
Test Start Thu Feb 18 13:17:32 GMT Standard Time 2010

=head1 DESCRIPTION

=head1 DESIGN/NOTES

Started with testreport.pl

tail /cygdrive/m/TclTestManager/Demo/DemoFeb2010/ofso_demo.log
tail /cygdrive/m/TclTestManager/Demo/DemoFeb2010/monitor_log.sh


=cut


sub openCSV {
my $outfile=shift;
my $refDetails=shift;

# open file, create and write header
# overwrite or append to outfile? overwrite.
( open( OUTFILE, ">$outfile" )) || die "file err: Can't open $outfile for write: $!";

# write header
my $bWriteComma = 0;
for my $ds ( @$refDetails ) {
#print "write var $ds->{var} val $ds->{val} regexp $ds->{regexp}\n";
($bWriteComma == 1) && print OUTFILE ", ";
if (defined($ds->{var})) { print OUTFILE "$ds->{var}"; }
$bWriteComma = 1;
}
print OUTFILE "\n";
close OUTFILE;
}

sub writeCSV {
my $outfile=shift;
my $refDetails=shift;

# open file, append
( open( OUTFILE, ">>$outfile" )) || die "file err: Can't open $outfile for append: $!";

# write data
my $bWriteComma = 0;
for my $ds ( @$refDetails ) {
#print "write var $ds->{var} val $ds->{val} regexp $ds->{regexp}\n";
($bWriteComma == 1) && print OUTFILE ", ";
if (defined($ds->{val})) { print OUTFILE "$ds->{val}"; }
$bWriteComma = 1;
}
print OUTFILE "\n";
close OUTFILE;
}


# TODO user passes in filename [optional outfile] [optional regexp/list of famille to take]
my $c = $#ARGV + 1;
print "ARGC=$c\n";
foreach my $i (0 .. $#ARGV) {
print "arg ARGV[$i]=$ARGV[$i]\n";
}

(defined $ARGV[0]) || die "args err: no outfile specified. \n$usage";
my $outfile = shift;
(defined $ARGV[0]) || die "args err: no infile specified. \n$usage";
my @infiles;
my $infilecount = 0;
while (defined($ARGV[0]) && -e $ARGV[0]) {
$infiles[$infilecount++] = shift;
print "args infile $infilecount is $infiles[$infilecount-1]\n";
}
($infilecount>0) || die "args err: no infile found. \n$usage";

my @data_spec;
my $data_spec_count = 0;
while (defined $ARGV[0]) {
# TODO: if match ^- option parse

# pair of data and regexp
(defined $ARGV[1]) || die "err: pair of data and regexp expected. \n$usage";
my $ds = {};
$ds->{var} = shift;
$ds->{regexp} = shift;
# mark first regexp as the one to trigger a write of values (e.g. 1st regexp is timestamp)
if ($data_spec_count == 0) { $ds->{write} = 1; }
push @data_spec, $ds;
$data_spec_count++;
print "data spec $data_spec_count is $data_spec[$data_spec_count-1]->{var} and $data_spec[$data_spec_count-1]->{regexp}\n";
#print "data spec 0 is $data_spec[0]->{var} and $data_spec[0]->{regexp}\n";
}

&openCSV ($outfile,\@data_spec);

#use File::Tail;
#my $file=File::Tail->new($infiles[0]);
#while (defined(my $line=$file->read)) {
# print "$line";
#}
open(INFILE, '-|', "tail -f $infiles[0]") or die "file err: Can't open $infiles[0]: $!";
my $lastline = "";
while () {
my $line=$_;
while (defined($_) && $line ne $lastline) {
#print "line: $_";
$lastline = $line;
for my $ds ( @data_spec ) {
#print "grep var $ds->{var} regexp $ds->{regexp}\n";
$_ = $line;
if (my ($val) = ($line =~ m/$ds->{regexp}/ ) ) {
print "GROPPED var $ds->{var} regexp $ds->{regexp} val $val\n";
# store value
$ds->{val} = $val;
# write out values when we get match on first regexp ...
if (defined($ds->{write})) {
print "GRO write\n";
&writeCSV ($outfile,\@data_spec);
}
last; # next; #break; continue;
}
}
}
}

return;

Questions about tcl for job interviews, thoughts on tcl quirks

Tricky question. "What is a simple question and answer that would give us confidence that they knew TCL?"

Try and set up interview so your candidate can write code. You can’t determine it with one question I think but you can if you get them to write a function. Incidentally The Gureilla Guide to Interviewing by Joel Spolsky FogCreek is very good on technical/software interviews.

One question might be what kinds of different data types are available by default in tcl?
Answer1: in tcl, “everything is a string” (but lots of people who know tcl mightn’t get this!)
Answer2: There is default support for list and array. And normal var types would be considered integers/floats/strings.
“everything is a string” means there is no type protection
Data types are supported by the way they are treated in procedures.


Write a procedure (that would require a while or for loop) to calculate something.
Show how this can be called and print the answer.
e.g. calculate the sum of a list

# my answer
proc calc_sum_of_list { list } {
set sum 0
foreach a $list {
# OR if you're a real tcler here use: incr sum $a
set sum [expr $sum + $a]
}
return $sum
}

# how this is called
set mList [list 1 2 3 4]
puts “sum of list result: [calc_sum_of_list $mList]”

# OR
set mList [list 1 2 3 4]
set result [calc_sum_of_list $mList]
puts “sum of list result: $result”


Simpler questions:

How do you set a variable?
Answer: set a 40

How do you increment an integer variable?
Answer: incr a

How do you decrement?
Answer: incr a -1

How to you print a variable?
Answer: puts $a
OR puts “a is $a”
OR puts [format “a is %d = 0x%08x, string %s” $a $a “string”]
OR ...



A fundamental tcl question: Are spaces needed in tcl in for/while/if loops?
Answer: YES tcl has a couple of really awkward quirks.

e.g. valid tcl:
set a 20
if {$a > 4} {
puts “$a is bigger than 4”
}

Invalid:
set a 20
if{$a > 4} {
puts “$a is bigger than 4”
}

invalid command name "if{20"

Invalid:
set a 20
if {$a > 4}
{
puts “$a is bigger than 4”
}

wrong # args: no script following "{$a > 4}" argument
invalid command name "
puts "$a is bigger than 4"
"



Another fundamental question: In tcl what brackets in comments might cause problems (especially if they’re not balanced in comments)?
Answer: Curley brackets {} cause problems. parenthesis () or square [] or anything else don’t cause problems in comments.

Pure and even very impure software people who haven't encountered tcl before have probably already WTFed several times and swooned away by now. Must be because tcl syntax was defined by one of those hardware people! :-P i.e. Possibly a very practical person who has made a very gluey language with a few glaring hairy things remaining in the syntax (possibly on purpose to scare/annoy computer science philosophers).
Please excuse me hardware and software people alike :)
Tcl language syntax overview
Why is tcl syntax so weird?on wiki.tcl.tk
syntax related pages on wiki.tcl.tk
Ousterhout
John Ousterhout on tcl history
wikipedia.org:Tcl

bash shell logrotate function

# Yet Another installation in the Yet Another script to do whatever series

# I have done logrotate in bash in at least two other flavours before. :-7

# bash shell logrotate function
function logroti () {
for f in $* ; do
echo logroti for f
if [ -e "$f" ] ; then
ROTFILES=$(find $f.[0-9] 2>/dev/null| sort -r )
for rf in $ROTFILES; do
BASE=${rf%.*}
NUM=${rf##$BASE.}
((NUM++))
echo logroti for rf=$rf BASE=$BASE NUM=$NUM new file $BASE.$NUM
mv -f $rf $BASE.$NUM
done
cp -f $f $f.0
cat /dev/null > $f
fi
done
}

# e.g. usage
logroti run_demo.log
while [ 1 ]; do
logrotate demo.log 2>&1 |tee -a run_demo.log
date |tee -a run_demo.log
echo "DEMO script start" |tee -a run_demo.log
tclsh RunDemo.tcl 2>&1 |tee demo.log
date |tee -a run_demo.log
echo "DEMO script exited" |tee -a run_demo.log
done

# the tcl script internally at top level has a catch {...} sError
# but some errors get past this! :)
# So am catching everything and can see how long things run for and what errors cause any problems