recover text from docx

A friend got into trouble with a broken usb stick and sole copy of notes in a now unopenable .docx file. Oh dear.

If you google for apps to recover corrupt .docx files there are a phletora of tools! Yeesh! Make backups people. You have to get burned once I suppose before you learn to make backups properly. I remember in the good old days, (well, the old days, (well, 1993ish, (not so old))) when I got a bit overconfident and put tons of content into a .doc for my final year project ... AND then discovered that the files I had worked on for hours were unopenable despite a few backups along the way. Argh!

So the winner of this impromptu recover text from .docx file IS:

http://sourceforge.net/projects/docx2txt/

 Tadahhhhh!




 Congrats to Sandeep Kumar.

Extra especially pleased I am that it is a perl script :) Heh heh :)
ANd actually looking inside it's nice and straightforward. Open up internals of docx as xml and parse out text

I initially tried xxd |less   
Then had a look with emacs. Murghh not good :-7
Then some .exe shareware tools, docXConvertor.exe  Damaged-DOCX2TXT.exe meh meh

log2csv.pl - grep data from any logfile format

Some people here were using LiveGraph, it looks a little grotty but works very very well. I had a quick look at using other things, custom graph generation in perl or perl + gnuplot. But LiveGraph gives a very neat solution with:
* GUI and immediate display of data by just pointing at .dat or .csv file
* immediate and easy tweaking of graph
* LIVE graph plotting
* Export graph to image using GUI
* Very nice generic automatic figure-format-out LiveGraph .csv/.dat file format

And hopefully, I think I did it before: * multiple sources of data on one plot

This script, log2csv.pl allows any textual log file with data in it to be fed to LiveGraph. So maybe I never have to write another data parsing + plotting script ever again! :)
log2csv takes:
* output .csv file name
* input log file name
* pairs of data name and regexp to grep out data value from log

#e.g.
log2csv.pl testdata.csv some.log \
D0 'WSS.*D00,PORT.*=\s*(\d+)\b' D1 'WSS.*D01,PORT.*=\s*(\d+)\b' \
PD_A 'PD:pd_a power:([-]*\d+[\.]*\d*)\b' \
PD_O 'PD:pd_o1a power:([-]*\d+[\.]*\d*)\b' \
PD_D1 'PD_D1:([-]*\d+[\.]*\d*)\b' \
PD_D2 'PD_D2:([-]*\d+[\.]*\d*)\b'

Run it on windows using cygwin (tail -f needed).

Here is the script:





#!/usr/bin/perl -w

=head1 NAME

log2csv.pl - process test log files, generate .csv file for LiveGraph

=head1 USAGE

=cut

my $usage = <
log2csv.pl [ ..]

e.g.
log2csv.pl testdata_22Feb_xx.csv /cygdrive/m/TclTestManager/Logs/TestRun_22Feb_*/*.log \\
D0 'WSS.*D00,PORT.*=\\s*(\\d+)\\b' D1 'WSS.*D01,PORT.*=\\s*(\\d+)\\b' \\
PD_A 'PD:pd_a power:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_O 'PD:pd_o1a power:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_D1 'PD_D1:([-]*\\d+[\\.]*\\d*)\\b' \\
PD_D2 'PD_D2:([-]*\\d+[\\.]*\\d*)\\b'
END

=head1 SYNOPSIS

e.g. parse this:

$ bash ./monitor_log.sh

log::logarray:aWSSinfo0(D00,ATTN) = 7.0
log::logarray:aWSSinfo0(D00,PORT) = 0
log::logarray:aWSSinfo0(D01,ATTN) = 7.0
log::logarray:aWSSinfo0(D01,PORT) = 1
MONITOR: WSS configured and check okay. D00 att:7.0 port:0 D01 att:7.0 port:1
MONITOR loop state:lock PD:pd_a power:-15.99 target:-16.00
TESTSTEP: FAIL fPDVal:-1.26 check PD_D1:-1.26 is about 7dBm, thresh:3 target:7 delta:8.26 thresh:3
TESTSTEP: FAIL fPDVal:-1.26 check PD_D2:-1.90 is about 7dBm, thresh:3 target:7 delta:8.26 thresh:3
MONITOR: OFSM processes check PASS
MONITOR PING test TX:1 RX:1 result:PASS


TODO: parse this:
TESTSTEP: FAIL waitForOlcControlState bResult=FAIL iTestTime=213 match(1,2):(0,0) check1:state:\s+lock\y
MONITOR: WSS init try:1

TODO: get time at every monitor loop, plot that, then can see when disconnected/waiting.
Test Start Thu Feb 18 13:17:32 GMT Standard Time 2010

=head1 DESCRIPTION

=head1 DESIGN/NOTES

Started with testreport.pl

tail /cygdrive/m/TclTestManager/Demo/DemoFeb2010/ofso_demo.log
tail /cygdrive/m/TclTestManager/Demo/DemoFeb2010/monitor_log.sh


=cut


sub openCSV {
my $outfile=shift;
my $refDetails=shift;

# open file, create and write header
# overwrite or append to outfile? overwrite.
( open( OUTFILE, ">$outfile" )) || die "file err: Can't open $outfile for write: $!";

# write header
my $bWriteComma = 0;
for my $ds ( @$refDetails ) {
#print "write var $ds->{var} val $ds->{val} regexp $ds->{regexp}\n";
($bWriteComma == 1) && print OUTFILE ", ";
if (defined($ds->{var})) { print OUTFILE "$ds->{var}"; }
$bWriteComma = 1;
}
print OUTFILE "\n";
close OUTFILE;
}

sub writeCSV {
my $outfile=shift;
my $refDetails=shift;

# open file, append
( open( OUTFILE, ">>$outfile" )) || die "file err: Can't open $outfile for append: $!";

# write data
my $bWriteComma = 0;
for my $ds ( @$refDetails ) {
#print "write var $ds->{var} val $ds->{val} regexp $ds->{regexp}\n";
($bWriteComma == 1) && print OUTFILE ", ";
if (defined($ds->{val})) { print OUTFILE "$ds->{val}"; }
$bWriteComma = 1;
}
print OUTFILE "\n";
close OUTFILE;
}


# TODO user passes in filename [optional outfile] [optional regexp/list of famille to take]
my $c = $#ARGV + 1;
print "ARGC=$c\n";
foreach my $i (0 .. $#ARGV) {
print "arg ARGV[$i]=$ARGV[$i]\n";
}

(defined $ARGV[0]) || die "args err: no outfile specified. \n$usage";
my $outfile = shift;
(defined $ARGV[0]) || die "args err: no infile specified. \n$usage";
my @infiles;
my $infilecount = 0;
while (defined($ARGV[0]) && -e $ARGV[0]) {
$infiles[$infilecount++] = shift;
print "args infile $infilecount is $infiles[$infilecount-1]\n";
}
($infilecount>0) || die "args err: no infile found. \n$usage";

my @data_spec;
my $data_spec_count = 0;
while (defined $ARGV[0]) {
# TODO: if match ^- option parse

# pair of data and regexp
(defined $ARGV[1]) || die "err: pair of data and regexp expected. \n$usage";
my $ds = {};
$ds->{var} = shift;
$ds->{regexp} = shift;
# mark first regexp as the one to trigger a write of values (e.g. 1st regexp is timestamp)
if ($data_spec_count == 0) { $ds->{write} = 1; }
push @data_spec, $ds;
$data_spec_count++;
print "data spec $data_spec_count is $data_spec[$data_spec_count-1]->{var} and $data_spec[$data_spec_count-1]->{regexp}\n";
#print "data spec 0 is $data_spec[0]->{var} and $data_spec[0]->{regexp}\n";
}

&openCSV ($outfile,\@data_spec);

#use File::Tail;
#my $file=File::Tail->new($infiles[0]);
#while (defined(my $line=$file->read)) {
# print "$line";
#}
open(INFILE, '-|', "tail -f $infiles[0]") or die "file err: Can't open $infiles[0]: $!";
my $lastline = "";
while () {
my $line=$_;
while (defined($_) && $line ne $lastline) {
#print "line: $_";
$lastline = $line;
for my $ds ( @data_spec ) {
#print "grep var $ds->{var} regexp $ds->{regexp}\n";
$_ = $line;
if (my ($val) = ($line =~ m/$ds->{regexp}/ ) ) {
print "GROPPED var $ds->{var} regexp $ds->{regexp} val $val\n";
# store value
$ds->{val} = $val;
# write out values when we get match on first regexp ...
if (defined($ds->{write})) {
print "GRO write\n";
&writeCSV ($outfile,\@data_spec);
}
last; # next; #break; continue;
}
}
}
}

return;

SalomeToXQual.pl

#!/usr/bin/perl -w

=head1 NAME

Read in Salomé_TMF exported .xml file.

Choose a Test Plan and set of tests to import.

Write out .xml or .csv that XQual can read.

=head1 SYNOPSIS

Salomé project INX8000-OLC

-> Data exchange format -> Export xml xml_export_olc_all.xml (file saved to Desktop)

Read in that file (as XML).

Read in Famille Nom, offer user choice of Famille Nom and/or SuiteTest Nom to choose.

Read in all wanted particular items matching Famille and SuiteTest.

Save to ; delimited .csv (awkward with excel) and " " indented for "with testplan" option.

Write fields out (some values merged, others generated/hardcoded).

=head1 SPEC

I tried doing this Salome_TMF export -> excel -> .csv -> XStudio.

But excel .csv output is annoyingly limited.

Possibly could get closer with openoffice but feh.

=head2 INPUT Salome_TMF .xml format

.

OLC

.

SU_A2 OLC

Area for tests which are parts of this phase of testing

SU_A2

=head2 Mapping Salomé .xml/.xls fields to XStudio format

A+BI+BJ (projet+id_famille+Nom) category; (set to 4) priority; (set to "") canonicalPath;(set to test script name) path

BM (id_test) index; (set to 1) implemented; BP (Nom) name;

CK+CL+CM step;param1,param2,...; CN check1,check2,...

=head2 XQual XStudio .csv and .xml import format

Format .xml or .csv or .csv with test plan

The .xml format (detailed below) for import I have not tested.

The IDs are in the .xml, how would we decide on those when importing?

So I'm using use .csv to import.

Tests and Testcases (without testplan)

category;implemented;priority;canonicalPath;path;indexTestcase1,indexTestcase2,...

OR Tests and Testcases (with testplan>, indentation spaces matter)

category;priority;canonicalPath;path(script name)

index;implemented;name

step;param1,param2,...;check1,check2,...

=head2 OUTPUT XStudio .csv in format

# the indentation whitespace matters

# test path must begin with /

# numeric fields must be numeric

# ColdRestartsOfTCS.tcl is the script for this example (one script for all test cases in category)

SU_A2;4;TestCanPathIsScriptNameMaybe;/TestPath/ColdRestartsOfTCS

1159;0;Cold Restarts of TCS with no input power

Action_1075 A0 Check bringup wait time for WSS;;"20 minutes, or longer. [HLD.OPTA.BLK_E.0010]"

Action_1076,A1,There is no optical output during this startup phase [Bringup slides] & R[HLD.OPTA.BLK_E.0011];;Using a power meter confirm

Action_1077,A2,"After warm-up phase complete if BI available, manufacture and calibration phase available then the following three tasks commence \n1) CC TX - CC laser and APR ON \n2) CC RX - Coarse RX Loop starts \n3) Data plane 1st powers and warms up \n";;Confirm these tasks commence.

Action_1090,A3,###############################################################################################################################################################################################################################################################;;Due to the fact that only part of the functionally is available cannot see without that stubbing that the bring can complete. This step attempts to highlight this issue.

=head2 running

I'm using cygwin's perl for now. By default XML/LibXML.pm is available.

#!/cygdrive/c/Perl/bin/perl -w

$ perl c:/Perl/scripts/SalomeToXQual.pl

Can't locate XML/LibXML.pm in @INC (@INC contains: c:/Perl/site/lib c:/Perl/lib .) at c:/Perl/scripts/SalomeToXQual.pl line 283.

=head2 DESIGN

http://perl-xml.sourceforge.net/faq/#quick_choice

=cut

use strict;

use XML::LibXML;

# defaults

my $filename = shift;

my $ofilename;

my $match = shift;

$filename="c:/Documents and Settings/james.coleman/Desktop/xml_export_olc_all.xml" if (!defined $filename);

#$match = "SU_A2 OLC" if (undef $match);

$match = "" if (!defined($match));

my $matchhash = $match;

$matchhash =~ s/[^\w\d]/_/g;

if (!defined $ofilename) {

$ofilename=$filename;

$ofilename =~ s/.xml$//;

$ofilename .= "_${matchhash}.csv";

}

print "$0 filename=$filename ofilename=$ofilename match=$match\n";

my $parser = XML::LibXML->new();

my $doc = $parser->parse_file($filename);

# TODO user passes in filename [optional outfile] [optional regexp/list of famille to take]

#my $c = $#ARGV + 1;

#print FILE "ARGC=$c\n";

#foreach my $i (0 .. $#ARGV) {

# print FILE "arg ARGV[$i]=$ARGV[$i]\n";

#}

open(FILE, '>', $ofilename) or die $!;

# ct = current test

my %ct;

my ($category,$priority,$canonicalPath,$path);

my ($index,$impl,$name);

my ($step,$param1,$param2,$check1,$check2);

$ct{'category'} = "0xdeadbeef";

$ct{'path'} = "0xdeadbeef";

$ct{'index'} = "0xdeadbeef";

$ct{'name'} = "0xdeadbeef";

$ct{'step'} = "0xdeadbeef";

$ct{'param1'} = "";

$ct{'param2'} = "";

$ct{'check1'} = "";

$ct{'check2'} = "";

$ct{'priority'} = 4;

$ct{'canonicalPath'} = "";

$ct{'impl'} = 0;

$ct{'project'} = $doc->findnodes('//Nom')->to_literal;

foreach my $f ($doc->findnodes('//Familles/Famille')) {

my($n) = $f->findnodes('./Nom');

print $n->to_literal, "\n";

next if ($match eq "");

$_ = $n->to_literal;

if (m/$match/) {

print "MATCH ", $n->to_literal, "\n";

$ct{'category'} = $n->to_literal;

$ct{'cathash'} = $ct{'category'};

$ct{'cathash'} =~ s/ /_/g;

### PATH MUST BEGIN WITH /

$ct{'path'} = "/IntuneTest/".$ct{'cathash'}."TestScript";

#my $s = $f->findnodes('./SuiteTests');

#print "Suite: ", $s->to_literal, "\n";

my($tests) = $f->findnodes('.//Tests');

#print "Tests: ", $tests->to_literal, "\n";

#category;priority;canonicalPath;path(script name)

# index;implemented;name

# step;param1,param2,...;check1,check2,...

print FILE "\n$ct{'category'};$ct{'priority'};$ct{'canonicalPath'};$ct{'path'}\n";

foreach my $t ($tests->findnodes('.//Test')) {

my @a0 = $t->attributes();

#$ct{'index'} = $t->getAttribute('test_id');

$ct{'index'} = $a0[0]->getValue();

$ct{'index'} =~ s/[^0-9]//g;

my $tn = $t->findnodes('./Nom');

$ct{'name'} = $tn->to_literal;

# no test name in .csv :( NO, actually yuou can do it (and must I think) - had a different problem

# AND two \n's are VITAL!

print FILE "\n $ct{'index'};$ct{'impl'};$ct{'name'}\n\n";

#print FILE "\n $ct{'index'};$ct{'impl'}\n\n";

# place test name in 1st test step

#print FILE " $ct{'name'};;\n";

### TODO Test Description not included

# my $desc = $tests->findnodes('./Description')) {

foreach my $step ($t->findnodes('./TestManuel/ActionTest')) {

my @stepa0 = $step->attributes();

$ct{'step'} = $stepa0[0]->getValue();

$ct{'step'} .= "," . $step->findnodes('./Nom')->to_literal;

$ct{'step'} .= "," . $step->findnodes('./Description')->to_literal;

$ct{'check1'} = $step->findnodes('./ResultAttendu')->to_literal;

#print FILE " $ct{'step'};$ct{'param1'},$ct{'param2'},;$ct{'check1'},$ct{'check2'},\n";

$ct{'step'} =~ s/[;,]/_/g;

$ct{'param1'} =~ s/[;,]/_/g;

$ct{'check1'} =~ s/[;,]/_/g;

print FILE " $ct{'step'};$ct{'param1'};$ct{'check1'}\n";

}

}

}

}

my $ts = time();

close FILE;

perl hexdump function (substr and x operator)

I can't quite believe I'm still writing hexdump functions.
I needed one again in perl and I had a basic one but fixed it up a bit more.

Interesting things: perl substr returns a warning/error if you request a substr outside range of the string. You should be able to persuade it to not produce warning. But I failed. perl v5.8.4 on solaris intel. From `perldoc -f substr`:
my $name = 'fred';
my $null = substr $name, 6, 2; # returns '' (no warning)

For padding make use of the `x` operator. `perldoc -q pad` `perldoc perlop`
http://perldoc.perl.org/perlfaq4.html#How-do-I-pad-a-string-with-blanks-or-pad-a-number-with-zeroes%3f
my $pad = " " x $padl;

Function hexdump is to be used in a file dumping different sections and in seperate calls hence possibility to pass in address. And generic pass in n - chars per line.

#!/usr/bin/perl -w

# n chars per line
sub hexdump {
    my $text = shift;
    my $addroffset = shift || 0; # optional
    my $n = shift || 20;
    my $pstr = "hack";
    my $addr = 0;

    # the extra awkward checks are to avoid warnings from substr
    until ($pstr eq "" || $addr>length($text) || !defined($pstr)) {
        $pstr = substr($text,$addr,$n);
        if (defined($pstr) && $pstr) {
           my $textaddr = sprintf "%08x", $addr + $addroffset;
           my $padl = 2 * ($n - length($pstr));
           my $texthex = join("", map { sprintf "%02x", $_ } unpack("C*",$pstr));
           my $pad =  " " x $padl;
           $pstr =~ s/[\x00-\x1f]/./g;
           $pstr =~ s/[\x7f-\xff]/./g;
           print "$textaddr: $texthex $pad $pstr\n";
        }
        $addr += $n;
    }
}

# one should check values of n (chars per line) 16, 20, others
# one should check hexdump(""); and 1 char, 2 chars up to n*2 + 1 chars at least
hexdump("1234567");
hexdump("MMMMMMMMMMMMOOOOOOOOOOOOOOORRRRRRR\nRRRRRRRRGGGGGGG\nGGGGGGEB1234567");
hexdump("MMMMMMMMMMMMOOOOOOOOOOOOOOORRRRRRR\nRRRRRRRRGGGGGGG\nGGGGGGEB1234567",2345);
hexdump("\x00\x02\x34MMMMMMMMMMMMOOOOOOOOOOOOOOORRRRRRR\nRRRRRRRRGGGGGGG\nGGGGGGEB1234567");
hexdump("\x78\x79\x7e\x7f\x80\x81\x82\xfe\xff\x00\x02\x34MMMMMMMMMMMMOOOOOOOOOOOOOOORRRRRRR\nRRRRRRRRGGGGGGG\nGGGGGGEB1234567");

my $file = shift;

if ( !open( FILE, "<$file" )) {
    warn "$file: $!";
    exit;
} else {
    print "Data from $file\n";
}

# read - text mode returns short lines 
#while ( ) {
#    hexdump $_;
#}

while ( read(FILE,$_,20)) {
    hexdump $_;
}

e.g. output:

$ ~/bin/hexdump.pl ~/bin/hexdump.pl 
00000000: 31323334353637                            1234567
00000000: 4d4d4d4d4d4d4d4d4d4d4d4d4f4f4f4f4f4f4f4f  MMMMMMMMMMMMOOOOOOOO
00000014: 4f4f4f4f4f4f4f525252525252520a5252525252  OOOOOOORRRRRRR.RRRRR
00000028: 525252474747474747470a474747474747454231  RRRGGGGGGG.GGGGGGEB1
0000003c: 323334353637                              234567
00000929: 4d4d4d4d4d4d4d4d4d4d4d4d4f4f4f4f4f4f4f4f  MMMMMMMMMMMMOOOOOOOO
0000093d: 4f4f4f4f4f4f4f525252525252520a5252525252  OOOOOOORRRRRRR.RRRRR
00000951: 525252474747474747470a474747474747454231  RRRGGGGGGG.GGGGGGEB1
00000965: 323334353637                              234567
00000000: 0002344d4d4d4d4d4d4d4d4d4d4d4d4f4f4f4f4f  ..4MMMMMMMMMMMMOOOOO
00000014: 4f4f4f4f4f4f4f4f4f4f525252525252520a5252  OOOOOOOOOORRRRRRR.RR
00000028: 525252525252474747474747470a474747474747  RRRRRRGGGGGGG.GGGGGG
0000003c: 454231323334353637                        EB1234567
00000000: 78797e7f808182feff0002344d4d4d4d4d4d4d4d  xy~........4MMMMMMMM
00000014: 4d4d4d4d4f4f4f4f4f4f4f4f4f4f4f4f4f4f4f52  MMMMOOOOOOOOOOOOOOOR
00000028: 5252525252520a52525252525252524747474747  RRRRRR.RRRRRRRRGGGGG
0000003c: 47470a474747474747454231323334353637      GG.GGGGGGEB1234567
Data from /home/james_coleman/bin/hexdump.pl
00000000: 23212f7573722f62696e2f7065726c202d770a0a  #!/usr/bin/perl -w..
00000000: 23206e206279206e0a7375622068657864756d70  # n by n.sub hexdump
00000000: 207b0a202020206d79202474657874203d207368   {.    my $text = sh
00000000: 6966743b0a202020206d792024616464726f6666  ift;.    my $addroff
00000000: 736574203d207368696674207c7c20303b202320  set = shift || 0; # 
00000000: 6f7074696f6e616c0a202020206d7920246e203d  optional.    my $n =
00000000: 207368696674207c7c2032303b0a202020206d79   shift || 20;.    my
00000000: 202470737472203d20226861636b223b0a202020   $pstr = "hack";.   
...
00000000: 3c46494c453e29207b0a23202020206865786475  ) {.#    hexdu
00000000: 6d7020245f3b0a237d0a0a7768696c6520282072  mp $_;.#}..while ( r
00000000: 6561642846494c452c245f2c32302929207b0a20  ead(FILE,$_,20)) {. 
00000000: 20202068657864756d7020245f3b0a7d0a           hexdump $_;.}.