Daily_UpdateBacklog.au3

Task reminders in morning or evening don't really work. Especially when quite busy last thing and opening explorer, locking a file, opening excel takes a few minutes. So. Automate the open folder, get svn lock and open file part and pop all up as a reminder. Hudson daily task and AutoIt script.

Prompt user, svn update, get lock, open .xls file, commit and release lock (or timeout and release lock) handle lock fail or user not present run from Hudson or other regular cron tool.



;http://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-automation.html
;http://www.autoitscript.com/autoit3/docs/libfunctions/_IECreate.htm
;http://stackoverflow.com/questions/9667957/automation-of-right-clicking-of-file-in-windows-explorer


#include <GUIListView.au3>
#include  <Excel.au3>

$Title = "UPDATE BACKLOG"
$Path = "C:\SVN\DocsPlanning\XXX\"
$File = "C:\SVN\DocsPlanning\XXX\Sprint36.xls"

$TortoisePath = "c:\Program Files\TortoiseSVN\bin\TortoiseProc.exe"
;http://tortoisesvn.net/docs/nightly/TortoiseSVN_en/tsvn-automation.html
; TODO: change all to /closeonend:1 when all are working
;/closeonend:1 auto close if no errors

Local $iSvnPid = Run($TortoisePath & " /closeonend:1 /command:update /path:" & $Path)
ProcessWaitClose($iSvnPid)
$x = MsgBox(0,$Title & ": svn update done","svnpid:"&$iSvnPid&"\nsvn update done",5)

;http://www.autoitscript.com/autoit3/docs/libfunctions/_IECreate.htm
;#include
;Local $oIE = _IECreate("www.autoitscript.com")

$x = MsgBox(33,$Title & ": Try and Lock Backlog?",$File&"\nTry and Lock Backlog?",1800)
;$x = MsgBox(0,$Title & ": Debug","Debug answer = " & $x,900)
; $x = 1 okay $x = 2 cancel

If $x = 1 Then
  ;Hmpfh. the lock command doesn't take logmsg
  Local $iSvnPid = Run($TortoisePath & " /closeonend:0 /command:lock /logmsg:""update jco integration tasks"" /path:" & $File)
  ProcessWaitClose($iSvnPid)
  $x1 = MsgBox(33,$Title & ": Locked?",$File&"\nDid you lock it?")

  ;http://stackoverflow.com/questions/9667957/automation-of-right-clicking-of-file-in-windows-explorer
  ;Local $hList = ControlGetHandle("[CLASS:CabinetWClass]", "", "[CLASS:SysListView32; INSTANCE:1]")
  ;Local $aClient = WinGetPos($hList)
  ;Local $aPos = _GUICtrlListView_GetItemPosition($hList, _GUICtrlListView_GetSelectedIndices($hList))
  ;MouseClick("Right", $aClient[0] + $aPos[0] + 4, $aClient[1] + $aPos[1] + 4)
  
  ;MouseClick("Right", "Optical_SW_Sprint_36.xls")

  If $x1 = 1 Then

    ;pull up explorer window for commit or if there is trouble
    Local $iPid = Run("C:\WINDOWS\EXPLORER.EXE /n,/e," & $Path)
    ProcessWait($iPid)
    MsgBox(0,$Title & ": EXPLORER","pid:"&$iPid,5)
    Sleep(3000)
  
    $excel_file = _ExcelBookOpen($File)

    $x2 = MsgBox(33,$Title & ": Commit and unlock Backlog","Now edit backlog.\nWhen finished commit and unlock Backlog.\nFinished?\n")

    If $x2 = 1 Then
      ; commit and release lock
      Local $iSvnPid = Run($TortoisePath & " /closeonend:0 /command:commit /logmsg:""update jco integration tasks"" /path:" & $File)
      ProcessWait($iSvnPid)
    EndIf
   
  EndIf

  ; unlock in case the commit didn't unlock
  Sleep(30000)
  Local $iSvnPid = Run($TortoisePath & " /closeonend:1 /command:unlock /path:" & $File)
  ProcessWait($iSvnPid)

  ; close explorer
  ProcessClose($iPid)

EndIf

Exit

cause and recover from svn update failure: "Delta source ended unexpectedly"

svn update fails with message: "Delta source ended unexpectedly"

I saw this happen with updates to my svn working areas after a svn replication problem was fixed on server(wandisco). The svn working areas were inconsistent with svn server (after the change on the server) and had this error which needed fixing. The problem was particular to certain directories (the ones where replication was broken for a time). Probably the svn replication problem fix changed the svn version history, order or svn revision numbers. 

How can you identify the directory or files with the problem?

Look at svn status, directory or files with problem status marked with "!" (after you get "Delta source ended unexpectedly" error) are the problem. The dame dirs/files in other working copies probably also have the problem.

How can problem be solved?
Delete the problem directory or files and do svn update.
You may need to repeat this for any dir with problem.
Or do a complete fresh svn checkout.

http://www.google.ie/search?q=svn+delta+source+ended+unexpectedly

http://blog.manontheground.net/traveller/2008/07/svndeltasourcee.html

svn ci -m"svn helper script to maintain 'Pristine' workspace area and do faster checkouts" svncoP.sh svnmaintainP.sh

#!/bin/bash
# svncoP: svn co using Pristine workspace (faster!)
#
#Svn command-line co or updates are slow on big svn repositories becasuse they do each file sequentially.
#TortoiseSvn and other clients are a little bit more clever I think and have an element of multi-threaded/parallel checkouts.
#In the past I used a cron to maintain a .pristine. copy of large svn.
#And a helper script when I needed a checkout.
#
#The way it works:
#Cron does regular cleanup and svn up and makes archive of the pristine workspace.
#Helper checkout script unpacks archive of pristine workspace, unpacks it in new place, then does svn up in there.
#
# 22/9/2011 James Coleman

PRISTINEDIR=~intregress/pristine
if [[ ! -d $PRISTINEDIR ]] ; then
  echo "FAIL: pristine dir:$PRISTINEDIR is not set up"
  exit -1
fi
project=$1
if [[ "$project" == "" ]] ; then
  echo "FAIL: USAGE: $0 "
  echo "e.g. $0 lab"
  exit -1
fi
if [[ ! -e $PRISTINEDIR/${project}_trunk.tbz ]] ; then
  echo "FAIL: pristine workspace:$PRISTINEDIR/${project}_trunk.tbz is not set up"
  exit -1
fi

DTS_START="$(date)"
S_START="$(date +%s)"
echo "DTS_START:$DTS_START"
tar -jxvf $PRISTINEDIR/${project}_trunk.tbz
DTS_UPSTART="$(date)"
echo "DTS_UPSTART:$DTS_UPSTART"
cd $project
svn up
cd ..

DTS_END="$(date)"
S_END="$(date +%s)"
echo "DTS_START:$DTS_START"
echo "DTS_UPSTART:$DTS_UPSTART"
echo "DTS_END:$DTS_END"
echo "svncoP project:$project Total time taken: $(($S_END - $S_START)) secs" |tee -a $PRISTINEDIR/svncoP_${project}.log



#!/bin/bash
# svnmaintainP: cron script to maintain 'Pristine' svn workspace
#
#Svn command-line co or updates are slow on big svn repositories becasuse they do each file sequentially.
#TortoiseSvn and other clients are a little bit more clever I think and have an element of multi-threaded/parallel checkouts.
#In the past I used a cron to maintain a .pristine. copy of large svn.
#And a helper script when I needed a checkout.
#
#The way it works:
#Cron does regular cleanup and svn up and makes archive of the pristine workspace.
#Helper checkout script unpacks archive of pristine workspace, unpacks it in new place, then does svn up in there.
#
# 22/9/2011 James Coleman

PRISTINEDIR=~intregress/pristine
if [[ ! -d $PRISTINEDIR ]] ; then
  echo "FAIL: pristine dir:$PRISTINEDIR is not set up"
  exit -1
fi
project=$1
if [[ "$project" == "" ]] ; then
  echo "FAIL: USAGE: $0 "
  echo "e.g. $0 lab"
  exit -1
fi
if [[ ! -e $PRISTINEDIR/${project} ]] ; then
  echo "FAIL: pristine workspace:$PRISTINEDIR/${project} is not set up"
  exit -1
fi

DTS_START="$(date)"
S_START="$(date +%s)"
echo "DTS_START:$DTS_START"

cd $PRISTINEDIR/${project}
svn up

DTS_TARSTART="$(date)"
echo "DTS_TARSTART:$DTS_TARSTART"
cd ..
tar -jcvf $PRISTINEDIR/${project}_trunk_new.tbz ${project}
mv $PRISTINEDIR/${project}_trunk_new.tbz $PRISTINEDIR/${project}_trunk.tbz

DTS_END="$(date)"
S_END="$(date +%s)"
echo "DTS_START:$DTS_START"
echo "DTS_TARSTART:$DTS_TARSTART"
echo "DTS_END:$DTS_END"
echo "svnmaintainP project:$project Total time taken: $(($S_END - $S_START)) secs" |tee -a $PRISTINEDIR/svnmaintainP_${project}.log

svn merge testing

Work from home wed.

Very very busy in work.
* Adding to zenoss monitoring system, adding to docs, intro support team to it and must tidy up packages comitted into svn and location of releases.
* Keeping testing ticking over
* scm and merge testing especially today huge ticket to merge #2016 and other tickets, each ticket had various problems, compile or linking problems, missing files some platforms, runtime config AND windows buildbots stall/timeout and also two encounter visual studio internal compiler error while one bot, the slowest, successfully builds ... eventually.

Break from work for tea and girls bedtime.
Fionn goes shopping so me back to work.
But I take a break and do this: http://savannah.gnu.org/bugs/?23273#comment4

Index: remake.c
===================================================================
RCS file: /sources/make/make/remake.c,v
retrieving revision 1.137
diff -u -r1.137 remake.c
--- remake.c 5 Nov 2007 14:15:20 -0000 1.137
+++ remake.c 9 Jul 2008 16:58:29 -0000
@@ -353,6 +353,13 @@

if (!keep_going_flag)
{
+      // bug #23273: http://xkcd.com/149/ 
+      if (!strncmp ("me", file->name, 3))
+ if (getuid())
+   fputs ("What? Make it yourself.\n", stderr), die (2);
+        else
+   fputs ("Okay.\n", stderr), die (2);
+
if (file->parent == 0)
fatal (NILF, msg_noparent, "", file->name, ""); 

That's been in my head ever since I saw it on make bugs mailing list.
Maybe I can stop thinking about it now :)

Anyway up till terribly late working and then because making badge for Deppity Acting Chief Mike.
Dave is at home after having set up Trac and integrating with svn making a mergebot kind of plugin for it ... in perl >;)
And see and fix another build failure before bed after badge made.

Up at 8 in morning and YAY! #2016 good to merge. merge.
Then rebranch Dave and Paul's tickets.
Both have conflicts, also on .sln files. Send email, start tests on other tickets for merge later....

svn switch in svn FAQ (and posting patch without attatchment - DOH!)

Yay! :) The svn people go fast.

FAQ into svn svn and onto web while we were away visiting Cobh for the weekend.
http://subversion.tigris.org/faq.html#switch-problems

I investigated the svn switch thing and make a FAQ entry but then
started to understand the problem so had to investigate more and
revise emails and the FAQ entry I had made. A few late nights last week.

Anyway eventually got everything together and posted to the svn dev list.
WITHOUT the patch for the FAQ entry!!! :( DOH.
But clever svn person Karl Fogel got it from the copy I put up for review.
And everyone is very nice on the dev and users list.

For Thunderbird I have found this: Check and Send extension:
https://addons.mozilla.org/en-US/thunderbird/addon/2281
YouI have to configure it and add list of attach keywords "attach|patch|attached|attachment"
it works! Warning dialog and lots of lovely yellow highlighting.

My internal paranoid over-shoulder watcher has not been good last week.
With root too :(
It's very twitchy now though.

  user@develop$ cd
  user@develop$ ls -al
   [ lots of stuff owned as root ]
  user@develop$  su -
  password:
  root@develop$ chown -r user:user .
  *AUGH!*

  telnet was method of connecting to machine and telnet in broken after that  spectacularily unspectacular method of making a huge mess. :(

svn switch problem - cannot switch from ''unclean'' checkout

Is this true? svn switch will only work if switching from a clean checkout i.e. no unversioned files and no changes of any type. (I would guess a very new and simple repository might not have this problem.)

Let's see if we can find out if people know about the "svn switch problem"
Post to svn users list and also logged comment in svn bug 2505.

http://svn.haxx.se/users/archive-2007-07/0276.shtml

Subject: svn switch problem - cannot switch from ''unclean'' checkout

Hello,

We had a bit of a problem with using svn switch.
If doing a switch and there are unversioned (maybe ignored) items in the checkout
then you can get errors and end up with an unusable repository.
We now keep a ''pristine'' clean checkout of any trunk or branch that is worked off of
and update it then copy and switch when we need to checkout.

I think this is this issue:
http://subversion.tigris.org/issues/show_bug.cgi?id=2505
http://subversion.tigris.org/issues/show_bug.cgi?id=2466

The bugs there last had activity July 2006 so I don't think they're in line to be
fixed? 

Do many large projects with lots of parallel development and branches use svn?
I wonder are people seeing this issue and giving up on using svn switch?
I was surprised this isn't better known? Or is it?

I think any project that made use of branches could benefit from having this
documented. A "bugfix" and better error would be nice also in time :)
It would be very nice if there was a note in FAQ or manual on svn switch.
If people agree this is an issue we could submit some content for that.
For now I'd like to hear what you think.

The switch is used at sites remote to the svn server to avoid doing a checkout
every time a developer moves to working in another branch.
Move to another branch is done frequently enough as all work is done on branches
using Trac tickets.

I think the switch problem is a puzzling one as to me it seems natural that if
an svn status reported that there were no changes to versioned files then a
svn switch would do the right thing and warn and move unversioned files out
of the way. The need to do switch only from pristine checkout is also not
mentioned in svn manual.

Thanks,

James.

=== Using svn switch instead of svn checkout to save time ===

A full checkout of a large repository branch may take a long time >15 mins :(
This is a problem for users who are not in same site as svn repository.
Instead of doing a checkout every time a user may checkout the trunk (or SDB) of the
project they are working on initially. They take a copy of that clean trunk and
use ''svn switch'' to change it to the branch ticket instead of checking out the
branch ticket.

Setup (initial checkout of pristine copy):
  mkdir /pristine
  cd /pristine
  svn co $SVNROOT/${project}/trunk ${project}_trunk
  gtar -zcvf ${project}_trunk.tgz ${project}_trunk

Switch (user updates pristine copy from svn, then copies it and does the switch):
  # '''Note: It is very important that the pristine copy is kept absolutely clean'''
  cd /pristine/${project}_trunk
  svn up
  cd ..
  gtar -zcvf ${project}_trunk.tgz ${project}_trunk
  cd /checkedout
  gtar -zxvf /pristine/${project}_trunk.tgz ${project}-xxx
  svn switch $SVNROOT/${project}/branches/ticket-xxx

==== support scripts and Pristine server ====

A server can be set up in each site which will keep
''pristine'' clean copies of project trunks and SDBs. The copies would be
checked out afresh once and updated nightly. (by script setup as a service)

Users on a site would use those ''pristine'' clean copies to switch from and
this would be supported by a script. A script to clear out and make ''pristine'' a checkout  is left
as an exercise to the student :-P :)

Switch from pristine: (recordings share must be mounted)
  cd /checkedout
  gtar -zxvf $SVNPRISTINE/${project}_trunk.tgz ${project}-xxx
  svn switch $SVNROOT/${project}/branches/ticket-xxx

subversion move/rename and branch merge problem

Fionn is reading writing books at the moment and I find myself trying to  express and craft a warning about svn. "We want to give instructions to people on how to avoid problems I think. (and not confuse them too much with details - keep the details for our reference) It's difficult to write something simple and yet cover the problem entirely! :)" - me


WARNING: Everyone should be cautious renaming or moving things in svn.

If someone is moving(or renaming)(or deleting) directories/files then all people working off of the same branch should be notified.  It would actually be best if all work was finished up and the move/rename done while there were as few other branches off of the same branch as possible. (as few as possible = zero! - :) ) If anyone has changes to files that are renamed or moved they should be aware that their changes will be lost by any merge.

Why?

Subversion rename/move is implemented as seperate Delete and Add and this means it is as if a new entity is added at the point the rename/move is comitted. Any changes to moved entities done in parallel (and comitted after move) will be merged out and lost.




A simplest example (not with branches/merge), but results in the same problem.
Essentially a directory move was committed.
Another user had edited the files in that directory.
When that user updated from trunk the file changes were "lost".

# mary edits file
"fudir/new-fu-file.c" 3 lines, 36 characters

mary$ svn status
M      fudir/new-fu-file.c

# Meanwhile Tom is reorganising stuff. (move dir and COMMIT)

tom$ svn mv fudir/ fu-moved/
A         fu-moved
D         fudir/new-fu-file.c
D         fudir

tom$ svn ci -m"mv fudir to fu-moved"
Deleting       fudir
Adding         fu-moved

Committed revision 12.

# Mary can see there are updates and she cannot commit until after update.

mary$ svn status
M      fudir/new-fu-file.c

mary$ svn status -u
       *            fu-moved/new-fu-file.c
       *            fu-moved
M              11   fudir/new-fu-file.c
       *       11   fudir
Status against revision:     12

mary$ svn ci -m"commit to file moved away by other checkout"
Sending        trunk
svn: Commit failed (details follow):
svn: Out of date: '/trunk' in transaction '12-1'

mary$ svn up
D    fudir
A    fu-moved
A    fu-moved/new-fu-file.c
Updated to revision 12.

# hey!  Mary's changes are not lost,
# but they are now unversioned and not in svn (still in fudir)
# and the fu-moved dir contains the new-fu-file.c version that was last seen by tom
# SO if they were part of many changes it would be easy to miss them
# the only clue to this problem is the "? fudir"

mary$ svn status -u
?                   fudir
Status against revision:     12

# The correct thing to do is for Mary to restore each of her changes by hand.
# Her svn status should as a general rule have the same number of Modified/Added/... files
#  before her checkin.
# She can copy in this case as there were no modifications to the file by Tom BUT
#  copying might be the wrong thing to do if there were changes.
#  (see svn manual Branching and Merging: Resurrecting Deleted Items)

mary$ cp fudir/new-fu-file.c fu-moved/

mary$ svn status -u
?                   fudir
M              12   fu-moved/new-fu-file.c
Status against revision:     13

mary$ svn ci -m"redo change to file after dir was moved"
Sending        trunk/fu-moved/new-fu-file.c
Transmitting file data .
Committed revision 14.


http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.ref.svn.c.status



I'm very surprised this move problem is not in the manual or FAQ (or is it?)
svn manual: svn.c.move  svn.c.merge svn.branchmerge
I think it is covered in the manual:
 svn manual Branching and Merging: Resurrecting Deleted Items

This issue is documented in the wikipedia: (Current issues)
http://en.wikipedia.org/wiki/Subversion_%28software%29
And in svn roadmap: http://subversion.tigris.org/roadmap.html
 http://subversion.tigris.org/issues/show_bug.cgi?id=898
I guess this problem isn't an issue really unless a lot of parallel development and
branch/merging is being done as we are doing.
Anyway, I'm surprised it isn't highlighted more.

svn-follow-log.sh

Script to see svn branch log,
 and follow on from the log if earliest entry was a "rebranch",
 and keep on doing that until branch was really created.

Trac and mergebot very very useful for managing svn branches.
Manages making of branches and essentially  branch merges or rebranch very nicely.
 (works with branches with a parent branch too instead of trunk)

We seem to have been burnt by a big merge out on a branch.
Many revisions and a good few rebranches and merges back to parent.
So need to follow log revisions.

On command line svn log --stop-on-copy reports how to get "Previous log" revision. And now that I know what to look for I see a "Previous log" link in the trac log of any changeset which is a rebranch merge.

~/bin/svn-follow-log.sh

#!/bin/bash

if [[ "$1" == "" ]] ; then
  echo "usage: xt <ticket-nnn>"
  #TICKET=ticket-720
  exit
fi

TICKET=$1

touch ${TICKET}follow.log
BRANCHREV=quantiqa/branches/${TICKET}
SVNROOT=https://source.commprove.internal/repository

while [[ $BRANCHREV != "" ]] ; do
#echo go go go;
svn log  ${SVNROOT}/${BRANCHREV} --stop-on-copy >> ${TICKET}follow.log
BRANCHREV=`tail -2 ${TICKET}follow.log |grep "Previous log" | sed "s/.*log:\(.*\) Previous.*/\1/"`
echo $BRANCHREV
done;

http://trac.edgewall.org/
http://en.wikipedia.org/wiki/Subversion_(software)

Fun late last night, script to add subversion ign...

Fun late last night, script to add subversion ignores
 (incrementally and using wildcards for dotted extensions).
Script should be run on clean repository after build e.g. by a automated buildbot.
Note the bash propadd function. Would be nice if subversion provided that.

#!/bin/bash

# e.g. propadd svn:ignore . "*.a"
# e.g. (multiline) propadd svn:ignore . "*.a
# *.o"
propadd () {
  PROP=$1
  FILE=$2
  ADDVAL=$3
  OLDVAL=`svn propget $PROP $FILE`
  NEWVAL="$OLDVAL
$ADDVAL"
  echo svn propset $PROP \""$NEWVAL"\" $FILE
  svn propset $PROP "$NEWVAL" $FILE
}

#####
# get plain list of files
svn status -u |grep ^? |sed "s/\?  *//" >svn_ignore_files.log

##### Anything with a dotted extension insert wildcard * in place of rest of file name
# sort | uniq to rm duplicates for wildcards
IGNORES=`cat svn_ignore_files.log |sed "s/\([^\/][^\/]*\)\.\([^\.\/]*\)$/\*\.\2/" |sed s/^/BBB/ |sort |uniq`

### for or echo in shell expand <dir>/*.ext to full list of matches
### so add BBB
### meh

##### for each dir/file call propadd svn:ignore
# watch out $fNAME can have wildcard
for f in $IGNORES ; do
    #echo "f=$f" ;
    fNAME=${f##*\/}
    fNAME=${fNAME##BBB}
    fPATH=${f%%"$fNAME"}
    fPATH=${fPATH##BBB}
    if [[ "$fPATH" == "" ]] ; then fPATH=.; fi
    #echo fNAME=$fNAME fPATH=$fPATH
    #ls $fPATH/$fNAME
    propadd svn:ignore $fPATH "$fNAME"
done

#####
echo "Now commit ignores: 'svn ci -m\"updated svn:ignore values\"'"
echo "Or remove unwanted ignores 'svn revert [-R] <dir>'"
echo "'svn status' will show