Thursday, 3 November 2016

rsync from android MTP . . freezy/hangy problems. s/rsync/crude script/

Samsung Galaxy Nexus.
3.8G of photos in internal storage.
Options are MTP "mounting" or PTP.
Ubuntu 14.xx


First attempt copying using file manager . . . started out okay. Reported 3.8G will be copied over in 7 mins . . 15 mins . . later 1 hour and x mins. And it was getting stuck / hanging up (looked like at the same point).

Unplug/replug USB did not bring back MTP connection after hang. Needed to go to phone USB computer connection Settings and un-check and then re-check MTP to recover it.


cd /run/user/1000/gvfs/
cd mtp* # e.g. mtp::host=%5Busb%3A001%2C017%5D/
cd "Internal storage/DCIM"
rsync -avzhP ./ ~/Pictures/201611FionnOldMobilePhone/DCIM/
see errors like this:

          1.59M 100%    1.67MB/s    0:00:00 (xfr#1834, to-chk=957/2800)
          1.98M 100%    1.25MB/s    0:00:01 (xfr#1835, to-chk=956/2800)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C017%5D/Internal storage/DCIM/Camera/IMG_20130808_123846.jpg": No data available (61)
          1.43M 100%    1.57MB/s    0:00:00 (xfr#1836, to-chk=955/2800)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C017%5D/Internal storage/DCIM/Camera/IMG_20130808_123902.jpg": No data available (61)
          1.58M 100%    1.21MB/s    0:00:01 (xfr#1837, to-chk=954/2800)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C017%5D/Internal storage/DCIM/Camera/IMG_20130808_124320.jpg": No data available (61)
          1.54M 100%    2.00MB/s    0:00:00 (xfr#1838, to-chk=953/2800)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C017%5D/Internal storage/DCIM/Camera/IMG_20130808_124435.jpg": No data available (61)
          1.90M 100%    1.54MB/s    0:00:01 (xfr#1839, to-chk=952/2800)
rsync: read errors mapping "/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C017%5D/Internal storage/DCIM/Camera/IMG_20130808_124530.jpg": No data available (61)
         32.77K   1%  126.98kB/s    0:00:17 
. . restarted . .
. . stuck again . . 

. . seemed to be getting stuck on the same files . . 
These errors seem to be encountered if rsyncing from files which get changed while sync happens.  I think android should not be changing them. I don't think MTP would expose corrupt disk problem ? Something like antivirus locking the files should not be problem. Anyway, MTP connection a bit flaky . . maybe something on the android side is locking files or MTP drivers are a bit funny.
Tried rsync with --remove-source-files on whole and on file selections (e.g. Camera/IMG_201606*) still got stuck. Each time after hang needed to go into droid unselect MTP and select again to connect again. At least files to be checked/transferred are being reduced now by removing them after they are synced.

    rsync -avzhP --remove-source-files ./ ~/Pictures/201611FionnOldMobilePhone/DCIM/

rsync was getting stuck on the same file all the time. cp seemed to have less of a hanging problem. more tolerant of read errors? So made a script to do copy and remove if copy successful.

    # Script using cp -p (and remove files as we go)
    cd /run/user/1000/gvfs/mtp*/Intern*/DCIM
    FILES=$(ls Camera/)
    for f in $FILES; do echo -n $f; cp -p Camera/$f $DEST/ && rm Camera/$f && echo success; done

This was run in a while true loop periodically recovering MTP connection when it hung.
      while true; do ~/bin/; sleep 1; done

The script saw some funny errors e.g. :
    IMG_20130815_145455.jpgcp: error reading ‘Camera/IMG_20130815_145455.jpg’: Input/output error
    cp: failed to extend ‘/home/jamesc/Pictures/201611FionnOldMobilePhone/DCIM/Camera/IMG_20130815_145455.jpg’: Input/output error
    . . .

And it would hang . . . periodically recovered MTP connection as already described.
The script evolved . . . 

    Binary files Camera/IMG_20160104_095626.jpg and /home/jamesc/Pictures/201611FionnOldMobilePhone/DCIM/Camera//IMG_20160104_095626.jpg differ

cd /run/user/1000/gvfs/mtp*/Intern*/DCIM
FILES=$(ls Camera/)
ls Camera/ |wc -l
for f in $FILES; do
   echo -n $f;
   if [[ ! -e $DEST/$f ]]; then
      # the timeout doesn't work on frozen mtp file
             # we run this full script in a while true; do ~/bin/; sleep 1; done loop
             # and periodically visit, if stuck unlock phone, deselect MTP and select it again
             # (to interrupt and restart MTP connection)
             # simply unplug & replug doesn't bring MTP connection back ok
      timeout 20s cp -p Camera/$f $DEST/ && rm Camera/$f && echo success;
      echo already done by earlier script which did not tidy up;
      diff -u Camera/$f $DEST/$f
      if [[ $? == 0 ]]; then
         rm Camera/$f && echo RM
         # doing this copy on the weird files causes nasty freezy freezes.
         #cp -p Camera/$f $DEST/$f && rm Camera/$f && echo success;
                # ALOT of files found to be a few bytes different.
                # another copy caused hanging . . . so if dest file kindof looks ok we just remove source
         identify $DEST/$f && rm Camera/$f && echo RM


    # a few bytes in size different
    ls -al Camera/IMG_20150117_171621.jpg $DEST/Camera/IMG_20150117_171621.jpg
      -rw------- 1 jamesc jamesc 1277901 Jan 17  2015 Camera/IMG_20150117_171621.jpg
      -rwxr----- 1 jamesc jamesc 1275306 Nov  3 01:09 $DEST/Camera/IMG_20150117_171621.jpg
    # copy again ? results in same size difference, date/time fixed up though
    cp -p Camera/IMG_20150117_171621.jpg $DEST/Camera/IMG_20150117_171621.jpg
    ls -al Camera/IMG_20150117_171621.jpg $DEST/Camera/IMG_20150117_171621.jpg
      -rw------- 1 jamesc jamesc 1277901 Jan 17  2015 Camera/IMG_20150117_171621.jpg
      -rw------- 1 jamesc jamesc 1275306 Jan 17  2015 $DEST/Camera/IMG_20150117_171621.jpg
    diff -u Camera/IMG_20150117_171621.jpg $DEST/Camera/IMG_20150117_171621.jpg
      Binary files Camera/IMG_20150117_171621.jpg and $DEST/Camera/IMG_20150117_171621.jpg differ

So EVENTUALLY by removing files we had already copied we had less and less to do and got through all files. Phew. And yay.

    du -h $DEST/.. --max-depth=0
      3.6G    /home/jamesc/Pictures/201611FionnOldMobilePhone/DCIM/Camera//..

Various other reading . . .