DebianDog - Wheezy

[fredx181]

Ah, so its because the pid of rsync is changing as copying proceeds? So CPPID won't be holding the correct rsync pid value when the 'while running loop' checks to see if rsync process finished yet (so it wrongly thinks rsync process is complete once the first rsync CPPID value is no longer detected as 'running' and thus moves on to the clean up stuff, which it shouldn't...).

Yes, maybe, it can be because there are multiple rsync pids and this:

Code: Select all

while running $CPPID; do

Probably holds only one.

Have to run now, get back tonight.

Fred

[mcewanw]

Probably not perfect, but I guess this would work for that line 125 instead of using while running $CPPID:

Code: Select all

while [ $(pidof rsync) ]; do

Of course if any other program was also using rsync as this was going on that would be a problem, but I guess that shouldn't be the case during remastering.

William

EDIT: I'm not convinced about the wait being a genuine solution - what is it waiting on? Certainly the command itself will introduce a delay that may just be long enough to let rsync truly finish, but that is no better than sleep. Or maybe the wait is waiting on some child process to complete or something, but I don't see what. I still think fixing the while loop rsync still running test is the logical solution. Or maybe the wait is working since it should be waiting on all currently running child processes to exit, which would be rsync processes I presume? - maybe...

EDIT2: Yes, I think that is why wait (without number after it) is working - it is waiting on all background processes (started earlier by that script) to complete, which is the rsync - but I feel that is a bit of a workaround rather than a solution to the original code fault, which remains (harmlessly?) in place.

EDIT3: It won't be quite 'harmless' anyway in the sense that the progress bar will more suddenly terminate even whilst the actual rsync processes are still running (albeit with the wait holding back the execution of the cleanup code that comes thereafter).

[saintless]

I think we can keep the included/upgarded yad ( 0.28 ) Fred.

Thanks, Toni, glad it's not because of a 'yad infection'.

Yes, I'm ready even to accept my both machines have some SDRAM or CPU bug and recompiling yad makes it work better on both. But we have many scripts to check and maybe fix.

Now the problem is located and I leave the best way to fix it to you and William. I will test whatever you decide on my machine. If it works I'm sure it will work on any other hardware.

Edit: BTW it took 6 months after the first report to find the real problem:
http://www.murga-linux.com/puppy/viewto ... 905#829905
Trust my machine test reports in the future

Toni

[fredx181]

Hi Toni, William

Trust my machine test reports in the future Smile

Yes, seriously, you are absolutely right, not accept mysteries as 'nothing can be done' to easy in the future.

Spend the whole evening trying to figure out what's the best for the remasterdog progress bar, interesting exercise but at the same time frustrating, (as almost always ) because didn't find good solution yet.
I think I agree with William that adding the 'wait' is a workaround (which works perfectly, btw) for a faulty progress bar code.
Continuing tomorrow...

Fred

[fredx181]

Hi Toni, William,

For testing, specially by our headchief tester Mr Toni ,remasterdog with a (hopefully) fixed progress bar, attached.
I tested mostly with working directory on a very slow pendrive, to simulate as much as possible your situation, Toni.

The problem was as I already thought because of not accurate comparing between source and destination (TOTAL in script is source, COPY is destination).
As the size of COPY at the end is a little larger than TOTAL, the progress bar closed to soon, with the result that the script continued, while the rsync processes were still busy.
The accuracy of comparing I tried, but couldn't make it right, so as a workaround, during the progress display there's a check if rsync processes are present, -sort of- the same as 'wait' after progress bar, but now during progress bar display.

The disadvantage of Toni's 'wait' solution is that in some situations there could be a long time that nothing happens/displays after the progress bar has closed, which scares people, (also including me, btw )

The progress code looks chaotic now, because of experiments (lots of commented out lines), will clean up later, first see if this works OK.

Btw, added also a cancel option, by closing progress window, needs testing too.

The other scripts with progress are probably easier to fix (without rsync) or maybe some won't need a fix at all.

PS, Please try not to get yourself down by some idiot, once in a while they appear somewhere, most people are not like that.

PS2: What I learned from this is that I should always take into account that a script can possibly be run from a very slow computer, and consider what might be the consequence of that.
And that progress bar codes are even more trickier than I thought!

Fred

[saintless]

Thanks Fred.
Works well on my machine few times with both yad versions from dd-repo.

Btw, added also a cancel option, by closing progress window, needs testing too.

Tested once and I get message window about copy out after cancel could take few minutes. No further message as "Done" or similar.
I'm not sure it is good idea to add extra functions to the progress bar.
Isn't it better and safer just to fix the bug without adding new option in the scripts? New option could lead to new bug and maybe it will appear after months.

Toni

[fredx181]

Hi Toni,

Code: Select all

I'm not sure it is good idea to add extra functions to the progress bar.
Isn't it better and safer just to fix the bug without adding new option in the scripts? New option could lead to new bug and maybe it will appear after months. 

Well, I just thought it's lacking cancel option, and since we are testing, better add it at the same time.
What's the point, you and I can test it and we can ask William to do also.

The message, btw, It's almost same William made on debdog-install after cancel, can be easily changed of course if you want.

Fred

[saintless]

Well, I just thought it's lacking cancel option, and since we are testing, better add it at the same time.

I don't mind Fred, but how can I confirm it works?
The working folder is deleted and I get the message it will take few minutes to copy out. If this is enough as testing leave it as it is.

Toni

[fredx181]

I don't mind Fred, but how can I confirm it works?
The working folder is deleted and I get the message it will take few minutes to copy out. If this is enough as testing leave it as it is.

You mean you would expect extra popup "Done" or similar?
Please tell, I'd like that you are satisfied with it
Maybe add to the message also that the working directory is deleted ?
As I said it's raw testing version, need to clean up progress code and change cancel message a little maybe.

Glad it works for you now, btw!

Fred

[saintless]

Yes, adding the working folder is deleted will be nice.

When the message says it will take few minutes to copy out it sounds like I have to wait few minutes doing nothing or waiting for further information after few minutes.

In my opinion if there is no problem to reboot the computer or do other work without waiting few minutes - then there is no need to give information it will take few minutes to copy out. Or add information no need to wait if you want to reboot.

If there is a problem to reboot before the copy out process is finished - then it is better to add "All done" message window when it is safe to reboot.

Toni

[fredx181]

Hi Toni,

When the message says it will take few minutes to copy out it sounds like I have to wait few minutes doing nothing or waiting for further information after few minutes.

Probably it's just to technical message and it's just enough to say that the copying has been cancelled and working dir will be deleted within a minute.

Fred

[saintless]

...copying has been cancelled and working dir will be deleted within a minute.

Yes, it will be enough as information. I like it.

Toni

[saintless]

Hi Fred.

We have important difference in sources.list in DD-Wheezy OpenBox and Jwm.
In Jwm smokey01.com is at top and in OpenBox it is at the end.
This means special packages from dd-wheezy repo willl not be installed with priority in OpenBox version.
I don't mind to change it as you prefer. This is the sources.list I plan to use in DD-Wheezy-Jwm:

Code: Select all

#DebianDog repository:
deb http://www.smokey01.com/saintless/DebianDog/Packages/ ./ 

deb http://ftp.us.debian.org/debian wheezy main contrib non-free
#deb-src http://ftp.us.debian.org/debian wheezy main contrib non-free

deb http://ftp.debian.org/debian/ wheezy-updates main contrib non-free
#deb-src http://ftp.debian.org/debian/ wheezy-updates main contrib non-free

deb http://security.debian.org/ wheezy/updates main contrib non-free
#deb-src http://security.debian.org/ wheezy/updates main contrib non-free

#Debian Multimedia
deb ftp://ftp.deb-multimedia.org wheezy main non-free

#Default-Original-Sources:
#deb http://http.debian.net/debian/ wheezy main
#deb-src http://http.debian.net/debian/ wheezy main
#deb http://security.debian.org/ wheezy/updates main
#deb-src http://security.debian.org/ wheezy/updates main
#deb http://http.debian.net/debian/ wheezy-updates main
#deb-src http://http.debian.net/debian/ wheezy-updates main

Toni

[fredx181]

Hi Toni,

We have important difference in sources.list in DD-Wheezy OpenBox and Jwm.

I will make it same as you above posted in next OB iso.

About progress bar scripts:
I am testing now to see if any other scripts have the same problem and couldn't find anything wrong yet.
I found good way to debug since I know what the real problem could be (the comparison between source and destination).
It's only about scripts that have the TOTAL > COPY comparing
I will test more but my Q now is:
Shall we change as less as possible?
From what I tested, remasterdog is an exception because of combination of the --exclude parameter for rsync and du commands and the fact that rsync has multiple process ID's.

In case I really don't find any problems:
If you want, for safety you always can add 'wait' after every progress bar code.

But just say the word if you like them all changed the way remasterdog is now.

PS, remasterdog gave problem because of removal tasks after copying.
That isn't the case for e.g. debdog-install (as copying is the last step of the script).

Edit: Can you test for example remastercow and remaster-kernel by adding:

Code: Select all

pidof cp

After progress code, and see if it shows anything?
For me it doesn't, so it seems OK.

Fred

[saintless]

Hi Fred.

Edit: Can you test for example remastercow and remaster-kernel by adding:

Code: Select all

pidof cp

After progress code, and see if it shows anything?
For me it doesn't, so it seems OK.

Seems OK on my machine also. Remastercow tested after installing 700Mb extra so it should be OK.

In my opinion we should change less as possible and testing cp command with progress bar doesn't show the problem I had with remasterdog.
But I like to hear if William has different opinion.

Maybe it is better to add wait just in case after each progress bar but this means rebuilding, reuploading and reinstalling many packages without good reason since I can't find a problem on my machine.

Toni

[fredx181]

Hi Toni,

The only script besides remasterdog I could find with the problem of progress bar closing earlier than the copy process finish is debdog-full-install.
Although it's not a real problem, as there's no cleaning done after the copying, I modified the progress code the same way as remasterdog..
(it needed a fix anyway for the y/n prompt when run from terminal, which is done now by "echo y | mkfs.ext4 /dev/$PART")

Attached debdog-full-install_remasterdog.tar.gz (remasterdog with cleaned up progress code and changed cancel message)

(and loadsfs-fuse, yes probably forgot to set library path /opt/lib)

Just for info, I didn't forget to set library path /opt/lib and I tested with 024-goffice-lupu-528JP-1-jessie.squashfs on Jwm-Jessie version and loadsfs-fuse works fine for me without sudo or gsu (logged in as puppy). So... Well done! , you created another mystery
But I'd suggest to move on to more important things instead trying to investigate this any further.

Ok... where was I, oh yes, making preparations for new iso release.

Fred

[mcewanw]

In my opinion we should change less as possible and testing cp command with progress bar doesn't show the problem I had with remasterdog.
But I like to hear if William has different opinion.
Toni

Hi Toni, Fred,

Well I just had a quick look at Fred's modified remasterdog and I note that he is no longer using CPPID at all and instead has an infinite while loop and another while loop inside that which tests for the existence of a $RSYNCPID value to determine whether to break out of the outer while loop... :-) whew... too little time just now and too much to get my head round.

I don't really know (which is a usual state for me sometimes - so means little) what was wrong with my own suggestion of

Code: Select all

while [ $(pidof rsync) ]; do

instead of:

while running $CPPID

at line 125 in the previous remasterdog, but I'm not surprise if there was some flaw I never thought of (and I never had time to try it, so miracle if it had been correct...).

So, in summary, if Fred's new version is working without problem, I'm happy with that - he knows his program better than I ever could! I hope you are correct Fred!

William

[fredx181]

Hi William,

I try always to avoid pidof , because (as you said already) in theory there can run another process of a program (in this case rsync)
Very little chance, of course, in case of remasterdog, but I have some sort of rule to always determine the exact pid(s).
Same goes for 'killall', it can be easy solution, but not a good practice to use in scripts IMO.
(but I'm sure there are exceptions).

Btw, I'm always struggling with while loops, I don't know, maybe the progress code can be made more simple or compact.

Edit: Below edits, for information, hopefully someone once might create a solid bash code for a copy progress bar (copy directory to another directory) by using yad, zenity or Xdialog, I didn't succeed yet without using some (ugly?) workarounds.

Edit1:

I note that he is no longer using CPPID at all and instead has an infinite while loop and another while loop inside that which tests for the existence of a $RSYNCPID value to determine whether to break out of the outer while loop...

Toni found out that adding 'wait' after progress code worked but disadvantage is that there could be a long delay after progress bar has closed.
My code, in fact, adds a wait while progress bar is still visible.

Edit2: I've tried "while [ $(pidof rsync) ]; do", btw, but it didn't solve the problem that the progress bar closed earlier than the rsync processes.
The problem was because of not accurate calculating the size of TOTAL (source) compared to COPY (destination) values.
(COPY is more than TOTAL at the end, so progress bar closes when reached at 100%, but in fact it's around 99% that has been copied at that moment, specially on slow computer or writing to slow media, such as a flashdrive)

Edit3: Forgot to mention; the first thing I tried to solve was the accuracy of the TOTAL and COPY values and found, by google search, some examples using find piped to du, but CPU usage became much higher because of that, so not a good option, I think.
So, making a perfect copy progress bar code without some workarounds seems not possible to me.

Fred

[saintless]

Hi Fred, William.
Repositories updated with latest changes in remasterdog and debdog-full-install
http://kazzascorner.com.au/saintless/De ... 3_i386.deb
http://kazzascorner.com.au/saintless/De ... 4_i386.deb
I moved the packages from Testing in Extra and from Updated inside Included to make the updaing easier for me in the future.

I will test more the latest changes. As long the script works on my machine without troubles I don't mind any way of fixing the rsync copy progress bar.

Toni

[mcewanw]

but I have some sort of rule to always determine the exact pid(s).
Same goes for 'killall', it can be easy solution, but not a good practice to use in scripts IMO.

Hi Fred,

Yes, I agree with your comment. I entirely avoid such killall type solutions in precord and pavrecord too (e.g. to kill running arecord processes) - it's very important to try and identify the exact pid of the exact processes you are interested in - anything else is a bit of an ugly fudge IMO, which can have serious unexpected side-effects (especially on a multi-user system...)

I actually didn't study your new code sufficiently to recognise it was determining the pids exactly, though it crossed my mind that that might be your reason for how you wrote your solution, which was what I was hoping for as an improvement too. Not at all easy to determine the exact pids for rsync I felt - the major complication in trying to fix the code really.

William