new save script
new save script
I'm currently rewriting some of the main Puppy scripts. I now have a completely new init-script and a completely new save-script.
The results of the modified save-script are really astonishing!
Here are the main differences:
First of all I use aufs instead of unionfs which allows for moving files between layers without remounting.
The save script neither copy or move a file if not necessary. I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents. Because reading of flash memory is so much faster than writing I first compare the files and copy or move the file only if it is different to the version stored in the save-file (or if it is a new file).
If a file is not opened by any process it is just moved to the save-file otherwise it is copied.
I also fixed a bug in the original script which made files reappear after reboot under some circumstances. (A file in the squashfs-layer reappeared if the file also existed in the save-file and was deleted.)
Now the benchmark results:
For an average work load the old script took about 1 minute to save the changes. This time increased for each new run of the script (every half an hour) because the ramdisk was never cleared.
The new script takes about 10 seconds for the first save and - if rerun - about 2 seconds ...
I will test the new script in everyday work during the next days (making more backups than usually ^^). If everything works as expected I will post the script here. The script should be 100%-compatible with the existing one - I use aufs in unionfs-compatibility-mode.
The results of the modified save-script are really astonishing!
Here are the main differences:
First of all I use aufs instead of unionfs which allows for moving files between layers without remounting.
The save script neither copy or move a file if not necessary. I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents. Because reading of flash memory is so much faster than writing I first compare the files and copy or move the file only if it is different to the version stored in the save-file (or if it is a new file).
If a file is not opened by any process it is just moved to the save-file otherwise it is copied.
I also fixed a bug in the original script which made files reappear after reboot under some circumstances. (A file in the squashfs-layer reappeared if the file also existed in the save-file and was deleted.)
Now the benchmark results:
For an average work load the old script took about 1 minute to save the changes. This time increased for each new run of the script (every half an hour) because the ramdisk was never cleared.
The new script takes about 10 seconds for the first save and - if rerun - about 2 seconds ...
I will test the new script in everyday work during the next days (making more backups than usually ^^). If everything works as expected I will post the script here. The script should be 100%-compatible with the existing one - I use aufs in unionfs-compatibility-mode.
- Dougal
- Posts: 2502
- Joined: Wed 19 Oct 2005, 13:06
- Location: Hell more grotesque than any medieval woodcut
Re: new save script
I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?Leachim wrote:I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents.
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind
Some say your nose
Some say your toes
But I think it's your mind
Re: new save script
I've tried every version of 3.x I can find...and none of them are as easy to remaster a LiveCD as 2.17.1 was.Dougal wrote:I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?Leachim wrote:I found out that a lot of files in the ramdisk are unmodified copies of their save-file-equivalents.
Hopefully someone can write a script to do it.....
What you did for 2.17.1 saves so much time and frustration.
Thanks
That would be super.Dougal wrote:I actually have 3.01 now, so I can go over it and update my remaster script.
When I use the current remaster script, it appears that none of the scripts I save in /root, things in /root/.gaim, in /etc/ are not saved during the remastering.
Thanks in advance for your script.
- ttuuxxx
- Posts: 11171
- Joined: Sat 05 May 2007, 10:00
- Location: Ontario Canada,Sydney Australia
- Contact:
Dougal yes your old remaster script, kills vesa in the 3.0 series which as we all know is a bad thing, It would be nice if you could have a erase cdrw/dvdrw before burning option.
its been a while since i used your remaster script. I get the 2 confused, I know the puppy remaster scrip misses the root/.mozilla , so you don't get the themes or extension and you manually have to move it, also it forgets the rox settings in the .icons folder and few others, about 5 settings in /root.
A new remaster script would be better then sliced bread. At least I think so
Keep up the excellent work Dougal
ttuuxxx
its been a while since i used your remaster script. I get the 2 confused, I know the puppy remaster scrip misses the root/.mozilla , so you don't get the themes or extension and you manually have to move it, also it forgets the rox settings in the .icons folder and few others, about 5 settings in /root.
A new remaster script would be better then sliced bread. At least I think so
Keep up the excellent work Dougal
ttuuxxx
http://audio.online-convert.com/ <-- excellent site
http://samples.mplayerhq.hu/A-codecs/ <-- Codec Test Files
http://html5games.com/ <-- excellent HTML5 games :)
http://samples.mplayerhq.hu/A-codecs/ <-- Codec Test Files
http://html5games.com/ <-- excellent HTML5 games :)
- Pizzasgood
- Posts: 6183
- Joined: Wed 04 May 2005, 20:28
- Location: Knoxville, TN, USA
I know that whenever you load a module, if it's in the zdrv_xxx.sfs file it gets copied into the pup_save.2fs file to avoid having to use zdrv_xxx.sfs next time. Maybe the same code is applying to situations where it isn't actually needed?I've encountered that before, namely with kernel modules: it seems like modules that are in pup_ro2 and modprobed are, for some reason, copied into pup_rw! Maybe they are somehow "touched" or something? Maybe it is something in the settings of unionfs?
[size=75]Between depriving a man of one hour from his life and depriving him of his life there exists only a difference of degree. --Muad'Dib[/size]
[img]http://www.browserloadofcoolness.com/sig.png[/img]
[img]http://www.browserloadofcoolness.com/sig.png[/img]
I had less time than expected this week. But today I rewrote the script again - there were some problems with folders. It seems that one cannot move a folder to the save-file if just a single file within the folder is held open by some process. I also changed some minor options in mounting aufs.
I will test again myself this week - I don't want to run somebody into frustration if the save-script fails - it's crucial!
The save-script is meant for pen-drives only - sorry no cd-seesions are created.
The update-time goes down to 0.15 seconds if nothing has to be done.
Here are some performance results:
I tested Firefox, Opera and Seamonkey.
The following values specify the run-times (including sync) of the save-script:
after start of the browser / repeated run of save-script / after shut-down of the browser
Firefox: 8,9s / 8,1s / 9,1s
Opera: 3,4s / 1,4s / 12,2s
Seamonkey: 4,0s / 3,1s / 11,3s
This values are hardware-specific, but their relative measurements should hold for other hardware, too.
The values for "repeated run of save-script" are a measurement for the amount of files held open by the browser. It would not make sense to remove open-files from the ramdisk. (I think it's also impossible to do so.)
Currently I have all 3 browsers open. Here is my ramdisk-usage (all values in KB):
0.5 drwx 2008-01-13 20:11 dev
0.0 drwx 2008-01-13 21:15 etc
0.0 drwx 2008-01-12 12:01 opt
2357.0 drwx 2008-01-13 20:23 root
0.0 drwx 2008-01-13 20:17 sbin
148.0 drwx 2008-01-13 23:45 tmp
0.0 drwx 2008-01-13 19:44 usr
396.0 drwx 2007-11-10 02:35 var
Remember that /var and /tmp are not copied back to the save-file!
/root contains some open files, so the remained in the ramdisk.
I will test again myself this week - I don't want to run somebody into frustration if the save-script fails - it's crucial!
The save-script is meant for pen-drives only - sorry no cd-seesions are created.
The update-time goes down to 0.15 seconds if nothing has to be done.
Here are some performance results:
I tested Firefox, Opera and Seamonkey.
The following values specify the run-times (including sync) of the save-script:
after start of the browser / repeated run of save-script / after shut-down of the browser
Firefox: 8,9s / 8,1s / 9,1s
Opera: 3,4s / 1,4s / 12,2s
Seamonkey: 4,0s / 3,1s / 11,3s
This values are hardware-specific, but their relative measurements should hold for other hardware, too.
The values for "repeated run of save-script" are a measurement for the amount of files held open by the browser. It would not make sense to remove open-files from the ramdisk. (I think it's also impossible to do so.)
Currently I have all 3 browsers open. Here is my ramdisk-usage (all values in KB):
0.5 drwx 2008-01-13 20:11 dev
0.0 drwx 2008-01-13 21:15 etc
0.0 drwx 2008-01-12 12:01 opt
2357.0 drwx 2008-01-13 20:23 root
0.0 drwx 2008-01-13 20:17 sbin
148.0 drwx 2008-01-13 23:45 tmp
0.0 drwx 2008-01-13 19:44 usr
396.0 drwx 2007-11-10 02:35 var
Remember that /var and /tmp are not copied back to the save-file!
/root contains some open files, so the remained in the ramdisk.
While rewriting the script this weekend I forgot to implement the "check before copy"-optimization.
I now added it again and have save- or better check-times of about 0.6 seconds even with lots of open applications.
So this awful forced pausing each half hour will have an end!
I will offer a complete (Puppy based) distribution including the new init- and save-scripts, dual-core-support, accelerated graphics and much more for download sometime next week.
I now added it again and have save- or better check-times of about 0.6 seconds even with lots of open applications.
So this awful forced pausing each half hour will have an end!
I will offer a complete (Puppy based) distribution including the new init- and save-scripts, dual-core-support, accelerated graphics and much more for download sometime next week.
The current version is for USB (flash) only, but should work with slight modifications with other aufs-layer schemes, too. I myself want to replace the ext2-save files with on the fly created sfs-files - which are sequentially written ant that is much more suitable for flash drives. So instead of creating a newly inserted sfs-layer it should make no big difference to create a new session on a CD.
_____________
I encountered some trouble with the save-script this week- maybe due to a bug in the aufs release I used.
After discussion with Junjiro Okajima, the author of aufs, I now use the newest version and till now the strange behaviour did not show up again.
_____________
I encountered some trouble with the save-script this week- maybe due to a bug in the aufs release I used.
After discussion with Junjiro Okajima, the author of aufs, I now use the newest version and till now the strange behaviour did not show up again.
I will post everything after a week of finding no more bugs ...
Last night (I had to install and patch the new 2.6.24 kernel as soon as it was available ...) I found (and fixed) a severe bug handling device nodes.
The good news: aufs and the save script work well with the new kernel, too. As I understand, Barry is going to use the new kernel for Dingo. I'm now using the aufs-build from last monday and lost no more files since then due to strange aufs behaviour. (Files were never really lost - the just disappeared in the current session and were all back after a reboot.)
Last night (I had to install and patch the new 2.6.24 kernel as soon as it was available ...) I found (and fixed) a severe bug handling device nodes.
The good news: aufs and the save script work well with the new kernel, too. As I understand, Barry is going to use the new kernel for Dingo. I'm now using the aufs-build from last monday and lost no more files since then due to strange aufs behaviour. (Files were never really lost - the just disappeared in the current session and were all back after a reboot.)