Release of overlay_init-0.1
Posted: Sun 07 Jan 2018, 07:52
At last, the release of overlay_init-0.1.
Unlike the previous versions in this project that were basically "proof of concept" trials of various facilities,
this version is meant to be an 'init' that can be used as a base for production.
Hence:
Like a normal Puppy, the DISTRO_SPECS file must now reside in "initrd.gz".
Like a normal Puppy, a wizard is run on first-shutdown to setup an appropriate "save" mechanism.
Main features, (apart from using overlayfs):
1. Freedom to use any sfs, stored anywhere, loaded both above and below the main puppy...sfs.
Gone is the concept of an "adrv", "ydrv", "fdrv" or "zdrv", there are now 3 sfs lists, SFS_ABOVE, SFS_MAIN and SFS_EXTRA.
These lists are added into the stack SFS_ABOVE on top, SFS_MAIN in the middle, and SFS_EXTRA at the bottom.
The only limitations are that the first sfs in the SFS_MAIN list is assumed to be the puppy...sfs and must load successfully, and if overlayfs support is a kernel module it must exist in the last sfs in the SFS_MAIN list.
So you can have as many sfs's above the puppy...sfs as you like, with any name you like.
e.g. SFS_ABOVE=':/puppy/overlay_mods.sfs,:adrv_artfulpup_17.11.sfs,:ydrv_artfulpup_17.11.sfs'
or SFS_MAIN=':puppy_xenial_7.0.8.6.sfs,:kernel_modules_4.9.71.sfs'
And these lists are managed by GUI utilities in the running system, you simply select the sfs file you want to add, and then move it up the list to it's desired position.
2. Freedom to store the savefolder anywhere, again controlled by a GUI utility in the running system.
This makes it easy to setup a "save" location on a different device, e.g. puppy...sfs on SSD, savefolder on HD.
The "Saveconfig" utility that automatically runs at first-shutdown, can be run at any time.
So, if you want to have a savefolder but at first-shutdown you don't have an appropriate partition available, you can choose "Archive" so that changes you have made during the session will not be lost.
Then later, when you have setup a Linux partition, you can run "Saveconfig" again and choose "Folder", then select the Linux partition and sub-directory.
On shuddown the current "save" data will be copied to the new savefolder, and the Puppy will reboot into pupmode=12 without losing any of your changes.
3. Support for Luks partitions as "save" targets:
Following on the above example, if at a later time you decide that the contents of your savefolder are private and need to be encrypted, you can choose a partition to sacrifice as a Luks partition, and run "Saveconfig" to setup the selected partition with Luks.
On shutdown the "save" data will be copied to the selected savefolder location inside the Luks partition.
On reboot "init" will prompt you for the password for the Luks partition and then boot into pupmmode=12.
Then you would need to remember to delete the old plain text savefolder.
4. This "init" is based on the idea that it only does what it's told to do.
It will not boot if you don't specify the location of the puppy...sfs with a "psfspart=" (or "pupsfs=") boot parameter, and a "psfsdir=" (or "psubdir=") boot parameter, if the puppy...sfs is in a sub-directory.
It will continue to boot in pupmode=5 until a sucessful run of "Saveconfig" tells it to do otherwise.
5. Mostly you tell "init" what to do by running a GUI utility in the running system.
Not only the examples above for sfs files and "save" configuring,
but also some "pfix=" boot parameters with "Pfix parameter manager", e.g. "nosave", that works in all pupmodes.
Just remember that these are "boot parameters", they change the behaviour of the next "boot", not the next "shutdown".
6. Simplified boot parameters.
Along with the "adrv", "ydrv" etc.. going, so has their boot parameters.
In fact all the ':' separated file specification boot parameters have gone.
The current file specifying boot parameters are:
"psfspart" -> contains a partition label, uuid, or name of partition containing puppy...sfs
aliases are "pdev1", "pdrv", "pupsfs"
"psfsdir" -> contains the relative path to sub-directory containing puppy...sfs
alias is "psubdir"
"psavepart" -> contains a partition label, uuid, or name of partition containing savefolder
alias is "psave"
"psavedir" -> contains the relative path to sub-directory containing savefolder
"psavepart" and "psavedir" are usually not specified as boot parameters, they are set by "Saveconfig" utility.
But they can be used to set a default "save" location, and hence the location of BOOT_SPECS.
7. There are many facilities in the current "init" script in woof-ce 'testing' that are not here.
Some of these are simply not possible due to the limitations of overlayfs, others are made too complicated by the limitations of overlayfs.
And some are simply things I never use, but could possibly be easily ported to this "init".
But I will leave the discussion of these to another time, (this is already a rather verbose post).
How to try it, using a Puppy with a working sfs_load:
0. Checks to do before you start:
Choose the Puppy you intend to use. It should be a recent woof-ce based Puppy.
Test the kernel of a running version of the Puppy by running the command "grep 'OVERLAY' /etc/modules/*" in a console.
A response that includes "CONFIG_OVERLAY_FS=m" or "CONFIG_OVERLAY_FS=y", indicates overlayfs support, which is necessary for this "init".
Then run the command "grep 'TMPFS' /etc/modules/*". A response containing "CONFIG_TMPFS_XATTR=y" is highly desirable.
Otherwise I recommend minimising activity while the read/write layer is in "tmpfs", (pupmode=5 and pupmode=21),
and moving to a savefolder (pupmode=12) as soon as possible.
You should also be able to run the tests against "etc/modules/*" in the Puppies zdrv.
1. Do a fresh frugal install of your target Puppy into a suitable writeable directory.
If you don't have a "test" partition on an internal device, a partition on a usb device is fine.
If using a usb device, a typical usb HD is usually much faster than a typical usb stick.
Make sure the "kernel" line for this install in your boot loader config contains
at least an appropriate "pupsfs=<partition>" parameter and a "psubdir=<sub-directory>" parameter if required.
Where "<partition>" is the LABEL or UUID of the target partition, and "<sub-directory>" is the path to the target directory.
It might be a good idea to boot this new install at this point just to test that it can.
If you do boot it, on first shutdown, choose "No" to avoid any saving.
2. Download http://www.fishprogs.software/puppy/ini ... it-0.1.tar to the new install directory.
Open a console window in the nenw install directory, and run command "tar xf overlay_init-0.1.tar".
This should produce 2 files, "initrd.overlay.gz" and "overlay_mods.sfs".
3. Load "overlay_mods.sfs" using sfs_load.
This will make a number of utilities specific to this 'init', available to the running Puppy.
4. Run the command "mk-timezone-file", to produce a "TIME_ZONE" file.
This text file, should contain the local time zone as "<TZ-format>|<zone file link info>".
If "mk-timezone-file" can't get the required information from the running Puppy, it gets it from "http://geoip.ubuntu.com/lookup", (thanks Lassar).
5. Run the command "overlay-setup <install-directory>", where "<install-directory>" is the full path to your install directory.
This will:
Extract the "DISTRO_SPECS" file from the "initrd.gz" file.
Store this "DISTRO_SPECS" file into "initrd.overlay.gz".
If a "TIME_ZONE" file exists store it into "initrd.overlay.gz".
Rename "initrd.gz" to "initrd.release.gz".
Rename "initrd.overlay.gz" to "initrd.gz".
Create a "BOOT_SPECS" file that will load "overlay_mods.sfs" above any other sfs's in the overlay stack.
6. Unload "overlay_mods.sfs" using sfs_load.
7. Boot this modified installation.
This should result in a first boot, change data in "Quick Setup" as required.
If you provided a "TIME_ZONE" file in step 5, the timezone should already be set correctly.
On first shutdown, a utility "Saveconfig" should run so you can choose a "save" mechanism.
"Archive" provides a "persistence but still all in ram" facility, that works on any filesystem.
"Folder" provides a "pupmode=12, savefolder on Linux partition" facility, an option to format an existing partition is provided.
8. Reboot to enjoy the new working puppy.
To modify the list of extra sfs's that "init" tries to mount, use "Extra-SFS manager".
To modify the list of system sfs's that "init" tries to mount, use "System-SFS manager".
To change some of the boot characteristics, use "Pfix parameter manager".
To change "save" to a different mechanism or change the location of the savefolder, use "Saveconfig".
Note1: If you use a Puppy that already boots with this "init" and it's "overlay_mods.sfs", then the utilities are already present, so steps 3 and 6 would be ommitted.
Note2: Utilities that store a partition identifier use a LABEL by preference, then UUID, then name. So if you use LABEL's on your partitions, make sure they are unique.
I use LABEL's on my partitions, as a UUID is rather meaningless when the stored data is viewed.
What needs to be tested:
1) That it does successfully boot current woof-ce based Puppies, i.e. they boot to the desktop.
2) Are there any Puppy utilities, other than the patched ones included in "overlay_mods.sfs", that are broken by this "init"?
Please don't bother telling me about "sfs_load" or "snapmergepuppy", these are incompatible with overlayfs.
3) That the booted Puppies work the same as when they are booted with their release "init".
While I have done many, many boots as part of this project, I haven't sustainably used the resultant Puppy.
I plan to setup a Puppy I can use as my daily workhorse and boot it with this "init".
gyro
Unlike the previous versions in this project that were basically "proof of concept" trials of various facilities,
this version is meant to be an 'init' that can be used as a base for production.
Hence:
Like a normal Puppy, the DISTRO_SPECS file must now reside in "initrd.gz".
Like a normal Puppy, a wizard is run on first-shutdown to setup an appropriate "save" mechanism.
Main features, (apart from using overlayfs):
1. Freedom to use any sfs, stored anywhere, loaded both above and below the main puppy...sfs.
Gone is the concept of an "adrv", "ydrv", "fdrv" or "zdrv", there are now 3 sfs lists, SFS_ABOVE, SFS_MAIN and SFS_EXTRA.
These lists are added into the stack SFS_ABOVE on top, SFS_MAIN in the middle, and SFS_EXTRA at the bottom.
The only limitations are that the first sfs in the SFS_MAIN list is assumed to be the puppy...sfs and must load successfully, and if overlayfs support is a kernel module it must exist in the last sfs in the SFS_MAIN list.
So you can have as many sfs's above the puppy...sfs as you like, with any name you like.
e.g. SFS_ABOVE=':/puppy/overlay_mods.sfs,:adrv_artfulpup_17.11.sfs,:ydrv_artfulpup_17.11.sfs'
or SFS_MAIN=':puppy_xenial_7.0.8.6.sfs,:kernel_modules_4.9.71.sfs'
And these lists are managed by GUI utilities in the running system, you simply select the sfs file you want to add, and then move it up the list to it's desired position.
2. Freedom to store the savefolder anywhere, again controlled by a GUI utility in the running system.
This makes it easy to setup a "save" location on a different device, e.g. puppy...sfs on SSD, savefolder on HD.
The "Saveconfig" utility that automatically runs at first-shutdown, can be run at any time.
So, if you want to have a savefolder but at first-shutdown you don't have an appropriate partition available, you can choose "Archive" so that changes you have made during the session will not be lost.
Then later, when you have setup a Linux partition, you can run "Saveconfig" again and choose "Folder", then select the Linux partition and sub-directory.
On shuddown the current "save" data will be copied to the new savefolder, and the Puppy will reboot into pupmode=12 without losing any of your changes.
3. Support for Luks partitions as "save" targets:
Following on the above example, if at a later time you decide that the contents of your savefolder are private and need to be encrypted, you can choose a partition to sacrifice as a Luks partition, and run "Saveconfig" to setup the selected partition with Luks.
On shutdown the "save" data will be copied to the selected savefolder location inside the Luks partition.
On reboot "init" will prompt you for the password for the Luks partition and then boot into pupmmode=12.
Then you would need to remember to delete the old plain text savefolder.
4. This "init" is based on the idea that it only does what it's told to do.
It will not boot if you don't specify the location of the puppy...sfs with a "psfspart=" (or "pupsfs=") boot parameter, and a "psfsdir=" (or "psubdir=") boot parameter, if the puppy...sfs is in a sub-directory.
It will continue to boot in pupmode=5 until a sucessful run of "Saveconfig" tells it to do otherwise.
5. Mostly you tell "init" what to do by running a GUI utility in the running system.
Not only the examples above for sfs files and "save" configuring,
but also some "pfix=" boot parameters with "Pfix parameter manager", e.g. "nosave", that works in all pupmodes.
Just remember that these are "boot parameters", they change the behaviour of the next "boot", not the next "shutdown".
6. Simplified boot parameters.
Along with the "adrv", "ydrv" etc.. going, so has their boot parameters.
In fact all the ':' separated file specification boot parameters have gone.
The current file specifying boot parameters are:
"psfspart" -> contains a partition label, uuid, or name of partition containing puppy...sfs
aliases are "pdev1", "pdrv", "pupsfs"
"psfsdir" -> contains the relative path to sub-directory containing puppy...sfs
alias is "psubdir"
"psavepart" -> contains a partition label, uuid, or name of partition containing savefolder
alias is "psave"
"psavedir" -> contains the relative path to sub-directory containing savefolder
"psavepart" and "psavedir" are usually not specified as boot parameters, they are set by "Saveconfig" utility.
But they can be used to set a default "save" location, and hence the location of BOOT_SPECS.
7. There are many facilities in the current "init" script in woof-ce 'testing' that are not here.
Some of these are simply not possible due to the limitations of overlayfs, others are made too complicated by the limitations of overlayfs.
And some are simply things I never use, but could possibly be easily ported to this "init".
But I will leave the discussion of these to another time, (this is already a rather verbose post).
How to try it, using a Puppy with a working sfs_load:
0. Checks to do before you start:
Choose the Puppy you intend to use. It should be a recent woof-ce based Puppy.
Test the kernel of a running version of the Puppy by running the command "grep 'OVERLAY' /etc/modules/*" in a console.
A response that includes "CONFIG_OVERLAY_FS=m" or "CONFIG_OVERLAY_FS=y", indicates overlayfs support, which is necessary for this "init".
Then run the command "grep 'TMPFS' /etc/modules/*". A response containing "CONFIG_TMPFS_XATTR=y" is highly desirable.
Otherwise I recommend minimising activity while the read/write layer is in "tmpfs", (pupmode=5 and pupmode=21),
and moving to a savefolder (pupmode=12) as soon as possible.
You should also be able to run the tests against "etc/modules/*" in the Puppies zdrv.
1. Do a fresh frugal install of your target Puppy into a suitable writeable directory.
If you don't have a "test" partition on an internal device, a partition on a usb device is fine.
If using a usb device, a typical usb HD is usually much faster than a typical usb stick.
Make sure the "kernel" line for this install in your boot loader config contains
at least an appropriate "pupsfs=<partition>" parameter and a "psubdir=<sub-directory>" parameter if required.
Where "<partition>" is the LABEL or UUID of the target partition, and "<sub-directory>" is the path to the target directory.
It might be a good idea to boot this new install at this point just to test that it can.
If you do boot it, on first shutdown, choose "No" to avoid any saving.
2. Download http://www.fishprogs.software/puppy/ini ... it-0.1.tar to the new install directory.
Open a console window in the nenw install directory, and run command "tar xf overlay_init-0.1.tar".
This should produce 2 files, "initrd.overlay.gz" and "overlay_mods.sfs".
3. Load "overlay_mods.sfs" using sfs_load.
This will make a number of utilities specific to this 'init', available to the running Puppy.
4. Run the command "mk-timezone-file", to produce a "TIME_ZONE" file.
This text file, should contain the local time zone as "<TZ-format>|<zone file link info>".
If "mk-timezone-file" can't get the required information from the running Puppy, it gets it from "http://geoip.ubuntu.com/lookup", (thanks Lassar).
5. Run the command "overlay-setup <install-directory>", where "<install-directory>" is the full path to your install directory.
This will:
Extract the "DISTRO_SPECS" file from the "initrd.gz" file.
Store this "DISTRO_SPECS" file into "initrd.overlay.gz".
If a "TIME_ZONE" file exists store it into "initrd.overlay.gz".
Rename "initrd.gz" to "initrd.release.gz".
Rename "initrd.overlay.gz" to "initrd.gz".
Create a "BOOT_SPECS" file that will load "overlay_mods.sfs" above any other sfs's in the overlay stack.
6. Unload "overlay_mods.sfs" using sfs_load.
7. Boot this modified installation.
This should result in a first boot, change data in "Quick Setup" as required.
If you provided a "TIME_ZONE" file in step 5, the timezone should already be set correctly.
On first shutdown, a utility "Saveconfig" should run so you can choose a "save" mechanism.
"Archive" provides a "persistence but still all in ram" facility, that works on any filesystem.
"Folder" provides a "pupmode=12, savefolder on Linux partition" facility, an option to format an existing partition is provided.
8. Reboot to enjoy the new working puppy.
To modify the list of extra sfs's that "init" tries to mount, use "Extra-SFS manager".
To modify the list of system sfs's that "init" tries to mount, use "System-SFS manager".
To change some of the boot characteristics, use "Pfix parameter manager".
To change "save" to a different mechanism or change the location of the savefolder, use "Saveconfig".
Note1: If you use a Puppy that already boots with this "init" and it's "overlay_mods.sfs", then the utilities are already present, so steps 3 and 6 would be ommitted.
Note2: Utilities that store a partition identifier use a LABEL by preference, then UUID, then name. So if you use LABEL's on your partitions, make sure they are unique.
I use LABEL's on my partitions, as a UUID is rather meaningless when the stored data is viewed.
What needs to be tested:
1) That it does successfully boot current woof-ce based Puppies, i.e. they boot to the desktop.
2) Are there any Puppy utilities, other than the patched ones included in "overlay_mods.sfs", that are broken by this "init"?
Please don't bother telling me about "sfs_load" or "snapmergepuppy", these are incompatible with overlayfs.
3) That the booted Puppies work the same as when they are booted with their release "init".
While I have done many, many boots as part of this project, I haven't sustainably used the resultant Puppy.
I plan to setup a Puppy I can use as my daily workhorse and boot it with this "init".
gyro