Multiple SFS support in initrd

[sunburnt]

Guys... Actually mksquashfs can append a Squash file!

I just tested it and it works fine, just write to the original SFS file as usual.


Thoughts: I like very much a config. image file and an apps. image file.
This isn`t hard to do, config. Save unions on /root, and app. on /usr.
But the problem is that the app. is separated from it`s configs.


So I made AppPkg, it`s RoxApp and AppDir compatible ( so no-union ).
So to copy an AppPkg everything associated copies with it, even internal libs.
Only links are made in the O.S.`s normal main file system.

Having a separate Save union layer to catch the needed links keeps it clean.
The link`s Save layer is: /tmp/AppPkg/links. It`s self-cleaning at shutdown.

I`ve tried to make chroot and fake-chroot work, but it`s a real bag of bolts.
But the method has some very nice aspects about it. Anyone good at it?

[mavrothal]

Patch file ADRV-YDRV_Woof did let me get a bit more into this. Although I did not change UMNTMAIN to UMNTRW it works.
And, oh, by the way: where is this mounted - I mean in which order at the pup_roX list (ro1, ro2, ro3 etc.)?

RSH,
Is not clear to me if you figured it by now but the whole trick is the splitting of "${UMNTMAIN}" to "${UMNTRW}${UMNTRO0}${ALAYER}${BLAYER}${UMNTRO1}"
The {a-z}drvs are mounted to /initrd/mnt/tmpfs{2-25} if there is enough RAM to hold them (ie 2x the total SFSs size) or to /initrd/pup_{a-z}.
All {a-z}drvs mounted this way in the initrd show as one so if you have the same files in 2 SFSs and try to mount them this way you may end up in trouble...

[mavrothal]

Thoughts: I like very much a config. image file and an apps. image file.
This isn`t hard to do, config. Save unions on /root, and app. on /usr.
But the problem is that the app. is separated from it`s configs.

Help me understand a bit.

It is likely that different pupplets will need different apps and configs. So an Apps and Config SFS are not going to be cross-pupplet. Correct?
If this is the case why you want to split apps and configs?
If on the other hand you want it cross-pupplet, how to you ensure that there are not missing/different *required* files among different pupplets or even versions of the same pupplet?

So if we are talking "one specific pupplet", what you want to accomplish is the equivalent of "selective" saving or remastering, but happening in every boot. Correct?
This could be easier accomplished by telling snapmergepuppy, which folders to save and which not with a config file. The problem is that some apps may put configs in places that you miss but I do not know if this can be solved in ay way if a developer ignores the standards.

Am I far off on what (not how) you want to do?

[sunburnt]

Hi mavrothal; You`re correct of course.
But really a set of apps will only work with a truly compatible Linux O.S. If a Puppy is truly Ubuntu Precise compatible, then those apps will all work.
Some cross compatibility is evident between Precise and Slack, but sketchy. AppPkg dirs. are named for the Linux O.S., distro. ver., and arch. of the app.

I didn`t say I wanted to split the apps. and configs. I said that was a problem with separate Save files.

I`m saying, why scatter files all over the O.S.? Put them in the app`s. dir. In fact I`m saying put everything to do with each app. in it`s AppPkg dir.
AppPkg is "no-union" "no-install" apps., so they operate independently from anywhere, HD, USB, LAN, etc. Copy or move it and its all still there.

What you suggest sounds more complex than simply unioning multiple Save files/dirs. To make a new main SFS just append select layers to the existing SFS.
Or if there are changes to the existing SFS file, then you`d make a new one. But this is the legacy Puppy way of doing things, it`s a little cumbersome.

One of the many union problems is the cpu and ram load goes up with each layer. So Barry sought to limit the number of layers for the sake of old PCs.
Union file systems will never be accepted into the kernel`s main code. Union mounts may possibly make it into the kernel`s code, they`re simpler.
Here is an excellent article about unions and their problems: http://lwn.net/Articles/324291/. They really are quite the rats nest of problem patching.

Far better to dump the union FS and design an O.S. that doesn`t need it. AppPkg apps are a critical part of this type of O.S. setup, it doesn`t use a union.
.

[nooby]

I`m saying, why scatter files all over the O.S.? Put them in the app`s. dir.
In fact I`m saying put everything to do with each app. in it`s AppPkg dir.
AppPkg is "no-union" "no-install" apps., so they operate independently from anywhere, HD, USB, LAN, etc. Copy or move it and its all still there.
...
Far better to dump the union FS and design an O.S. that doesn`t need it.
AppPkg apps are a critical part of this type of O.S. setup,
it doesn`t use a union.
.

Not that I understand any of this but it sounds cool.
Keep at it looks promising.

[mavrothal]

What you suggest sounds more complex than simply unioning multiple Save files/dirs. To make a new main SFS just append select layers to the existing SFS.
Or if there are changes to the existing SFS file, then you`d make a new one. But this is the legacy Puppy way of doing things, it`s a little cumbersome.

Could be, but I do not see why.
Is basically jemimah's idea having a base and an apps sfs taken one step further for additional flexibility.
Let's say base(no X), desktop managers (plus X), apps, or base (plus X) minimal apps, extended apps.
In this scheme , someone could append additional fils to one or more of the SFSs as needed.
However, I do not really think it should be used with 5-50 sfs files placed together in the union both for reasons you mentioned and because without careful splitting/loading you may end up with a broken system.

Far better to dump the union FS and design an O.S. that doesn`t need it. AppPkg apps are a critical part of this type of O.S. setup, it doesn`t use a union.
.

If you really want to go with the multiple SFSs approach do look at Tinycore (as I mentioned earlier). Besides its traditional TCZs it now has the standalone CMSs. TC has its share of issues but when it comes to having apps in squashed filesystems and loading them as needed, it keeps the best of SDL and is unmatched I believe.

However, is hard to see why puppy's current layered system is "broken" and the need we must cover here, other than easier testing of pupplets as they come out.
Would you give me a "real life use case" that the current puppy system (with 1-3 SFSs) fails?

[Q5sys]

However, is hard to see why puppy's current layered system is "broken" and the need we must cover here, other than easier testing of pupplets as they come out.
Would you give me a "real life use case" that the current puppy system (with 1-3 SFSs) fails?

I can report my experiences with multiple SFS use...
I routinely use anywhere from 15~19 SFS packages on my lighthouse64 install. And I've been running those numbers for well over a year and not had issue. In the past I've been as low as 10 and as high as 29. I have yet to encounter any issues.

[mavrothal]

However, is hard to see why puppy's current layered system is "broken" and the need we must cover here, other than easier testing of pupplets as they come out.
Would you give me a "real life use case" that the current puppy system (with 1-3 SFSs) fails?

I can report my experiences with multiple SFS use...
I routinely use anywhere from 15~19 SFS packages on my lighthouse64 install. And I've been running those numbers for well over a year and not had issue. In the past I've been as low as 10 and as high as 29. I have yet to encounter any issues.

I know this tread is getting long but do you load 15-20 SFS on the union in the initrd or with SFS_load/boot manager?
If not, should we then conclude that your statement implies that the current puppy setup (with one main sfs) is just fine?

[Q5sys]

However, is hard to see why puppy's current layered system is "broken" and the need we must cover here, other than easier testing of pupplets as they come out.
Would you give me a "real life use case" that the current puppy system (with 1-3 SFSs) fails?

I can report my experiences with multiple SFS use...
I routinely use anywhere from 15~19 SFS packages on my lighthouse64 install. And I've been running those numbers for well over a year and not had issue. In the past I've been as low as 10 and as high as 29. I have yet to encounter any issues.

I know this tread is getting long but do you load 15-20 SFS on the union in the initrd or with SFS_load/boot manager?
If not, should we then conclude that your statement implies that the current puppy setup (with one main sfs) is just fine?

Ah yea... i wasnt specific there. Those SFS packages are loaded with the current SFS bootmanager system.
As for unioning in the inird
Slackbones has 2 SFS files inside the initrd which I have not had issues with.

[sunburnt]

Tiny Core Linux does not use SFS files that are unioned, it uses Squash files.
Tiny Core does not use a union, it uses thousands of links in the main O.S.
It`s newer SCM files are very like AppPkg, RoxApp, or AppDir, no union.


The main arguments are in the article I posted a link to. It speaks for itself.


Q5sys; But what`s the performance on a low ram old PC with 20 layers?
This type of setup has limits to it`s usefulness. But not no-union apps.

[gcmartin]

Hi @Mavrothal and thanks for this discussion. This thread provides insights that developers wrestle with when working in Puppyland to produce distros. And its helpful, I'm sure to those who also build in that it presents some ideas of interest in good methods for the people they envision are the targets for their distros and their work.

... do you load 15-20 SFS on the union in the initrd or with SFS_load/boot manager ...

I saw your post about loading 10-15 SFS in LightHouse64.

TaZoC designed this feature about 2 years ago. It happens at boot-time and it will load any OR all SFS it finds in root at boot.

The easiest view of this is to download his LightHouse64 Mariner edition (requires 64bit PC) burn a CD/DVD and boot it without ANY parms.

It will boot and provide you a selection for all the SFSs it sees and when arrival after boot, you will have a desktop comprised of all of the SFSs you allowed at boot time. Further, you can add more SFSs to the root of the CD/DVD and on subsequent boot, the boot subsystem will intelligently present them, as well for selection.

Finally, upon completion of your running session, at shutdown, the LH64 system will allow one to create a save-file of all of their work to minimize subsequent boot needs,

Should you choose to investigate his approach, when you boot the CD/DVD hit F3 and F2 to see some of the boot selection offering and explanations available to users.

One thing of note is that ALL of the SFSs that he provides in Mariner are one he has tested to ensure compatibility. Thereby eliminating "unexpectations" at boot.

Here to help

[mavrothal]

Tiny Core Linux does not use SFS files that are unioned, it uses Squash files.
Tiny Core does not use a union, it uses thousands of links in the main O.S.
It`s newer SCM files are very like AppPkg, RoxApp, or AppDir, no union.


The main arguments are in the article I posted a link to. It speaks for itself.


Q5sys; But what`s the performance on a low ram old PC with 20 layers?
This type of setup has limits to it`s usefulness. But not no-union apps.

I guess I was not clear enough pointing to the fact that the TC system is not a unionfs but that's because I did not realize that you just want a non-uninion filesystem for puppy.
Unionfs do have their problems as pointed in the article you mentioned, however I doubt any of the ones mentioned in the article have anything to do with 99.999% of puppy usage.
Regarding low specs machines, believe me I know about them and union is the least of their problems.

I do not really say that unionfs is a "must" for puppy(full installs are also available) but I would really like to see what "use case" we are trying to cover here. This may focus and help the discussion a bit more.

[Q5sys]

Q5sys; But what`s the performance on a low ram old PC with 20 layers?
This type of setup has limits to it`s usefulness. But not no-union apps.

I've never done any actual testing... so I cant really make a reliable statement on that.
I've used 5 to 10 on my netbook... but i've never done any comparisons with it.
Also keep in mind that Lighthouse is 64bit... so 'low ram old pcs' cant run it.

[mavrothal]

TaZoC designed this feature about 2 years ago. It happens at boot-time and it will load any OR all SFS it finds in root at boot.

From what I can see in TaZoC's init code, his mods just do "automatically" what bootmanager/SFS_load does on a case by case (manual) setting.
These multiple SFSs are loaded in the union as mount points but are *not* loaded in RAM as the main SFS does.
What the adrv/ydrv patch does is to load multiple SFSs of the unionfs in the RAM as one (if enough RAM available).
So TaZoC's mods are very nice but not relevant. Actually his mods could work in parallel to the ones proposed here as they have no overlap.

As mentioned many times so far, the goal of adrv/ydrv is to facilitate *development* of different flavors/customizations of a given puppy, not to change the way puppy works.

[Q5sys]

TaZoC designed this feature about 2 years ago. It happens at boot-time and it will load any OR all SFS it finds in root at boot.

From what I can see in TaZoC's init code, his mods just do "automatically" what bootmanager/SFS_load does on a case by case (manual) setting.
These multiple SFSs are loaded in the union as mount points but are *not* loaded in RAM as the main SFS does.
What the adrv/ydrv patch does is to load multiple SFSs of the unionfs in the RAM as one (if enough RAM available).
So TaZoC's mods are very nice but not relevant. Actually his mods could work in parallel to the ones proposed here as they have no overlap.

As mentioned many times so far, the goal of adrv/ydrv is to facilitate *development* of different flavors/customizations of a given puppy, not to change the way puppy works.

Actually LHP64 can load all SFS into ram. You have the option to run them from disk or load to ram.

[sunburnt]

Puppy will always be a union type O.S., that`s how Barry wants it.
And after all, it is his baby, so that`s just how it should be... Right?

Users find little wrong with unions, but they don`t put it all together.
My thought was more to new Linux O.S.s that drop the legacy crap.

The real Q: If it`s not really needed, then why have it at all?
It`s just an unnecessary complication that enables blending loose files
and Squash files into a sudo-writable file system. There`s better ways.

[R-S-H]

RSH,
Is not clear to me if you figured it by now but the whole trick is the splitting of "${UMNTMAIN}" to "${UMNTRW}${UMNTRO0}${ALAYER}${BLAYER}${UMNTRO1}"
The {a-z}drvs are mounted to /initrd/mnt/tmpfs{2-25} if there is enough RAM to hold them (ie 2x the total SFSs size) or to /initrd/pup_{a-z}.
All {a-z}drvs mounted this way in the initrd show as one so if you have the same files in 2 SFSs and try to mount them this way you may end up in trouble...

Hi, mavrothal.

Meanwhile I did change UMNTMAIN to UMNTRW, so the code is now equal. But I'm a bit confused - still after reading all new posts in this thread.

Inside the code of your patches, the adrv uses loop2

Code: Select all

mount -r -t squashfs -o noatime /dev/loop2 /pup_a

the ydrv uses loop3

Code: Select all

mount -r -t squashfs -o noatime /dev/loop3 /pup_y

Inside saluki code loop3 isused for zdrv

Code: Select all

mount -r -t squashfs -o noatime /dev/loop3 /pup_z

and so is in my LazY Puppy init script.

Do I have to increase the loops for each (a,b,c)drv like loop4, loop5, loop6 ond so on?

Also in saluki code adrv and zdrv go to tmpfs2

Code: Select all

cp -af ${MNT_ZFILE}${ZFILE} /mnt/tmpfs2/

The ydrv in your patch goes to tmpfs3

Code: Select all

cp -af ${MNT_YFILE}${YFILE} /mnt/tmpfs3/

Do I have to create these folders inside initrd.gz or at boot up from within the init script?

Thanks

RSH

P.S.

All {a-z}drvs mounted this way in the initrd show as one so if you have the same files in 2 SFSs and try to mount them this way you may end up in trouble...

Could you please explain a bit more? Does it mean to have those two icons for fbpanel (one inside main sfs the other one inside the winmgr sfs) could be responsible for an actual issue (can't load sfs files using sfs_load at mount point pup_ro9 and up)?

[mavrothal]

Actually LHP64 can load all SFS into ram. You have the option to run them from disk or load to ram.

Never used LH64 so I do not know for a fact.
I just glanced at the code and although I can see COPYEXTRASFS2RAM I can not see how more that one SFS is loaded as UMNTRO.
I probably miss something, but can you actually see RAM usage increasing as much as the SFS size when more than 1 extra SFSs are loaded with the copy2am option? (I know you have the RAM for that test )

[mavrothal]

My thought was more to new Linux O.S.s that drop the legacy crap.

Go ahead and we'll help if we can.

[mavrothal]

Inside the code of your patches, the adrv uses loop2

Code: Select all

mount -r -t squashfs -o noatime /dev/loop2 /pup_a

the ydrv uses loop3

Code: Select all

mount -r -t squashfs -o noatime /dev/loop3 /pup_y

Inside saluki code loop3 isused for zdrv

Code: Select all

mount -r -t squashfs -o noatime /dev/loop3 /pup_z

and so is in my LazY Puppy init script.

If you use zdrv bump it to use loop 4

Do I have to increase the loops for each (a,b,c)drv like loop4, loop5, loop6 ond so on?

Yes

Also in saluki code adrv and zdrv go to tmpfs2

Code: Select all

cp -af ${MNT_ZFILE}${ZFILE} /mnt/tmpfs2/

The ydrv in your patch goes to tmpfs3

Code: Select all

cp -af ${MNT_YFILE}${YFILE} /mnt/tmpfs3/

Do I have to create these folders inside initrd.gz or at boot up from within the init script?

In the initrd

All {a-z}drvs mounted this way in the initrd show as one so if you have the same files in 2 SFSs and try to mount them this way you may end up in trouble...

Could you please explain a bit more? Does it mean to have those two icons for fbpanel (one inside main sfs the other one inside the winmgr sfs) could be responsible for an actual issue (can't load sfs files using sfs_load at mount point pup_ro9 and up)?

I can not verify that but is likely. Try to remove one and check