alternative puppy build system

[s243a]

@jamesbond...

sorry for being so
frank, woof-Next is useless -- or limited to creating a CLI distro. I do not have the
knowledge to fix this. You do. Again, please fix this?

TIA.

Best regards, respectfully submitted, etc.

You have a very odd attitude IMO musher0. I suppose you did say, please, but does come across as more than a bit pushy. Can't imagine that tack working.

Thanks for your comment, wiak.

What can I say? I have literally spent a week on the woof-Next with very little result.
This is all I have been doing for the past week. I am retired so I have time.

It seems I am the only one really working from the woof-Next ATM. We haven't heard
from foxpup in a while, and as I understand it, your are trying for an adaptation of the
Void distro, and wanderer is experimenting with a Puppy-tinycore hybrid.

I'm working on my fork which may or may not get pulled back into woof-next:

https://github.com/s243a/woof-CE/network

granted mine isn't ready for testing yet and I don't have the linux skills that jamesbond does to help you with your sid issues.

Oddly I can see more of my commits on jamesbond's network page than my own...Probably because I forked more of the woof-CE repo than need to

https://github.com/jamesbond3142/woof-CE/network

[wanderer]

yes

and i am waiting for s243a woof-next
(no pressure s243a just having fun)

and am working on my version
which i intend to merge into s243a version

musher0
i think you should work on this system
puppy tinycore devuan + other sources
since it builds ok already
and would save you some grief

wanderer

[wiak]

@musher: Though jamesbond did bring up his long ago made woof-next system he did say that he himself would not be working on it further but rather just bringing it to the intention of anyone who might find it useful. I imagine, assuming he has time and can even remember how woof-next works, he might reluctantly take a look when someone totally stuck on some issue. But he never said he would, as far as I know, so you just need to hope and not push I'd say - I can understand your frustration though. I haven't myself looked at woof-next, and won't be doing so, but it is probably relatively complex and the Xorg being used (from Devuan?) maybe has some problem on your system? If you have it working via straight Devuan, all I can suggest is you look into how it is configured on there and arrange for the same on the woof-next version, but I'm sure you've considered that and can't find where the problem is. When you think about it, it could be very difficult for jamesbond to suggest a fix unless he is using the same hardware system - that is always an issue on new distro releases of course - someone has to fix the bugs - the developer actually only can do so much (in making the distro work on his own machine and as universal as he can imagine from past practices).

That is certainly where strength in numbers is useful. One man building a distro can be extremely frustrating since roadblocks do often keep emerging and like I say it is quite possible jamesbond would not have a ready answer to give - though perhaps you'll end up lucky. Otherwise, sometimes we do have to abandon our developments, if no workarounds can be identified (so, yes, it is not at all always fun or even satisfying...).

wiak

[wanderer]

hi musher0

abandon that non working system
and help me with my version
delta puppy
it has many desktops besides jwm
that you have mentioned you like
i think you will find it very satisfying
and not frustrating at all

wanderer

[wanderer]

and now an update of delta puppy

delta means change
and its symbol is a triangle

if we cannot change puppy
we cannot improve it

in my opinion
there are 3 major flaws in the old puppies

1. the boot process is unnecessarily complex

2. it is not modular

3. it uses the layered filesystem which is inferior to the symlink system

i am not interested in perpetuating these mistakes

delta puppy corrects these 3 flaws

i use the tinycore system as a starting point
but
it will be a puppy because i will integrate
puppy code
puppy parts
and retain the puppy concept
as appropriate

yes i am using things that have been created by others
as barry k did when he created puppy
he used what others created
when he felt it would work best
and so will i

tinycore has its own repository
with just about everything that one might need
delta puppy can use that

i also use woof-next
to download and build sfs files from the devuan repository
as well as to make the isos

i use uextract to unsquash
and mksquashfs to squash
debs pets and sfs files
from puppy and other sources
to use in delta puppy

delta puppy has a directory in root called /apps
all the added non tinycore apps will be mounted there
the sfs files themselves will be kept in the cde or tce dir
so they are easy to find and the system is orderly

when you want to load an app
you will mount it to the /apps directory
and symink it to root with the busybox cp -ais command

when you want to unload the app
you will rename the app in the /apps directory
so the symlinks will no longer point to it

the dead symlinks are of no consequence
but will all be removed when you reboot

so now i have explained the basic system

i am making an iso for the first stage
and will post it in the repository soon

it will be called the d5.iso
because it is a continuation
of the development that started with the d1.iso

this is the system
that i will make into an alternative build system for puppy

all comments and help are welcome

wanderer

[s243a]

and now an update of delta puppy

delta means change
and its symbol is a triangle

if we cannot change puppy
we cannot improve it

in my opinion
there are 3 major flaws in the old puppies

1. the boot process is unnecessarily complex

I think this is a matter of opinion.

2. it is not modular

Agreed but tinycores core.gz suffers from the same issue.

3. it uses the layered filesystem which is inferior to the symlink system

I think this is also a matter of opinion.

i also use woof-next
to download and build sfs files from the devuan repository
as well as to make the isos

Iinteresting, so if we turn off dependency tracking then we for instance can create an sfs containing exactly the Xorg packages that we want.

To get the list we could first build a larger system using dependency tracking see which packages are installed and then decide which Xorg packages that we want.

p.s. thanks for the info on how your system works

[wanderer]

hi s243a

thanks for the input
i hope some of my post is interesting to you

regards

wanderer

[wanderer]

hi s243a and wiak

as i understand it
you guys are trying to break the build process
into more manageable pieces of code for each stage

i too am looking at woof-next devuan
to do the same thing
since i know that woof-next devuan
builds a working iso
i know everything is there
i just have to figure out what each piece does

if you have the time and inclination
please keep me updated on your progress
as i feel this is definitely the way to go

regards

wanderer

[s243a]

hi s243a and wiak

as i understand it
you guys are trying to break the build process
into more manageable pieces of code for each stage

i too am looking at woof-next devuan
to do the same thing
since i know that woof-next devuan
builds a working iso
i know everything is there
i just have to figure out what each piece does

if you have the time and inclination
please keep me updated on your progress
as i feel this is definitely the way to go

regards

wanderer

I broke out the list of required packages to show the specific packages required rather than letting the package manager just pull them in as dependencies. Some packages that I thought we might be able to live without are blocked using the "%dummy" directive (these are just guesses).

Code: Select all

%dummy libexpat1      #Dependency for dbus, libegl1-mesa, libgbm1
%dummy libwayland-client0 #Dependency for libegl1-mesa, libgbm1
%dummy libwayland-server0 #Dependency for libegl1-mesa, libgbm1
libxau6            #Instead of libXau (from tinycore) 
                   #Needed for libxcb1, xserver-xorg-core
libxxf86vm1    #Dependency of x11-utils, dependency for libgl1-mesa-glx 
libgbm1         #Required for xserver-xorg-core, libegl1-mesa
libxshmfence1 #Required for libegl1-mesa, libgl1-mesa-glx, xserver-xorg-core

/woof-distro/x86/tiny_devuan/ascii/basesfs#L94

After a while, around where I started looking at video drivers I thought that I didn't want to guess the dependencies so I turned back on dependency tracking.

Code: Select all

###Turn on dependency Tracking ################
%depend           
xserver-xorg-video-all
%dumy xserver-xorg-video-all #Let's pick the specific drivers that we want
xserver-xorg-video-vesa 
xserver-xorg-video-vmware #This also works in virtualbox and qemu
xserver-xorg

/woof-distro/x86/tiny_devuan/ascii/basesfs#L162

I've tried to order the packages within:
/woof-next/woof-distro/x86/tiny_devuan/ascii/basesfs#L162

based on the order they are needed when building the system. We can in the script add intermediate points to generate a sfs, and then have the working directory of the build script continue in a higher file system layer. This isn't implemented yet but I think would fit into your philosophy.

I know that I'm just guessing at which dependencies we might be able to live without but at this point I'm just testing the build system and not focusing on the final product.

[wanderer]

thanks s243a

very interesting

i have been trying to change the basefs in stages
to just run x jwm and aterm
but without tinycore packages

got it up to hanging on the chroot
but getting closer

wanderer

[wanderer]

nah

thats a dead end
just bloats up ridiculously

i will go back to seeing how tinycore does it

wanderer

[s243a]

nah

thats a dead end
just bloats up ridiculously

i will go back to seeing how tinycore does it

wanderer

Hello wanderer. If you look in:

Code: Select all

/var/lib/dpkg/info

you can see what packages are installed. You can use the %dummy directive to block any of these packages from being installing.

[wanderer]

thanks s243a

i'll look into that

wanderer

[wiak]

For the most part, I feel that it is pointless to "avoid bloat" in terms of blocking what the upstream distro feels are reasonable/recommended dependencies. The exception to me is firmware - so a system for only matching firmware for a particular system would avoid a lot of unnecessary bloat. Otherwise, in general, users tend to add new packages and the dependencies blocked out probably end up needed anyway. My own strategy isn't really to avoid such eventual bloat, but rather just to provide a small original download if possible - just a convenience in that sense since it is bound to become bloated once apps and the usually required facilities get added

Also, I'm not so interested in final polished distro creation, since I already have that in the XenialDog64 I use - rather I want little distros to make developement experiments convenient (can easy break them and start fresh again without caring about damage to a polished distro, if you see what I mean).

Yes, tiny core is kind of like that, though it does aim for less bloat overall than I care about (since if I was building up my small distro I wouldn't want its apps limited by too much dependency cutting.

So, yeah, the things I am working on are not intended for full polished distro new Pups or whatever. They are just playthings for dev experiments really (though can be developed to become more of course).

Anyway, if the core provided is small then it can certainly load more easily into RAM, which can be useful if maybe building a system to only run one or two apps.

wiak

[s243a]

I recall people on this forum commenting on the size of llvm. I noticed that if I install xserver-xorg-video-vmware, that it pulls in llvm as a dependency. My guess is that it won't work without llvm, and this is a what-if road I don't want to go down.

vmware video driver is only usefull for a virtual machine but new distros are often tested in a virtual machine. I noticed that dpup strech provided all the video drivers:
https://packages.debian.org/stretch/xse ... -video-all

but historically vesa worked on most systems. SliTaz has a nice command line tool that will auto configure xorg for your and download the necessary video driver. This way one doesn't need to have unused video drivers on their system.

Anyway here is my related debugging output:

Code: Select all

+ local pkg=xserver-xorg-video-vmware
+ OIFS=' 	
'
+ IFS='|'
++ grep -m1 '^xserver-xorg-video-vmware|' repo-ascii-i386/pkgdb
+ set -- xserver-xorg-video-vmware 1:13.2.1-1+b1 xserver-xorg-video-vmware_13.2.1-1+b1_i386.deb http://deb.devuan.org/merged/pool/DEBIAN/main/x/xserver-xorg-video-vmware/xserver-xorg-video-vmware_13.2.1-1+b1_i386.deb optional x11 b3cce869687ac00ca3397a5cdefbc2a8 'libc6, libdrm2, libx11-6, libxatracker2, libxext6, xorg-video-abi-23, xserver-xorg-core,'
+ '[' -z xserver-xorg-video-vmware ']'
+ IFS=' 	
'
+ PKG=xserver-xorg-video-vmware
+ PKGVER=1:13.2.1-1+b1
+ PKGFILE=xserver-xorg-video-vmware_13.2.1-1+b1_i386.deb
+ PKGPATH=http://deb.devuan.org/merged/pool/DEBIAN/main/x/xserver-xorg-video-vmware/xserver-xorg-video-vmware_13.2.1-1+b1_i386.deb
+ PKGPRIO=optional
+ PKGSECTION=x11
+ PKGMD5=b3cce869687ac00ca3397a5cdefbc2a8
+ PKGDEP='libc6, libdrm2, libx11-6, libxatracker2, libxext6, xorg-video-abi-23, xserver-xorg-core,'
+ set +x
...
+ local pkg=libxatracker2
+ OIFS=' 	
'
+ IFS='|'
++ grep -m1 '^libxatracker2|' repo-ascii-i386/pkgdb
+ set -- libxatracker2 13.0.6-1+b2 libxatracker2_13.0.6-1+b2_i386.deb http://deb.devuan.org/merged/pool/DEBIAN/main/m/mesa/libxatracker2_13.0.6-1+b2_i386.deb optional libs 9f0a071210bba17c901b3cfe99306174 'libc6, libdrm-nouveau2, libdrm2, libexpat1, libgcc1, libgcrypt20, libllvm3.9, libsensors4, libstdc++6,'
+ '[' -z libxatracker2 ']'
+ IFS=' 	
'
+ PKG=libxatracker2
+ PKGVER=13.0.6-1+b2
+ PKGFILE=libxatracker2_13.0.6-1+b2_i386.deb
+ PKGPATH=http://deb.devuan.org/merged/pool/DEBIAN/main/m/mesa/libxatracker2_13.0.6-1+b2_i386.deb
+ PKGPRIO=optional
+ PKGSECTION=libs
+ PKGMD5=9f0a071210bba17c901b3cfe99306174
+ PKGDEP='libc6, libdrm-nouveau2, libdrm2, libexpat1, libgcc1, libgcrypt20, libllvm3.9, libsensors4, libstdc++6,'
+ set +x
...
+ local pkg=libllvm3.9
+ OIFS=' 	
'
+ IFS='|'
++ grep -m1 '^libllvm3.9|' repo-ascii-i386/pkgdb
+ set -- libllvm3.9 1:3.9.1-9 libllvm3.9_3.9.1-9_i386.deb http://deb.devuan.org/merged/pool/DEBIAN/main/l/llvm-toolchain-3.9/libllvm3.9_3.9.1-9_i386.deb optional libs fdb091aebdaa36253618b88100962a8b 'libc6, libedit2, libffi6, libgcc1, libstdc++6, libtinfo5, zlib1g,'
+ '[' -z libllvm3.9 ']'
+ IFS=' 	
'
+ PKG=libllvm3.9
+ PKGVER=1:3.9.1-9
+ PKGFILE=libllvm3.9_3.9.1-9_i386.deb
+ PKGPATH=http://deb.devuan.org/merged/pool/DEBIAN/main/l/llvm-toolchain-3.9/libllvm3.9_3.9.1-9_i386.deb
+ PKGPRIO=optional
+ PKGSECTION=libs
+ PKGMD5=fdb091aebdaa36253618b88100962a8b
+ PKGDEP='libc6, libedit2, libffi6, libgcc1, libstdc++6, libtinfo5, zlib1g,'
+ set +x

[s243a]

For the most part, I feel that it is pointless to "avoid bloat" in terms of blocking what the upstream distro feels are reasonable/recommended dependencies. The exception to me is firmware - so a system for only matching firmware for a particular system would avoid a lot of unnecessary bloat. Otherwise, in general, users tend to add new packages and the dependencies blocked out probably end up needed anyway.

Philosophically that makes a lot of sense but one can always remove blocks (or dummy packages) and install the actual package. Since my vision is a layered approach to the distro, blocks can be removed in higher layers to get the missing dependencies, and if one is looking for a leaner system then they can stick to using only the lower layers. Also lower layers could reside in ram but the higher layers could reside on the hard in such a way that only the actual files needed get pulled into ram at the time of processing.

[s243a]

thanks s243a

i'll look into that

wanderer

My last test produces a rootfs of
65.3 MiB (uncompressed)
33MB (Compressed as an sfs)

That said since I cut out a lot of things I don't what will work and what won't work.
/woof-distro/x86/tiny_devuan/ascii/basesfs (May 20, 2019 8:09pm MDT

[wiak]

thanks s243a

i'll look into that

wanderer

My last test produces a rootfs of
65.3 MiB (uncompressed)
33MB (Compressed as an sfs)

That said since I cut out a lot of things I don't what will work and what won't work.
/woof-distro/x86/tiny_devuan/ascii/basesfs (May 20, 2019 8:09pm MDT

Is that with xorg? That would certainly be very impressive. I couldn't get near producing such a small rootfs with xorg using Void Linux (not that I'm trying to since as I posted above I'm not concerned with cutting Void Linux dependencies - tiny core design philosophy has different setup of course, where install size particularly does matter).

wiak

[wanderer]

hi s243a and wiak

since we all seem to be working on the same problem

i would like to advocate for tinycore

not only as a base

but as a teaching tool

we could then translate the information
into instructions for woof-next
and have it build the iso
from puppy and devuan (or tinycore) parts
if we wanted to

the core.gz is 8 megs for the console
and if you open it you can identify all the parts easily
you could even reduce this size
by taking parts out of the core.gz

tinycore with xvesa
has all the parts and dependencies broken down into tcz
and loaded into the cde folder

if you load xorg
it will load it and load all the dependencies
it even has a dependency list

i have been taking debs
and using uextract to unsquash
and mksquashfs to squash them
and then symlinking them to root in the core.gz
this works just like loading an sfs file
with the layered file system

this also works with puppy pets and sfs files

as well as puppy.sfs files made by woof-next

you can load the corresponding tcz to a deb
and tinycore will load all the dependencies
and you can find their deb equivalents

as wiak noted
with the tinycore system bloat is not relevant
since you only need to load what you want at the time

tinycore has all the apps we would ever need in its repository
and can make a completely full polished distro
which i do for my uses

also we have the tinycore team to help with development

and many other advantages

i also wonder if dcore debian stretch sce maker
can be pointed to the devuan repositories

what do you think

wanderer

[s243a]

thanks s243a

i'll look into that

wanderer

My last test produces a rootfs of
65.3 MiB (uncompressed)
33MB (Compressed as an sfs)

That said since I cut out a lot of things I don't what will work and what won't work.
/woof-distro/x86/tiny_devuan/ascii/basesfs (May 20, 2019 8:09pm MDT

Is that with xorg? That would certainly be very impressive. I couldn't get near producing such a small rootfs with xorg using Void Linux (not that I'm trying to since as I posted above I'm not concerned with cutting Void Linux dependencies - tiny core design philosophy has different setup of course, where install size particularly does matter).

wiak

It does but it remains to be scene whether or not I installed enough Xorg packages. I'll do a chroot test first.

hi s243a and wiak

since we all seem to be working on the same problem

i would like to advocate for tinycore

My current experiment is with the tinycore base but with devaun packages. I'm not sure which approach will be smaller because I think that tinycore9 has slightly newer libraries than devaun ascii but at the same time tinycore is designed to be small.

Code: Select all

     not only as a base 

     but as a teaching tool 

      we could then translate the information 
      into instructions for woof-next 
      and have it build the iso 

We could start writing wiki entries. There is the official puppylinux wiki or maybe we can use github as a wiki.

from puppy and devuan (or tinycore) parts
if we wanted to

Yes we should focus on flexibility.

Code: Select all

      the core.gz is 8 megs for the console 
      and if you open it you can identify all the parts easily 
      you could even reduce this size 
      by taking parts out of the core.gz 

We could but this isn't priority because we can just include the entirety of core.gz as a package. In my latest commit, I moved the device files out of the core.gz package because they are only needed if we are using the produced rootfs as an initrd.

In some boot modes we will want to do this in others we won't.

Code: Select all

      tinycore with xvesa 
      has all the parts and dependencies broken down into tcz 
      and loaded into the cde folder 

So with regards to woof-next if you are building from the devaun repo then regarding the video driver dependencies of "xserver-xorg" create a dummy package (using the %dummy directive) for either "xserver-xorg-video-all" or "xorg-driver-video" and then install xserver-xorg-video-vesa