alternative puppy build system

[wanderer]

hi s243a and wiak

yes great works guys

i am concentrating on a tinycore-puppy base hybrid

(i am adding puppy functions to tc-config
which is the equivalent of the puppy init file
in the tinycore core.gz)

with puppy tinycore devuan and other source packages

i think that will yield the most modular and flexible system

wanderer

[wanderer]

syslinux.cfg

and

tc-config

are where boot code options can be added

wanderer

[wanderer]

hi guys

this is a just a thought for the future

if we had

1. a package that had the base dependencies of a distro

2. and a package that had the package manager of the distro

3. and a package that pointed to the repositories of the distro

we could load these packages

and then access that particular distros repositories

we could have different package groups
for each distro

so our system could be a multi distro build system

wanderer

[wanderer]

hi guys

this system has an unlimited number of options

but i would like to get a simple iso together
just to show proof of concept
and as a starting point

i guess i will make one
and post it on the first post repository link

any ideas that i should keep in mind

wanderer

[s243a]

Some gotchas that I worked through.

1. The "%rm" directive is dangerous. I moved it from line#350 to after the "%cutdown" directive on line #368. The reason being is after this point the /dev folder in the rootfs should be unmounted and consequently I won't delete device files in the host file system. As a second measure, I commented this directive out because the way I am building things there isn't really any device files to remove in the rootfs.

Note that the bind mounts of /dev and related unmounts are part of my fork and not the original woof next. I do a lazy unmount after at the end of the cutdown function.

Code: Select all

unbind_ALL_lazy

/builders/deb-build.sh#L280

Code: Select all

unbind_ALL_lazy(){
  unbind_PROC_lazy; unbind_SYS_lazy; unbind_DEV_lazy

}
unbind_PROC_lazy(){
  if [ "$(mount | grep "$CHROOT_DIR/proc")" != "" ]; then
    umount -R -l $CHROOT_DIR/proc
  fi
}
unbind_SYS_lazy(){
  if [ "$(mount | grep "$CHROOT_DIR/sys")" != "" ]; then
    umount -R -l $CHROOT_DIR/sys
  fi
}
unbind_DEV_lazy(){
  if [ "$(mount | grep "$CHROOT_DIR/dev")" != "" ]; then
    umount -R -l $CHROOT_DIR/dev
fi

/builders/chroot_and_tmpfs_fns.sh#L135

and a forced unmount prior to creating the sfs

Code: Select all

%makesfs)
    set -x
    unbind_ALL
	shift # $1-output $@-squashfs params
	echo Creating $1 ...
make_sfs "$@" ;;

/builders/deb-build.sh#L603

Code: Select all

bind_ALL(){
  bind_PROC; bind_SYS; [ $INTERACT_DEV -eq 1 ] && bind_DEV
}
unbind_PROC(){
  if [ "$(mount | grep "$CHROOT_DIR/proc")" != "" ]; then
    umount -R -f $CHROOT_DIR/proc
  fi
}
unbind_SYS(){
  if [ "$(mount | grep "$CHROOT_DIR/sys")" != "" ]; then
    umount -R -f $CHROOT_DIR/sys
  fi
}
unbind_DEV(){
  if [ "$(mount | grep "$CHROOT_DIR/dev")" != "" ]; then
    umount -R -f $CHROOT_DIR/dev
  fi
}

/builders/chroot_and_tmpfs_fns.sh#L117

I should also add a check to make sure the unmount of these binds was successful before making the sfs.

P.S. for safety reasons I added a "press enter to continue" prompt at the end of the cutdown function.

Code: Select all

read -p "Press enter to continue"

/builders/deb-build.sh#L287

for this to work I had to use file descripts in the process_pkglist funciton

Code: Select all

mkdir -p /tmp/deb-build/
exec 15<> /tmp/deb-build/fd_15

/builders/deb-build.sh#L574

and I turned the related loop into a style which isn't supported by ash. Using bash as the intereter may or may not be necessary and it certainly won't be necessary if we remove said "press enter to continue" prompt.

2. the file basesfs, probably needs a blank line at the end of it.. My tests indicate that the file basesfs needs a blank line at the end of it because the last line isn't being read. I would like other people to verify this and if they can figure out why then please let me know. I've reported the issue.

3...more as I try to remember them.

[s243a]

On another note, dpkg doesn't seem to want to install either bash or dash without libselinux.so.1. I could use the "%bootstrap" directive to install both bash and dash, but I think libselinux1 might be important so I'll install/configure it properly.

However, if I add libselinux1 via the %bootstrap detective the library libselinux.so.1 doesn't appear in the search path. It is located in the folder /lib/i386-linux-gnu

My /etc/profile seems to be coming from my package tinycore9_base_gz.

I'll define the LD_LIBRARY_PATH variable here as follows:

Code: Select all

#s243a, add LD_LIBRARY_PATH variable
LD_LIBRARY_PATH=/lib:/usr/lib:/root/my-applications/lib:/usr/local/lib
if [ -d /lib/i386-linux-gnu ]; then #added by s243a
  LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/lib/i386-linux-gnu" 
fi 

and see what happens

Hmmm...other things that might be relevant to this are /etc/ashrc /root/.ashrc and /etc/ld.so.conf

Also I wonder if I can specify this profile info in the chroot command.

Related info:
http://forum.tinycorelinux.net/index.php?topic=11758.0
https://unix.stackexchange.com/question ... ld-so-conf
https://unix.stackexchange.com/question ... how-to-add
https://askubuntu.com/questions/350068/ ... -libraries

[s243a]

The issue was since I wasn't using a login shell the host environment was bing used. Consequently, I just defined the necessary variables in the host environment but I should also try it with a login shell.

On another issue. In need to delete /bin/tar from the tinycore9_base_gz package because dpkg requires the full version. Later on though I will make /bin/tar a script that looks for the full version and if it doesn't exist calls busybox.

[wanderer]

wow s243a

fantastic work

major coding

im trying to absorb all this

also read your post on root-skeleton

thanks for all your work

wanderer

[s243a]

I tried sc0ttman's package manager in a chroot. I got the following errors (not all at the same run):

================

Code: Select all

+ WORKDIR=/root/pkg
+ '[' -z '' ']'
++ grep 'export PKG_TAB_COMPLETION=true' /root/.bashrc
+ '[' 'export PKG_TAB_COMPLETION=true' = '' ']'
+ '[' '!' -d /tmp/pkg/ ']'
+ '[' '!' -d /tmp/pkg/ ']'
+ PKG_NAME_ALIASES=...
+ '[' '!' -f /tmp/pkg//pkg_aliases ']'

/Pkg/blob/master/usr/sbin/pkg#L188

Code: Select all

+ '[' '!' -f /root/.pkg/firstrun -a --repo-update = '' ']'
+ set -a
+ '[' '!' -f /root/.packages/Packages-devuan-ascii-main -o '!' -f /root/.pkg/sources ']'
+ mkdir -p /root/.pkg
+ mkdir -p /root/.packages
+ '[' '!' -f /root/.pkg/sources ']'
+ '[' -f /root/.packages/Packages-puppy-noarch-official ']'
+ '[' '!' -f /root/.packages/Packages-puppy-noarch-official ']'
+ error 'Repo files not found. Check \e[95m/root/.packages\e[0m'
+ echo -e '\e[91mError:\e[0m Repo files not found. Check \e[95m/root/.packages\e[0m'
Error: Repo files not found. Check /root/.packages

/Pkg/blob/master/usr/sbin/pkg#L7361

Code: Select all

+ cleanup 1 1
+ rm -f '/tmp/pkg/missing_dep*' '/tmp/pkg/installed_pkg*' '/tmp/pkg/all_dep*' /tmp/pkg/DEP_DONE /tmp/pkg/list_deps_busy /tmp/pkg/pkg_file_list /tmp/pkg//dependents_list /tmp/pkg/DEP_DONE /tmp/pkg/ldd_file_list

Code: Select all

Warning: No repo files found. Setting the default repo to 'noarch'.

/Pkg/blob/master/usr/sbin/pkg#L7359

Code: Select all

/ # pkg --repo
cp: '/root/.packages/Packages-puppy-noarch-official' and '/root/.packages/Packages-puppy-noarch-official' are the same file
Warning: No repo files found. Setting the default repo to 'noarch'.
grep: /root/.pkg/sources-user: No such file or directory
grep: /root/.pkg/sources-user: No such file or directory
grep: /root/.pkg/sources-user: No such file or directory
grep: /root/.pkg/sources-user: No such file or directory
grep: /root/.pkg/sources-user: No such file or directory
Success: 'noarch' added.
Re-running: pkg --repo

/Pkg/blob/master/usr/sbin/pkg#L7366


=======================

Anyway, my guess is that if the file /root/.pkg/firstrun exists then pkg tries to install the distro information from the existing package manager (i.e. petget). I think there is a bug here. I think that if these files don't exist then pkg should delete the "firstrun" file and use the existing repo information:
/root/.pkg/pkgrc
/root/.pkg/sources
root/.pkg/sources-all

So anyway, how I corrected this (not tested yet) is to move the first run file from /root/.pkg/firstrun to a new package I created called:
woof-code/rootfs-packages/puppy_ppm_configs_devaun_ascii
which containes the repo info from Sailor Enceladus's "puduan-7.0.0a1 (Devuan Ascii) woof-ce testing"

Now regarding the size, some packages wern't installed in my previous run, due to missing dependencies for dpkg (and/or the post install scripts). I've added the following packages:
libattr1 and libacl1 in commit 5889251829d8ab13cd8188500247b2824e836af6
tar in commit f6f129af688d1f4d50275f16c9b9856a6a4a40c4
libbz2-1.0 in commit: c0e75d62d10e99ae3bfe8c5d6d83870fed5dfae4
liblzma in commit da6f23c90f2c018f7c6ba1b5cb00e047e223d3dd
coreutils in commit: ffc53898bcfc97d687250e802eb07a6536b49e2b

If we want a smaller system we can move these dependencies to the adrive or another sfs.

I also also added "midnight commandar's" (AKA mc) dependencies and I'm now installing mc (and it's dependencies) using the "%bootstrap" directive due to the following type of errors:

Code: Select all

dpkg: error: parsing file '/var/lib/dpkg/status' near line 1994 package 'libxxf86vm1':
 mixed non-coinstallable and coinstallable package instances present; most probably due to an upgrade from an unofficial dpkg

I like dpkg for building a system because it has more info about dependencies (minimum version and alternatives), however, I think for the package manager in the final build I will probably prefer pkg because it will give more flexibility and won't refuse to install things due to these types of inconsistencies. In my opinion dpkg is too easy to break. This is good if you want to protect a working system but a pain in the but when trying to get the the final set of applications that one wants.

The section where I added mindnight commander and dependencies is:

Code: Select all

%bootstrap 
libssh2-1 #Required for mc
libgpm2 #Required for mc
e2fslibs #Required for mc
libslang2 #Required for mc
mc #Replaces rox-filer          # this will pull-in gtk2
%dpkgchroot

/woof-distro/x86/tiny_devuan/ascii/basesfs#L364

Currently, I"m using midnight commander instead of rox. The reason being is that I want a system that works well in a console environment and also because it is possible to use jwm without a desktop. In a higher layer I'll add rox (and/or pcmanfm).

[wanderer]

hi s243a

i am following along as best i can

great work

wanderer

[s243a]

wow s243a

fantastic work

major coding

im trying to absorb all this

also read your post on root-skeleton

I think you mean this thread, "Extract Specific Files from woof-code/rootfs-skeleton/"

thanks for all your work

wanderer

Thankyou . I'm hoping to get it a point where others can take over because I want to return to TazPup64 but I think that woof-next has a lot of potential and I would like to see it get to a point where people are excited about it

[wanderer]

hi s243a

well i certainly am excited about it

and i think when we begin to make isos

that will make it real to everyone

once again thanks for your awesome work

wanderer

[s243a]

It appears that sc0ttman's pkg requires the following dependencies:
findutils, grep, sed

but I could probably create busybox wrappers for these to strip out any unsupported options. For now I'll include them.

Installing individual packages doesn't seem to be that roboust Maybe have an option to boostrap on fail. See what I ended up missing! in quotes.

I was also missing two dependency for midnight commander:
libglib2 and libgdk-pixbuf2.0-0
**These were on my list but didn't install so I'm bootstrapping them instead.

and various dependences (either didn't install or wern't included on my list) for xdialog which are:
libpangocairo, libpango-1.0-0, libcairo2 libxcomposite1, libxdamage1, libxfixes3, libpangoft2-1.0-0, libfontconfig1, libxrender1, libxi6, libxrandr2 and libxcursor1, libthai0, libmount1, libharfbuzz0b, libexpat1, libdatrie1, libgraphite2-3

I installed aterm because that was the terminal that wander was using but it just seems to be a wrapper for https://packages.debian.org/stretch/aterm]urxvt so I'll install that. I don't feel like going though the dependency for this at the moment so I'll install it after I turn on the dependency tracking.

on another note, I need the following file to be able to update the repo database via sc0ttman's package manager (i.e. pkg).
/woof-code/rootfs-packages/PKG/usr/local/petget/0setup

I also copied more files into the following directory:
/woof-code/rootfs-packages/puppy_ppm_configs_devaun_ascii/var/packages

from Sailor Enceladus's "puduan-7.0.0a1 (Devuan Ascii) woof-ce testing"

Anyway midnight commander is now working for me and I'm able to use PKG to install packages. However, PKG won't download the dependencies until I update the file/s which gives the pet specs of the installed packages.

Edit 1: I might have a binary incompatbility problem with the tinycore libc vs the devaun asci version:

Code: Select all

/ # Xdialog
Xdialog: relocation error: /usr/lib/i386-gnu/libXfixes.so.3: symbol __stack_chk_guard, version GLIBC_2.4 not defined in file ld.so.1 with link time reference

hmm...what to do?

[s243a]

It appears that sc0ttman's pkg requires the following dependencies:
findutils, grep, sed

but I could probably create busybox wrappers for these to strip out any unsupported options. For now I'll include them.

Installing individual packages doesn't seem to be that roboust Maybe have an option to boostrap on fail. See what I ended up missing! in quotes.

I was also missing two dependency for midnight commander:
libglib2 and libgdk-pixbuf2.0-0
**These were on my list but didn't install so I'm bootstrapping them instead.

and various dependences (either didn't install or wern't included on my list) for xdialog which are:
libpangocairo, libpango-1.0-0, libcairo2 libxcomposite1, libxdamage1, libxfixes3, libpangoft2-1.0-0, libfontconfig1, libxrender1, libxi6, libxrandr2 and libxcursor1, libthai0, libmount1, libharfbuzz0b, libexpat1, libdatrie1, libgraphite2-3

I installed aterm because that was the terminal that wander was using but it just seems to be a wrapper for https://packages.debian.org/stretch/aterm]urxvt so I'll install that. I don't feel like going though the dependency for this at the moment so I'll install it after I turn on the dependency tracking.

on another note, I need the following file to be able to update the repo database via sc0ttman's package manager (i.e. pkg).
/woof-code/rootfs-packages/PKG/usr/local/petget/0setup

I also copied more files into the following directory:
/woof-code/rootfs-packages/puppy_ppm_configs_devaun_ascii/var/packages

from Sailor Enceladus's "puduan-7.0.0a1 (Devuan Ascii) woof-ce testing"

Anyway midnight commander is now working for me and I'm able to use PKG to install packages. However, PKG won't download the dependencies until I update the file/s which gives the pet specs of the installed packages.

Edit 1: I might have a binary incompatbility problem with the tinycore libc vs the devaun asci version:

Code: Select all

/ # Xdialog
Xdialog: relocation error: /usr/lib/i386-gnu/libXfixes.so.3: symbol __stack_chk_guard, version GLIBC_2.4 not defined in file ld.so.1 with link time reference

hmm...what to do?

A lot of the missing packages noted above were due to them failing to be installed via the function dpkgchroot_install(). My solution is to do a bootstrap intall if this fails:

Code: Select all

DPKG_CHROOT_FALLBACK='%bootstrap'

/builders/deb-build.sh#L262
and

Code: Select all

	
if [ "$DPKG_CHROOT_FALLBACK" = '%bootstrap' ] &&
	[ ! -e "$CHROOT_DIR$ADMIN_DIR/info/${PKG}.list" ]; then
	  bootstrap_install
fi

/builders/deb-bd.sh#L320

[s243a]

I added the ability to sync the dpkg info that comes from the "control" file to the petspec database.

The function that does the work is:
sync_pet_specs_fm_dpkg.sh

This is include in the config file as follows:

Code: Select all

%addpkg sync_pet_specs_fm_dpkg
%chroot sync_pet_specs_fm_dpkg.sh
%addpkg puppy_ppm_configs_devaun_ascii
%addpkg PKG
%chroot pkg -ru #Update the packge repos

/woof-distro/x86/tiny_devuan/ascii/basesfs#L386

We need this info for PKG to work properly. Otherwise it doesn't know which dependencies to download and just gives up.

[wanderer]

hi s243a

still following along

awesome work

wanderer

[s243a]

on another note, I need the following file to be able to update the repo database via sc0ttman's package manager (i.e. pkg).
/woof-code/rootfs-packages/PKG/usr/local/petget/0setup

I was going though 0setup and I also need debdb2pupdb and find_cat which are both binaries.

DEBDB2PUPDB="$(pwd)/support/debdb2pupdb"
FIND_CAT="$(pwd)/support/find_cat"
#note, 3builddistro copies it into rootfs-complete/usr/local/petget when building a pup.

/rootfs-packages/PKG/usr/local/petget/0setup#L71

which are both binary files. Here is the makefile for it:
/initrd-progs/pkg/w_apps_static/w_apps/Makefile#L5

[wanderer]

hi s243a

could you give a very short overview
of what you are doing

i am following along
but it is not completely clear to me
all that you are doing

and thanks for doing all this great work

awesome

wanderer

[s243a]

hi s243a

could you give a very short overview
of what you are doing

I'll try to do this later. I need to focus on each step so that I can document what I'm doing but I'll try to do a summarized overview later.

However, that said. In addition to this thread you can see my commits at:
https://github.com/puppylinux-woof-CE/w ... :woof-next

[s243a]

I decided that I wanted to make the basesfs specification simpler and let the package manager pull in the dependencies. Therefore the following file is now shorter:

/woof-distro/x86/tiny_devuan/ascii/basesfs

the longer more detailed one I renamed as:

/woof-distro/x86/tiny_devuan/ascii/basesfs-detailed

I found out that sc0ttman's package manager (i.e. pkg) needs wget. I think the devaun repositories don't like the --spider option in the busybox version of wget.

I modified deb-build.sh so that one could use the functions within it without running the script:

Code: Select all

params() {
	case "$1" in
		--help|-h) echo "Usage: ${0##*/} [--help|-h|pkglist]" && exit ;;
		"") ;;
		--sourced)
		   SOURCED=1
		   shift 
		   params "$@"
		;;
		*) PKGLIST="$1"
	esac
}

### main
params "$@"
if [ -z "$SOURCED" ] || [ $SOURCED -ne 1 ]; then
  sanity_check
  prepare_dirs
  add_multiple_repos $DEFAULT_REPOS
  echo Flattening $PKGLIST ...
  flatten_pkglist $PKGLIST > $FLATTEN
  if [ -z "$DRY_RUN" ]; then
	process_pkglist $FLATTEN
  else
	cat $FLATTEN
  fi
fi

/builders/deb-build.sh#L713

As an example of how one might use this new feature can be found at:
/woof-next/examples/workdir/make_sfs_fm_rootfs.sh

I also did try running the ISO I built in qemu. I ended up in a tinycore shell and I had no idea how to switch to root user. I"ll look into this.