Fatdog64 ISO Builder

[belham2]

@belham2: congrats on your success. The parameter you passed to build-iso is incorrect though.

If you want to build small initrd only - "./build-iso.sh initrd small"
If you want to build an ISO file with small initrd - "./build-iso.sh iso small"

________________________

If you want to get complicated, here are more variations:

INITRD AND ISO
==========
If you want to build an BIOS-only (no UEFI support) ISO file with small initrd - "./build-iso.sh iso small isolinux"
If you want to build an BIOS (no UEFI support) with grub4dos ISO file with small initrd - "./build-iso.sh iso small grldr"
If you want to build a standard (=huge) initrd only - "./build-iso.sh initrd"
If you want to build an ISO with standard (=huge) initrd - "./build-iso.sh iso"
If you want to build an BIOS-only ISO with standard (=huge) initrd - "./build-iso.sh iso std isolinux"
etc

SFS only
======
If you want to build only the basesfs - "./build-iso.sh base"
If you want to build only the devx sfs - "./build-iso.sh devx" (note: will need to add DEVX_ROOTFS_DIR= to build-iso.conf). Edit sfs-devxlist as needed.
If you want to build only the nls sfs - "./build-iso.sh nls" (note: will need to add NLS_ROOTFS_DIR= to build-iso.conf). Edit sfs-nlslist as needed.
If you want to build basesfs, devx, nls at the same time, - "./build-iso.sh multi"
If you want to build 32bit compat lib sfs - "./build-iso.sh base32" (note: will need BASE32_ROOTFS_DIR= in your build-iso.conf, and also edit sfs-base32list as needed).

If you want to build only the unsplit basesfs (unsplit means all the header files, docs, nls etc are in the same sfs) - "./build-iso.sh single" - but you will need to populate sfs-pkglist

Other notes
=======
If you already built a basesfs and want to build an initrd/iso, and don't want to rebuild the basesfs again, pass USE_EXISTING=1 to env before calling build-iso.sh, e.g.

USE_EXISTING=1 ./build-iso.sh iso

All these additional notes should go into the README or INFO or something, I know. I'll do that in the next release ... hopefully


_________________________

Thank you, James!! I'm going to do a couple more builds over the next few days trying variations of the parameters you listed here. Much thanks again for making this possible for someone like me

[proebler]

.. ok, I too can now report success with the iso-builder after sorting out the problems I had.
It is a queer sort of story.
I had put the build-iso files and directories on a bootable USB stick, which -of all things- I once set up to to boot to a (Win98) DOS console. The USB stick is 16GB, has a lot of free capacity, formatted to FAT32, so I use it simply as a storage device.

.. now, ROX cannot copy symlinks to a Win/DOS device, they are incompatible there !

I must have got (ROX) error warnings, when I copied initrd to the fatdog-initrd directory.
But I ignored the warnings when I checked, and -superficially at least- , found that things 'looked ok'.
(I have ignored warnings from ROX before with apparent impunity)
Well, in this instance, the warnings came with good reason.
I discovered, that the fatdog-initrd directory was missing the 'run' and the 'init' links and quite probably many others.
No wonder then, that the result was kernel panic.
There were other errors as well, possibly unrelated to the above, but rather to limiting RAM (?).
I found that more than half the files in pkgcache were empty (0byte) files, so I had to re-download them.

After that: Success, I have an iso without LibreOffice, Gimp, games; instead it has Gnumeric, Abiword, fotoxx and Qemu to do testing !

Some observations I have made:

The initrd which has to be stripped of kernel-modules.sfs and fd64.sfs and then copied into the fatdog-initrd directory must not be repacked . It has to be copied unpacked. (I didn't repack, but during my investigations also tried it repacked, which does not work)
Perhaps Note 3 in the README should say so. (see also the earlier posted step 13 of jamesbond's detailed instructions)

Something to watch when building successive iso's, is that the build directory does not get cleared after each build.
I got caught by that, when after building the full iso using the default sfs-baselist, I used a reduced, different sfs-baselist. The resulting build was again the full iso.
Perhaps the build directory ought to be cleared automatically (?).
BUT I have in fact used that retained build to later make some additions to a build/iso (again running ./build-iso.sh iso). In a way, an editing of the fd64.sfs, I suppose (?), I'm not sure, it worked.

This is it,
thanks again for this tool and the support.
proebler

[jamesbond]

Congrats proebler.

I will update note 3 in the README, yes (I thought it was obvious enough, but apparently not). Another note I will add also is that you will need a properly Linux filesystem to build. Definitely no FAT32, not NTFS.

EDIT:
RE: build-dir -- the build directory containing temporary rootfs is not cleared - yes, it is not. By default those are built in /tmp and are cleared, but if you redirect it elsewhere, I assume you would want to keep it (for further checking etc). Perhaps not a very good assumption. For now you would just have to remember to clear it after every build.

[Terry H]

Firstly, thanks to the fatdog team for making the fatdog64 iso builder available.

I have had 3 attempts to run the fatdog-iso-builder. On the 3rd attempt I got what appears to be a fully working iso. The problem is that 11 (of 612) packages were not downloaded as the message stated that they already existed. THe PC I am using is a desktop with AMD FX-6100 processor with 10 GB RAM.

The following is a summary of the 3 attempts I have made so far:

Attempt # 1
I downloaded and extracted fatdog-builder-iso and package list. Created build & build-iso folders on ext4 partition. Did all the copying and setup as specified in the jamesbond's qemu example (post #27) I messed up with the initrd, as after unpacking and removing the 2 items from the unpacked initrd, I repacked and copied that to the fatdog-initrd folder. Upon running ./build-iso.sh iso. The job did not download approximately 11 packages as they were flaged as already existing. The same packages were also skipped in the build. The iso that was produced failed to boot with a kernel panic. I presume this was due to my initrd mistake, not the skipped packages.

To test the generated iso's from each attempt I did a new frugal install with savefile=none.

Attempt # 2
Deleted the contents of the build folder, created the build-iso folder. I retained the fatdog-builder-iso and package list archives. Deleted everything else and did the setup again. My mistake this time was forgetting to extract the package list and include it in the fatdog-builder folder. The job ran but only downloaded 37 packages and created an iso. I was able to boot and login to a console session as root.

Getting a bootable iso was encouraging.

Attempt # 3
Same as Attempt # 2, but included package list. This time showed 612 packages to be downloaded. As per attempt # 1, the same packages were not downloaded and skipped in the build.

The skipped packages were:

Code: Select all

slapt-get-0.10.25-x86-64-1
libnettle-3.1.1-x86-64-1
e2fsprogs-1.42.12-x86-64-1
parted-3.2-x86-64-1
lua-5.2.3-x86-64-1
libacpi-0.2-x86-64-1
libnotify-0.5.2-x86-64-1
libgnutls-3.3.18-x86-64-1
bluez-4.10.1-x86-64-1
obexd-0.46-x86-64-1
xcb-util-keysyms-0.4.0-x86-64-1

The resulting iso seems to be fully functional. On testing this new frugal install, I was able to run slapt-get and parted from the command line. These were 2 of the packages which were in the skipped group. Checking for other libs showed that they were also included.

If possible could one of the fatdog team please advise how the packages may have been flagged as already existing and subsequently skipped, but subsequently seemingly being included in the output fatdog.iso.

Thanks in advance for any assistance.

[Terry H]

To check my results I decided to run the build on my wifes laptop, which hasn't had fatdog on it before. It has grub4dos on it, but didn't have any ext3/4 partitions. I shrank a data partition in windows and then created an ext3 partition using a frugal installed tahr pup I already had installed on it.

I downloaded the fatdog710 iso and created a new frugal install on the new partition. I then downloaded the fatdog-build and packagelist archives and did the setup to run the build system on this PC.

The result was the same, as the final run in my previous post. The same packages were not downloaded. I did a new frugal install using the new fatdog.iso with savefile=none. Runs fine, connected to internet using wifi, ran slapt-get from CLI update and upgrade. Updated 6 packages. I ran parted -l from cli and listed partitions. So all is good.

I still would like advice as per my previous post, however, it now appears to me that, the results I have received are normal. In the mean time I am going to make a few changes to the package list and run again. Mainly going to remove the browser and run a browser as an sfs.

[mcradventures]

Can this be used to make essentially the latest Fatdog but with the 3.14 kernel? I only ask because I like the sound of Fatdog but it doesn't run on the old laptop I have, but Tahr pup does, which uses an older kernel. I have narrowed down the issue to the kernel as it seems no "modern" distro with the 4.x kernel will boot on the laptop. I've spent two days trying to get ANY linux distro on this laptop and the only successful one was Tahr pup, but it lacks some features that Fatdog seems to have and I can't figure out any way to add those features to Tahr pup.

If this can't do it, is there any way to do it? Is there a possibility of Fatdog being released officially with an older kernel for older hardware? If curious, the laptop is a Dell Vostro 1000.

[step]

I don't know how to answer your specific questions directly, sorry. But have you considered running Fatdog 702 on your Dell? 702 did run on k3.14 until it was upgraded. Here's the kernel repo for 702 http://distro.ibiblio.org/fatdog/kernels/700/. Are you looking for specific features of 710 that aren't available in 702? If so, which ones?

[step]

@Terry H,

I suspect that indeed all the packages were downloaded, and that the "already downloaded" messages occur on subsequent download attempts, which depend on the inner workings of the ISO builder.

I have traced one exemplar package, parted-3.2, as it goes through the script. What I see is that it is downloaded successfully, then about 50 seconds later the script wants to download it again but finds it in the cache and outputs "already downloaded", then after some more time the script wants to install it, and finds it "already installed" (in my system), and so on.

I could make the whole trace available, if you really need it, but perhaps this excerpt is enough to show what's going on in the life of parted-3.2. And I guess the other packages go through the same drill.

Note that parted-3.2 takes two different IDs, 450 and 468, in the download list. There are duplicates in the list, probably because the script errs on the safe side. No package is downloaded more than once. The script "announces" 612 intended downloads, but it actually downloads only 601 packages. It means that 11 packages are duplicated, which is the number of packages in your list.

11 = 612 -1 - $(ls pkgcache/*.txz | wc -l) # -1 because pkgcache starts with a locally created package, pkgcache/bash-default-shell-1.0-noarch-1.txz

Code: Select all

+ : 'read pkg(parted-3.2-x86_64-1) Thu Jan 26 8:30:39'
+ pkg=parted-3.2-x86_64-1
+ set -- parted-3.2-x86_64-1
+ pkg=parted-3.2-x86_64-1
+ '[' -z parted-3.2-x86_64-1 ']'
+ PKGS=450
+ echo -n '450 of 612: '
450 of 612: + '[' -s pkgcache/parted-3.2-x86_64-1.txz ']'
+ download_file parted-3.2-x86_64-1.txz
+ echo downloading parted-3.2-x86_64-1.txz ...
downloading parted-3.2-x86_64-1.txz ...
+ '[' parted-3.2-x86_64-1.txz = parted-3.2-x86_64-1.txz ']'
+ wget -c -o /dev/null -O pkgcache/parted-3.2-x86_64-1.txz http://distro.ibiblio.org/fatdog/packages/710/parted-3.2-x86_64-1.txz
+ check_md5 parted-3.2-x86_64-1.txz
+ echo -n 'Checking integrity parted-3.2-x86_64-1.txz ... '
Checking integrity parted-3.2-x86_64-1.txz ... + read goodsum
++ awk -v pkg=./parted-3.2-x86_64-1.txz '$2==pkg {print $1;exit} END {print "NONE"}' pkgcache/CHECKSUMS.md5
+ read filesum
++ md5sum pkgcache/parted-3.2-x86_64-1.txz
++ awk '{print $1}'
+ '[' a65b74ddf8bf314430f15bde4cf5793f = NONE ']'
+ '[' a65b74ddf8bf314430f15bde4cf5793f '!=' a65b74ddf8bf314430f15bde4cf5793f ']'
+ echo good.
good.
+ return 0

-------------------------- about 50 seconds later -------

+ : 'read pkg(parted-3.2-x86_64-1) Thu Jan 26 8:31:24'
+ pkg=parted-3.2-x86_64-1
+ set -- parted-3.2-x86_64-1
+ pkg=parted-3.2-x86_64-1
+ '[' -z parted-3.2-x86_64-1 ']'
+ PKGS=468
+ echo -n '468 of 612: '
468 of 612: + '[' -s pkgcache/parted-3.2-x86_64-1.txz ']'
+ echo parted-3.2-x86_64-1 already exist, not downloaded.
parted-3.2-x86_64-1 already exist, not downloaded.
+ check_md5 parted-3.2-x86_64-1.txz
+ echo -n 'Checking integrity parted-3.2-x86_64-1.txz ... '
Checking integrity parted-3.2-x86_64-1.txz ... + read goodsum
++ awk -v pkg=./parted-3.2-x86_64-1.txz '$2==pkg {print $1;exit} END {print "NONE"}' pkgcache/CHECKSUMS.md5
+ read filesum
++ md5sum pkgcache/parted-3.2-x86_64-1.txz
++ awk '{print $1}'
+ '[' a65b74ddf8bf314430f15bde4cf5793f = NONE ']'
+ '[' a65b74ddf8bf314430f15bde4cf5793f '!=' a65b74ddf8bf314430f15bde4cf5793f ']'
+ echo good.
good.
+ return 0

[Terry H]

@step

Thanks so much for running the trace and the explanation. That response is more than satisfactory. With subsequent runs I had become satisfied that it was OK. The parted-3.2 example is sufficient.

Apart from checking that the affected applications ran, I had also checked the pkgcahe and saw the packages were present following the build. Unfortunately duplicate entries in the sfs-baselist didn't cross my mind. I've read through this thread several times now and I was surprised that this hadn't been queried previously in the almost year that this thread has been existing.

Hopefully this will be assistance to new users who want to run the Fatdog64 ISO Builder in the future. This systm is really an excellent inclusion to this community.

[rufwoof]

Made some notes ... followed/filled them in and ran the build script to build a base (cli) system only with wifi net connect ... and all ran quickly and smoothly.

/tmp threw me a bit. I boot frugally all running in ram on a 4GB system and could perhaps have gotten away with just leaving that as it, but I moved /tmp to a hdd based /tmp as part of the build. Subsequently I've seen a earlier post in this thread that details how to set /tmp to a different location within the build script.

My notes ...

Code: Select all

http://murga-linux.com/puppy/viewtopic.php?t=105329

The Fatdog64 ISO Builder.

It enables you to build a custom ISO from the ground up, from a given list of packages.
No remastering is involved. It is the same builder that we use ourselves to build the
official ISO.

Another way to explain it: Fatdog64 ISO Builder is the Fatdog's version of Woof. If you
have seen how the "next-generation" branch of Woof-CE (aka woof-ng), you would see a lot
of similarities since the woof-ng builder was originally based from this.

Downloaded the latest build script tarball from
http://distro.ibiblio.org/fatdog/iso/builder/
ran xz -d to decompress it and tar xvf ... to extract it (under a hdd based 
folder in my case within a folder /mnt/sda1/fatdog-build that I created)

===============

Get a recent ISO, extract out
its initrd, put it in a folder called `fatdog-initrd` and use that
folder to replace the one here. Note: use the extracted content,
and do not re-pack it again.

If you forgot to do this you will be gently reminded when you boot
the resulting ISO
Make sure you remove `kernel-modules.sfs` and `fd64.sfs` from the
extracted initrd to save space as they are not used and will be
replaced with by the build process anyway.

deleted all files/folders in fatdog-initrd then ..
cd /mnt/sda1/fatdog-build/fatdog-iso-builder/fatdog-initrd
cat /mnt/sda1/initrd | cpio -id # to extract
removed both fd64.sfs and kernel modules sfs


=================

All the working files will be created in this directory unless you
specify otherwise (via `build-iso.conf`). Thus make sure that you put
this in a filesystem large enough to contain all the downloaded
packages and the final output.

It will use `tmp` as temporary directory by default; so you will also
need plenty of space there. How big the space is needed depends on how
many packages you want to build.

Fatdog64 800 requires at least 2.5GB free in /tmp.
Newer version will require more space.

If you have less than 2.5GB in /tmp, you need to specify directory
location for storing temporary files.


mkdir -p /mnt/sda1/tmp
mount -M /tmp /mnt/sda1/tmp  # Move from tmpfs (limited/ram under frugal boot) to hdd based
# resize current allocation to /tmp (ram limit)
# mount -o remount,size=10G /mnt/sda1/tmp ... when it came to the actual
run I didn't need to do that as under fatdog tmpfs was sized to 16GB). The
following is also the df -h as I ran when making the notes under voidlinux ... 
# so df -h shows
# df -h
Filesystem      Size  Used Avail Use% Mounted on
inram           3.4G  115M  3.3G   4% /mnt/layers/RAM
/dev/sda1       447G  146G  279G  35% /mnt/sda1
overlay_result  3.4G  115M  3.3G   4% /
run             1.7G  696K  1.7G   1% /run
dev             1.7G  4.0K  1.7G   1% /dev
shm             1.7G   60M  1.7G   4% /dev/shm
cgroup          1.7G     0  1.7G   0% /sys/fs/cgroup
tmpfs            10G  8.0K   10G   1% /mnt/sda1/tmp


=====================

To build BIOS-only ISO with huge (standard/default) initrd, using syslinux bootloader:
`./build-iso.sh iso std isolinux`

To build a standard BIOS/UEFI ISO (like standard Fatdog64):
`./build-iso.sh iso`

I built the first of the above two noted choices. Ran through very quickly, went off to make coffee and by the time I returned it was back at the command prompt (had finished). So single digit minutes build time.

I opened the .iso that had been produced and copied out the initrd and
vmlinuz and set up grub4dos menu.lst to boot those files, along with
network connect ...

# menu.lst
color white/blue black/cyan white/black cyan/black
timeout 10
default 0

title Fatdog
root (hd0,0)
kernel /fatdog-build/vmlinuz net=wpa2:VM123456-2G:abcd1234:wlan0:dhcp pkeys=uk
initrd /fatdog-build/initrd

and it booted OK. At login prompt entered userid 'root' and it auto logged in
upon pressing <enter>

vmlinuz filesize around 6MB
initrd filesize (with kernel modules and main sfs integral to that) around 116MB

Font was small so once logged in I cd /lib/boot/consolefonts and ran
setfont ./big.psf.gz 
... to load a larger font


ping google.com
... worked OK ... etc

[rufwoof]

Edited sfs-baselist to add mc (and its required dependencies) and screen (multiplex terminal) so that snippet of code section now looks like ...

Code: Select all

vim-8.0.586-x86_64-1 
##grub-bios # build later, together with the other bootloaders
mc-4.8.22-x86_64-1 
gpm-1.20.7-x86_64-1 
libssh2-1.8.0-x86_64-1 
glib-2.58.1-x86_64-1 
openssl-1.1.0j-x86_64-1 
libpcre-8.42-x86_64-1 
screen-4.6.2-x86_64-1 


### end of CLFS, start of CBLFS and beyond

I also changed build-iso.sh to use lz4 high compression instead of xz (builds a lot faster and only a modest increase in filesize (due to less tight compression)

SQUASHFS_PARAM=${SQUASHFS_PARAM--comp lz4 -Xhc}

A nice little bootup for cli junkies (ssh'ers). Great how fatdog supports just specifying your wireless ssid and password as part of the kernel boot parameters and starts up already wireless net connected.

[rufwoof]

Also added fbterm and netsurf-fb .. and the final filesize for initrd isn't that much larger. Doesn't work as-is however (well fbterm does, but not netsurf-fb)

Code: Select all

### fbterm
fbterm-1.7.0-x86_64-1 
freetype2-base-2.9.1-x86_64-1
fontconfig-2.13.0-x86_64-1 
libpng-1.6.34-x86_64-1 
# font-cantarell-0.0.25-noarch-1  # horrible looking font
font-terminus-ttf-4.46.0-noarch-1 # nicer font or ... (use fbterm --font-size=32)
#font-inconsolata-2.001-noarch-1 # nice, but mc needs -a to display properly
### netsurf
netsurf-fb-3.8-x86_64-1 
gtk2-2.24.32-x86_64-1 
librsvg-2.40.20-x86_64-1 
expat-2.2.5-x86_64-1 
libpng12-1.2.59-x86_64-1 
libjpeg-turbo-1.5.3-x86_64-1 
curl-7.64.0-x86_64-1 
libSDL-1.2.15-x86_64-1 
libxcb-1.13.1-x86_64-1 		# ?? might not be required ??
xcb-util-0.4.0-x86_64-1 
xcb-util-image-0.4.0-x86_64-1 
xcb-util-keysyms-0.4.0-x86_64-1 
xcb-util-renderutil-0.3.9-x86_64-1 
xcb-util-wm-0.4.1-x86_64-1 
libidn2-2.0.4-x86_64-1 
librtmp-2018.05-x86_64-1 
libXau-1.0.8-x86_64-1 
libXdmcp-1.1.2-x86_64-1 
harfbuzz-1.7.6-x86_64-1 
libunistring-0.9.8-x86_64-1 
libgnutls-3.6.2-x86_64-1 
libnettle-3.4-x86_64-1 
libgmp-6.1.2-x86_64-1 
libgraphite2-1.3.11-x86_64-1 
libp11kit-0.23.9-x86_64-1 
libtasn1-4.13-x86_64-1 
libpcre-8.42-x86_64-1 
libffi-3.2.1-x86_64-1 
tslib-1.16-x86_64-1 
# starts to black screen with redirected error message indicating ...
# Message translations failed to load
# Unable to initialise the font system

[rufwoof]

Built the standard model, but then removed the fd64.sfs from initrd and also extracted the kernel modules sfs and reformed it with no compression (-noI -noD -noF -noX mksquashfs parameters)

Then extreme compressed that initrd (xz -e --check=crc32), which yielded a 49MB initrd.xz alongside a 6MB vmlinuz

Set grub4dos to boot that, to initrd cli (lateshell boot parameter) together with wireless net connection

Code: Select all

title Fatdog8
root (hd0,0)
kernel /fatdog-build/vmlinuz net=wpa2:VM123456-2G:abcd1234:wlan0:dhcp pkeys=uk lateshell
initrd /fatdog-build/initrd.xz

Booted fine and from the initrd's cli I could ssh into other boxes, mount drives ..etc. First thing I tend to do when booted is ...
cd /lib/boot/consolefonts
setfont ./big.psf.gz
... as that's a more comfortable font size for me compared to the default tiny text.

Downloaded a fd64.sfs into the / folder of that initrd cli session and typed 'exit' to exit the initrd cli and resume bootup and it booted to the standard Fatdog gui desktop.

The small size of that setup (6MB vmlinuz, 49MB initrd.xz) fits better with PXE booting (slow tftp data transfer). PXE booting does require a hardwired LAN (ethernet) so if your laptop has no ethernet connection you'd have to boot that vmlinuz/initrd.xz using usb or CD/DVD, but once booted to initrd cli with net connected you can pretty much pull down the larger fd64.sfs (main system) from pretty much anywhere. A reasonable choice for centralised administration of the main OS (multiple users sharing the same fd64.sfs build).

Note that it was better (smaller initrd.xz filesize) to do it that way ... non compressed kernel modules sfs file inside initrd and compress initrd - than it was to use high compression for the kernel modules and gzip the initrd (that yielded a 66MB initrd.gz filesize). Saved around 17MB (combined vmlinuz + initrd of 55MB versus 72MB).

Of course you don't have to use the fd64.sfs as the main system choice, could be any main sfs, from anywhere, provided it was compatible with the vmlinuz/initrd.

[jamesbond]

Yup, how do you like the flexibility?

[belham2]

Yup, how do you like the flexibility?

LOL

Yup, I tell my lifelong-one-mouseclick-Microsoft-using friends that if they ever have hesitations about coming to the puppy/ddog/fatdog-verse, to just come to 'mother murga' & read rufwoof's posts across various threads to see how easy it all is.

Unfortunately I never hear from them again......

[rufwoof]

Yup, I tell my lifelong-one-mouseclick-Microsoft-using friends that if they ever have hesitations about coming to the puppy/ddog/fatdog-verse, to just come to 'mother murga' & read rufwoof's posts across various threads to see how easy it all is.

Unfortunately I never hear from them again......

Don't blame me for having no friends.

[rufwoof]

Yup, how do you like the flexibility?