Alternative way to build Ubuntu / Debian Puppy [RETIRED]
kernel
What is the reason for not having all modules built-in to the kernel? The kernel should have better performance too.jamesbond wrote:For this to work, the kernel must be compiled with enough kernel modules to be able to find ZDRV and load it. It means loopback, aufs, squashfs, filesystem drivers, usb drivers (if we want to boot from usb) *must* be compiled to the kernel, not as a module. :
- Iguleder
- Posts: 2026
- Joined: Tue 11 Aug 2009, 09:36
- Location: Israel, somewhere in the beautiful desert
- Contact:
Re: kernel
They're huge - a kernel package is ~50 MB with the modules. This means, the distro RAM usage starts at the uncompressed size of the kernel package.stemsee wrote:What is the reason for not having all modules built-in to the kernel? The kernel should have better performance too.jamesbond wrote:For this to work, the kernel must be compiled with enough kernel modules to be able to find ZDRV and load it. It means loopback, aufs, squashfs, filesystem drivers, usb drivers (if we want to boot from usb) *must* be compiled to the kernel, not as a module. :
Having all the modules in the main SFS doesn't have any downsides. A kernel with all the modules jamesbond specified should be 2-3 MB.
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]
Just generated the woof-next branch in woof-CE.
Basically moved all woof building scripts to woof-code/woof2-scripts_and_files and changed merge2out not to copy over all the distro-package-...
It does not have jamesbond's changes so it can not build anything right now, but does have jamesbond as a woof-CE member
Basically moved all woof building scripts to woof-code/woof2-scripts_and_files and changed merge2out not to copy over all the distro-package-...
It does not have jamesbond's changes so it can not build anything right now, but does have jamesbond as a woof-CE member
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==
Soon I'll work on modifying kernel kit to package the modules in an sfs (zdrv) so we can make use of the simple fatdog style initrd. Anyone think of anything else to be into the kernel image? So far we have squashfs, aufs, ext2,3,4, .....other fs? Really "other fs" can be up to the builder; eg: if one wants to try booting from hfs+ then that must be a builtin. Same for reiser, btrfs, f2s etc.
Puppy Linux Blog - contact me for access
VFAT for sure.01micko wrote:Soon I'll work on modifying kernel kit to package the modules in an sfs (zdrv) so we can make use of the simple fatdog style initrd. Anyone think of anything else to be into the kernel image? So far we have squashfs, aufs, ext2,3,4, .....other fs? Really "other fs" can be up to the builder; eg: if one wants to try booting from hfs+ then that must be a builtin. Same for reiser, btrfs, f2s etc.
f2fs if possible too.
What about booting from NTFS volumes?
Although is sacrilegious to run linux in a Mac ( ) there is a bunch of 32bit intel core duo (not core2duo) Macs that are stuck with OSX 10.6 and not supported anymore. So go for it.
I guess the other question is SCSI/IDE/SATA etc.
Another option would be to have the entire kernel in the initrd so you do not really need to wary about what is where. XOpup does that but it has really few modules.
Should not really affect boot times since this file will be loaded from somewhere and will likely decrease the kernel memory since a bunch of unused modules will not be in vmlinuz.
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==
- Iguleder
- Posts: 2026
- Joined: Tue 11 Aug 2009, 09:36
- Location: Israel, somewhere in the beautiful desert
- Contact:
All we need is:
- Drivers for USB mass storage devices
- All hard drive drivers (they're tiny)
- SCSI CDROM drivers
- Drivers for all file systems we want to be able to boot from - ext2, ext3, ext4, FAT16/32, NTFS (?), XFS, reiserfs (for unlucky users who installed an old distro and kept the file system when they moved on to Puppy) and btrfs ... isn't f2fs too young?
- Squashfs and Aufs
This is the classic Puppy recipe - it should produce a 2-3 MB kernel.
I think sticking to a LTS kernel is the way to go, because pretty often, Aufs gets messed up with the latest and greatest (segfaults, in the kernel ). I've been using 3.10.x since it received its LTS status and it's simply awesome.
EDIT: 01micko - here's my kernel building script for 3.10. It's very Puppy-like, except the fact it's for Linux-libre and doesn't have Aufs. It took some time and additional gray hair to get UEFI to work (i.e, it boots but you get a black screen because it doesn't have the UEFI framebuffer module ), so it's a good reference configuration.
EDIT 2: I have a static toybox and many static daemons I wrote on my own (i.e syslogd and klogd). Functionality-wise, they should be compatible with the BusyBox ones. Maybe we could use them in the initrd instead of BusyBox. They're much lighter and smaller.
- Drivers for USB mass storage devices
- All hard drive drivers (they're tiny)
- SCSI CDROM drivers
- Drivers for all file systems we want to be able to boot from - ext2, ext3, ext4, FAT16/32, NTFS (?), XFS, reiserfs (for unlucky users who installed an old distro and kept the file system when they moved on to Puppy) and btrfs ... isn't f2fs too young?
- Squashfs and Aufs
This is the classic Puppy recipe - it should produce a 2-3 MB kernel.
I think sticking to a LTS kernel is the way to go, because pretty often, Aufs gets messed up with the latest and greatest (segfaults, in the kernel ). I've been using 3.10.x since it received its LTS status and it's simply awesome.
EDIT: 01micko - here's my kernel building script for 3.10. It's very Puppy-like, except the fact it's for Linux-libre and doesn't have Aufs. It took some time and additional gray hair to get UEFI to work (i.e, it boots but you get a black screen because it doesn't have the UEFI framebuffer module ), so it's a good reference configuration.
EDIT 2: I have a static toybox and many static daemons I wrote on my own (i.e syslogd and klogd). Functionality-wise, they should be compatible with the BusyBox ones. Maybe we could use them in the initrd instead of BusyBox. They're much lighter and smaller.
[url=http://dimakrasner.com/]My homepage[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]
[url=https://github.com/dimkr]My GitHub profile[/url]
Thanks Iguleder - using your DOTconfig with 3.12.21 with aufs added. Just raw, as in I haven't checked if new modules are available. Compiling now.
I'll add James' firmware to a kernel-modules.sfs and see if the thing boots - more soon.
EDIT: BTW 3.12 is LTS too and what slacko64 was using before the 3.13 experimental release.
EDIT: failed to boot, initramfs problem, anyway rebuilding using fatdog-630-rc2 config ..which is what i had on hand ..should "just work"
EDIT: FD config booted but no keyboard at console for me. I honestly don't know if it worked in fatdog itself at console on this machine so I'll have to check. (LATER: checked .. yes it did work in FD-630-rc2 k3.12.6)
NEXT Morning
EDIT: depmod call in /etc/rc.d/rc.sysinit fails so that's why kernel modules didn't load. I ran depmod on the kernel modules before squashing and now it works . I should be able to mod kernel kit appropriately to produce a fatdog style kernel.
Ignore the $DISTRO_TARGETARCH.. uname certainly wants us to know that this is x86_64, 3 times
I'll add James' firmware to a kernel-modules.sfs and see if the thing boots - more soon.
EDIT: BTW 3.12 is LTS too and what slacko64 was using before the 3.13 experimental release.
EDIT: failed to boot, initramfs problem, anyway rebuilding using fatdog-630-rc2 config ..which is what i had on hand ..should "just work"
EDIT: FD config booted but no keyboard at console for me. I honestly don't know if it worked in fatdog itself at console on this machine so I'll have to check. (LATER: checked .. yes it did work in FD-630-rc2 k3.12.6)
NEXT Morning
EDIT: depmod call in /etc/rc.d/rc.sysinit fails so that's why kernel modules didn't load. I ran depmod on the kernel modules before squashing and now it works . I should be able to mod kernel kit appropriately to produce a fatdog style kernel.
Code: Select all
Linux puppypc29976 3.12.21 #1 SMP Sun Jun 8 07:25:04 EST 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/DISTRO_SPECS
DISTRO_NAME='ubuntu Puppy'
DISTRO_VERSION=7.0
DISTRO_BINARY_COMPAT='ubuntu'
DISTRO_FILE_PREFIX='ubuntu'
DISTRO_COMPAT_VERSION='ubuntu'
DISTRO_XORG_AUTO='yes'
DISTRO_TARGETARCH='x86'
DISTRO_DB_SUBNAME='ubuntu'
DISTRO_PUPPYSFS=puppy.sfs
DISTRO_ZDRVSFS=kernel-modules.sfs
Puppy Linux Blog - contact me for access
I just did a fresh install of slacko-5.7 to test out kernel-kit modifications on 32 bit and the Ubuntu install using tahr bins. I have just pulled james' latest so it will have the mods he put in (so far I haven't updated since initial pull).
If all goes well I'll post kernel kit here for testing and if that goes well I'll push it to woof-CE/testing. Kernel is compiling now. (again 3.12.21). Kernel kit will have both configs. BTW, it's 4G not PAE so I can test on my old centrino lap top. That can be easy changed in the config for those who like PAE, and beauty of the FatDog kernel method is simplicity in switching kernels. Just replace vmlinuz and kernel-modules.sfs and reboot.
Mavrothal.. it might actually be time soon for a new git repo.. woof-NG
If all goes well I'll post kernel kit here for testing and if that goes well I'll push it to woof-CE/testing. Kernel is compiling now. (again 3.12.21). Kernel kit will have both configs. BTW, it's 4G not PAE so I can test on my old centrino lap top. That can be easy changed in the config for those who like PAE, and beauty of the FatDog kernel method is simplicity in switching kernels. Just replace vmlinuz and kernel-modules.sfs and reboot.
Mavrothal.. it might actually be time soon for a new git repo.. woof-NG
Puppy Linux Blog - contact me for access
Progress ! !
This new kernel, initrd and kernel-modules.sfs combination, has to be the most significant breakthrough in Puppy Development for ages. The time I have wasted fiddling around with DISTRO-SPECS in main.sfs and initrd and swapping modules around and separating modules for initrd, and/or compressing/decompressing *.ko.gz mods, and all the other stuff was utterly unnecessary! And this method will, hopefully, prove it!
Kudos to you. Anything I can do let me know!
Edit: Any chance of ensuring all puppys build with the ability to 'xinput create-master second' , only Lighthouse64 has that ability and after research and experimentation I can shed no light no the issue. It is the one thing which gives superiority to Debian based systems in my opinion.
Kudos to you. Anything I can do let me know!
Edit: Any chance of ensuring all puppys build with the ability to 'xinput create-master second' , only Lighthouse64 has that ability and after research and experimentation I can shed no light no the issue. It is the one thing which gives superiority to Debian based systems in my opinion.
Ok, 32 bit kernel is a success and posting from it now, just added dhcpcd to the default, hacked build-iso.sh to grab the kernel from my personal repo and added firefox and we are away.
More later.. dinner time!
Code: Select all
# uname -a
Linux puppypc27288 3.12.21 #1 SMP Sun Jun 8 15:24:18 EST 2014 i686 athlon i686 GNU/Linux
Puppy Linux Blog - contact me for access
Time for some code
Diff to build iso (grabs kernel)
NB: The third hunk of the diff will not be necessary for FatDog (isohybrid part)
Apply that and run with KERNEL_URL as an arg (will work as I have uploaded that kernel)
I'll attach kernel-kit adjusted to build a fatdog64 (or 32) bit kernel.
Diff to build iso (grabs kernel)
Code: Select all
--- build-iso.sh.orig 2014-06-03 20:54:06.660580810 +1000
+++ build-iso.sh 2014-06-08 18:02:47.560002531 +1000
@@ -10,6 +10,7 @@ ISO_ROOT=${ISO_ROOT:-$OUTPUT_DIR/iso-roo
PUPPY_SFS=${PUPPY_SFS:-puppy.sfs}
KERNEL_VERSION=${KERNEL_VERSION:-3.12.9}
PARENT_DISTRO=${PARENT_DISTRO:-ubuntu} # or debian
+KERNEL_URL=${KERNEL_URL:-http://distro.ibiblio.org/fatdog/kernels/700}
WOOF_ISO_ROOT=${WOOF_ISO_ROOT:-boot}
WOOF_INITRD=${WOOF_INITRD:-boot/initrd-tree0}
@@ -29,7 +30,7 @@ install_boot_files() {
install_kernel() {
for p in vmlinuz kernel-modules.sfs; do
! [ -e $ISO_ROOT/$p ] &&
- wget -P $ISO_ROOT -c http://distro.ibiblio.org/fatdog/kernels/700/$p-$KERNEL_VERSION
+ wget -P $ISO_ROOT -c $KERNEL_URL/$p-$KERNEL_VERSION
mv $ISO_ROOT/$p-$KERNEL_VERSION $ISO_ROOT/$p
done
}
@@ -58,7 +59,7 @@ make_iso() {
-volid "Puppy-Linux" \
-iso-level 4 -D -R \
-b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table $ISO_ROOT/
- isohybrid -o 64 "$OUTPUT_DIR/$OUTPUT_ISO"
+ isohybrid "$OUTPUT_DIR/$OUTPUT_ISO" #-o 64
}
### main
#
Apply that and run with KERNEL_URL as an arg (will work as I have uploaded that kernel)
Code: Select all
KERNEL_VERSION=3.12.21-slacko32FD4G KERNEL_URL=http://01micko.com/packages/ ./build-iso.sh
- Attachments
-
- kernel-kit-ng-0.2.tar.gz
- (158.43 KiB) Downloaded 372 times
Puppy Linux Blog - contact me for access
next kernel-kit
Question on kernel-kit-ng01micko wrote:I'll attach kernel-kit adjusted to build a fatdog64 (or 32) bit kernel.
''# DO NOT set it to 0 (zero) >> cooked machine
JOBS=-j6''
Having -J0 results in cooked cpu, dual core cpu has 4 threads (-j4) kernel-kitt is set default to ''JOBS=-j6'' will that not cook a machine also unless it has 3 or more cores?
Also, ../dist/sources/patches directory missing/not created only a patches file, hence no patches to apply ... copied manually. Maybe that is how it is supposed to be ... what do I know??
Kernel compiled using new kit = 3.14.5-32FD-pae and sfs i don't have the new initrd which I imagine needs an amended init pointing to the modules.sfs so I haven't tested it. Also the DOTconfig has some significant changes, for example deselected RTC, and included additional fs, i486. Also, added firmware. The kernel gets dumped in dist/packages/ outside of the linux_kernel-*/boot directory.
kernel
https://drive.google.com/file/d/0B4GhZV ... edit?usp=1
modules.sfs
https://drive.google.com/file/d/0B4GhZV ... edit?usp=1
Looks good.01micko wrote: I'll attach kernel-kit adjusted to build a fatdog64 (or 32) bit kernel.
added dm_crypt in vmlinuz just in case
Aufs mm/fremap.c patching failed in 3.12
Also in line 396 the sfs sufix is in the middle
Another little issue is that will not empty or stop and warn idf the output directories are not empty so if you build 2 karnels (different conf or $FD) may end up in a mix up.
== [url=http://www.catb.org/esr/faqs/smart-questions.html]Here is how to solve your[/url] [url=https://www.chiark.greenend.org.uk/~sgtatham/bugs.html]Linux problems fast[/url] ==
@stemsee
-
@mavrothal
Thanks for reports.
Well, I don't think jobs are the same as threads.. I should have commented it however I have been using -j6 forever on my athlon X2. Cooked machine is a possibility but more likely is the thermal cut out protection.''# DO NOT set it to 0 (zero) >> cooked machine
JOBS=-j6''
Having -J0 results in cooked cpu, dual core cpu has 4 threads (-j4) kernel-kitt is set default to ''JOBS=-j6'' will that not cook a machine also unless it has 3 or more cores?
There is intentionally only 1 patch in patches. Others are generated in the script and archived in dist/ for future reference.Also, ../dist/sources/patches directory missing/not created only a patches file, hence no patches to apply ... copied manually. Maybe that is how it is supposed to be ... what do I know??
There are 2 ways you can test a kernel resulting from the FD build.Kernel compiled using new kit = 3.14.5-32FD-pae and sfs i don't have the new initrd which I imagine needs an amended init pointing to the modules.sfs so I haven't tested it. Also the DOTconfig has some significant changes, for example deselected RTC, and included additional fs, i486. Also, added firmware. The kernel gets dumped in dist/packages/ outside of the linux_kernel-*/boot directory.
- 1. build a distro using jamesbond's scripts
2. build a 64 bit kernel and replace the FatDog64 kernel
-
@mavrothal
Yes I am aware. This needs to be reported to Junjiro and that's why I changed from "exit 1" to a pause and inspect routine.Aufs mm/fremap.c patching failed in 3.12
Intentional to be compatible with jamesbond's build-iso.sh. As stated up the page, the diff needs to be applied so you can use the KERNEL_URL as a param. The suffix gets stripped off in the script; vmlinuz gets the same treatment.Also in line 396 the sfs sufix is in the middle
Yes, I suppose I should add a "clean" param to the script and a note in README. I don't want to automate that, so much hand holding is ok but it's the kernel we are talking about here and if a user doesn't know what they are doing then they shouldn't be using the scripts. It doesn't check for space eitherAnother little issue is that will not empty or stop and warn idf the output directories are not empty so if you build 2 karnels (different conf or $FD) may end up in a mix up.
Thanks for reports.
Puppy Linux Blog - contact me for access
great
I built a debian with jamesbond's scripts 'build-deb'. However I realise that i misunderstood kernel-kit-ng... just have to recompile kernel and then insert for ./3builddistro
So to clarify. The kernel does not need to be in the ../dist/linux_kernel-*/boot directory nor does that directory need to dir2tgz-ed and tgz2pet -ed and added to local-repositories for inclusion in the build ?
So to clarify. The kernel does not need to be in the ../dist/linux_kernel-*/boot directory nor does that directory need to dir2tgz-ed and tgz2pet -ed and added to local-repositories for inclusion in the build ?
James' script doesn't touch 3builddistro. I believe he only used the woof-CE framework so that the generic puppy scripts (including the initrd) could be imported to the build. After deb-build.sh run build-iso.sh. This pulls in the kernel from the fatdog ibiblio repo. My diff just adds an option to pull in a kernel from elsewhere and I uploaded mine to my repo to test.
To use your kernel, just comment the install_kernel line (near the bottom), add a read command directly under it, which effectively pauses the script so that you can insert your kernel (into iso/iso-root/) , continue the script and an iso will be built with your kernel. You could be more elegant than my hacky method if you want to spend the time to add the code pulling in a kernel from a local directory.
To use your kernel, just comment the install_kernel line (near the bottom), add a read command directly under it, which effectively pauses the script so that you can insert your kernel (into iso/iso-root/) , continue the script and an iso will be built with your kernel. You could be more elegant than my hacky method if you want to spend the time to add the code pulling in a kernel from a local directory.
Puppy Linux Blog - contact me for access