Cutting edge kernel for 2.17

Under development: PCMCIA, wireless, etc.
Message
Author
User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

#21 Post by Dougal »

kirk wrote: The Xine FAQ page recommends 1000hz.
Yes, they would, obviously... as would any multimedia related projects.

What I'm trying to think of if the best balance that would fit the general-purpose Puppy, not a multimedia oriented one (or new-HW oriented one)...

I think the point (made in the last comment in that post I linked) about HW components not being tested beyond 100HZ might be worth keeping in mind... especially when we're using old HW.
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

kirk
Posts: 1553
Joined: Fri 11 Nov 2005, 19:04
Location: florida

#22 Post by kirk »

Yes 1000hz may be too high and may cause noise on some hardware, or maybe the tickless option takes care of that. Don't know. A safe bet would probably be the new 300hz option.

User avatar
Dougal
Posts: 2502
Joined: Wed 19 Oct 2005, 13:06
Location: Hell more grotesque than any medieval woodcut

#23 Post by Dougal »

I went to the powerTop site a few days ago and noticed somehting that's actually shown in their example:

Code: Select all

CONFIG_USB_SUSPEND=y
What's the ugliest part of your body?
Some say your nose
Some say your toes
But I think it's your mind

User avatar
Sit Heel Speak
Posts: 2595
Joined: Fri 31 Mar 2006, 03:22
Location: downwind

#24 Post by Sit Heel Speak »

tempestuous, please enlighten me. Why will this patched kernel work only on a full hard drive install?

It would be interesting to compare the Ingo Molnar scheduler-patched (mainline kernel) Puppy, against the Con Kolivas scheduler-patched (patches recently rejected from mainline, in favor of Ingo Molnar's).

There was a big flame war over the decision to put Mr. Molnar's scheduler in mainline, with Linus Torvalds coming down in favor of Mr. Molnar. Mr. Kolivas then announced he'd had it with kernel programming, he was totally burnt out -- and that the 2.6.22 set of his "-ck" scheduler patches will be the last, ever, from him.

Mr. Kolivas's patch-set has quite an ardent following, over on the Gentoo discussion forum. It is said to be far superior to anything else for multimedia e.g. mplayer.

Incidentally, I've emailed Mr. Kolivas, to suggest that he take a look at Puppy.
Last edited by Sit Heel Speak on Sat 28 Jul 2007, 00:35, edited 1 time in total.

Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

#25 Post by Leachim »

I think the only problem with a non-hdd-install is, that you also have to rebuild the initrd with the appropriate new kernel modules.

I did this for the "cutting-edge-2.17-kernel" (2.6.21.5) and use it everyday under Puppy 2.15CE. As a coincidence I plan to apply Con Kolivas' patches tomorrow - maybe to a 2.6.22-kernel - let's see!

When I get results, I will post them here!

User avatar
Sit Heel Speak
Posts: 2595
Joined: Fri 31 Mar 2006, 03:22
Location: downwind

#26 Post by Sit Heel Speak »

Leachim wrote:I think the only problem with a non-hdd-install is, that you also have to rebuild the initrd with the appropriate new kernel modules.
Ah. I see. Of course. So then, you hand-copied all your necessary kernel modules over?

It is only within the past week that I've managed to get Gentoo up and running. With its super-powerful "Portage" package management system, this should make the job of converting to a -ck patched kernel and creating the necessary new versions of the loadable kernel modules, a snap. The most difficult part will be to, if I guess your meaning correctly, (laboriously, manually) copy over all the new auto-generated-by-Gentoo versions of the kernel modules, to my Puppy initrd sourcedir.

As an aside, I must comment, Gentoo certainly is powerful--but, to a noob like me, rather daunting to initially get going. Going from Puppy to Gentoo is like stepping from an Acura into the cockpit of a fighter jet--except, you have to build the jet yourself first. I could not have done it had I not first had a year of Linux practice with Puppy. It took me four days to figure out how to get something more than a vanilla Gentoo onto my disk and make it boot to a command line. Another day to figure out how to make my old Rage Pro 128 video card correctly configure in xorg.conf so that X would start (Gentoo, unlike Puppy, takes nothing for granted). And still another day to get a window manager (fluxbox) to correctly load. And still another day to get icewm up and running.

It is easy to understand why a recently-banned forum member, who advocated Gentoo, could not help himself but exhibit a superior and condescending attitude toward us lesser mortals. I hope that, now that I have Gentoo up and running, I won't be afflicted with the curse of arrogance. For example, I can see where, with Gentoo, it should be possible to compile from scratch a complete distro on my Pentium MMX-233 machine with 32MB of RAM, but with the actual heavy lifting (i.e. the compiling) done by the P3 and P4 machines on my home network. Theoretically, this should take only about eight hours or less, possibly very much less. The built package could then be copied over into the Puppy subdir layout. This would result in a 32MB-capable (if a full hd install) Puppy with everything up-to-date and all the dependencies perfectly resolved. Slow yes, but it would run.
Leachim wrote:I did this for the "cutting-edge-2.17-kernel" (2.6.21.5) and use it everyday under Puppy 2.15CE. As a coincidence I plan to apply Con Kolivas' patches tomorrow - maybe to a 2.6.22-kernel - let's see!

When I get results, I will post them here!
The -ck patch set is said to do wonders for mplayer. Especially on an SMP system. If I glean correctly, mplayer is the only popular single-threaded multimedia player out there, and it kicks ass under -ck.

I look forward to your results! If I get the time this weekend, I too will try to create a Puppy 2.17 with the -ck patch set.

And if that works, I may try to create a Puppy where the internal filesystem of the .sfs files is reiserfs4!

(As Napoleon and Hitler both said...on to Moscow!)

User avatar
Sit Heel Speak
Posts: 2595
Joined: Fri 31 Mar 2006, 03:22
Location: downwind

#27 Post by Sit Heel Speak »

Ach. It just dawned on me:

When talking about "famous present-day high-performance patches by Ingo Molnar," there are two of them.

One is his "real time patch," which lets hardware interrupts grab non-interruptible priority.

The second is his "completely fair scheduler (CFS) patch." Actually the CFS will be not a patch, but rather part of the mainstream Linux kernel, beginning with 2.6.23.

The "real time patch" may be dangerous with a root-only distro like Puppy. Any program which grabs real-time priority and then goes into an endless loop will lock your machine--big red switch time. But, since Puppy is easily recoverable, so what? Might as well give it a go.

The big yelling-match between the followers of Molnar and the followers of Kolivas is essentially an argument over the relative merits of Mr. Molnar's Completely Fair Scheduler (CFS) versus Mr. Kolivas's Staircase Deadline (SD) scheduler. As far as I can tell, the real-time patch is a separate animal. So the actual question when deciding what is the "ultimate ferocious Puppy kernel" is, which is faster?...between

1. Con Kolivas's Staircase Deadline Scheduler + Ingo Molnar's RT patch

and

2. Ingo Molnar's Completely Fair Scheduler + Ingo Molnar's RT patch

This is the real heavyweight-champion-of-the-world match.

I've set up hard disk installs of 2.17 on three separate partitions of the same disk...will keep one normal, make the second CFS+RT, and the third SD+RT...and test them against each other. In the morning.

Can anyone suggest some decisive benchmarks?

tempestuous
Posts: 5464
Joined: Fri 10 Jun 2005, 05:12
Location: Australia

#28 Post by tempestuous »

There is an interesting interview with Con Kolivas about why he quit -
http://apcmag.com/6762/interview_with_c ... x_performa
Yes, Ingo Molnar's CFS patch is the officially accepted CPU scheduling enhancement, but in theory it's supposed to be based on Con's patches.
Sit Heel Speak wrote:I've set up hard disk installs of 2.17 on three separate partitions of the same disk...will keep one normal, make the second CFS+RT, and the third SD+RT...
From my experience, I think you will have difficulty applying any two of these patches in combination.
Sit Heel Speak wrote:The -ck patch set is said to do wonders for mplayer. Especially on an SMP system.
MPlayer's performance is already good on my old P2-350 with old 16MB AGP graphics card. It plays DVD's faultlessly.

But if I need to play blu-ray DVD's or high-definition digital TV in the future, I will have to upgrade. No amount of kernel tweaking is going to make my P2-350 able to decode 25fps of MPEG2 video at 1920x1080 pixels.

Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

#29 Post by Leachim »

Sit Heel Speak wrote:
Leachim wrote:I think the only problem with a non-hdd-install is, that you also have to rebuild the initrd with the appropriate new kernel modules.
Ah. I see. Of course. So then, you hand-copied all your necessary kernel modules over?
No, I'm much to lazy, doing such things by hand ... ^^

I wrote myself a script that updated all existing kernel modules (you will have to replace the paths with your initrd/kernel-directories):

Code: Select all

#!/bin/bash

KERNEL_VERSION=2.6.21.5

set -e

function update_gz_module()
  {
  while read ABSNAME
  do
    FILENAME=$(basename $ABSNAME)
    FILENAME=${FILENAME%.gz}
    SRCNAME=$(find lib/modules/$KERNEL_VERSION -name "$FILENAME")
    echo updating $ABSNAME from $SRCNAME
    ABSNAME=${ABSNAME%.gz}
    cp $SRCNAME $ABSNAME
    rm $ABSNAME.gz
    gzip -9 $ABSNAME
  done
  }

function update_module()
  {
  while read ABSNAME
  do
    FILENAME=$(basename $ABSNAME)
    SRCNAME=$(find lib/modules/$KERNEL_VERSION -name "$FILENAME")
    echo updating $ABSNAME from $SRCNAME
    cp $SRCNAME $ABSNAME
  done
  }

find mnt/lib/modules/$KERNEL_VERSION -name "*.ko.gz" | update_gz_module
find mnt/lib/modules/$KERNEL_VERSION -name "*.ko" | update_module

User avatar
Sit Heel Speak
Posts: 2595
Joined: Fri 31 Mar 2006, 03:22
Location: downwind

#30 Post by Sit Heel Speak »

A hypothetical question:

Supposing...that I can't get the patches to work under Puppy. But, I can get them to work under Gentoo.

To make a Puppy out of it is not a matter of a simple drop-in replacement for vmlinuz plus the kernel modules. I must construct a complete Puppy .iso variant starting from a working Gentoo installation.

I have zero experience whatsoever with scripting. And so I must ask, how feasible would it be to do this:

Write a script. It will run in the following environment:

--Puppy 2.17 is booted and running from a USB flash key sda1. The script will be started from a terminal window.

--a full hard disk install of Puppy 2.17, as created by Puppy Universal Installer, unmodified, is sitting on hda2.

--on hdb2 is a Gentoo install as similar as I could make it, in which I have compiled, under Gentoo (to accommodate SD+RT), every single app present (except for the Puppy-specific ones, e.g. PUI, PFind, PMount) in Puppy 2.17 (but, compiled from source code versions carefully chosen with the enormous help of Gentoo's package management system). In other words, it's a Gentoo-ized duplicate of Puppy 2.17. You may consider it to have the identical directory structure as Puppy 2.17 (I am savvy enough to adjust for discrepancies).

What I want, is a script which will inspect every file and symlink of the (inactive-at-the-moment) full hard disk (option 2) Puppy 2.17 install on hda2.

For each file and symlink, it will look over on the Gentoo install on hdb2 and, if a matching file or symlink is present, copy it over onto a freshly-formatted new partition, hdd2. In short, I'm building the new FrankenPuppy on hdd2.

Then, it will copy over all the Puppy init scripts and config files and Puppy-specific apps (e.g. PMount, PFind, PUI etc.) from hda2 not to hdd2 but rather to hdd3. This way I will be able to go through them one-by-one and check that they will all point and work correctly, before I hand-copy them to the new FrankenPuppy-.iso-source-full-hard-drive-install-on-hdd2.

You can leave all X11R7 subdirs and /etc/X11/xorg.conf alone, i.e. do not copy these over from Gentoo-hdb2.

Yes, I realize it will have to be hand-tweaked afterward. But, I think I can do it.

Is such a script feasible?
Last edited by Sit Heel Speak on Sun 29 Jul 2007, 18:53, edited 1 time in total.

Leachim
Posts: 229
Joined: Sun 27 May 2007, 23:04

#31 Post by Leachim »

You should not forget to copy the new kernel-include-files and all kernel modules to the new system. (The initrd contains just a small subset of the kernel modules.)

I do have a dual boot with the old and the new kernel - both using the same save-file. So I always had a fall-back if things went a bit unexpected ...

I made all installations (including building the new kernel) from my USB-stick-based system. I must admit: I never did a hdd-install of Puppy!

There is also no need for iso-images. I simply copy all needed files to the root directory of my usb-stick.

User avatar
Sit Heel Speak
Posts: 2595
Joined: Fri 31 Mar 2006, 03:22
Location: downwind

#32 Post by Sit Heel Speak »

All righty. I figured it out and am now a "blooded" Puppy kernel custom-compiler. I noticed that the rt patch put the modules not in 2.6.21.5 but rather in 2.6.21.5-rt20 so I copied them over.

When I apply the cfs patch before rt, the cfs patch does complete OK, but if I subsequently apply the rt patch, then the patch-rt program throws many errors, and I don't know C so can't understand them. So I'll draw in my horns and forego experimentation with cfs for now, until Puppy acquires a 2.6.23 kernel. btw I was mistaken about cfq: what cfs does is patch cfq to improve it. So, they *are* the same, sorta.

The Con Kolivas scheduler patch also makes the rt patch throw errors.

So what I've done is, one partition is an ordinary P217 build, the second is an rt-patched build like tempestuous's (output of ps ax shows the softirq processes and hardware irq's), and the 3rd is a Con Kolivas deadline scheduler build but not rt-patched.

At 1000Hz in both the latter builds Puppy is real snappy. I'll try them both over the next few weeks and see if one is better than the other. Whew. I feel like I ought to advertise myself for hire. I suspect if I really wanted to learn C I could make cfs and ck work with rt. Perhaps later, when my eyeballs stop burning.

John Doe
Posts: 1681
Joined: Mon 01 Aug 2005, 04:46
Location: Michigan, US

#33 Post by John Doe »

Timer Frequency Tests

Method:
I took barry's patched source and compile with tempestuous's july 13 config (*see "Config Notes"). No additional patches (realtime, scheduling or suspend...etc).

Then I recompiled with this contol and only changed the timer frequency.

Results:
Boot times on 75mhz are as follows:
100mhz 1:57
250mhz 1:58
300mhz 2:26
1000mhz 2:52

Conclusion 1
I'm concluding that the other patches make it worse for old hardware, based on my other testing of the experimental kernel that was in this thread.

Conclusion 2
Hyperthreading and dual processor support seem to have no ill effects on my oldest available hardware and bring out the best in the new.

*Config Notes

Using Leachim's script I identified serveral config options that where included in barry's config and not the july 13 version.

I added them and they were for the following modules:

istallion.ko
riscom8.ko
stallion.ko
i2c-elektor.ko
irport.ko
ni50l0.ko
xircom_tulip_cb.ko

screenshots below to show before and after with hyperthreading and an after shot of the 75 going for fun.
Attachments
75mhz-newConfig.png
(47.64 KiB) Downloaded 738 times
3ghz-newConfig-small.png
(79.3 KiB) Downloaded 739 times
3ghz-oldConfig-small.png
(90.35 KiB) Downloaded 724 times
DOTconfig-k2.6.21.5-cuttingedge02AUG07.gz
(17.58 KiB) Downloaded 304 times

User avatar
Sit Heel Speak
Posts: 2595
Joined: Fri 31 Mar 2006, 03:22
Location: downwind

#34 Post by Sit Heel Speak »

1. What is the system info program we are seeing?

2. Has anyone else besides me, had trouble unpacking .zip archives when using a realtime-patched kernel (no problem though with ordinary 2.17, and no problem with ck-patched)?

3. At present, I'm way beyond the upper limit of my "Peter Principle competence" here...however...my newbie question is...

I wonder if the loadable kernel modules are adjusted by the choice of kernel patches + timer tick + scheduler et cetera at kernel-compile time.

I wonder if any other (e.g. header, library) files get likewise adjusted during the kernel compilation.

And consequently, I wonder to what extent all applications link into the kernel modules and/or specially-adjusted headers et al at their own compile time...and/or refer during their make to the kernel .config...and therefore require recompilation if the kernel is hacked.

Furthermore I wonder to what extent timings must be adjusted in the init scripts, when using such a kernel.

If app code is indeed so influenced, then results using only a hacked kernel, without simultaneously recompiling the entire distro and/or adjusting the init scripts, are not meaningful.

John Doe
Posts: 1681
Joined: Mon 01 Aug 2005, 04:46
Location: Michigan, US

#35 Post by John Doe »

Sit Heel Speak wrote:1. What is the system info program we are seeing?
Hardinfo, it's this cool app Barry and some others found (i read that it's going in Puppy from now on).

http://www.murga-linux.com/puppy/viewtopic.php?&t=20211
Sit Heel Speak wrote:2. Has anyone else besides me, had trouble unpacking .zip archives when using a realtime-patched kernel (no problem though with ordinary 2.17, and no problem with ck-patched)?
can't address this.
Sit Heel Speak wrote:3. At present, I'm way beyond the upper limit of my "Peter Principle competence" here...
don't worry, me too. i'm just sort of grasping some of this as i go and "standing on the shoulders of giants".
Sit Heel Speak wrote:however...my newbie question is... I wonder if the loadable kernel modules are adjusted by the choice of kernel patches + timer tick + scheduler et cetera at kernel-compile time.
one would really have to review the patch to see the files that are effected. i still can't run patch unless i read directions. the first time i patched something i read the file names and line numbers out of the patch and copied and pasted the C (it was a REALLY short one).
Sit Heel Speak wrote:I wonder if any other (e.g. header, library) files get likewise adjusted during the kernel compilation.
i don't know for certain, but it seems highly unlikely that a kernel patch would change anything but code for the kernel (that could include any headers for the kernel).
Sit Heel Speak wrote:And consequently, I wonder to what extent all applications link into the kernel modules and/or specially-adjusted headers et al at their own compile time...and/or refer during their make to the kernel .config...and therefore require recompilation if the kernel is hacked.
i've run across very few apps that require kernel references (only those that require loadable modules of some type). there are some, but the main things that would need updated in Puppy are a couple modem/usb-modem, wireless and "misc" modules. there are other modules for the kernel that aren't "in" the main kernel source. the afformentioned items are such. they break. for instance in my testing here, my modem is no longer recognized because the module on my machine that was compiled by someone with Puppy 2.17 source and config against that kernel image will not load with the one i recompiled.

apps are different in that they are compiled against gcc which is a runtime for C. i don't think the runtime has any kernel references. other than talking back and forth through nodes.
Sit Heel Speak wrote:Furthermore I wonder to what extent timings must be adjusted in the init scripts, when using such a kernel.
init is a shell script that runs against busy box which runs ontop of gcc. it's two layers out from the kernel and seems to work fine from my tests. :-)
Sit Heel Speak wrote:If app code is indeed so influenced...
i don't think it is.

p.s. @all, if i got anything wrong, please correct me.

John Doe
Posts: 1681
Joined: Mon 01 Aug 2005, 04:46
Location: Michigan, US

#36 Post by John Doe »

John Doe wrote:
Sit Heel Speak wrote:If app code is indeed so influenced...
i don't think it is.
....although it does seem to effect performance "just a bit".

SeaMonkey load times under straight 2.17 and 2.17 with aug 02 config:

old: ~1:56
new: ~2:13

keep in mind I'm scraping the bottom of the barrel for my test environment (now using Full HD install):

http://www.murga-linux.com/puppy/viewtopic.php?t=15946

at the level i'm testing these times are VERY exagerated and probably exponentially lessened with a proper processor and memory.

tempestuous
Posts: 5464
Joined: Fri 10 Jun 2005, 05:12
Location: Australia

#37 Post by tempestuous »

As an endnote to this exercise, I just compiled a cutting-edge kernel for Puppy v3.00 using the Completely Fair Scheduler patch, but I now realise how much MORE work will be required to completely remaster Puppy with this new kernel. There will need to be a lot of third-party modules compiled, including the all-important aufs module. Then the initrd.gz cpio archive will need to be recreated.

Then I realised that the latest available kernel, 2.6.23, already includes the Completely Fair Scheduler enhancements, so my efforts would offer no better features than what will be available with the next major upgrade of the official Puppy.

So I have compiled the kernel and standard modules, and will take the project no further.
If anyone wants to try this new kernel, let me know and I can upload the files.

kirk
Posts: 1553
Joined: Fri 11 Nov 2005, 19:04
Location: florida

#38 Post by kirk »

Makes you appreciate what Barry does. I Built one of the 2.6.23-rc-mm and rebuilt the liveCD with it. Didn't have a free partition. Not much fun, and I didn't build all the extra modules that Barry includes. I just wanted to see if Gxine would look better with 1000hz. Didn't seem to make any difference. The mm series included the latest unionfs2. I think Barry's back with Unionfs now. And I read that the mksquashfs developer is making another push for kernel inclusion. Maybe the next kernel won't need any patches.

pyrael
Posts: 13
Joined: Tue 23 Oct 2007, 22:02

#39 Post by pyrael »

tempestuous,

(of course) after posting in the beginner's section, I came across this thread :P

I'd love to try out that kernel:D I have a pentim4 503j (HT) that I use for folding at home (amongst others) and I'd love to have the thing use both sides in HT mode :P

Just a suggestion though, I have seen other Live distros give options for multiple kernels (32 bit, 32 bit with smp and 64 bit) I admit I have no Idea how involved this would be, but it could be a nice addition to a future release. The distro that pops in my head ATM is the LSF live CD which you can boot on a 64 bit machine by typing linux64 at the welcome screen.

Post Reply