Devel

Slitaz Kernel Performance and Scheduling Options (27 posts)

  • Profile picture of David David said 1 year, 4 months ago:

    Hi Everyone,

    This is my first post so I want to say thank you all for such an awesome distribution! It is fast, compact and customizable — exactly what most linux hackers need. I have found some performance issues that seem to be related to process scheduling for both the redis-benchmark and some internal python testing runs. I get somewhat sporadic performance on both the redis benchmark (generally giving 90Kops/s, 60Kops/s or 30Kops/s depending on where the client and server processes land) and also internal tests (running in 5s, 3s or 1.5s sporadically).

    With my internal python testing I went and compiled the latest python and numpy versions from scratch, but without improvement. With the redis benchmark I was able to fix cpu locations using taskset to get consistent 90Kops/s performance. I think this is a scheduling issue. I know this isn’t ubuntu (yay!), but the same tests perform consistently well with a minimal ubuntu install (500+MB, ick!). Comparing the kernel compile options between the two and browsing through the kernel code I think it is one of the following two options (maybe both) that are not set:

    CONFIG_RT_GROUP_SCHED : This is real-time group scheduling. There are several segments of the process scheduler that depend on this being set. Which includes making code run by root/kernel have rt scheduling requirements and reserving processing time for root/kernel. Having these specifications could allow more intelligent scheduling. This should probably be on for server applications and it doesn’t hurt desktop performance either (99.7% free cpu on lubuntu and slitaz). The defaults leave the vast majority of processing available and it seems that the RT needs for audio/video would be improved for the desktop and reserved processing time for the system would improve servers.

    CONFIG_SCHED_SMT : This is the option to enable hyperthreading. I’m not sure how this differs between the new AMD “cores” and regular Intel HT — whether just leaving this on would affect compatibility with older CPUs or give a performance hit on non Intel CPUs. I think the scheduler identifies whether or not it should be used or it doesn’t have any noticeable negative performance impact, just like SMP for old cpus without multi-core. Obviously if it breaks compatibility with old systems its a no-go, but I don’t think that is the case.

    When I tried to recompile my kernel there were a whole bunch of options that I didn’t have a clue about and I ended up with a kernel panic. So I was wondering if anyone else has any little nagging benchmark problems? Anyone recompiled their kernel with everything the same except these two options turned on? Personally, I think these things should be turned on in the standard kernel, what do you think?

    –David

  • Profile picture of David David said 1 year, 4 months ago:

    UPDATE:
    I had the opportunity to test the exact same redis benchmark on a dual core AMD machine and got consistent good results (yay tazusb!). So it seems like the difference is the CONFIG_SCHED_SMT option… I get the same performance on the dual core AMD machine using Ubuntu 12.04 with the generic 3.2 kernel even though the compile options include CONFIG_SCHED_SMT. So it appears that, like SMP for multi-core, CONFIG_SCHED_SMT does not cause significant overhead on non multi-threaded systems. Although I am not sure what the effect would be on the newest bulldozer/piledriver AMD chips (with their flavor of pseudo-core hyperthreading) — I think the general consensus was linux distributions didn’t have the same kernel problems that Windows 7 did when the bulldozer architecture was released. Does anyone else have experience with the CONFIG_SCHED_SMT kernel option? From the results that I got (on both AMD dual core and old hyper-threaded i3) it looks like this option should be turned on. CONFIG_RT_GROUP_SCHED was off with good results on the AMD so I’m not sure about that.

  • Profile picture of David David said 1 year, 4 months ago:

    UPDATE: Compiled with CONFIG_SCHED_SMT enabled… No difference. Recompiling with CONFIG_RT_GROUP_SCHED.

  • Profile picture of kultex kultex said 1 year, 4 months ago:

    Hi David,

    when you are already working on compiling a new kernel and you have new boards, can you also activate pae?

    I think, its just

    config_highmem enabled

  • Profile picture of David David said 1 year, 4 months ago:

    Yeah, kultex, I can do that (will need it myself soon). The ram limit without that set is 4GB, right? Do you know anything about the HIGHMEM4G and HIGHMEM64G options? It looks like the current cooking default is CONFIG_HIGHMEM4G (CONFIG_NOHIGHMEM off; CONFIG_HIGHMEM=y). So I think HIGHMEM64G=y (and 4G off) will allow PAE to map up to 64GB (or less), but that won’t work for non-PAE cpus. So maybe a slitaz-linux-modern would be helpful as a kernel package option? That gives PAE-64GB, scheduling to handle HT or non-HT (maybe this should be default) and some other (say <5 yrs, but stable performance improving) options.

    One more thing — I'm trying to move to an in memory compile process to speed things up, but it keeps crashing on me. Directly on hardware on one machine and from a VM on another machine (i3 and athlon cpus, respectively). The crashes instantly halts both machines — completely shut off, no warning. So going from my VM that's stable with a .vdi HD — it's a sloooow process. Has anyone had problems compiling the kernel from a live CD (all in ram)?

    Is there a linux-slitaz (slitaz kernel) maintainer? Are these posts in the right area?

    –David

  • Profile picture of kultex kultex said 1 year, 4 months ago:

    its the right place

    I think, slitaz will never have pae by default – so also cooking has no pae kernel -it would not boot on my Portege R100

    I never did a SliTaz Kernel, but I did some debian kernels, but cannot say anything until now with a new slitaz kernel

    just compiles some single kernel modules, which I needed and this worked like a charm

    thats the siltaz kernel – mostly done I think by Pankso

    http://hg.slitaz.org/wok-stable/file/50647c0be30b/linux/stuff/linux-2.6.37-slitaz.config

    I am a bit astonished to see that CONFIG_HIGHMEM=y

    so have to digg in deaper

  • Profile picture of David David said 1 year, 4 months ago:

    Yeah, I definitely agree that slitaz shouldn’t have PAE on the default slitaz-linux package (it wouldn’t boot on half of the systems I use). The CONFIG_HIGHMEM4G=y is the hack for 4G instead of up to 3G (CONFIG_NOHIGHMEM=y is like the WinXP/etc setting). CONFIG_HIGHMEM64G=y looks to be the true PAE from this reference:

    http://oreilly.com/linux/excerpts/9780596100797/kernel-configuration-option-reference.html

    Would a linux-slitaz-modern.tazpkg be helpful? That has kernel options/updates that are stable but only supported on CPUs less than ~4 years old be useful? There is a serious performance issue with the cooking kernel on my i3 that I’m trying to solve. If the solution isn’t backward compatible to 386/486 maybe a linux-slitaz-modern would be helpful so that people with more recent processors can run “tazpkg get-install linux-slitaz-modern” to get the most out of newer configurations?

    Pankso is one of the big bosses (the big boss?) right?

    –David

  • Profile picture of kultex kultex said 1 year, 4 months ago:

    yes, but there is no option in the config for HIGHMEM64G, that waht makes me a little bit worrying

    There are more interested in a PAE kernel, I think the way how it could work – we have to make a iso with pae kernel to test, that we share throug dropbox ore whatever.

    When whe have one, that is good working, we ask pankso or pascal to set it up in wok and create a slitaz-pae.iso

    for cooking there exists already a 64bit kernel,

    http://hg.slitaz.org/wok/file/8b9b79fb08c1/linux/stuff/linux-slitaz.config64

    I think, that you have just to install all the linux64 stuff, you need

    http://pkgs.slitaz.org/search.sh?package=64

    Thomas

  • Profile picture of christophe christophe said 1 year, 4 months ago:

    I tried to do a pae kernel a while ago
    Though i did not investigate what was not working (it booted and i did some things but some applications were not working as expected).

    I used the default configuration tool (i think it is accessible trough the cnfigure shell) and going through the menu what to change was pretty straightforward, if I am not mistaking (but that is already a while ago).
    What I did not investigate is if i broke something else.
    In another words, i think it configured on the basis of some default file which may not be the defaut slitaz config file.
    Seems to me the safe side would be to use the default slitaz config file, which is availabe from any iso, copy it as the default in the configuration directory, then use the configuration tool to change whater needs to be changed rather than directly editing the file (i suspect some changes in the menu will in fact affect several variables/directives of the config file).

    I have no experience in compiling a kernel or anything whatsoever (well, nothing younger than 15 years ago),this is just my 2 cents on the short try i gave.

    Sorry i cannot spend more time on this now than this few comments….

  • Profile picture of Trixar_za Trixar_za said 1 year, 4 months ago:

    Kultex & Christophe: From how both are you are pushing for PAE – to the point of hijacking threads and getting other people to do the work for you, I think both of you have gone a bit overboard.

    So let me say this: You cannot magically make a program use PAE that wasn’t designed to use it. In fact there is no magic compile option that suddenly super boosts Linux systems and programs like you guys try to find every couple of months. It doesn’t exist. What does exist is a combo of compile options when used the right way can give the proper boost – but none that work alone. This is also why I think David isn’t noticing any improvements by enabling one or the other option in the kernel compile. Maybe it needs both to work effectively.

  • Profile picture of David David said 1 year, 4 months ago:

    I did get a significant improvement! With the CONFIG_RT_GROUP_SCHED option enabled (also had the CONFIG_SCHED_SMT option on, but that individually did not help). I just got ~20% improvement on the redis benchmark and my python benchmark. This was on athlon and i3 architectures. I am still having a scheduling issue on the i3 so that I get a 100%, 50% and 25% performance depending on where the processes land. Although with the CONFIG_RT_GROUP_SCHED turned on I saw 20% improvement on all three core scheduling possibilities (110%, 55%, 27.5%).

    I agree PAE will not improve performance. It just allows you to use more than 4G of ram with a 32bit compiled kernel — but requires a recent processor that supports it. I couldn’t really test it out. I only have a 2G and 4G machine. Nothing with >4G.

    I’m going do some more looking at the config options for scheduling to see if I can get the same consistency as the ubuntu kernel, but I’m running out of options. It maybe something that improved with the 3.6 kernel.

    –David

  • Profile picture of kultex kultex said 1 year, 4 months ago:

    trixar nobody is pushing anybody – it was just a question!

    I will look for it tomorrow, and I think it is allowed to discusssomething before do something and I know, that programs are working – both with normal and pae kernel – avlinux is the example – you can try it out

    and I think I will try it out tomorry and will see, if I will have the same troubles like David

    @David I hope you did not have the feeling of being abused

  • Profile picture of David David said 1 year, 4 months ago:

    @kultex no worries. I’m going to get to my kernel optimizations before yours anyway! :) I will be interested in a PAE because my next rig will have 8G or 16G.

  • Profile picture of Trixar_za Trixar_za said 1 year, 4 months ago:

    I’ll admit that our kernel needs work. Maybe this will help: http://kernelnewbies.org/

  • Profile picture of David David said 1 year, 4 months ago:

    Update: well, that 20% performance improvement was me going off memory and accidentally comparing numbers vs VM. I am recording the results of every attempt now (along with the /proc/config.gz). I did get about a 50% improvement in HT scheduling for python and consistently high throughput on redis-benchmark on an i3. Also had no degradation on the athlon X2. That was with CONFIG_RT_GROUP_SCHED and CONFIG_SCHED_SMT enabled. I am trying a slew of options to further improve HT scheduling (but keep i386 support and non-HT performance).