Boot single UEFI image to shell

tux_compHarald Hoyer from Red Hat has come up with a demonstration of booting the Linux kernel to a command prompt on a UEFI device using a single image. It is the userland proof of concept for the work done by Kay Sievers and David Herrmann in gummiboot. They created a UEFI loader which starts a linux kernel with an initrd and a kernel cmdline, all stuffed-in as the COFF section of the executable.

Hover did this using a shell script which creates a rescue image on Fedora using a rescue kernel and rescue initrd. The kernel cmdline parameter rd.retry is set to 20 seconds to wait for devices. Finally it drops to a shell as the root device is specified as “root=/dev/failme” which does not exist. You can fsck your devices, mount them and repair your system from this shell.

To try it out on Fedora, run:

# yum install gummiboot binutils
# wget https://raw.githubusercontent.com/haraldh/mkrescue-uefi/master/mkrescue-uefi.sh
# chmod +x mkrescue-uefi.sh
# bash mkrescue-uefi.sh BOOTX64.EFI

Copy the generated BOOTX64.EFI to a UEFI-formatted bootable FAT32 USB volume under EFI/BOOT/ and point your BIOS to boot from the USB stick. If you are not sure how to create the FAT32 volume, refer to my earlier article on creating a UEFI bootable ArchBang USB.

[Source]

intel_frequency: manipulate Intel GPU frequency

tux_compBenjamin Widawsky has pushed a new tool to change the frequency of Intel GPUs from userland. This is not a new functionality as this could be done using sysfs but this tool brings it to end users. The tool should be useful for debugging performance issues or benchmarking. It does not allow any GPU overclocking or any functionality not allowed via sysfs. Also note that the tool underwent minimal testing at the time of writing.

Usage

The name of the tool is intel_frequency. It should be run as the root user or sudoer. Options:

  • Get current, min, max and efficient frequencies
    $ sudo intel_frequency --get=cur,min,max,eff
    OR
    $ sudo intel_frequency -g
    cur: 200 MHz
    min: 200 MHz
    RP1: 200 MHz
    max: 1200 MHz
  • Another way to get a particular frequency
    $ sudo intel_frequency -geff
    RP1: 200 MHz
  • Lock frequency to an absolute value
    $ sudo intel_frequency --set min=300
  • Set a custom max (or min) frequency
    $ sudo intel_frequency --custom max=900
  • Lock frequency to the most efficient frequency
    $ sudo intel_frequency -e
    $ sudo intel_frequency -g
    cur: 200 MHz
    min: 200 MHz
    RP1: 200 MHz
    max: 200 MH
  • Lock frequencies to the max frequency
    $ sudo intel_frequency --max
    $ sudo intel_frequency -g
    cur: 1200 MHz
    min: 1200 MHz
    RP1: 200 MHz
    max: 1200 MHz
  • Lock frequencies to the min frequency (DEBUG only)
    $ sudo intel_frequency --min
    $ sudo intel_frequency -g
    cur: 200 MHz
    min: 200 MHz
    RP1: 200 MHz
    max: 200 MHz

The patch is available here in the Intel Graphics mailing list, waiting to be added to the Intel GPU tools package (at the time of writing).

NeteXt’73: paid optimizations for Linux kernel

tux_compNeteXt’73 is a collection of patches on the latest released Linux kernel for improved performance and power management, currently available for Ubuntu and its derivatives. All the information regarding this development is in Polish so the information we could collect (after seeing the original post about it on Phoronix) is through Google Translate and looking at the source code of the project on GitHub.

The optimizations are around CPU microcode (Intel and AMD64), Linux firmware package and Advanced Power Management. The package provides GUI based tool to tweak several parameters as well as to set your own preferences for these components and the kernel. The modifications have been tested with the famous Phoronix Test Suite and has shown better performance and power savings than the stock kernel.

The source has integrated code to make payments through the Paypal gateway. So it seems that once you make the payment the features will be unlocked. The price is around $15.

To install the packages from PPA:

$ sudo add-apt-repository ppa:netext/netext73
$ sudo apt-get update
$ sudo apt-get install netext73

Note that the packages include benchmarking tools as well.

Webpage: NeteXt’73

lazytime: beyond atime and relatime

tux_compHistorically Linux maintains 3 timestamps for a file following POSIX standards:

1. ctime: last time metadata or content was changed
2. mtime: last time content was modified
3. atime: last time file was accessed

Note that ctime and mtime are often same,
unless only the file attributes are updated.
In that case only ctime gets updated.

atime is the problematic one among these 3. Repeated disk access to update the access time change can affect Continue reading lazytime: beyond atime and relatime

Fastsocket: linearly scalable sockets for multicore systems

cool_penguin_smallFastsocket is a Linux kernel module and a userspace shared library combination that implements almost linearly scalable TCP sockets for very fast network throughput. It takes advantage of multiple cores. As per the analysis from the developers of Fastsocket – “stock Linux kernel does not scale well when CPU core number is above 4. It is even worse that the throughput could collapse when there are more than 12 CPU cores”. Fastsocket, however, scales linearly up to 24 cores.

Fastsocket is designed and implemented in the Linux kernel (currently kernel-2.6.32-431.17.1.el6 of CentOS-6.5) so that applications do not notice it – it is transparent to socket-using applications and no code change is required to use Fastsocket. It has already been deployed in the SINA production environment and is used with Haproxy to provide HTTP proxy service.

Performance claims:

  • Fastsocket increases throughput of Nginx and Haproxy (measured by connections per second) by 290% and 620% on a 24-core machine, compared to the base CentOS-6.5 kernel.
  • With Fastsocket, Hyper-Threading can make an extra 20% performance increase.
  • With Fastsocket, NIC that support Flow-Director(like Intel 82599) can increase the throughput by 15% if the server works as a proxy(like Haproxy).

Let’s explore how Fastsocket can be used in the custom Linux kernel on CentOS-6.5.

COMPILATION

  1. Checkout the code from GitHub
    # git clone https://github.com/fastos/fastsocket.git
  2. Compile the Fastsocket kernel
    # cd fastsocket/kernel
    # make defconfig
    # make
    # make modules_install
    # make install
  3. Compile the userspace library
    # cd library
    # make

    This will generate libfsocket.so in the same directory

USAGE

  • To use Fastsocket as a kernel module, modify grub to use the Fastsocket kernel generated in Step 2 and reboot into it. Then insert the module:
    # modprobe fastsocket
  • Verify that the module is inserted
    # lsmod|grep fastsocket
  • Run the nic.sh script (comes with Fastsocket source under scripts directory) t make necessary config changes to the system:
    # cd fastsocket
    # scripts/nic.sh -i eth0 //if eth0 is your default network interface
  • To use Fastsocket in your application use the shared library from Step 3. Example usage:
    # cd library
    # LD_PRELOAD=./libfsocket.so nginx
  • The application runs normally if the share library is not pre-loaded:
    # nginx

USE CASE

Applications like Haproxy, Nginx (with accept mutex disabled) and Lighttpd work well with Fastsocket. The following scenarios benefit the most from Fastsocket:

  • The machine has at least 8 CPU cores.
  • Large portion of the CPU cycles is spent in network softirq and socket related system calls.
  • Short TCP connections are heavily used.
  • Application uses non-blocking IO over epoll as the IO framework.
  • Application uses multiple processes to accept connections individually.

Webpage: Fastsocket

Boost system entropy on Linux

tux_compEntropy in simple terms means the degree of randomness. A high entropy is desired in a system. Cryptographic applications may block on a system with low entry and lead to situations like a slow wlan connection. A low entropy may also lead to generation of predictable values in security-related algorithms. Low entropy may occur due to low workloads, like on a headless server or a virtual machine doing the same tasks.

To check the amount of entropy in a system, run:

$ watch -n .1 cat /proc/sys/kernel/random/entropy_avail

If the value is < 1000 it’s time to take corrective measures.

Haveged

Haveged is an easy-to-use, unpredictable random number generator based upon an adaptation of the HAVEGE algorithm. Haveged maintains a 1M pool of random bytes used to fill /dev/random whenever the supply of random bits in /dev/random falls below the low water mark of the device.

To install Haveged on Ubuntu:

$ sudo apt-get install haveged

rng-tools

The rngd daemon acts as a bridge between a Hardware TRNG (true random number generator) such as the ones in some Intel/AMD/VIA chipsets, and the kernel’s RNG (pseudo-random number generator). It tests the data received from the TRNG using the FIPS 140-2 (2002-10-10) tests to verify that it is indeed random, and feeds the random data to the kernel entropy pool. This increases the bandwidth of the /dev/random device, from a source that does not depend on outside activity.  It may also improve the quality (entropy) of the randomness of /dev/random.

To install on Ubuntu:

$ sudo apt-get install rng-tools

To configure, edit the file /etc/default/rng-tools and add the following as the last line:

HRNGDEVICE=/dev/urandom
OR
HRNGDEVICE=/dev/random

Finally, restart the daemon:

$ sudo /etc/init.d/rng-tools restart

zswap: a Linux kernel feature

tux_compSwapping is the technique of moving process data (likely to be used again) to an external device from the system RAM and bringing it back to RAM when required for execution. Linux normally uses a volume on the hard disk as swap space. Though this seems redundant in the latest machines with multiple GBs of RAM (at least for normal usage), the swap volume plays a major role in the performance of older machines. While swapping enables the operating system to implement virtual memory technique (where a process believes it has a lot of RAM to use), a major bottleneck is the difference in speed of RAM and hard disks. RAM is much faster than a hard disk. So in some cases, a lot of swapping can slow down your system.

zswap is a built-in feature in the Linux kernel that can compress the swap pages in a dynamically allocated cache and store it in the RAM. It is a trade-off between CPU cycles and slow I/O due to swap. In general, the rw speed of the compressed cache is faster than that of the swap I/O. Data is swapped to the disk on LRU (Least Recently used) basis only when the RAM is filled. It was introduced in kernel 3.11 and is an experimental feature at the time of writing.

The potential benefits of zswap are:

  • Desktop/laptop users with limited RAM capacities can mitigate the performance impact of swapping.
  • Overcommitted guests that share a common I/O resource can dramatically reduce their swap I/O pressure, avoiding heavy handed I/O throttling by the hypervisor. This allows more work to get done with less impact to the guest workload and guests sharing the I/O subsystem.
  • Users with SSDs as swap devices can extend the life of the device by drastically reducing life-shortening writes.

To enable zswap, pass the following kernel boot parameter:

zswap.enabled=1

There are two configurable options in /sys/modules/zswap/parameters:

max_pool_percent: maximum percentage of memory that the compressed pool can occupy
max_compressed_page_size: maximum size of an acceptable compressed page. Any pages that do not compress to be less than or equal to this size will be sent to the swap device.

To see how much data is being stored in zsawp cache and swap:

$ cat /sys/kernel/debug/zswap/stored_pages [number of compressed pages stored in zswap]
$ cat /sys/kernel/debug/zswap/flushed_pages [number of pages written back from zswap to the swap device]

There are some other files under /sys/kernel/debug/zswap/ which provide more information.

hidepid: hide a process from other users

tux_compThe Linux kernel 3.2+ has added an option to hide processes from other users as a part of kernel hardening. Only the user who owns the process and the root can see the process.

hidepid has been under discussion as early as late 2011. The git commit log is here. It is essentially an option to mount the virtual /proc filesystem. hidepid can have 3 values:

  • 0: default behaviour
  • 1: users may not access any /proc/<pid>/ directories, but their own. Sensitive files like cmdline, sched*, status are protected against other users.
  • 2: hidepid=1 plus all /proc/PID/ will be invisible to other users. It hides the process euid and guid.

To use it this option, edit your /etc/fstab as sudoer and add/edit the proc filesystem entry line as below:

proc /proc proc defaults,hidepid=2 0 0

You can remount /proc live (as root) as well:

# mount -o remount,rw,hidepid=2 /proc

Some apps might break when you use this option. You need to append the gid (group id) option to the fstab mount options to fix this. Notes from the commit log:

gid=XXX defines a group that will be able to gather all processes’ info (as in hidepid=0 mode). This group should be used instead of putting nonroot user in sudoers file or something. However, untrusted users (like daemons, etc.) which are not supposed to monitor the tasks in the whole system should not be added to the group.

Add the owner user’s group id to /etc/fstab as below:

proc /proc proc defaults,hidepid=2,gid=mygrp 0 0

Fast compile kernel with all modules needed for your box

tux_compWith Ubuntu, you can always get the latest packaged kernel from the Ubuntu mainline kernel-ppa and install it on your system. However, the kernel with be a debug enabled kernel with many more modules than your device actually needs and will be of a considerable size. For example the installed size of my non-debug custom compiled kernel is ~21MB but the one from mainline is >200MB. There is a way to build only the modules loaded currently in your system but it will compile ONLY the loaded kernel modules. Arch Linux has a way to compile all the kernel modules ever loaded in the system using modprobe-db but it is not available for Ubuntu. Here’s a easy way to do it on Ubuntu. I am not going into the details of what packages you need to compile the kernel on Ubuntu and stuff as you can easily look it up in Google and apt-get the necessary packages. In addition, it is advisable to run all commands as root. Steps:

  1. Reboot your system and go into BIOS/UEFI settings. Make sure all devices are enabled.
  2. Attach external devices you use frequently, like USB2, USB3, SD Card, DVD, USB dongle etc. This is to ensure the relevant modules are loaded. For example I use an encrypted volume so I need dm_crypt module loaded. If you know the module already you can directly insert it. For example:
    # insmod dm_crypt
  3. Get the latest kernel source. The easiest way is to get it from Linus’s Git repository on GitHub:
    # git clone https://github.com/torvalds/linux.git
  4. Go into the linux directory and copy your existing running kernel configuration. It’s better if you install a mainline kernel closer to the latest version. The differences in configuration will be less and it wil help in Step 5.
    # cp -vi /boot/config-`uname -r` .config
  5. Check and mitigate the differences between your running kernel configuration and the newer one:
    # make oldconfig
  6. Update the configuration to compile only the loaded modules in the running kernel:
    # make localmodconfig
  7. Make any further configuration changes:
    # make menuconfig
    OR
    # make xconfig
  8. Compile and package. As you are compiling only a few modules compilation time will be less. Make it parallel as well.
    # apt-get install dpkg-dev
    # make -jn deb-pkg
    //where n = 1 + number of CPU cores you have
  9. The packages will be generated above the linux directory. You will find both the debug and non-debug kernels. Install the non-debug one along with headers and libc packages:
    # dpkg -i linux-headers-xxx.deb linux-image-xxx.deb linux-libc-dev_xxx.deb
  10. Reboot and select your new kernel from GRUB to boot into it!

The Eudyptula Challenge (and more)

tux_compThe Eudyptula Challenge is a series of programming challenges to teach the process of Linux kernel development. The challenges are ordered in terms of increasing complexity starting with a very basic “Hello World” kernel module and gradually moving on to getting a patch accepted in the mainline kernel. It is created by an anonymous hacker who goes by the name of Little Penguin.

To join the challenge, you need to send an email to little at eudyptula-challenge.org in plain-text format. HTML formatted emails will be rejected just as it happens in the Linux kernel mailing list. An email client like mutt or Sylpheed is ideal for the job. New joiners will get an email with the first challenge and the details on how the challenge is run. There aren’t any rules for participating. However, the creator warns against taking any help from outside or posting answers online as the participants are expected to solve the challenges themselves.

Basic understanding of C is the only qualification to join the challenge. No earlier experience in Linux kernel programming is necessary. The main objective of the program is to bring up new kernel developers.

Similar challenge

Project Euler: Learn programming through mathematical challenges. It is a growing list of challenges with new ones getting added periodically.

Webpage: The Eudyptula Challenge