Create huge files on Linux instantly

tux_compThere are those times during development when you may need to fill up a partition quickly to do some stress testing or measure network speed etc. Instead of looking for big files (even in TBs or beyond) you can use some cute utilities on Linux to do this. You will also be able to control the exact size of the file. Here are 3 ways to do this easily:

  1. Good old dd (slower option as it fills up the file)
    $ dd if=/dev/zero of=out.img bs=1G count=1
  2. Sparse file of size 5GB using dd. No data, only file metadata is updated.
    $ dd of=out.img bs=1M seek=5120 count=0
  3. Create a 1GB file instantly (and use up disk space too)
    $ fallocate -l 1G out.img
  4. Sparse file using truncate
    $ truncate -s 1G out.img

The reverse operation is to clear a file instantly:

  1. Just a redirection!
    $ >out.img
    //This is also useful if you want to delete a huge file on Linux, clear and then use rm.
  2. truncate the file
    $ truncate out.img

Multitail: handle multiple logs at once

multitailIf you need to work with multiple dynamic logs at once, tail might not be enough to follow all of them. Enter Multitail – open multiple logs in split windows or search in all windows at once or get notifications on log changes. Multitail is customizable to the maximum extent possible.

Features

  • Highly customizable log views: multiple windows, horizontal or vertical split, search in all wondows at one, suppress or filter certain lines and so on…
  • Run external tools when a regular expression matches
  • Scrollback, set number of lines to display, suppress blank lines
  • Set update frequency
  • Colour schemes
  • Act as a visual pipe of syslog server
  • Monitor stdin
  • Set marks
  • Wrap, truncate, show only left or right for long lines
  • On new content beep, show a flashing screen, a popup or do nothing
  • Set configurations from the cmdline

Installation

To install multitail on Ubuntu, run:

$ sudo apt-get install multitail

Usage

Multitail has a large number of options. To view all of them, run:

$ multitail -h

Webpage: Multitail

Enable extra compression formats on Ubuntu

compress_compWhile any Linux distro supports the standard Linux compression formats like gzip, bzip2 etc. out of the box today, most newbies wonder how to use formats like rar, 7z etc. on Linux. The good news is, while you need to install a third party app like 7zip or WinRAR on Windows, on Ubuntu everything just translates down to integration with the standard archive management tool – File Roller. It comes with the same flexibility and functionality that you get on Windows, like compression levels, spanned archives etc. Continue reading Enable extra compression formats on Ubuntu

Turn screen off on Ubuntu [from console]

ubuntu_logo_81x81A good way to save power is to turn the screen off when you can to leave the system by itself for sometime. Unlike my old laptop my new laptop doesn’t have a dedicated key to do that. An easy way to achieve this is to run the following command:

$ perl -e 'select(undef,undef,undef,.1)' && xset dpms force off

Assign a keyboard shortcut for it on Unity and you can easily turn off the monitor even if you don’t have a dedicated key. You can set it in the following file on LXDE:
~/.config/openbox/lxde-rc.xml

Linux binary editors and file diff viewers

Systems programmers need a hex editor frequently. Here are some options:

  1. A file can be viewed/edited in hex mode from vi using the :%!xxd and :%!xxd -r combination but it is not too strong and needs careful editing.
  2. An efficient and easy to use cmdline tool for editing files in hex mode is ncurses-hexedit which is available in Synaptic package manager on Ubuntu. The editor has many options like search, go to location, insert, delete bytes etc. which are more than enough for regular usage. After installing the package, run:
    $ hexeditor filename
  3. hexedit
  4. lfhex
  5. le
  6. hexcurse
  7. wxHex Editor has a GUI and is multiplatform. It can support massive files.
If you are looking for a hex file compare utility cum editor:
  1. VBinDiff: can compare and edit files.
  2. dhex: only diff viewer.
  3. cmp: options.
  4. xdelta

Cmdline Linux multimedia editing

tux_compOften I need to edit multimedia for various reasons. Handbrake is a great friend for quality transcoding but there are frequent instances when I need Swiss-Army knives. For example, removing the non-English audio streams, subtitles from MKV files, extracting audio from one file and inserting into another, saving only a selected region from a video etc. All of these can be done in Linux (resulting in great quality output) with a little bit of understanding of powerful cmdline utilities like mencoder, ffmpeg and mkvtoolnix (in my experience, mencoder and ffmpeg produce better output than mkvtoolnix). You can fine-tune each and every aspect of any multimedia file and control them in the output. They are much faster than any GUI based utility as well. I have used all of these on Ubuntu.

An example of extracting some streams (a video, audio and subtitle) from an MKV file and creating a new file:

$ ffmpeg -i input.mkv -acodec copy -vcodec copy -scodec copy output.mkv -map 0.2 -map 0.3 -map 0.1

Another example of cutting a part of a video file using mencoder:

$ mencoder -ss 01:19:30 -endpos 17:45 -oac copy -ovc copy input.avi -o output.avi

Do the same with ffmpeg:

$ ffmpeg -i input.avi -ss 01:19:30 -t 17:45 -acodec coy -vcodec copy output.avi

Once you use these tools for an hour, you would probably never consider installing bulky GUI applications for multimedia editing once again.

sort top output

terminalThe top command is used to capture the live status of Linux process. It is a very useful command for debugging process related issues. By default the top command is very dynamic and changes so much that it is very difficult to track a single process. To find out the rogue process there i a way to sort the output of top. Most important fields to sort by are %CPU and %MEM. For other fields refer to man top, section 3c.

To sort by CPU usage, run top and then press <Shift-p>.
To sort by memory usage <Shift-m>.

import: take screenshots on Linux

The import utility provided by ImageMagick is a powerful utility to take screenshots. The simplest usage is:

$ import mydesktop.png

And click on the window you want to capture or select the region you want to capture.

There are many available options such as:

  1. Take a screenshot ofthe desktop and resize to 50% after some delay
    $ import -window root -resize 50% -delay 500 screen.png
  2. Take a screenshot ofthe desktop and resize to 800×600
    $ import -window root -resize 800x600

Check the man import for full list of options.

To record a video of the desktop, try:

$ ffmpeg -f x11grab -s wxga -r 25 -i :0.0 -sameq desktop_video.mpg

If you are looking for information on how to record the screen on Ubuntu take a look here.

The smart find and grep utilities

search_compfind and grep are most useful friends when you are connected to an unfamiliar remote Linux box via a terminal or trying to find some specific files or a particular string in any file.
For example, to find all movies matching the name “evening”, of type MKV and size between 650MB to 750MB in the current directory recursively, you can run:

# find . -iname '*evening*.mkv' -size +650M -size -750M

To find the c files accessed and modified in the last 10 minutes:

$ find /home/david -amin -10 -name '*.c'
$ find /home/david -mmin -10 -name '*.c'

Look for string in .c and .cpp files:

$ grep -nr string --include=*.c --include=*.cpp

Check out the man pages for many more useful options.

Run a http server from any directory on Linux

cool_penguin_smallMany a time we need to share files over the LAN. There is a very simple way of doing it when you are on Linux using python2. cd to the directory the contents of which you want to share and run:

//Python 2
$ python -m SimpleHTTPServer 8181

//Python 3
$ pyhton3 -m http.server 8181

Here, 8181 is the arbitrary port number you want to use for your HTTP server. The files can now be accessed from any system in the LAN using a browser at

http://YOUR_IP_ADDRESS:8181

Another interesting utility is nweb.

Tested it between my laptop and mobile over the wireless and worked fine. Happy sharing!