Monitoring logs: tail -f vs less +F

cool_penguin_smallWe came across a very useful application of less on Reddit today. In general, the most popular method to monitor changing logs is to use tail -f or tailf. However, when you see something interesting in the logs, you may need to come out of tail and open the file in vim to have a closer look. Continue reading Monitoring logs: tail -f vs less +F

GameConqueror: track & modify a variable in memory

GameConqueror is the GUI for scanmem, a powerful debugger to locate and modify a variable in an executing process. In addition, it can also track by value or string. The author has provided a tutorial that explains the working of scanmem and how to use it for hacking games but here’s a more regular use of GameConqueror.

gameconqueror_comp

To attach a running process click on the Computer icon. A new window with the list of all running processes will pop-up, select the one you want to check out. I have attached GameConqueror to Pidgin running on my system and I want to check where the particular string teststring I’m sending to my friend in chat is loaded in memory. I typed in teststring in the Value field. Hover on the ? and it will show you all available options. You can change Search Scope, Data Type etc. as you need.

Double click on the particular address with the string you want to inspect in the left column and it will be added to the list below. You can also right click and select Add to cheat list. For example, in my case I want to inspect the address 0x7f5583a7a820. To check the address in more detail (as in the image below), right click on the entry (or row) and select Browse this address. Click on the Pencil icon to edit memory and change a value directly.

address_comp

As you can see, GameConqueror is a very powerful debugger for running processes (even compiled without debug symbols) as well as a useful utility to analyze compromised processes or reverse engineering.

Webpage: GameConqueror

mtrace: quickly check memory leaks

gnumtrace provides a quick way to check memory leaks with the malloc() family of functions like malloc(), calloc(), free(), realloc() etc. at runtime. It’s included in the GNU C Library. mtrace is very easy to use and hence very handy too. We will check a simple program with mtrace for memory leaks on Ubuntu. Here’s the sample program:

//Filename: mtest.c

#include <stdio.h>
#include <stdlib.h>
#include <mcheck.h>

int main()
{
    char* pStr;

    //Enable memory leak trace.
    //Must be called before calling malloc() etc.
    mtrace();

    pStr = (char*) calloc(1, 10); 

    //Disable memory tracing before exit
    muntrace();

    return 0;
}

For informative results with mtrace we must compile the program in debug mode:

$ gcc -g -o mtest mtest.c

To get memory leak information, you have to provide the path to the file where mtrace should write the results. Use the MALLOC_TRACE macro:

$ MALLOC_TRACE=./mtest.txt ./mtest

Let’s see the contents of mtest.txt:

$ cat mtest.txt 
= Start
@ ./mtest:[0x4005e9] + 0xfda460 0xa
= End

Though the contents of the file mtest.txt may seem cryptic (if there’s a leak), it’s easy to interpret. The format is:

@ executable:[address in memory map for the leaked allocation] + "memory address leaked" "bytes leaked"

The following shows the line number in your source code where the leak of 10 bytes (0xa) happens:

$ addr2line -e mtest 0x4005e9
/home/user1/testprogs/mtest.c:12

Which says memory allocated in the following line is leaked:

pStr = (char*) calloc(1, 10);

But wait, mtrace does all that for you in a much easier way. Time to use mtrace!

$ mtrace mtest mtest.txt

Memory not freed:
-----------------
           Address     Size     Caller
0x0000000000fda460      0xa  at /home/user1/testprogs/mtest.c:12

KGTP: GDB as a kernel module

tux_compKGTP (stands for Kernel GDB Tracepoint Module) is a provision for opening a GDB remote interface to analyze issues in real time on production systems running the Linux kernel or applications (including Android). It can be compiled as a kernel module and inserted into the running kernel to hook into it remotely. It is also possible to debug userspace programs using GDB tracepoint and other functionality. Even if the board doesn’t have GDB pre-installed or a remote debug interface, KGTP can debug it in offline debugging mode. KGTP supports x86, x86_64, MIPS and ARM architectures and Linux kernel versions from and above 2.6.18. To use KGTP, the kernel should be compiled with some specific debugging related options enabled.

Detailed Documentation: HOWTO

Webpage: kgtp

A good article on Linux kernel debugging:
How to use kdump for Linux Kernel Crash Analysis

LiME: dump RAM

tux_compWhile there are many memory dump tools available on Linux, LiME uses an approach closer to the kernel that can be extended to other platforms (like Android) those use the Linux kernel. LiME is a highly useful forensic data analysis tool. LiME runs as a loadable kernel module (just a regular .ko file) which can capture volatile memory data. Being a kernel module it reduces the entropy generated due to userspace and kernel space interaction more than any userspace tool. LiME is the first tool than can capture memory from Android. It can work in two modes to capture memory: over the network (TCP) or to any storage (like SD Card). To use LiME on Android cross-compiling it might be required depending on the target hardware architecture.

LiME documentation

Webpage: LiME

colormake: colorize make output

terminalI wrote about ccze in an earlier article. colormake is a similar tool that colorizes the output of make to help you detect warnings and errors easily. It is actually a wrapper over make and calls make internally. You can just run colormake instead of make or add the following line to your .bashrc and run make as usual:

alias make='colormake'

To simplify things further, it can also truncate long lines so that they do not wrap. To do that invoke it in either of the following ways:

$ colormake-short
OR
$ colormake --short

To install colormake on Ubuntu:

$ sudo apt-get install colormake

Multitail: handle multiple logs at once

multitailIf you need to work with multiple dynamic logs at once, tail might not be enough to follow all of them. Enter Multitail – open multiple logs in split windows or search in all windows at once or get notifications on log changes. Multitail is customizable to the maximum extent possible.

Features

  • Highly customizable log views: multiple windows, horizontal or vertical split, search in all wondows at one, suppress or filter certain lines and so on…
  • Run external tools when a regular expression matches
  • Scrollback, set number of lines to display, suppress blank lines
  • Set update frequency
  • Colour schemes
  • Act as a visual pipe of syslog server
  • Monitor stdin
  • Set marks
  • Wrap, truncate, show only left or right for long lines
  • On new content beep, show a flashing screen, a popup or do nothing
  • Set configurations from the cmdline

Installation

To install multitail on Ubuntu, run:

$ sudo apt-get install multitail

Usage

Multitail has a large number of options. To view all of them, run:

$ multitail -h

Webpage: Multitail