GCC Workshop at GRC, IIT Bombay

Mustafa (my manager) knew about my current fascination with learning the _real_ basics of computers (electronics, kernel, compilers, etc.), so he arranged for me to attend the GCC workshop held by IIT Bombay this week. My first reaction was that I would be wasting my time there since I didn't know the first thing about compiler theory and was a novice at best in assembly language programming. He said it wouldn't hurt to try. So I had to try.

I got a chance on the day before the workshop to read up some about things like IRs, RTLs, etc. It was enough that I would not be completely lost from day 1. But I did not attend day 1 at all, thanks to the countrywide strike that crippled all public transport (and some people too). So I spent that day working and also trying to cover what would otherwise have been covered in day 1 -- the various phases of compilation, passes, gray box probing to find out some more about intermediate outputs, etc. I got hold of last year's slides, so it was a little easier.

So I finally made it to days 2, 3 and 4. The first thing I noticed during the lecture sessions was that the professor really knew his stuff. He was well acquainted with the internal layout of gcc and was able to explain it well enough that I really _got_ it. Overall I come out of the sessions today with much more knowledge about gcc than I could ever gain in 4 days on my own. Here are some observations that I made during the course of this session.

The professor really knew his stuff. I say this again so that it does not look like I am ignoring that. There are also a lot of really talented individuals at the GRC, who are doing some pretty interesting research based on gcc. The trouble though is that there seem to have been no efforts whatsoever to share these ideas upstream.

One such idea is the Generic Data Flow Analyzer (GDFA). It is a patch to gcc that provides a data flow analyzer, which can be used to find and eliminate dead code or unused variables. It adds a gimple pass to the compilation sequence and intends to replace the current dead code elimination and unused variable elimination passes with the same code called with different parameters. While the idea is pretty interesting, the sad thing is that there are no signs of an attempt to push this idea upstream. All I could find was an announcement to the gcc mailing list, but no request for comments or for inclusion of the patch.

This is only one of many more ideas that are brewing in the GRC in the minds of some very talented people. But one felt that these ideas were being used only to get degrees and nothing was being done to actually test their feasibility in live production level code. It would be nice to see some of these ideas actually presented upstream with a genuine interest in getting them incorporated.

To conclude, it is a pretty good session for those who want to get started with learning compilers and gcc internals.


Yahoo chat future and libyahoo2

Philip had recently posted on libyahoo2-users that Yahoo is planning to open its instant messaging platform API to the public. It has been delayed a bit since then, but it is surely due.

So how does this change things for libyahoo2 or any other FOSS implementations? For one, the fun of digging through binary data and trying to make sense of it will be gone ;) But on a serious note, we can hope to have some more consistency in behaviour and support will definitely improve. I'm not very keen about the fact that the support will be over HTTP, but I guess it works well for them. For now we can only wait for their announcement before we know what the entire thing looks like. If it is anything like the messages that the current official yahoo! messengers send, then it's only really a wrapper around their old pain of a protocol. But this does not really use JSON, so it is likely that they're writing a fresh implementation. In any case, there is still time for it and in that time, we have some decent work going on on the libyahoo2 code base.

In other news, Kai Zhang has been working on implementing chat room support for libyahoo2 as his Fedora Summer Coding project. His code can be found here. Other than the brief comments in the git logs, everything seems to be quite ok. A bulk of the feature set is already in, so that is pretty good progress. Once the entire feature set is completed and tested, I will have them included in the main libyahoo2 source tree. Following that will be a release and a rebase on Fedora. This will be a good rebase compared to the ugly one the last time around, where I broke all API compatibility in an effort to revamp the authentication support.


Hacking on assembly code: Dynamic memory allocation on stack?

So I started dabbling with assembly language programming a couple of days ago. This was the next logical step in the "going lower down" move I have been doing ever since I started writing programs in Visual Basic some years ago (there, I admitted it). Since then I went through C#, Java, C++, C and now finally assembly. And it is fun to watch a program die in so many innovative ways. It is helping me understand the internals of a program much better.

One of the first things I learnt about assembly programming was that I needed to use completely different syscall numbers and instructions for x86_64 as compared to i386. For example, the syscall number for exit on i386 is 1 while on x86_64 it is 60. Same goes for write -- 4 on i386 and 1 on x86_64. I spent half an our trying to figure out why my program was calling fstat on x86_64 while a similar program built with --32 would work fine.

Crossing all these hurdles, I finally wrote a slightly more complicated (but still useless) program than a hello world. This is a program that takes in an integer string through the command line, converts it to an integer, converts it back to string and prints it back out. Pretty useful huh :)

Now for the interesting part in the code. I always thought of dynamic memory allocation as something you can only do through the OS using the brk() and/or mmap() syscalls. Generally we do this indirectly through malloc() and friends. But what I ended up doing in my program is allocating memory on the stack on the fly. Here's the code snippet:

	movb $0x0a, (%rsp)
	decq %rsp
	movq $0, %rdx
	divq %rdi
	addq $0x30, %rdx

	# Hack since we cannot 'push' a byte
	movb %dl, (%rsp)
	decq %rsp

The complete code along with the makefile is at the end of this post. You can build it if you have an x86_64 installation. What I do above is simply:

  1. Read a digit from the number
  2. Move the stack pointer ahead to make room for a byte
  3. Store the ascii representation of that number into that byte

I could not use the push instruction itself, since it can only push 16, 32 or 64 bit stuff on to the stack (with pushw, pushl, pushq). If you push a single byte value, it will be stored in one of the above sizes, not in just 1 byte. What I wanted was to create a string on the fly without limiting myself to a fixed size array, so this seemed to be the only approach. While this works, I still need to find out a few more things about this:

  1. Is it safe?
  2. If it is safe, then is there a similar way to do this in C without embedding assembly code? This would be really cool, especially in usage scenarios such as the above. Admitted that the above scenario is pretty useless in itself, but I'm sure there must be similar examples out there that are at least a little more useful.

The code:

.section .data
	.ascii "Usage: printnum-64 <the number>\n"
	usagelen = . - usage

.section .text
.globl _start

# Convert a string representation of an integer into an int
.type _get_num, @function
	push %rbp
	movq %rsp, %rbp
	movq 0x10(%rbp), %rdx
	mov $0x0, %rcx
	mov $0x0, %rax
	# Iterate through the string
	movb (%rdx), %cl
	cmp $0x0, %rcx
	je call_done

	subq $0x30, %rcx
	imulq $0xa, %rax
	addq %rcx, %rax
	incq %rdx
	jmp nextchar

# Convert a number into a printable string
.type _print_num, @function
	push %rbp
	movq %rsp, %rbp
	movq 0x10(%rbp), %rax
	movq $0x0a, %rdi

	# Hack since we cannot 'push' a byte and increment
	# %rsp by only 1. push will push whatever it has as
	# a 16, 32 or 64 bit value (pushw, pushl, pushq)
	movb $0x0a, (%rsp)
	decq %rsp
	movq $0, %rdx
	divq %rdi
	addq $0x30, %rdx

	# Hack since we cannot 'push' a byte
	movb %dl, (%rsp)
	decq %rsp

	cmp $0x0, %rax
	jne next_digit

	movq %rsp, %rbx
	addq $0x1, %rbx
	movq %rbp, %rcx
	subq %rsp, %rcx

	push %rcx
	push %rbx
	push $0x01
	call _write
	jmp call_done

# Wrap around the write system call
.type _write, @function
	push %rbp
	movq %rsp, %rbp
	movq 0x10(%rbp), %rdi
	movq 0x18(%rbp), %rsi
	movq 0x20(%rbp), %rdx
	movq $0x01, %rax
	jmp call_done

# I always do this when I am done with a function call
	movq %rbp, %rsp
	pop %rbp

#Program Entry point
	# Command line arguments:
	# The parameter list is:
	# argc: The number of arguments
	# argv: The addresses of all arguments one after the other
	#       They can be popped out one by one
	pop %rax
	cmp $0x2, %rax
	jne error

	# Pop out the first arg since it is the program name, but
	# keep the second so that it can be fed into the next function
	pop %rax

	call _get_num
	push %rax
	call _print_num
	jmp exit
	push $usagelen
	push $usage
	push $0x2
	call _write
	movq $0xff, %rax
	movq %rax, %rdi
	movq $60, %rax

The makefile:

	as --32		$(target).s	-o $(target).o
	ld -melf_i386	$(target).o	-o $(target)

	as $(target).s	-o $(target).o
	ld $(target).o	-o $(target)

If you save the source as foo.s, you can build it with:

make target=foo 64

Lots and lots of work

The past week has been quite hectic, with a lot of juggling between different things I have been wanting to do. So here's what I had on my mind:

  • I have been looking to learn more about compilers. I goofed off in college and missed out on the same course that was taught twice. I always understood enough to fool my teachers into thinking I knew it all, but not enough to really know it all. Or some for that matter. So now I want to make up for it.
  • I had not touched ayttm and libyahoo2 for quite a while. So I wanted to do something there
  • Kushal had asked me if I could package libraw for Fedora because some random app needed it. He asked me because I knew autotools and I could autotoolize the project before I package it.
  • Rahul pointed out this cool little command line audio player called gst123. I had been looking to write something like this for some time now but I just could not wrap my head around gstreamer. I tried it and immediately fell in love. I just had to package it for Fedora.
  • Work at my day job. Lots and lots of work.
  • Work at my day job. Lots and lots of work. Yes, it is worth mentioning twice

And so here's what I actually ended up doing over the week:

  • I had bought 3 books to study compilers. They're just lying there since I haven't had enough time to actually start studying.
  • Nothing on ayttm and libyahoo2. Not enough time
  • Packaged libraw and submitted for a package review. There is no activity on that bug report yet, but there was some action before it. Libraw upstream does not like autotools, so I had to hand-write a configure script to detect stuff. I also looked up and tried out the app for which Kushal wanted me to package libraw. The app is Shotwell, a photo management program. And it is good; I'm starting to use it for my photographs now. I'm glad I decided to package libraw for it.
  • I packaged gst123. The package has been approved and I have already submitted an update for F-13. I did this while on a bus from Pune to Mumbai :D

    gst123 is a really cool app, try it out. It might not play internet radio streams right out of the box (my use case), but you can easily pipe/grep/cut your way to getting it to work. Here's how I play the radio stream from Absolute Radio:

    gst123 `curl -s http://network.absoluteradio.co.uk/core/audio/ogg/live.pls?service=vcbb | grep File1 | cut -d '=' -f 2`

    See, it's so easy!

Oh yeah, work at my day job. Lots and lots of work.


Fedora Activity (half) Day

The FAD (Fedora Activity Day) was announced over a month ago with an intention to get some real work done during an event. I really only had a chance to participate in 1/4th of the FAD (1/2 day on Saturday), since I had to fly to Bangalore on Saturday evening to spend the weekend (or whatever was left of it) with family. But that was enough to get whatever I wanted out of the event.

Being pretty much a newcomer into the Fedora community, there wasn't much that I could think of to directly contribute but I wanted to do something. I really only maintain 1 package, which also does not have much traffic, so I wasn't exactly brimming with ideas. Rahul helped me there by asking me to do an Autotools workshop. I was also looking forward to meeting some of the guys I had met at FOSS.in last year; Susmit, Hiemanshu and Sayamindu. I could not meet Hiemanshu (did he come at all?), but it was good to meet Susmit and Sayamindu after quite a long time.

We started the day with my autotools workshop; I hope at least someone found it useful. I demonstrated the process of autotoolizing a simple C program using the same example I used during my Fedora classroom session earlier this month: linkc. The main reason I keep choosing this program is that I am too lazy to find or write anything on my own. The other reason is that the program helps to cover quite a few things at one go -- it is small, it has an external dependency, a subdirectory and some distributable files. So all those things win over the fact that the app just doesn't work as advertised. Oh well...

Once the only "session" of the day was over, everyone announced their aims for the two days while Sankarshan distributed some swag (t-shirts, stickers and buttons). After that it was pretty much everyone working on their own stuff. Me too.

Only a couple of days before FAD, Ray van Dolson added me as a co-maintainer for libyahoo2 in Fedora so that we could share the workload of doing releases/bug fixes. After discussion with him, I decided to do a libyahoo2 release into rawhide during the event. So I finally had something that I could do, which was much closer to Fedora.

I knew that the release would break freehoo, a console messenger for yahoo since libyahoo2 1.0.0 broke all backward compatibility, so I set about fixing that. The result was a bug report with a patch to fix freehoo to build with the latest libyahoo2. Finally, I also changed ayttm to dynamically link against libyahoo2 instead of cloning the code base all the time. There was absolutely no incentive in maintaining two code bases for it, so it finally had to go.

By the time the ayttm change was done, it was time to leave. But before that, Kushal asked me to take a look at libraw to see if I could pitch in with something there. So I will be looking at autotoolizing it and packaging it for Fedora. I was supposed to do it today, but all of my day was spent in playing catch-up with work at my day job. Maybe I'll have more time tomorrow for it.


Finding connections to a specific port with SystemTap

Earlier in the week I was asked by someone if I know a way to monitor applications trying to connect to the telnet port. The obvious answer was netstat/lsof, but the problem was that this application would be up for merely seconds and he did not know when/where it started up. All he had was his telnetd log telling him about the connections. So I decided to go the SystemTap way and came up with this:

/* snoop.stp */
function sockaddr_to_port:long(sck:long)
        short ret = 0;
        struct sockaddr *sock = (struct sockaddr *) (long) THIS->sck;

        memcpy(&ret, sock->sa_data+1, 1);
        memcpy(((char *)&ret)+1, sock->sa_data, 1);

        THIS->__retvalue = ret;

global testport = 0

probe begin(0) {
        if (testport == 0) {
                printf("Usage staprun snoop.ko testport=<portnum>\n");

probe syscall.connect {
        port = sockaddr_to_port($uservaddr)

        if (port == testport) {
                printf ("%d: %s trying to connect to port %d\n", gettimeofday_s(), execname(), testport);
                printf ("\tPID: %d\n", pid());
                printf ("\tUID: %d\n", uid());
                printf ("\tEUID: %d\n", euid());
                printf ("\tParent: %s\n", pexecname());
                printf ("\tPPID: %d\n", ppid());

So this really is a very simple tap on the connect syscall to be able to intercept all connect requests on the system. The sockaddr_to_port is a little more interesting. It takes the first two bytes of sockaddr->sa_data and swaps them to get them to the correct byte order to return as the port number. The reason for the swap is that network byte ordering is essentially big endian (The most significant byte first) while x86 computers are little endian. So the port number 23, at byte level would be seen on your computer as:


while in network byte order it is seen as:


Once we have the port number, the rest is easy pickings with SystemTap giving us easy functions to collect the executable name, pid, parent process name, etc. As for the date, one may easily convert it to human readable form using:

date -d @<my date in seconds>

Now that we have all the pieces in place, we can build the above script into a kernel module using:

stap -vvvg -r `uname -r` snoop.stp -m snoop

The output of this is the kernel module named snoop.ko. The -g signifies the "Guru mode". We need it since our function sockaddr_to_port is embedded C and we need to tell stap that we really know what we're doing. I like putting in a lot of v's in the first place so that I get build errors right away. This is essentially building a kernel module. The -r and -m are for kernel version number and module name respectively. Once built, you can deploy the module using:

staprun snoop.ko testport=23

This loads the module into the kernel and waits to print messages (whenever the program decides to connect to telnet) to standard output. If you look into the output of lsmod, you will find that snoop is a loaded module.

Packages you will need to implement all of this:

  • systemtap
  • systemtap-runtime
  • kernel-debuginfo
  • kernel-devel
  • kernel-debuginfo-common

If you're using the PAE kernel then you need the kernel-PAE-debuginfo and kernel-PAE-devel instead of kernel-debuginfo and kernel-devel. If you're only looking to deploy a binary systemtap module, you will only need systemtap-runtime. Yes, I can run a module built on one system, on another system provided they have the same kernel version and architecture. But be careful of what you run, always inspect the source to make sure you're not running anything malicious.


Checking Reliance Netconnect prepaid account usage

So I've been using the Reliance Netconnect usb dongle since I shifted to Magarpatta City. Since work is so close to home now, I really only need internet access at home to check emails when I wake up (yes, I am an addict). Being a prepaid account, I wanted to know how I could monitor my usage. I asked the vendor and he told me to go to the Usage menu. I told him I use Linux. He insisted that I ought to be going to the usage menu.

I gave up asking him.

And it was a good thing I did, because the information was quite easily available online. All you had to do was go to this URL and enter your MDN, i.e. the number of your netconnect dongle. But that was very cumbersome, so I hacked up this little script so that usage monitoring is now just a command away:


cleanup_temp () {
	rm -f $tempfile
	exit 2

if [ $# -lt 1 ]; then
	echo "Usage: $0 <reliance netconnect number>"
	exit 1

trap cleanup_temp SIGINT
trap cleanup_temp SIGTERM
trap cleanup_temp SIGHUP


nc reliancenetconnect.co.in 80 > $tempfile <<ECHO
POST /RNetconnect/RNC/Netconnect_Authentication.jsp HTTP/1.1
Content-type: application/x-www-form-urlencoded
Content-Length: 14


grep "and your Netconnect" $tempfile | links -dump

rm -f $tempfile

And if you want to make things even simpler, modify the above to read the number from an environment variable and export that variable in your .bashrc.


libyahoo2 is now 1.0.0

There have been a number of code changes in libyahoo2 since some time now, with Gennady Feldman and later me pitching in. So it was finally time to do a release. My primary motivation for this was to remove libyahoo2 code base from ayttm and have it link dynamically to libyahoo2. I hope Ray van Dolson packages libyahoo2 for Fedora soon, so that I can pester Minto to package ayttm with that patch too.

The YMSG16 support changes in 1.0.0 will break things for many people using libyahoo2, but that was necessary. I wanted to avoid tying libyahoo2 to a specific crypto library for SSL use and also reduce the overhead of maintaining that code base. The result of this change is that it removed all of the networking code from libyahoo2 as well, which reduced maintenance overhead even more.

Another big change was the removal of the ability to bypass the callback struct and link functions directly. It would make things difficult when you're trying to find out in your configure script if the library is installed. One would have to write a test program in the autoconf, which would have to have all the callbacks implemented, to be able to compile successfully. That is painful. Also, The approach is much more extensible compared to earlier the approach.

There are a number of things that we still need to get done in libyahoo2:

  • Chat room support
  • Webcam support— I am not sure if it is broken since I don't have a webcam to test, but I want to make sure anyway
  • Are mail notifications working?
  • Voice support

I will be submitting a Fedora Summer Coding idea for libyahoo2 soon to add chat room support as a mentor. The major skills required for this would be C and the ability to read and understand wireshark dumps.


Viewing signal numbers in gdb

Recently someone asked me if there was a way, with gdb, to find out for which signal a particular signal handler was called. The catch was that there were no debugging symbols, but the binary was not stripped either. That way I could set breakpoints and do some general debugging on the flow of the program but not view variable values directly. So the backtrace does not show the signal number that was passed into the signal handler and I would have to do some magic to get at that value. So here's a little demo to explain what I did:

So I had a program that looked like this:

#include <signal.h>
#include <stdio.h>

void foo(void)
    printf ("that's all folks!\n");

void handler(int sig)

int main()
    signal(SIGIO, handler);

I built the program, started it under gdb and set a break point at foo(). I then ran it and from another terminal, did a killall -IO a.out. here's what I got:

(gdb) bt
#0  0x00000000004004dc in foo ()
#1  0x00000000004004f8 in handler ()
#2  <signal handler called>
#3  0x000000000040050d in main ()

Notice that the argument value for the handler() function is not shown, which is our problem. This is because there is no way that gdb would know anything about the arguments without any debug symbols, i.e. the number of arguments, the size of each argument, etc. But we do. A signal handler has only one argument and it is an integer. It is the signal number.

To go any further, we need to check out the stack frame of handler(). As you can see in the stacktrace above, it is frame 1, so here's what we get:

(gdb) info frame 1
Stack frame at 0x7fff84277c30:
rip = 0x4004f8 in handler; saved rip 0x301e2301b0
called by frame at 0x7fff84277c38, caller of frame at 0x7fff84277c10
Arglist at 0x7fff84277c20, args:
Locals at 0x7fff84277c20, Previous frame's sp is 0x7fff84277c30
Saved registers:
  rbp at 0x7fff84277c20, rip at 0x7fff84277c28

So we know now that the arglist starts at 0x7fff84277c20, so the signal number argument should be in this region. Since it is a 32 bit value, we do some basic math and go down the stack by 4 bytes to see what word is stored at 0x7fff84277c1c:

(gdb) x/1 0x7fff84277c1c
0x7fff84277c1c:    0x0000001d

Do some more college math and convert 0x0000001d to decimal and we get 29. Lookup signum.h (search for it in /usr/include) and you'll see that it is SIGIO, the signal I sent in my kill. W00t!!

One may use this method to also get local variables of that function. It is just that we will have to do some guessing to find out the sequence of the locals in the stack if one does not have access to the source code. It should not be that hard though. This may also be one of the bases for those try to use buffer overflows to overwrite some value in the stack and hence modify the flow of a program. I'm yet to figure out how one could overwrite stuff far enough to drop from the program right into a shell. Learning computer science bit by bit now after wasting 6+ years of formal schooling in it :)


gource in Fedora

My first package submission was finally completed a few days ago with everything from ticket overload, personal issues to my US trip making sure that I had little time to get things done on time. I had started this in the first week of December and it took all of 2 months and some days to get my stuff together and put out an rpm package for gource. It is finally done now and you can install gource on your Fedora 12 box (or later if you're nutty like me) with just a

~# yum install gource

I've built gource only for F-12 and later since F-11 does not have the required version of ftgl. That is OK I guess since F-11 is way too old anyway ;)