Moving on

I have finally moved my website content over to dreamhost. The Net4 account has been utter nonsense, with its idiotic control panel and silly pricing. For more or less the same price, I am hosting two websites on dreamhost with unlimited email aliases, subdomains, mysql database, host of apps and a really good control panel. Livejournal has been fairly sucky too lately with its random downtimes, ads and loads of comment spam.

Everything looks pretty much the same right now, but I'll soon start making changes. I will send requests to update planet links (at least wherever I know I am syndicated) once I have everything in place. I need to write up a bit on the couple of sessions I'll be doing at FudCon Pune


Preparing for FUDCon -- Fedora Activity Day

September has been a very busy month for me with Siddhi going to UK for her studies, Mom moving in with us and my role change at work. In all of this confusion, a Fedora Activity Day (FAD) was planned for 10th September. I had to skip it because I had to go to Mumbai to spend the week with Siddhi before she left for Manchester. (Un)luckily, that FAD was rescheduled because of some last moment issues at COEP and it finally happened today at our office.

This FAD was planned to be something of an initiation for students so that they could do much more at the FUDCon than just attend talks. Going with that theme, I pitched in to do a rerun of my autotools demo that I had done last year, and even in the Fedora classroom. It is one of the basic things that a Fedora packager ought to know and if we're doing sessions on packaging, we might as well throw in some autotools foo.

There were a bunch of talks planned, mostly by Rahul Sundaram and Shakthi Kannan and then some by me, Prasad, Kashyap and Shreyank. But only a few of those talks actually happened. Essentially, since I came in, only I, Shakthi and Rahul spoke at all.

To begin with, I was late. According to the schedule, the talks were to happen in two meeting rooms in parallel and me and Shakthi were to do the autotools demo in parallel in both rooms. But when I reached office, I saw that there were enough to fit into one large meeting room, so scheduling talks was going to be easier.I had missed Rahul's first talk and he had already begun talking about packaging. Rahul walked everyone through the process of packaging the hello world app. After joking around a bit with Pai and Kushal, I joined Kushal and Shakthi in helping those who got stuck in the packaging demo.

Lunch followed the rpm walkthrough and we literally destroyed the pizzas that came in - there weren't enough in the end and we had to order a few more pizzas.

While the pizzas were being delivered, Shakthi walked everyone through version control using git. The main theme was managing love letters to various bollywood superstars using git. Pretty entertaining :)

After the second lunch break (where more pizzas were quickly destroyed), I was to start my demo on autotools. Before it started, Pai and Kashyap had a couple of quick 2 minute talks. Kashyap because he was asked by a number of attendees on how upstream, Fedora and RHEL were related. He showed a quick 1 minute video in which Paul Frields explained how bits come in from upstream into Fedora and finally into RHEL and how engineers from Red Hat are involved at every stage with the community. Pai wanted to gauge audience interest in databases and their role in ERPs -- there were a fair number of people who were interested, so we might see a talk from him on those lines at the next FAD.

My autotools walkthrough ended up being the last thing on the agenda because I (just like the talks before me) took more time than was allotted to me to finish. We finished with some good feedback and suggestions for improvement from the audience.

The best takeaway from this session was the involvement of all of the students who attended today's FAD. The schedule was quite gruelling and all of the walkthroughs were very intense. Despite that it was refreshing to see all of them putting their heads down and giving it their best, asking good questions and not giving up at any stage. I am hoping that we will have a similar if not better response in our next FAD in October.


My first program on my Samsung Galaxy S

No, it is not a java program, it is in assembly, which was possible because I recently flashed Cyanogenmod on my phone. Here it is:

.global _start
	mov r7, #1
	mov r0, #42
	swi #0

The program simply returns 42 and exits. I used my own cross-built binutils (--target=armv-android-eabi) to build this and simply copied it over to my phone, chmod and executed it. You need to put it outside your sdcard environment however, since you cannot execute anything in /sdcard. A Stackoverflow thread helped figure out the right opcode for the syscall.

It took me all of 6 hours to figure this out. 5 minutes to figure out the above code and 5 hours and 55 minutes to realize and fix the last line from

swi 0
swi #0


Changing the default loader for a program in its ELF

I was following an email thread in the libc-help mailing list, where the OP asked how one could force a program to load with a different loader, i.e. a loader from a different build of glibc. The answer, which is available if you follow the thread, is that it is necessary to rebuild the program to do this, with the --dynamic-linker switch to the linker. The best answer for the thread was to do this without a rebuild by making a wrapper script that invokes the program with an alternate loader, like:

/path/to/other/ld-linux.so program_name

So the story should have ended there. But it got me thinking; how hard could it be to edit the program to change the hard-coded loader value in the program? So I decided to find out.

The first try was as simple as replacing the string with my new loader string in a test program, which is basically just a Hell Oh World program. To do this, I opened the binary in emacs and changed to hex mode using M-x and then hexl-mode. Next I just wrote over the string with the absolute path of the new loader. I saved my changes and ran the program:

[siddhesh@localhost ~]$ ./a.out
bash: ./a.out: cannot execute binary file

Obviously I had done something wrong. So I had to do a bit of reading to understand what all was to change. One very obvious thing about this was that the length of the loader string was being recorded somewhere -- I was prety sure I had not overwritten anything and I had not changed the offset of the string at all. So I started reading the System V ABI documentation, which includes information on the ELF header. The ELF header documents where the program header table is, and the size of each program header table entry. one could read this very easily using the elfutils readelf command. So here's what I got:

[siddhesh@localhost ~]$ readelf -h ./a.out
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x4003e0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          2472 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         30
  Section header string table index: 27

So the important information here is the start of program headers and the size of each program header. In the case above, the program header starts 64 bytes into the file and then every 56 bytes after that was a program header. You can also read the program headers using elfutils:

[siddhesh@localhost ~]$ readelf -l ./a.out

Elf file type is EXEC (Executable file)
Entry point 0x4003e0
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001c0 0x00000000000001c0  R E    8
  INTERP         0x0000000000000200 0x0000000000400200 0x0000000000400200
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000000068c 0x000000000000068c  R E    200000
  LOAD           0x0000000000000690 0x0000000000600690 0x0000000000600690
                 0x00000000000001ec 0x0000000000000200  RW     200000
  DYNAMIC        0x00000000000006b8 0x00000000006006b8 0x00000000006006b8
                 0x0000000000000190 0x0000000000000190  RW     8
  NOTE           0x000000000000021c 0x000000000040021c 0x000000000040021c
                 0x0000000000000044 0x0000000000000044  R      4
  GNU_EH_FRAME   0x00000000000005e8 0x00000000004005e8 0x00000000004005e8
                 0x0000000000000024 0x0000000000000024  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     8

 Section to Segment mapping:
  Segment Sections...
   01     .interp 
   02     .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame 
   03     .ctors .dtors .jcr .dynamic .got .got.plt .data .bss 
   04     .dynamic 
   05     .note.ABI-tag .note.gnu.build-id 
   06     .eh_frame_hdr 

The relevant section here is the INTERP section. Each program header is basically this struct:

typedef struct {
        Elf32_Word p_type;
        Elf32_Off p_offset;
        Elf32_Addr p_vaddr;
        Elf32_Addr p_paddr;
        Elf32_Word p_filesz;
        Elf32_Word p_memsz;
        Elf32_Word p_flags;
        Elf32_Word p_align;
} Elf32_Phdr;

Here, the relevant values of the string size are p_filesz and p_memsz. If you look into the values from the readelf, you'll see that the size is 0x1c, which is the length of the string, including the trailing null character. So that is what I had to change. Now I had to find this header in the hex editor. The header is the one beginning with PT_INTERP, which has value 0x03:

87654321  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789abcdef                                                                                                               
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000  .ELF............
00000010: 0200 3e00 0100 0000 e003 4000 0000 0000  ..>.......@.....
00000020: 4000 0000 0000 0000 a809 0000 0000 0000  @...............
00000030: 0000 0000 4000 3800 0800 4000 1e00 1b00  ....@.8...@.....
00000040: 0600 0000 0500 0000 4000 0000 0000 0000  ........@.......
00000050: 4000 4000 0000 0000 4000 4000 0000 0000  @.@.....@.@.....
00000060: c001 0000 0000 0000 c001 0000 0000 0000  ................
00000070: 0800 0000 0000 0000 0300 0000 0400 0000  ................
00000080: 0002 0000 0000 0000 0002 4000 0000 0000  ..........@.....
00000090: 0002 4000 0000 0000 1c00 0000 0000 0000  ..@.....(.......
000000a0: 1c00 0000 0000 0000 0100 0000 0000 0000  (...............

I just changed the highlighted 1c to 28, which is the hex value of the string length of my interpreter (/home/siddhesh/sandbox/lib/ld-2.8.90.so). This and the change in the string was all that was needed to change the interpreter for the program. This has one really basic flaw, which is that if the interpreter string is longer than the current available space in the header, I will end up overwriting other data, thus corrupting the binary. So if you're being adventurous, be sure that your interpreter path is small enough to fit into the available space.


Getting rid of annoyances... the hard way

Lately I have been tinkering around with code in shotwell, which is the photo manager I use to sort pictures I click. As a result I built an svn snapshot and set about making my changes. On firing up the freshly built app, I saw an "Updating libraries..." progress bar pulsing away for quite some time. it just would not go away. I didn't wait long enough for it and ended the app to modify my code further. I started the app again. The progress bar comes up again and does not leave.

This had been happening for a few days and I kept ignoring it because I was working on a different part of the code (porting shotwell to cairo). I finally decided to find out what was going on today and I started shotwell in debug mode, which is simply:

SHOTWELL_LOG=1 SHOTWELL_LOG_FILE=:console: ./shotwell

... and it struck me that I had been lazy the very first time and did not bother to set the default directory for shotwell to scan. Shotwell simply assumed it to be my home directory and that was it. Scanning my home directory obviously was taking very long since it has a little over 30GB of data. Anyway, I moved my photos to a different location but it turns out that shotwell cannot read them anymore despite pointing it to the new location. I will have to modify its database to update the file paths.

So the unique thing about shotwell is that it does not modify the photo files. It stores all edits into a database correlated with the file path. So all I had to do was to edit this database (which is $HOME/.shotwell/data/photos.db, an sqlite3 database) and change the file paths to the new location.

Now here's the big warning: NEVER do such changes when you're having a conversation with someone else or are distracted by anything else.

I ended up writing an incorrect query... and then cancelling it midway, thus leaving the database in a pretty messy state. So in the end the only reliable information I had was the name of the image files. I then hacked up a little script to correct this:


# Set your photo path here

sqlite3 photo.db "select filename from PhotoTable" |
	while read file; do
		basename=$(basename "$file")
		fullname=$(find $photopath -name "$basename")

		if [ -n "$fullname" ]; then
			echo "Updating $fullname"
			sqlite3 photo.db "update PhotoTable set filename='$fullname' where filename='$file'"

So those who want to move their shotwell photo databases to another location, here's one (fairly roundabout) way to do this. Enjoy!


FOSS.in day 2 - The Fedora miniconf day

I concluded my previous post yesterday, on my way to FOSS.in to attend day two. Before I left, I saw Gopal saying w00t! to Pradeepto on twitter and wondered what was going on. Turns out, Pradeepto was coming to FOSS.in!

When I arrived at the venue, I found that we had (once again) captured a table in the Expo area. We spread out the DVDs and buttons and waited for people to come over. Saleem was obsessed about spreading the DVDs in a perfect line and arranging the buttons to form the 'f' of Fedora. of course, people were constantly picking the buttons from the 'f' and ruining his design. Fedora goodies were a hit on the first day and we had quite a few people picking them up on the second day too. There were a few people asking for 64-bit DVDs; we might have to note that in future conferences and hopefully till then the flash nonsense will sort itself out. to me, that seems to be the only hurdle right now for a completely trouble-free installation of a 64 bit system.

We had a few people coming over asking for help with their installations and some of us helped out till lunch time. I then met Pradeepto and Sujith along with a bunch of KDE folks for lunch. Post-lunch, we had the Fedora miniconf.

The miniconf was sort of not very well timed for me, since I wanted to attend artagnon's talk on the git object store. Rahul and Amit Shah kicked off the proceedings and Amit gave the first talk of the session. Rahul announced the workouts -- the Fedora for Kids spin by Aditya Patawari and Fedbura Bugzapping by /me. I went up to the first floor and waited for prospective contributors. Two people came in, fish_sticks and jijo. I walked them through setting up their Fedora account, bugzilla account and the sign-up for bugzapping. I also walked them through triaging bugs and the different scenarios they would face. They then started setting up virtual machines so that they could reproduce bugs in emacs. Finally, we could not get much done at all, since neither of them could get their virtual machines up and running in the time we had. Both were running Ubuntu -- it's kinda hard reproducing Fedora bugs in Ubuntu ;)

While this was going on, I was also discussing packaging with some college students from Vellore. Their college had a CMS that students had designed and they were presenting it in the FOSS.in expo. I think it was Pragyaan CMS; I can't remember the name. I volunteered to help thm get started in packaging it for Fedora. I did not want to do it myself since I personally do not have any use for a CMS. They were again trying to set things up to try out a small test rpm package using a tutorial. That was kinda hard going though since they too were using Ubuntu.

In the end the best I could do was point them to some documentation and give some tips on how they could get started and where they could get help. I think they finally joined Aditya's workout, which seemed to be a very crowded affair -- possibly the most popular workout in FOSS.in. I'm not sure how it went though, since I was busy talking to Philip and artagnon later. Oh, I had another person approach me for bugzapping when I had wrapped up and picked up my stuff -- he was a Fedora user for a change ;) We could not really sit down and work on something at that point since it was a little late, so we talked about how he could sign up and start contributing. I hope he finally does sign up for bugzapping.

The day ended with a security talk by James Morris. I made a quick exit after that since I had promised my wife that I will have dinner with her.

I will not be attending day 3, since I have a few things to wrap up before I leave for a week long holiday in north India tomorrow. It's unfortunate that I will be missing Aanjhan's closing keynote as a result. Like the last FOSS.in, I will remember this event for friends -- all the people you get to meet only once a year as well as new people you meet and exchange ideas with. It was definitely not as crowded as last year though. Too bad this will probably be the last FOSS.in, but I hope there is a similar conference somewhere similarly accessible next year.


FOSS.in Day 1

So I made it to what apparently is the last FOSS.in. It was late as usual and so I did not miss much by reaching late. There was a bit of confusion about big grown men like myself coming in as student delegates, but that was cleared out fairly quickly when one of the organizers explained to the volunteers that miniconf speakers were to be admitted as student delegates. I am to conduct a workout today on bugzapping in Fedora today (day 2).

The beginning of FOSS.in was definitely less emphatic than before. Kishore Bhargava had replaced Atul Chitnis with the kick-off ceremony. That was quickly compensated however, by a really interesting talk by Danese on Wikipdia and its technical architecture. There was talk of an Indian mirror. That will be a very cool thing if it materializes. I'm sure one of the IITs can set aside bandwidth and hardware for them.

The wikipedia talk was followed by two completely technical talks; on page cache optimization in VMs by balbir Singh and the other by Lennart Poettering on his latest interest -- systemd. Balbir Singh's talk was quite interesting, where he ex[plained the problem of page cache duplication between guests and hosts in a VM environment. He went on to explain a few possible approaches to solving this. Lennart's talk didn't give anything new for me since I had been following the Fedora devel discussions.

Me, Shreyank and Rahul Sundaram then spent some time pulling artagnon's leg. We went back to the main hall in time to see Philip Tellis finish his talk. Rahul then followed that with his talk on failures in Fedora and what we learned from them.

We finally moved on for our Fedora dinner. Dimitris Glezos wanted to have kababs and we were told of a place where we could get good kababs. That was good, except for the fact that we didn't know where it was. After running around Forum mall looking for the place, we finally settled for Firangi Pani inside Forum mall. All that most of us ended up having was drinks and starters (kababs of course). Lots of them. That accompanied with Lennart and Dimitris discussing how they could get translations from Transifex to systemd with all of the commit log intact. We'll probably hear more about it once they actually reach a conclusion on that.

Now I'm sitting in the bus writing this post and preparing for the workout that I will be doing today. I hope I get a few bugzappers interested today.


Back to the OS class: Memory Allocation

A lot of us in India learn OS concepts from textbooks. The concepts really go directly from the textbooks into examination papers for most, including me. We would grab on to key phrases like "semaphores", "paging", "segmentation", "stack" and so on, and never really stop to wonder how this all is *really* implemented. Some of us aren't interested since all we care about is getting a well paying job while others find goofing around to be a more attractive alternative. Either way, very few really get it.

Four years since I finished college, six since I last took an OS class, and I can say now that I finally got it. Somewhat.

Recently there was a very interesting case I hit upon, which got me wondering how memory allocation was managed by the workhorse of memory allocation, the malloc() function. The malloc function is implemented in the glibc library for Linux (and also for other Unix systems if you want). Its major responsibility (along with its friends, free, mallopt, etc.) is to do accounting of memory allocated to a process. The actual *giving* and *taking* of memory is done by the kernel.

Now the fun part is that on a typical *nix system, there are two ways to request memory from the OS. One is using the brk() system call and the other is by using the mmap() system call. So which one should glibc use to allocate memory? The answer is, both. What malloc does is that it uses brk() to allocate memory for small requests and mmap() for large requests.

So what is the difference between mmap and brk you ask? Well, every process has a block of contiguous memory called the data area. The brk() system call simply increases one end of the data area and hence increases size, allocating memory to the process. To free, all it does is decrease the same end of the data area. This operation is quite fast most of the time. On the other hand, the mmap system call picks up a completely different portion of memory and maps it into the address space of the process, so that the process can see it. Additionally, the mmap call also has to put zeroes in the entire memory area it is about to allocate so that it does not end up leaking information of some old process to this one. This makes mmap quite slow.

So why have mmap at all if it is so slow? The reason is the inherent limitation that brk() has due to the fact that it only grows one way and is always contiguous. Take a case where I allocate 10 objects using brk and then free the one that I had allocated first. Despite the fact that the location is now free, it cannot be given back to the OS since it is locked by the other 9 objects. One way that malloc works around this is by trying to reuse these free spaces. But what if the size of the object I am about to allocate next is larger than any of these freed "holes"? Those holes remain and the process ends up using more memory than it really needs. This is "internal fragmentation".

So to minimize the effect of this internal fragmentation, glibc limits allocation of small objects to brk(). Larger objects are allocated with mmap(). A threshold was set at 128KB, so objects smaller than it are allocated using brk and anything larger is allocated using mmap. The assumption is that smaller object requests would come more often, so the little fragmentation is worth the improvement in speed. Oh, and as for the reuse of the memory holes, it does the "best fit algorithm" -- remember that phrase? ;)

But with more recent versions of glibc (around 2006 actually, so not *very* recent), this threshold limit is dynamic. glibc now tries to adjust to the memory allocation pattern of your program. If it finds that you are allocating larger objects and freeing them soon, it will then increase the threshold, expecting you to allocate larger objects and free them more often. There is of course an upper limit of 32 MB to this. So anything larger than 32 MB will *always* be allocated using mmap. This is quite awesome since it speeds up malloc quite a bit. But it obviously comes with the price of potentially larger memory holes.

There is so much more to this, like the actual details of the way accounting of the brk'ed memory is done, obstacks, arenas. The fun seems to be only beginning.


Yahoo chat room support now in libyahoo2 trunk!

Yahoo chat room support is now in trunk. Many thanks to Kai (Kay) Zhang for this effort, which he made as part of his Fedora Summer Coding project. The code still needs more testing, so I also need to start working on the chat room implementation in ayttm.

I expected git-svn to commit the patchset along with the history in Kay's git repository, but unfortunately that did not happen. So those who want to see the history of commits for this may take a look at his libyahoo2 github repository


libyahoo2 to get chat rooms soon

Kay has been working on the libyahoo2 project as part of the Fedora Summer Coding initiative. He's been working on chat room support and things are looking quite good so far. Kay has finished working on the core functionality of logging in, joining and leaving rooms; only the chat room list functionality is remaining. He was a little shy of interacting with libyahoo2 upstream earlier, but he has been working on it since.

Looking at the pace of the project so far, we could have Kay's code merged into upstream very soon. Following this, me or Ray van Dolson will do a release in Fedora.