Weblog on robot bachelor paper - spring 2002

Real-time image processing of an iPAQ based robot (iBOT)

Steffen Larsen, Steffen Nissen & Sidsel Jensen

zool@diku.dk, lukesky@diku.dk & purps@diku.dk

University of Copenhagen, Department of Computer Science

NONE looked in since 05-08-2002
We turned in a small description of our paper.
You can read the full report in pdf format here.

The equipment we are gonna need for the project are the following :

1 COMPAQ iPAQ 3660 - sponsored by COMPAQ !
1 PC Card Expansion Bay
1 Serial Cable
1 LEGO Mindstorms set
1 Winnov VideumCam Traveler

We received the promised iPAQ 3660 from COMPAQ and installed the Linux BootLoader on it.
Here is a link to how that is done - www.handhelds.org. Its a step by step guide.

We also did a small presentation of the project in the robotics study group

We met at zools place, and started building the LEGO robot itself.

Here is a quick prototype, tested with weight...

Here is the almost "finished" robot. It still lacks the camera.

Since then we have strengthened the wheels a bit, the axis is very small compared to the size of the wheels.

We need 3 engines in all. One for each side of wheels, so they can be controlled independently of each other and one for the rotation of the camera.

Today we received the camera :-).

We also borrowed a LEGO Mindstorms Remote Control (thanx Rasmus !), that we are going to use, to record some of the infrared commands send to and from the robot.

Today we received the two missing parts !

We received the COMPAQ Serial Cable, which is needed for transferring Linux onto the iPAQ, and we received the PC Card Expansion Pack.

We hurried to get the Linux Familiar Distribution installed on the iPAQ. We chose to install version 0.5.1. Installation instructions can be found here. Please notice that the instruction is for the 36xx models ONLY !

Here you see it booting up for the first time.

Unfortunately is doesnt automatically install X on the iPAQ. For that you need to install a package called "task-x : All the required bits to get a working X server, and clients up", but this requires network access. So currently the process is stalled here untill we get hold of a PCMCIA network card, which shouldn't take too long :).
Today we did A LOT !

We started out with the mission of getting X on the iPAQ, and succeeded, after some troubles, since we couldn't get the Xircom PCMCIA netcard working :)

We ended up manually installing all the packages :/. You can find a list of the packages installed here. You can install a new package by using the Zmodem capabilities of your serial console terminal program and executing the command : "ipkg install 'package name'" once the program is downloaded. You can find an on-line list of all the packages available at :


Here you can see a picture of VIM starting up for the first time.
"You need the right tools to get the job done ;o"

We upgraded the bootloader to v.2.18.16 instead of the v.2.18.01 we had.
This was primarily because of a problem we had with the iPAQ. It wouldn't recognize the COMPAQ Expansion Bay - better known as the sleeve.

After working on this for a while, we decided it was time to get some help. So we went to irc.openprojects.net. Here are two IRC channels on which most of the iPAQ Linux Developers are. The two channels are called #familiar and #handhelds.org.

Since this was obviously a bug, we submitted a bug report to one of the developers, and soon after, we received a new h3600_generic_sleeve.o (thanks to Jamey Hicks), which should replace the old one in the directory "/lib/modules/2.4.16-rmk1/kernel/drivers/pcmcia". It worked like a charm, and suddenly the cardctl returned valid data:
 sh-2.03# cardctl ident
Socket 0:
  product info: "Winnov", "VideumMPC", "Rev 1.0", ""
  manfid: 0x01a5, 0x0000
  function: 254 ((null))
Socket 1:
  no product info available
If you have the same problem, the symptoms are the following :
sh-2.03# run_sbin_hotplug:78 * hotplug_path=/sbin/hotplug
h3600_sleeve_eject-9- sleeve_dev->driver=00000000 sleeve removed
run_sbin_hotplug:78 * hotplug_path=/sbin/hotplug
h3600_sleeve_eject-9- sleeve_dev->driver=00000000 sleeve removed
h3600_sleeve_insert: vendorid=0xffff devid=0xffff
h3600_sleeve_insert: unrecognized sleeve
h3600_sleeve_insert: *** Error - sleeve insert without sleeve remove ***
h3600_sleeve_insert: vendorid=0xffff devid=0xffff
h3600_sleeve_insert: unrecognized sleeve
Which basically says, that if you remove and reinsert the sleeve, it says it dosn't recognize it. Please notice the vendorid = 0xffff ! This PC Card Expansion Bay, is a single slot PCMCIA sleeve from COMPAQ.

The video card is almost working !
You need to look at this page http://pads.east.isi.edu/ipaq.htm, to find the installation guide.
We installed this package : "winnov-modules-2.4.16-rmk1_hh15_arm.ipk"

We found a really good small explanation as to how to rotate the display to portrait or landscape http://www.handhelds.org/pipermail/familiar/2001-June/000396.html
Here is a small picture of the iPAQ running X (blackbox window manager):

We have also rebuild the robot a bit. Here is the newest picture of it :

Sidsel, played around with the camera, and this resulted in the first visible pictures taken from the iPAQ:

Today, we started working on a cross-compilation environment.
We need this for our development environment.

We found a pre-compiled gcc for the iPAQ at the heandhelds site.
We successfully translated the vcat and vctrl programs for the video device, but we had trouble translating the xcaptest program, which involved some X11 components.

We successfully compiled the program xcaptest for the iPAQ.
The program shows the capture of the camera on the LCD screen of the IPAQ.

We also signed up for this year RoboCup contest held at DTU.

Today we defended our synopsis, describing the project. You can find our slides HERE.

We also modified the robot a bit, so now the camera is mounted.

It seems that we are going to run into some problems with lighting.
If the sun is shining the pictures get VERY bright.

We met at luke's place and started working on the LegOS and IR part.

First obstacle was to get legOS on the RCX. Fortunately there is a good guide HOWTO do that on Debian Linux. Second obstacle is to get some form of Infrared Communication between the IR tower and the RCX and between the RCX and the iPAQ.

We also decided that being able to rotate the camera on a motor is going to be too big a challenge, so we will modify the robot, so that the camera is firmly mounted, and then just turn the robot itself, in order to see in different directions.

Otherwise we might have a problem of centering the camera.

We met at luke's place, and continued working on the Infrared communication problem.
We found out, that there is no need for LegOS anyway. The things we wanna make the RCX do, are so simple that we might as well do it in the LEGO language. -- So we did.

We programmed a small prog, that accept message 1, 2 and 3 on the infrared.

Message 1 makes the engines stop
Message 2 makes the engines go forward
Message 3 makes the engines go backwards
anything hitting the two touch sensors on the front, sends the message 1 or 2 out through the iR.

We can now control the RCX with the LEGO Remote Control (available through the Ultimate Accessories Kit)

Today we met at the Robotics study group and heard a little bit about the other robotics projects.
We also gave a status about our own project.

We were given 2 references to two articles about an edge detection algorithm, that might be really good for us to use.

The articles are : "International Journal of Computer Vision, nr. 30(2), 1998 : Tony Lindeberg - Feature Detection with Automatic Scale Selection"

and "Tony Lindeberg - Edge Detection and Ridge Detection with Automatic Scale Selection"

We met at luke's place and continued working on iR. It finally works !

You can read a lot more about the LEGO iR device at :

Basically the format is as follows :

0x55 0xFF 0x00 D1 ~D1 D2 ~D2 ... Dn ~Dn C ~C

The header is "0x55 0xFF 0x00". Next comes the packet data and the complentary of the data. The packet ends with a Checksum and the complementary of the checksum. The checksum is very simple, since its the sum of the packet data eg.
C = D1 + D2 + D3 + ... + Dn

If you want to send the message of say 12, the packet looks like this :
55 ff 00 f7 08 12 ed 09 f6 

The header. F7 indicates that the next part is a message.
08 indicates that it is a 8 bit message.
Next comes the message number eg. 12 and the complentary of 12
09 and f6 is the tail of the packet - the checksum.

But, each byte is NOT 8 bits. Its 11 bits, since it has a start bit, a stop bit and a parity bit.

The iR device works at 38kHz and has a pulse of 417 us (1/2400 baud)

We ended up getting it to work with lirc - Linux InfraRed Remote Control - which is a universal remote control program.
Unfortunately all "normal" remote controls has a pause in its signal, so you can identify if each button pressed more than once.
The RCX discards a message, if the same message is called twice or more. Also and more importantly - it doesnt pause between messages.

This gave us a real headache - until we realized the problem.

The LIRC tools thought the pause was the 0xFF of the header, since it was a long silence.
This resulted in, that when we used the automatically generated config file, it only sent some of the header. It never sent the real data !
Once the problem was identified, we changed the config file to use raw_codes, and used the program mode2, to record a full message sent from the RCX. And finally it worked !

We can now use the lirc tool irexec - to execute programs and certain commands, once we receive a certain message on the iR device.
So we can react to data coming from the RCX, but more importantly we can remote control the RCX from the iPAQ.

Today was a long day. We did a lot of work.
We finished the program located on the LEGO RCX.
We also met with the robotics studygroup to give a small status.

Afterwards we went to Lyngby to see the premisis of the DTU RoboCup 2002 contest.
We took some pictures, that we are going to use for the image processing.
We are particullary interested in the lighting conditions in the building.

The good news : We can easilly go down (and up) the ramp.
The bad news : We cannot go down the stairs.
You can see the pictures from our excursion here.

Today we started working seriously on the real-time image processing of the robot.

We added a 2 dimensional Gaussian blur function to the code.
It works with a spreading of 8 and a 15x15 kernel, which is folded over the image.

We found a really good explanation for how Gauss is done at this URL : http://www.dai.ed.ac.uk/HIPR2/gsmooth.htm

Today we journeyed into a mathematic hell.
None of us have taken any serious mathematic courses and none of us have had any introduction to Computer Vision courses, so we were on our own.

We were trying to implement a ridge detection algorithm from Tony Lindebergs article, but we ran into all sorts of trouble understanding the mathematics.

At the end of the day we did have the basics right though.

The ridge detection works as a edge detector, thus making us able to see the white and black tape.
A local minima is equal to black tape, and a local maxima is equal to white tape.

However these calculations use a lot of floating point arithmetic, and the iPAQ is not too good at computing those.
So we are gonna need a lot of optimization, to keep our frame rate up at a decent level.

These articles/lecture notes helped us a lot :

Yet another very long day...

Finally we got something usable out of our long labour -
We can now identify a white line and a black line. *wheee*

The pictures are (from left): The original picture, The picture after adding the Gauss filter, the picture indicating with dots where the tape is. White dots indicate a black ridge or edge and black dots indicate a white line or ridge.

Basically some very long debugging days !

Today was the pre-qualification for RoboCup.

Unfortunately our robot decided it wanted to do the exact opposite, of what we wanted!
Hopefully things will run a lot better on Wedensday !

Here are some pictures from today.

Oh, and a journalist came by and asked us a lot of questions.
It was later published on this website : www.sletten.dk

RoboCup 2002 is over. *sniff*

Its was really fun, even though our robot didn't perform to good.
We managed to get in the news though. You can see the clip here (best viewed with real player)

You can see results from the competition here.
You can watch some pictures from the day here

We turned in the bachelor report. You can find it here :
  • Report in pdf format (986KB)

    Images will be availble from here.