Practical Reverse Engineering Part 2 - Scouting the Firmware

  • Part 1: Hunting for Debug Ports
  • Part 2: Scouting the Firmware
  • Part 3: Following the Data
  • Part 4: Dumping the Flash
  • Part 5: Digging Through the Firmware

In part 1 we found a debug UART port that gave us access to a Linux shell. At this point we’ve got the same access to the router that a developer would use to debug issues, control the system, etc.

This first overview of the system is easy to access, doesn’t require expensive tools and will often yield very interesting results. If you want to do some hardware hacking but don’t have the time to get your hands too dirty, this is often the point where you stop digging into the hardware and start working on the higher level interfaces: network vulnerabilities, ISP configuration protocols, etc.

These posts are hardware-oriented, so we’re just gonna use this access to gather some random pieces of data. Anything that can help us understand the system or may come in handy later on.

Please check out the legal disclaimer in case I come across anything sensitive.

Full disclosure: I’m in contact with Huawei’s security team; they’ve had time to review the data I’m going to reveal in this post and confirm there’s nothing too sensitive for publication. I tried to contact TalkTalk, but their security staff is nowhere to be seen.

Picking Up Where We Left Off

Picture of Documented UARTs

We get our serial terminal application up and running in the computer and power up the router.

Boot Sequence

We press enter and get the login prompt from ATP Cli; introduce the credentials admin:admin and we’re in the ATP command line. Execute the command shell and we get to the BusyBox CLI (more on BusyBox later).

-----Welcome to ATP Cli------
Login: admin
Password:    #Password is ‘admin'
BusyBox vv1.9.1 (2013-08-29 11:15:00 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.
# ls
var   usr   tmp   sbin  proc  mnt   lib   init  etc   dev   bin

At this point we’ve seen the 3 basic layers of firmware in the Ralink IC:

  1. U-boot: The device’s bootloader. It understands the device’s memory map, kickstarts the main firmware execution and takes care of some other low level tasks
  2. Linux: The router is running Linux to keep overall control of the hardware, coordinate parallel processes, etc. Both ATP CLI and BusyBox run on top of it
  3. Busybox: A small binary including reduced versions of multiple linux commands. It also supplies the shell we call those commands from.

Lower level interfaces are less intuitive, may not have access to all the data and increase the chances of bricking the device; it’s always a good idea to start from BusyBox and walk your way down.

For now, let’s focus on the boot sequence itself. The developers thought it would be useful to display certain pieces of data during boot, so let’s see if there’s anything we can use.

Boot Debug Messages

We find multiple random pieces of data scattered across the boot sequence. We’ll find useful info such as the compression algorithm used for some flash segments:

boot msg kernel lzma

Intel on how the external flash memory is structured will be very useful when we get to extracting it.

ram data. not very useful

SPI Flash Memory Map!

And more compression intel:

root is squashfs'd

We’ll have to deal with the compression algorithms when we try to access the raw data from the external Flash, so it’s good to know which ones are being used.

What Are ATP CLI and BusyBox Exactly? [Theory]

The Ralink IC in this router runs a Linux kernel to control memory and parallel processes, keep overall control of the system, etc. In this case, according to the Ralink’s product brief, they used the Linux 2.6.21 SDK. ATP CLI is a CLI running either on top of Linux or as part of the kernel. It provides a first layer of authentication into the system, but other than that it’s very limited:

Welcome to ATP command line tool.
If any question, please input "?" at the end of command.

help doesn’t mention the shell command, but it’s usually either shell or sh. This ATP CLI includes less than 10 commands, and doesn’t support any kind of complex process control or file navigation. That’s where BusyBox comes in.

BusyBox is a single binary containing reduced versions of common unix commands, both for development convenience and -most importantly- to save memory. From ls and cd to top, System V init scripts and pipes, it allows us to use the Ralink IC somewhat like your regular Linux box.

One of the utilities the BusyBox binary includes is the shell itself, which has access to the rest of the commands:

BusyBox vv1.9.1 (2013-08-29 11:15:00 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.
# ls
var   usr   tmp   sbin  proc  mnt   lib   init  etc   dev   bin
# ls /bin
zebra        swapdev      printserver  ln           ebtables     cat
wpsd         startbsp     pppc         klog         dns          busybox
wlancmd      sntp         ping         kill         dms          brctl
web          smbpasswd    ntfs-3g      iwpriv       dhcps        atserver
usbserver    smbd         nmbd         iwconfig     dhcpc        atmcmd
usbmount     sleep        netstat      iptables     ddnsc        atcmd
upnp         siproxd      mount        ipp          date         at
upg          sh           mldproxy     ipcheck      cwmp         ash
umount       scanner      mknod        ip           cp           adslcmd
tr111        rm           mkdir        igmpproxy    console      acl
tr064        ripd         mii_mgr      hw_nat       cms          ac
telnetd      reg          mic          ethcmd       cli
tc           radvdump     ls           equipcmd     chown
switch       ps           log          echo         chmod

You’ll notice different BusyBox quirks while exploring the filesystem, such as the symlinks to a busybox binary in /bin/. That’s good to know, since any commands that may contain sensitive data will not be part of the BusyBox binary.

Exploring the File System

Now that we’re in the system and know which commands are available, let’s see if there’s anything useful in there. We just want a first overview of the system, so I’m not gonna bother exposing every tiny piece of data.

The top command will help us identify which processes are consuming the most resources. This can be an extremely good indicator of whether some processes are important or not. It doesn’t say much while the router’s idle, though:


One of the processes running is usbmount, so the router must support connecting ‘something’ to the USB port. Let’s plug in a flash drive in there…

usb 1-1: new high speed USB device using rt3xxx-ehci and address 2
++++++sambacms.c 2374 renice=renice -n +10 -p 1423

The USB is recognised and mounted to /mnt/usb1_1/, and a samba server is started. These files show up in /etc/samba/:

# ls -l /etc/samba/
-rw-r--r--    1 0        0             103 smbpasswd
-rw-r--r--    1 0        0               0 smbusers
-rw-r--r--    1 0        0             480 smb.conf
-rw-------    1 0        0            8192 secrets.tdb
# cat /etc/samba/smbpasswd
nobody:0:XXXXXXXXXXXXXXXXXXX:564E923F5AF30J373F7C8_______4D2A:[U ]:LCT-1ED36884:

More data, in case it ever comes in handy:

  • netstat -a: Network ports the device is listening at
  • iptables –list: We could set up telnet and continue over the network, but I’d rather stay as close to the bare metal as possible
  • wlancmd help: Utility to control the WiFi radio, plenty of options available
  • /etc/profile
  • /etc/inetd
  • /etc/services
  • /var/: Contains files used by the system during the course of its operation
  • /etc/: System configuration files, etc.

/var/ and /etc/ always contain tons of useful data, and some of it makes itself obvious at first sight. Does that say /etc/serverkey.pem??

Blurred /etc/serverkey.pem


It’s common to find private keys in embedded systems. They could be RSA private keys used for mutually-authenticated TLS connections with a server, variables buried in a file to be loaded by an application, etc. By accessing 1 single device via hardware you may obtain the keys that will help you eavesdrop encrypted connections, attack servers, end users or other devices in the fleet.

This key could be used to communicate with some server from Huawei or the ISP, although that’s less common. On the other hand, it’s also very common to find public certs used to communicate with remote servers.

In this case we find 2 certificates next to the private key; both are self-signed by the same ‘person’:

  • /etc/servercert.pem: Most likely the certificate for the serverkey
  • /etc/root.pem: Probably used to connect to a server from the ISP or Huawei. Not sure.

And some more data in /etc/ppp256/config and /etc/ppp258/config:


These credentials are also available via the HTTP interface, which is why I’m publishing them, but that’s not the case in many other routers (more on this later).

With so many different files everywhere it can be quite time consuming to go through all the info without the right tools. We’re gonna copy as much data as we can into the USB drive and go through it on our computer.

The Rambo Approach to Intel Gathering

Once we have as many files as possible in our computer we can check some things very quick. find . -name *.pem reveals there aren’t any other TLS certificates.

What about searching the word password in all files? grep -i -r password .

Grep Password

We can see lots of credentials; most of them are for STUN, TR-069 and local services. I’m publishing them because this router proudly displays them all via the HTTP interface, but those are usually hidden.

If you wanna know what happens when someone starts pulling from that thread, check out Alexander Graf’s talk “Beyond Your Cable Modem”, from CCC 2015. There are many other talks about attacking TR-069 from DefCon, BlackHat, etc. etc.

The credentials we can see are either in plain text or encoded in base64. Of course, encoding is worthless for data protection:

$ echo "QUJCNFVCTU4=" | base64 -D

WiFi pwd in curcfg.xml

That is the current WiFi password set in the router. It leads us to 2 VERY interesting files. Not just because of their content, but because they’re a vital part of how the router operates:

  • /var/curcfg.xml: Current configuration file. Among other things, it contains the current WiFi password encoded in base64
  • /etc/defaultcfg.xml: Default configuration file, used for ‘factory reset’. Does not include the default WiFi password (more on this in the next posts)

Exploring ATP’s CLI

The ATP CLI includes very few commands. The most interesting one -besides shell- is debug. This isn’t your regular debugger; debug display will simply give you some info about the commands igmpproxy, cwmp, sysuptime or atpversion. Most of them don’t have anything juicy, but what about cwmp? Wasn’t that related to remote configuration of routers?

debug display cwmp

Once again, these are the CWMP (TR-069) credentials used for remote router configuration. Not even encoded this time.

The rest of the ATP commands are pretty useless: clear screen, help menu, save to flash and exit. Nothing worth going into.

Exploring Uboot’s CLI

The bootloader’s command line interface offers raw access to some memory areas. Unfortunately, it doesn’t give us direct access to the Flash IC, but let’s check it out anyway.

Please choose operation:
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.
You choosed 4
Stopped Uboot WatchDog Timer.
4: System Enter Boot Command Line Interface.
U-Boot 1.1.3 (Aug 29 2013 - 11:16:19)
RT3352 # help
?       - alias for 'help'
bootm   - boot application image from memory
cp      - memory copy
erase   - erase SPI FLASH memory
go      - start application at address 'addr'
help    - print online help
md      - memory display
mdio   - Ralink PHY register R/W command !!
mm      - memory modify (auto-incrementing)
mw      - memory write (fill)
nm      - memory modify (constant address)
printenv- print environment variables
reset   - Perform RESET of the CPU
rf      - read/write rf register
saveenv - save environment variables to persistent storage
setenv  - set environment variables
uip - uip command
version - print monitor version
RT3352 #

Don’t touch commands like erase, mm, mw or nm unless you know exactly what you’re doing; you’d probably just force a router reboot, but in some cases you may brick the device. In this case, md (memory display) and printenv are the commands that call my atention.

RT3352 # printenv
ramargs=setenv bootargs root=/dev/ram rw
addip=setenv bootargs $(bootargs) ip=$(ipaddr):$(serverip):$(gatewayip):$(netmask):$(hostname):$(netdev):off
addmisc=setenv bootargs $(bootargs) console=ttyS0,$(baudrate) ethaddr=$(ethaddr) panic=1
flash_self=run ramargs addip addmisc;bootm $(kernel_addr) $(ramdisk_addr)
load=tftp 8A100000 $(u-boot)
u_b=protect off 1:0-1;era 1:0-1;cp.b 8A100000 BC400000 $(filesize)
loadfs=tftp 8A100000 root.cramfs
u_fs=era bc540000 bc83ffff;cp.b 8A100000 BC540000 $(filesize)
test_tftp=tftp 8A100000 root.cramfs;run test_tftp
ethact=Eth0 (10/100-M)

Environment size: 765/4092 bytes

We can see settings like the UART baudrate, as well as some interesting memory locations. Those memory addresses are not for the Flash IC, though. The flash memory is only addressed by 3 bytes: [0x00000000, 0x00FFFFFF].

Let’s take a look at some of them anyway, just to see the kind of access this interface offers.What about kernel_addr=BFC40000?

md `badd` Picture

Nope, that badd message means bad address, and it has been hardcoded in md to let you know that you’re trying to access invalid memory locations. These are good addresses, but they’re not accessible to u-boot at this point.

It’s worth noting that by starting Uboot’s CLI we have stopped the router from loading the linux Kernel onto memory, so this interface gives access to a very limited subset of data.

SPI Flash string in md

We can find random pieces of data around memory using this method (such as that SPI Flash Image string), but it’s pretty hopeless for finding anything specific. You can use it to get familiarised with the memory architecture, but that’s about it. For example, there’s a very obvious change in memory contents at 0x000d0000:

md.w 0x000d0000

And just because it’s about as close as it gets to seeing the girl in the red dress, here is the md command in action. You’ll notice it’s very easy to spot that change in memory contents at 0x000d0000.

Next Steps

In the next post we combine firmware and bare metal, explain how data flows and is stored around the device, and start trying to manipulate the system to leak pieces of data we’re interested in.

Thanks for reading! :)

Practical Reverse Engineering Part 1 - Hunting for Debug Ports

  • Part 1: Hunting for Debug Ports
  • Part 2: Scouting the Firmware
  • Part 3: Following the Data
  • Part 4: Dumping the Flash
  • Part 5: Digging Through the Firmware

In this series of posts we’re gonna go through the process of Reverse Engineering a router. More specifically, a Huawei HG533.

Huawei HG533

At the earliest stages, this is the most basic kind of reverse engineering. We’re simple looking for a serial port that the engineers who designed the device left in the board for debug and -potentially- technical support purposes.

Even though I’ll be explaining the process using a router, it can be applied to tons of household embedded systems. From printers to IP cameras, if it’s mildly complex it’s quite likely to be running some form of linux. It will also probably have hidden debug ports like the ones we’re gonna be looking for in this post.

Finding the Serial Port

Most UART ports I’ve found in commercial products are between 4 and 6 pins, usually neatly aligned and sometimes marked in the PCB’s silkscreen somehow. They’re not for end users, so they almost never have pins or connectors attached.

After taking a quick look at the board, 2 sets of unused pads call my atention (they were unused before I soldered those pins in the picture, anyway):

Pic of the 2 Potential UART Ports

This device seems to have 2 different serial ports to communicate with 2 different Integrated Circuits (ICs). Based on the location on the board and following their traces we can figure out which one is connected to the main IC. That’s the most likely one to have juicy data.

In this case we’re simply gonna try connecting to both of them and find out what each of them has to offer.

Identifying Useless Pins

So we’ve found 2 rows of pins that -at first sight- could be UART ports. The first thing you wanna do is find out if any of those contacts is useless. There’s a very simple trick I use to help find useless pads: Flash a bright light from the backside of the PCB and look at it from directly above. This is what that looks like:

2nd Serial Port - No Headers

We can see if any of the layers of the PCB is making contact with the solder blob in the middle of the pad.

  1. Connected to something (we can see a trace “at 2 o’clock”)
  3. 100% connected to a plane or thick trace. It’s almost certainly a power pin, either GND or Vcc
  4. Connections at all sides. This one is very likely to be the other power pin. There’s no reason for a data pin in a debug port to be connected to 4 different traces, but the pad being surrounded by a plane would explain those connections
  5. Connected to something

Soldering Pins for Easy Access to the Lines

In the picture above we can see both serial ports.

The pads in these ports are through-hole, but the holes themselves are filled in with blobs of very hard, very high melting point solder.

I tried soldering the pins over the pads, but the solder they used is not easy to work with. For the 2nd serial port I decided to drill through the solder blobs with a Dremel and a needle bit. That way we can pass the pins through the holes and solder them properly on the back of the PCB. It worked like a charm.

Use a Dremel to Drill Through the Solder Blobs

Identifying the Pinout

So we’ve got 2 connectors with only 3 useful pins each. We still haven’t verified the ports are operative or identified the serial protocol used by the device, but the number and arrangement of pins hint at UART.

Let’s review the UART protocol. There are 6 pin types in the spec:

  • Tx [Transmitting Pin. Connects to our Rx]
  • Rx [Receiving Pin. Connects to our Tx]
  • GND [Ground. Connects to our GND]
  • Vcc [The board’s power line. Usually 3.3V or 5V. DO NOT CONNECT]
  • CTS [Typically unused]
  • DTR [Typically unused]

We also know that according to the Standard, Tx and Rx are pulled up (set to 1) by default. The Transmitter of the line (Tx) is in charge of pulling it up, which means if it’s not connected the line’s voltage will float.

So let’s compile what we know and get to some conclusions:

  1. Only 3 pins in each header are likely to be connected to anything. Those must be Tx, Rx and GND
  2. Two pins look a lot like Vcc and GND
  3. One of them -Tx- will be pulled up by default and be transmitting data
  4. The 3rd of them, Rx, will be floating until we connect the other end of the line

That information seems enough to start trying different combinations with your UART-to-USB bridge, but randomly connecting pins you don’t understand is how you end up blowing shit up.

Let’s keep digging.

A multimeter or a logic analyser would be enough to figure out which pin is which, but if you want to understand what exactly is going on in each pin, nothing beats a half decent oscilloscope:

Channel1=Tx Channel2=Rx

After checking the pins out with an oscilloscope, this is what we can see in each of them:

  1. GND and Vcc verified - solid 3.3V and 0V in pins 2 and 3, as expected
  2. Tx verified - You can clearly see the device is sending information
  3. One of the pins floats at near-0V. This must be the device’s Rx, which is floating because we haven’t connected the other side yet.

So now we know which pin is which, but if we want to talk to the serial port we need to figure out its baudrate. We can find this with a simple protocol dump from a logic analyser. If you don’t have one, you’ll have to play “guess the baudrate” with a list of the most common ones until you get readable text through the serial port.

This is a dump from a logic analyser in which we’ve enabled protocol analysis and tried a few different baudrates. When we hit the right one, we start seeing readable text in the sniffed serial data (\n\r\n\rU-Boot 1.1.3 (Aug...)

Logic Protocol Analyser

Once we have both the pinout and baudrate, we’re ready to start communicating with the device:

Documented UART Pinouts

Connecting to the Serial Ports

Now that we’ve got all the info we need on the hardware side, it’s time to start talking to the device. Connect any UART to USB bridge you have around and start wandering around. This is my hardware setup to communicate with both serial ports at the same time and monitor one of the ports with an oscilloscope:

All Connected

And when we open a serial terminal in our computer to communicate with the device, the primary UART starts spitting out useful info. These are the commands I use to connect to each port as well as the first lines they send during the boot process:

Boot Sequence

Please choose operation:
   3: Boot system code via Flash (default).
   4: Entr boot command line interface.

‘Command line interface’?? We’ve found our way into the system! When we press 4 we get a command line interface to interact with the device’s bootloader.

Furthermore, if we let the device start as the default 3, wait for it to finish booting up and press enter, we get the message Welcome to ATP Cli and a login prompt. If the devs had modified the password this step would be a bit of an issue, but it’s very common to find default credentials in embedded systems. After a few manual tries, the credentials admin:admin succeeded and I got access into the CLI:

-----Welcome to ATP Cli------

Login: admin
Password:    #Password is ‘admin'

BusyBox vv1.9.1 (2013-08-29 11:15:00 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

# ls
var   usr   tmp   sbin  proc  mnt   lib   init  etc   dev   bin

Running the shell command in ATP will take us directly into Linux’s CLI with root privileges :)

This router runs BusyBox, a linux-ish interface which I’ll talk about in more detail in the next post.

Next Steps

Now that we have access to the BusyBox CLI we can start nosing around the software. Depending on what device you’re reversing there could be plain text passwords, TLS certificates, useful algorithms, unsecured private APIs, etc. etc. etc.

In the next post we’ll focus on the software side of things. I’ll explain the differences between boot modes, how to dump memory, and other fun things you can do now that you’ve got direct access to the device’s firmware.

Thanks for reading! :)

Picking a Practice Lock in Slow Motion

Some time ago I recorded a slow motion clip of myself picking a cut-away lock. This kind of lock is used for lock picking practice, as it lets you see the pins and springs that make the lock work.

In this clip we can see the racking method. Racking a lock is the Rambo approach to non-destructive lock picking: You jam a racking pick into the lock (half diamond, snowman, saw… Pretty much any pick but a hook) and move it around; you play with the force applied in the tension wrench until all the pins lock in place.

This technique works beautifully here because we don’t have any security pins in the lock. Security pins usually require a more precise approach and need to be carefully picked pin by pin.

Slow motion

Regular speed

This is the same video at real time speed. As you can see, the whole process is pretty quick.

Explaining the video

Too much pressure in the tension wrench caused the first pin to overset (the lower part of the first pin got stuck between the cilinder -the piece that turns when the lock is opened- and the case).

After reducing the force applied in the wrench we can see the pin drop and lock into position. The lock is now ready to be opened.

Emulating a USB Mouse with mbed to Cheat at CookieClicker

Some time ago, I was testing mbed’s USBMouse and USBKeyboard, and used CookieClicker for the proof of concept.

The idea of these libraries is that the microcontroller will tell the computer “Hey, I’m a mouse” or “I’m a keyboard”, and we will program it to send the key presses or movements we want. This can be used for all kinds of reasons, such as security attacks: “Hey, I’m a keyboard. Launch the terminal, execute these commands, close the terminal.”, but it can also be used for fun. Here, the mbed will identify itself as a mouse, and send tons of left clicks without any delay between them.

Thanks to the mbed’s libraries, you don’t need to configure the low level stuff for USB communication, and the code is as simple as this:

#include "mbed.h"
#include "USBMouse.h" //The library to work as a mouse

USBMouse mouse; //Declare the object mouse
DigitalIn myInput(p5); //Set an input to control when to send clicks

int main() {
    myInput.mode(PullDown); //Internal pull-down in the input pin
    while (1) { //Forever:
        if( //If the input button/switch is enabled
  ; //click

And here’s a video of the device in action:

Note: The computer does not actually lag like in the video. That was because of the screen-recording software.

For the sake of keeping this post short, I’m not going to post schematics, but it’s a really simple circuit: The switch is connected between the pin p5 and Vout, and the USB was an old USB cable cut in half, soldered to some header pins, and connected to the mbed’s USB pins as explained here.

In my quest for fun, I also wrote a program with the USBKeyboard library to send the key presses 0 to 9 time and time again. I replaced the switch with a button, started a StarCraft 2 game, created 10 random control groups, and kept the button pressed a few times during the beginning of the game. This way, we can inflate our APM (Actions Per Minute) automatically. Here is the result:

StarCraft sky high APM

I’ll have to try it again, pressing the button during much more time (That 736 APM is just the average APM for all the game), just to see how high it can go. Could it be possible to overflow a variable somewhere? We’ll see…

Non-Persistent XSS with a Curious Attack Vector

For those of you who don’t know it, “memondo network” is the spanish company behind a bunch of websites for memes, funny gifs, etc. And they have, according to alexa, a quite large amount of traffic (a couple of their sites being in the top 3000 and in the spanish top 200).

Well, a couple of days ago I found an XSS vulnerability in their search system with a curious attack vector, so let’s take a look at it:

The vulnerable pages were http://www.$(SITE).com/buscar/0/

When you tried to search something -“DEVDEV” in our example- this GET request was sent:

XSS output vector

After playing a bit with the search parameter, the first output of the value (displayed to the user) seemed to be properly filtered, but the page navigation buttons -prev, next- were not, so we should be able to inject code there:

But that injection is tricky… The vulnerable parameter is a link, and it’s being processed by the server before echoing it to make it URL-friendly, which means that any space would become %20 and any slash would screw the attack up. That implies that you could get the code executed when you decoded those values manually in the source code, but that would not be a feasible attack.

As far as i got, I could not get a way to bypass that problem, so I had to think about a different attack vector… I couldn’t use tags with parameters (because of the space between the tag name and the first attribute) and I could not inject a script because of the slash in </script>, so…. What about adding attributes to the tag being injected?

It was dirty, it was anything but subtle and it required another step of social engineering, but it did work and it might fool someone out there. Here’s a quick example of how it could be done:

Shitty example of social engineering

And when the victim’s mouse hovers over the link…

Javascript successfully injected!

And that’s as far as I got. I did not wait to have anything prettier or better and just reported it like that. I sent the email yesterday at 22.12, they were very friendly about it and it had already been fixed by 12.00 today, so good job on their part :)