Profiler[1] - You can have your PIE and eat it too!
You did read part 0, right?
tl;dr, position-independent executables (PIE) plus address space randomization (ASLR) threw off my profiler’s ability to convert function call addresses to human readable function names, file names and line numbers.
The Solution(s)
Well, there are a few easy ways out of this problem that wouldn’t require any changes to my profiler at all.
We could disable ASLR as described here:
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space
Or, we could prevent gcc from building PIE with -no-pie:
gcc -g -no-pie test.c
But, we don’t want the easy way out! Can we just have the profiler support PIE?
To support PIE, we need to know where the kernel decided to randomly place our executable each run. Thankfully, Linux provides a huge amount of information about each process in the /proc psuedo-filesystem. This is a special filsystem that the kernel presents to userland through the /proc mount point.
I highly suggest you checkout man proc for lots of juicy details.
There is a directory under /proc for every process where the directory name is the pid. There is also a special directory called “self” which is a symlink to a process’s own /proc/[pid] directory.
ls -lash /proc/self
0 lrwxrwxrwx 1 root root 0 Sep 7 18:36 /proc/self -> 17895
In that output, 17895 is actually the pid of the ls command itself.
The file that interests me the most is /proc/[pid]/maps. It contains a list of each mapped memory region in a process’s virtual address space.
We can use the same idea to peek at the maps file.
cat /proc/self/maps
562d32a8e000-562d32a96000 r-xp 00000000 08:02 16121892 /bin/cat
562d32c95000-562d32c96000 r--p 00007000 08:02 16121892 /bin/cat
562d32c96000-562d32c97000 rw-p 00008000 08:02 16121892 /bin/cat
562d34028000-562d34049000 rw-p 00000000 00:00 0 [heap]
7f63f39a9000-7f63f3d87000 r--p 00000000 08:02 3793644 /usr/lib/locale/locale-archive
7f63f3d87000-7f63f3f6e000 r-xp 00000000 08:02 13172799 /lib/x86_64-linux-gnu/libc-2.27.so
7f63f3f6e000-7f63f416e000 ---p 001e7000 08:02 13172799 /lib/x86_64-linux-gnu/libc-2.27.so
7f63f416e000-7f63f4172000 r--p 001e7000 08:02 13172799 /lib/x86_64-linux-gnu/libc-2.27.so
7f63f4172000-7f63f4174000 rw-p 001eb000 08:02 13172799 /lib/x86_64-linux-gnu/libc-2.27.so
7f63f4174000-7f63f4178000 rw-p 00000000 00:00 0
7f63f4178000-7f63f419f000 r-xp 00000000 08:02 13172770 /lib/x86_64-linux-gnu/ld-2.27.so
7f63f436b000-7f63f436d000 rw-p 00000000 00:00 0
7f63f437d000-7f63f439f000 rw-p 00000000 00:00 0
7f63f439f000-7f63f43a0000 r--p 00027000 08:02 13172770 /lib/x86_64-linux-gnu/ld-2.27.so
7f63f43a0000-7f63f43a1000 rw-p 00028000 08:02 13172770 /lib/x86_64-linux-gnu/ld-2.27.so
7f63f43a1000-7f63f43a2000 rw-p 00000000 00:00 0
7ffcd8283000-7ffcd82a4000 rw-p 00000000 00:00 0 [stack]
7ffcd82c6000-7ffcd82c9000 r--p 00000000 00:00 0 [vvar]
7ffcd82c9000-7ffcd82cb000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
The first 3 entries show us where /bin/cat was randomly placed in memory. So, let’s take a closer look at those rows. (I’ve added column names for convenience):
address perms offset dev inode pathname
562d32a8e000-562d32a96000 r-xp 00000000 08:02 16121892 /bin/cat
562d32c95000-562d32c96000 r--p 00007000 08:02 16121892 /bin/cat
562d32c96000-562d32c97000 rw-p 00008000 08:02 16121892 /bin/cat
Each entry for /bin/cat has a different set of permissions. Notice the first mapping has x or execute permissions. This is the most important mapping for my profiler because it’s where all the machine code for the functions reside. The addresses I’m trying to pass to addr2line fall within this range.
As for the other two mappings, one is read-only which contains any constants in the executable. The other one has read/write permissions and contains mutable variables like globals.
Now, not only does /proc/[pid]/maps show us where the executable /bin/cat was mapped in memory, but it also shows where library files were mapped.
7f63f3d87000-7f63f3f6e000 r-xp 00000000 08:02 13172799 /lib/x86_64-linux-gnu/libc-2.27.so
Given an address and the /proc/[pid]/maps file, we should be able to find which binary (executable or library) the address resides in and convert the address to be relative to that binary. Then, you can pass the relative address and the correct binary to addr2line and get human readable FUNCTION:FILE:FILENO information!
Let’s do it once by hand with this simple program:
#include <stdio.h>
int main()
{
printf("%p\n", main);
printf("Press any key to exit...\n");
getchar();
return 0;
}
Compile and run it (don’t forget to compile with debug information):
gcc -g test.c
./a.out
0x5631e31086da
Press any key to exit...
The executable will continue running until we press a key, which gives us time to check the /proc/[pid]/maps file for the running a.out.
cat /proc/$(pidof a.out)/maps
5631e3108000-5631e3109000 r-xp 00000000 08:02 11714756 /home/kevin/a.out
5631e3308000-5631e3309000 r--p 00000000 08:02 11714756 /home/kevin/a.out
5631e3309000-5631e330a000 rw-p 00001000 08:02 11714756 /home/kevin/a.out
5631e4b64000-5631e4b85000 rw-p 00000000 00:00 0 [heap]
7f3e7d188000-7f3e7d36f000 r-xp 00000000 08:02 13172799 /lib/x86_64-linux-gnu/libc-2.27.so
7f3e7d36f000-7f3e7d56f000 ---p 001e7000 08:02 13172799 /lib/x86_64-linux-gnu/libc-2.27.so
...
The address our program printed was 0x5631e31086da which falls within the first mapping. That makes sense because it’s the executable mappping.
We can convert the virtual address to be relative to the executable binary by subtracting the mappings starting address from the address we printed.
printf "0x%x\n" $(( 0x5631e31086da - 0x5631e3108000 ))
0x6da
Then, we can pass that relative address to addr2line:
addr2line -fse a.out 0x6da
main
test.c:3
And, voila! We took a randomized virtual address from a position indepenent executable and got something human readable from addr2line.
This is exactly the approach I switched over to for my profiler. A program being profiled now automatically saves the contents of /proc/[pid]/maps to a metadata file on disk when it first starts. The metadata file is used later by the profiler to map the addresses.
It can now map address to both executable and library files (aassuming they have debug information). It was never able to do that for libraries before!
Future Improvements
This is a huge improvement over what I had, but there are still some shortcomings. I only store the /proc/[pid]/maps file once when a program starts. However, not all shared libraries are loaded when a program starts.
For example, dlopen() can be used to load a shared library at any point during execution. Once the library is loaded, it will appear in /proc/[pid]/maps, but at that point it’s too late.
Food for thought!