systems

Join us July 23rd at 4PM ET (8PM UTC), as Djordje Todorovic joins me to discuss the state of debug information in LLVM and the work that is happening to improve it. The event will be livestreamed at Triplefault. LLVM is a popular compiler backend that is the basis for toolchain across the Apple ecosystem, popular game consoles and more. Compared to compilers such as gcc, the quality of debug information in the presence of optimizations has historically been very poor for LLVM-based compilers....

To make a long story short, it’s almost easy, now that tracing is available on commodity hardware, especially with a small library to handle platform-specific setup. Here’s a graph of the empirical distribution functions for the number of calls into OpenSSL 1.0.1f it takes for Hypothesis to find Heartbleed, given a description of the format for Heartbeat requests (“grammar”), and the same with additional branch coverage feedback (“bts”, for Intel Branch Trace Store)....

Debug information is used by symbolic debuggers and the Backtrace platform to reconcile process executable state to the source-code form that is familiar to most software engineers. This information is responsible for mapping memory addresses and register values to function names, source code locations and describing variables. Some environments choose to omit debug information. Of the many reasons, the typically valid reasons are disk utilization and arguably, intellectual property protection....

Have you ever had an assert get triggered only to result in a useless core dump with missing variable information or an invalid callstack? Common factors that go into selecting a C or C++ compiler are: availability, correctness, compilation speed and application performance. A factor that is often neglected is debug information quality, which symbolic debuggers use to reconcile application executable state to the source-code form that is familiar to most software engineers....

Earlier this year we published a post titled Implementing A Debugger: The Fundamentals. This post gave an overview of debuggers, what they do, and how they work. In today’s post, we build upon this knowledge and talk about our journey of extending Backtrace’s debugger to support Go. Intro If you have the time and haven’t read Implement A Debugger: The Fundemantals, we highly recommend you do so. This post builds upon terms and knowledge discussed there....

We’ve previously posted a few blogs covering memory management, memory bugs, and post-mortem memory analysis. Now, we are excited to add another memory debug tool to Backtrace’s offering: stale pointer analysis. In this post, I will explain how this analysis works, its strengths, and its limitations. Introduction Among many memory errors, stale pointer is one of the subtlest and most difficult to debug. It happens when a memory region is freed or reallocated but the old references, aka aliases, to the memory are not updated properly....

ROP exploitation in Linux The exploitation techniques used in stack-smashing have evolved right alongside the security mitigations that were created to prevent them. At one point in time an attacker was able to craft an exploit which overwrites the saved return address so that it points into a stack location where the shellcode was injected. This technique has been foiled by various security mechanisms over the years, but it was primarily DEP (data execution prevention) that created the necessity for an exploit to return somewhere other than the stack....

If you read our previous posts of Memory Management Bugs: An Introduction and Post-Mortem Memory Debugging, you have an idea how Backtrace may help debugging various memory errors. In this post, I will illustrate how it works under the hood with the example of TCMalloc, one of the memory allocators supported by Backtrace. TCMalloc Internals The primary goal of TCMalloc is high performance of memory allocations, especially for memory intensive applications in multi-threaded environment....

We previously explored FreeBSD userspace coredumps. Backtrace’s debugging platform supports FreeBSD kernel coredumps too, and their traces share many features. They are constructed somewhat differently, and in the process of adding support for them, we found a way to improve performance for automated programs accessing them. Read on to learn how information is extracted from a FreeBSD kernel core, and how we improved performance for this mechanism. Generating kernel core files A kernel core is typically only generated in exceptional circumstances....

This is the first of a two-part series describing the implementation of a generic debugging tool. Part one covers the core internals of a debugger; part two will focus on extending a debugger to support a specific programming language – Go. Implementing a debugging tool may seem like a monumental task. gdb, one of the most popular debuggers, is over 500,000 SLOC (according to cloc); it’s not exactly easy to pick up and read through, and it has many auxiliary features not fundamental to a debugger’s purpose....