Building a Go Debugger

Earlier this year we published a post titled Implementing A Debugger: The Fundamentals. This post gave an overview of debuggers, what they do, and how they work. In today’s post, we build upon this knowledge and talk about our journey of extending Backtrace’s debugger to support Go. Intro If you have the time and haven’t read Implement A Debugger: The Fundemantals, we highly recommend you do so. This post builds upon terms and knowledge discussed there....

Post-Mortem Analysis: Stale Pointer Detection

We’ve previously posted a few blogs covering memory management, memory bugs, and post-mortem memory analysis. Now, we are excited to add another memory debug tool to Backtrace’s offering: stale pointer analysis. In this post, I will explain how this analysis works, its strengths, and its limitations. Introduction Among many memory errors, stale pointer is one of the subtlest and most difficult to debug. It happens when a memory region is freed or reallocated but the old references, aka aliases, to the memory are not updated properly....

ROP exploitation detection

ROP exploitation in Linux The exploitation techniques used in stack-smashing have evolved right alongside the security mitigations that were created to prevent them. At one point in time an attacker was able to craft an exploit which overwrites the saved return address so that it points into a stack location where the shellcode was injected. This technique has been foiled by various security mechanisms over the years, but it was primarily DEP (data execution prevention) that created the necessity for an exploit to return somewhere other than the stack....

Post-Mortem Heap Analysis: TCMalloc

If you read our previous posts of Memory Management Bugs: An Introduction and Post-Mortem Memory Debugging, you have an idea how Backtrace may help debugging various memory errors. In this post, I will illustrate how it works under the hood with the example of TCMalloc, one of the memory allocators supported by Backtrace. TCMalloc Internals The primary goal of TCMalloc is high performance of memory allocations, especially for memory intensive applications in multi-threaded environment....

Performance Improvements for FreeBSD kernel debugging

We previously explored FreeBSD userspace coredumps. Backtrace’s debugging platform supports FreeBSD kernel coredumps too, and their traces share many features. They are constructed somewhat differently, and in the process of adding support for them, we found a way to improve performance for automated programs accessing them. Read on to learn how information is extracted from a FreeBSD kernel core, and how we improved performance for this mechanism. Generating kernel core files A kernel core is typically only generated in exceptional circumstances....

Implementing a Debugger: The Fundamentals

This is the first of a two-part series describing the implementation of a generic debugging tool. Part one covers the core internals of a debugger; part two will focus on extending a debugger to support a specific programming language – Go. Implementing a debugging tool may seem like a monumental task. gdb, one of the most popular debuggers, is over 500,000 SLOC (according to cloc); it’s not exactly easy to pick up and read through, and it has many auxiliary features not fundamental to a debugger’s purpose....

Post-Mortem Memory Debugging

In our previous post Memory Management Bugs: An Introduction, we discussed common errors when dealing with manual memory management. These types of errors are some of the most time-consuming and difficult to identify and resolve. At Backtrace, we’ve built automated analysis and classification into our platform to help highlight important signals and reduce the pain associated with these types of errors and more. This post introduces Backtrace’s memory allocator analysis and highlights use-cases it serves that existing technologies do not....

Memory Management Bugs: An Introduction

Many languages provide the ability to manually allocate and deallocate memory. For some workloads, this level of control over memory management enables superior memory utilization and performance. However, manual memory management comes at the cost of enabling a wide-class of bugs involving memory safety. Languages such as C and C++ are infamous for allowing difficult-to-debug issues involving mis-use of dynamic memory allocation. High-level languages are still susceptible to these classes of bugs if they are interoperating with system libraries that are built on unmanaged languages....

Announcing First-Class Go Support

Backtrace is an end-to-end debugging platform that software engineering and support teams use to improve software reliability by automating and enhancing the error management process. We are pleased to announce first-class support for Go with the latest release of our platform. With Backtrace, discover where goroutines were created and why they might be waiting; capture the contents and wait queues of channels; get deep insights into the runtime scheduler and, with an upcoming release, the garbage collector....

Exploiting ELF Expansion Variables

ELF security is somewhat esoteric, and the related vulnerabilities are often very interesting. Today we will be discussing a security issue that lives within the ELF loading code of unpatched IllumOS kernels, and is not specific to any given architecture. The vulnerability may very likely extend to other operating systems which support ELF, and have legacy roots. The Backstory During testing of a new product feature, I discovered that the IllumOS kernel did not protect suid executables against $ORIGIN variable expansion, providing a vector for privileged code execution....