Backtrace Engineering

Backtrace is an end-to-end debugging platform that software engineering and support teams use to improve software reliability by automating and enhancing the error management process. We are pleased to announce first-class support for Go with the latest release of our platform. With Backtrace, discover where goroutines were created and why they might be waiting; capture the contents and wait queues of channels; get deep insights into the runtime scheduler and, with an upcoming release, the garbage collector....

ELF security is somewhat esoteric, and the related vulnerabilities are often very interesting. Today we will be discussing a security issue that lives within the ELF loading code of unpatched IllumOS kernels, and is not specific to any given architecture. The vulnerability may very likely extend to other operating systems which support ELF, and have legacy roots. The Backstory During testing of a new product feature, I discovered that the IllumOS kernel did not protect suid executables against $ORIGIN variable expansion, providing a vector for privileged code execution....

Symbolic debuggers are one of the most important tools in the programmer’s toolkit, but also one of the most overlooked pieces of technology. They have to work in some of the harshest conditions, supporting a huge set of programming languages and aggressive transformations by compilers. What makes them work? And when don’t they work? In this presentation, we will take you on a journey to some of the darkest and most confusing pits of systems programming involving debug formats, compilers and process control....

We are happy to announce the first release of Coresnap, a suite of tools that intercept and aggregate coredumps as they occur on Linux and FreeBSD systems. With Coresnap, both operations and software engineers benefit from having a holistic view of the state of faults on their systems and across their environments. Backtrace assistive debugging analyzes these dumps to make sure the state most relevant to the fault is not missed by incident responders and engineers....

At Backtrace we built and are continually building security and forensics features into our product that rely on understanding the structural nuances of ELF binary internals, and process memory infection techniques. This article outlines some of the core concepts that are being applied in our technology today. For well over a decade attackers have been installing memory resident backdoors, rootkits, and parasites of various kinds into userland processes. The goal is to inject executable code into an existing process, to alter its functionality, while remaining stealth and keeping the attackers activity surreptitious....

A core represents the state of a process at a point in time. It contains all the information that an engineer needs in order to inspect the process and its state even after the process has exited. This information includes thread information, mapped memory, register state and more. By using a debugger with the core file, engineers can interact with and inspect the state of the process as if they had attached a debugger to the process at the time when the core file was generated....

This article introduces a general technique for achieving single-writer non-blocking hash tables at low to negligible cost. The resulting hash table requires no barriers (fences) or locked instructions on architectures such as x86/x86-64. Read operations are lock-free and write operations are fast and bounded. Insertion and deletion are wait-free. Probe sequence mutation is linearized for the common case, bounded and fast. Architectures with relaxed memory models still require barriers. Non-blocking data structures and their benefits often come at the cost of increased latency because they require additional complexity in the common case....

A few months ago, a friend noted that they saw a significant increase in the time required to read /proc/<pid>/maps in Linux due to a change from a few years ago. A patch was introduced to the Linux kernel in 2012 (> 3.2) that marked thread stacks in /proc/<pid>/maps output. Previously, these regions were indistinguishable from other anonymous memory. Unfortunately, with this additional output there is a high cost for applications that utilize maps....

Backtrace I/O is building a turn-key infrastructure platform to detect, aggregate, analyze and collaboratively fix software bugs of all types for even the most demanding software applications. We are taking a unique approach to the problem, from how backtraces are generated to how crashes are stored and analyzed. As engineers working on enterprise software, backtraces are exceptionally useful to us. In production, backtraces can provide key insights to real-world performance. In bug reports, backtraces and basic environmental data are usually the only thing engineers have to go on for determining and fixing the root cause of a crash....

At Backtrace I/O, we are setting out to improve software debugging through state-of-the-art, purpose-built technology for engineers, by engineers. Our first goal is to provide a platform that vastly improves the painful post-mortem crash analysis experience of today’s demanding applications. The team is hard at work and we are excited to share some of the fruits of our labor soon. We just completed our Series Seed with an incredible team of investors and advisors: Icon Venture Partners, Tribeca Venture Partners, 60 East Technologies, Brian O’Kelley, Jeffrey M....