systems

In our previous post Memory Management Bugs: An Introduction, we discussed common errors when dealing with manual memory management. These types of errors are some of the most time-consuming and difficult to identify and resolve. At Backtrace, we’ve built automated analysis and classification into our platform to help highlight important signals and reduce the pain associated with these types of errors and more. This post introduces Backtrace’s memory allocator analysis and highlights use-cases it serves that existing technologies do not....

Many languages provide the ability to manually allocate and deallocate memory. For some workloads, this level of control over memory management enables superior memory utilization and performance. However, manual memory management comes at the cost of enabling a wide-class of bugs involving memory safety. Languages such as C and C++ are infamous for allowing difficult-to-debug issues involving mis-use of dynamic memory allocation. High-level languages are still susceptible to these classes of bugs if they are interoperating with system libraries that are built on unmanaged languages....

Backtrace is an end-to-end debugging platform that software engineering and support teams use to improve software reliability by automating and enhancing the error management process. We are pleased to announce first-class support for Go with the latest release of our platform. With Backtrace, discover where goroutines were created and why they might be waiting; capture the contents and wait queues of channels; get deep insights into the runtime scheduler and, with an upcoming release, the garbage collector....

ELF security is somewhat esoteric, and the related vulnerabilities are often very interesting. Today we will be discussing a security issue that lives within the ELF loading code of unpatched IllumOS kernels, and is not specific to any given architecture. The vulnerability may very likely extend to other operating systems which support ELF, and have legacy roots. The Backstory During testing of a new product feature, I discovered that the IllumOS kernel did not protect suid executables against $ORIGIN variable expansion, providing a vector for privileged code execution....

Symbolic debuggers are one of the most important tools in the programmer’s toolkit, but also one of the most overlooked pieces of technology. They have to work in some of the harshest conditions, supporting a huge set of programming languages and aggressive transformations by compilers. What makes them work? And when don’t they work? In this presentation, we will take you on a journey to some of the darkest and most confusing pits of systems programming involving debug formats, compilers and process control....

We are happy to announce the first release of Coresnap, a suite of tools that intercept and aggregate coredumps as they occur on Linux and FreeBSD systems. With Coresnap, both operations and software engineers benefit from having a holistic view of the state of faults on their systems and across their environments. Backtrace assistive debugging analyzes these dumps to make sure the state most relevant to the fault is not missed by incident responders and engineers....

At Backtrace we built and are continually building security and forensics features into our product that rely on understanding the structural nuances of ELF binary internals, and process memory infection techniques. This article outlines some of the core concepts that are being applied in our technology today. For well over a decade attackers have been installing memory resident backdoors, rootkits, and parasites of various kinds into userland processes. The goal is to inject executable code into an existing process, to alter its functionality, while remaining stealth and keeping the attackers activity surreptitious....

A core represents the state of a process at a point in time. It contains all the information that an engineer needs in order to inspect the process and its state even after the process has exited. This information includes thread information, mapped memory, register state and more. By using a debugger with the core file, engineers can interact with and inspect the state of the process as if they had attached a debugger to the process at the time when the core file was generated....

This article introduces a general technique for achieving single-writer non-blocking hash tables at low to negligible cost. The resulting hash table requires no barriers (fences) or locked instructions on architectures such as x86/x86-64. Read operations are lock-free and write operations are fast and bounded. Insertion and deletion are wait-free. Probe sequence mutation is linearized for the common case, bounded and fast. Architectures with relaxed memory models still require barriers. Non-blocking data structures and their benefits often come at the cost of increased latency because they require additional complexity in the common case....

A few months ago, a friend noted that they saw a significant increase in the time required to read /proc/<pid>/maps in Linux due to a change from a few years ago. A patch was introduced to the Linux kernel in 2012 (> 3.2) that marked thread stacks in /proc/<pid>/maps output. Previously, these regions were indistinguishable from other anonymous memory. Unfortunately, with this additional output there is a high cost for applications that utilize maps....