Backtrace I/O is building a turn-key infrastructure platform to detect, aggregate, analyze and collaboratively fix software bugs of all types for even the most demanding software applications. We are taking a unique approach to the problem, from how backtraces are generated to how crashes are stored and analyzed. As engineers working on enterprise software, backtraces are exceptionally useful to us. In production, backtraces can provide key insights to real-world performance. In bug reports, backtraces and basic environmental data are usually the only thing engineers have to go on for determining and fixing the root cause of a crash. Backtraces with insufficient information quickly become useless and lead engineers to draw incorrect conclusions that lead to much wasted time and effort. Today’s tools can fail to perform for demanding applications and we would like to fix that.

In order to build the best platform for crash analysis we know we must begin with the best tracers and crash data formats. In this post we will explore some of the performance advantages of our tracers compared to other tools and announce some of our first shareware.

A major component of our platform is the advanced tracer and tracing framework, which provide features such as temporal analysis, modern crash deduplication (in cooperation with our database) and more.

The advanced tracer relies on our core backtrace library and today we’re happy to provide public access to prerelease binaries of the lightweight core library tracer bt, the same library our advanced tracer builds on. We do not expect this version of the lightweight tracer to be bug-free. You can find the lightweight tracer at our Shareware page. Please contact us at signup@backtrace.io to request access to our platform and advanced features as we make pieces available. If you have any feature requests or bug reports, please contact us at team@backtrace.io so we can make things happen.

The lightweight tracer is a barebones modern replacement for the traditional pstack and gstack system utilities. The purpose of the tool is to exercise our core client-side debugging libraries without any advanced features. These core libraries include fast DWARF, executable format and backtrace generation support including a framework for non-blocking server-side debugging. In this version, we are providing support for FreeBSD and Linux on x86-64 (plenty of other platforms are on the way). Read on for performance results and usage.

Performance

Let’s begin by demonstrating performance on several complex real-world applications. The metrics we primarily care about are max resident memory usage and time to completion. We compare gstack (a GDB wrapper on RHEL), lldb (of the LLVM project), glider (of GIMLI) and the Backtrace I/O bt tool. The significant performance differential is due to the specialized tracer functionality in our core libraries.

These tests were completed on Debian 7.5.0. The Linux 3.2.0 kernel is installed and the processor is a 4-core Intel Core i7-2600K box at 3.40GHz. The machine had 16GB of RAM and we made sure relevant caches were warmed up. GDB 7.4.1 was used. It has been verified that GDB 7.7 on the same box exhibits nearly identical performance. The trunk (217927) version of LLDB was used and compiled with optimizations enabled.

Even though the bt tool supports DWARF2, DWARF3 and DWARF4, the usage of DWARF3 is enforced to allow glider to run for comparison. GCC has switched to DWARF4 since version 4.8, which we support.

The memory usage column refers to peak resident memory in megabytes and total time refers to time to generate backtrace in seconds. Both glider and bt will access all stack reachable variables and crawl through them by default. Only a summary backtrace (versus a full detailed backtrace) is requested from gstack (gdb) and thread backtrace all (lldb) in these tests unless otherwise stated. Results are from several runs, after things have warmed up. Both bt and gdb are on parity with line number mapping information while glider is inaccurate. I found that LLDB failed to reliably unwind past signal frames on Linux and that some variable forms were unsupported. The performance numbers are also without any of the fancier optimizations our advanced tracer utilizes.


IceWeasel (Firefox)

The following are relevant statistics for Mozilla IceWeasel 24.8.0 with 670 mapped segments and 37 threads. There is approximately 1.3GB worth of DWARF3 debug data across these objects.

iceweaselMemory UsageTotal Time
gdb1313.50MB15.41s
glider1128.08MB2.82s
lldb1950.71MB54.45s
bt217.62MB00.29s

LibreOffice

The following are relevant statistics for LibreOffice 3.5 with 726 mapped segments and 4 threads. There is approximately 2.1GB worth of DWARF3 debug data across these objects (with only a small subset immediately relevant to the trace). LLDB failed to load several debug files in this scenario and so the data has been omitted. I may revisit lldb performance with LibreOffice at a future date.

libreofficeMemory UsageTotal Time
gdb1355.69MB19.13s
glider360.37MB05.90s
bt91.08MB00.14s

Chromium

The following are relevant statistics for Chromium 35.0.1916.114 with 466 mapped segments and 1 thread. There is approximately 2.6GB worth of debug data in a single executable here.

chromiumMemory UsageTotal Time
gdb2634.71MB54.00s
glider3017.08MB05.63s
lldb3810.00MB02:10.65s
bt461.15MB00.61s

Impact of Thread Count

Modern large-scale processes can scale in the thousands if not tens of thousands of threads, hundreds of gigabytes of memory usage and tens of thousands of memory mappings. The following is from an artificial crash scenario across a variable number of threads at 60000 mappings and a stack frame depth of 10 for every thread. The program was compiled with DWARF3 for comparison to glider. GCC 4.7.2 was used for compilation at -O2 optimization level. Some variables were optimized away completely. Unfortunately, we did not have time to scale the test up further but users have reported 3 second trace times on a 30K+ thread 500GB+ RSS process with almost 200K map entries. Both glider and bt emit more information than GDB bt full output as they actually crawl the stack and chase pointers. Every frame looks similar to the following (bt output):

  [  5] crash                    recurse (crash.c:224)
     state = (parameter) reference(0, 0x143a280)
       -><> = 
          .e = 
             .n_thread = 4
             .delay = 1000000
             .fault_depth = 419714448
             .stack_depth = 0
             .fault_tid = 0
             .state = 
                .threads = reference(0x143a298, 0x143a250)
                   -><> = 139947633866496
          .function = reference(0x143a2a0, 0x4011b0, scenario_align_fault)
             -><> = function(0x4011b0, scenario_align_fault)
          .state = reference(0x143a2a8, 0)
          .id = 0
     counter_pointer = reference(0x60259c, 0x60259c, global_counter)
       -><> = 44
     tls_local = --
     pointer_to_array_of_8 = reference(0x602540, 0x602540, global_8)
       -><> = 
          [0] <> = 121
          [1] <> = 122
          [2] <> = 123
          [3] <> = 124
          [4] <> = 125
          [5] <> = 126
          [6] <> = 127
          [7] <> = 128
     array_of_pointer = --
     mesohard = 
       [0] array
          [0] <> = reference(0x7f481900dcc0, 0x4141414141414141)
             -><> = --
          [1] <> = reference(0x7f481900dcc8, 0x4141414141414141)
             -><> = --
          [2] <> = reference(0x7f481900dcd0, 0x4141414141414141)
             -><> = --
          [3] <> = reference(0x7f481900dcd8, 0x4141414141414141)
             -><> = --
          [4] <> = reference(0x7f481900dce0, 0x4141414141414141)
             -><> = --
          [5] <> = reference(0x7f481900dce8, 0x4141414141414141)
             -><> = --
          [6] <> = reference(0x7f481900dcf0, 0x4141414141414141)
             -><> = --
          [7] <> = reference(0x7f481900dcf8, 0x4141414141414141)
             -><> = --
     extreme_pack = 
       .potato = 1
     example_enum = 1 (CRASH_ENUM_ORANGES)
     example_variable = 1
     example_register = --
     example_const = --
     example_long_double = --
     example_double = --
     pack = 
       .apples = 1
       .oranges = 1
       .pineapples = 2
       .grapes = 0
     example_union = 
       .u8 = 254
       .u16 = 65534
       .u32 = 4294967294
       .u64 = 18446744073709551614
       .junk = 
          .a = 18446744073709551614
          .b = 0
     enum_array = 
       [0] <> = 0 (CRASH_ENUM_APPLES)
       [1] <> = 2 (CRASH_ENUM_PINEAPPLES)
       [2] <> = 5 (!)
       [3] <> = 1 (CRASH_ENUM_ORANGES)
     md_array = 
       [0] array
          [0] <> = 1
          [1] <> = 1
          [2] <> = 0
       [1] array
          [0] <> = 2
          [1] <> = 2
          [2] <> = 4
     example_deref_me = reference(0x602560, 0x602560, example_deref_base)
       -><> = 917992.869600
     string_array = --
     simple_array = --
     example_complex = --
     complex_result = 4.000000 + i4.000000
     float_complex = 1.000000 + i3.000000
     long_complex = 1.000000 + i3.000000
     negative_double = -1.000000
     md_array_static = 
       [0] array
          [0] <> = 1
          [1] <> = 4
       [1] array
          [0] <> = 2
          [1] <> = 5
       [2] array
          [0] <> = 3
          [1] <> = 6
     md_3d_array = 
       [0] array
          [0] array
             [0] <> = 1
             [1] <> = 2
          [1] array
             [0] <> = 3
             [1] <> = 4
          [2] array
             [0] <> = 5
             [1] <> = 6
       [1] array
          [0] array
             [0] <> = 7
             [1] <> = 8
          [1] array
             [0] <> = 9
             [1] <> = 10
          [2] array
             [0] <> = 11
             [1] <> = 12
       [2] array
          [0] array
             [0] <> = 13
             [1] <> = 14
          [1] array
             [0] <> = 15
             [1] <> = 16
          [2] array
             [0] <> = 17
             [1] <> = 18
     md_3d_1d_array = 
       [0] array
          [0] array
             [0] <> = 1
             [1] <> = 2
             [2] <> = 3
     md_4d_array = 
       [0] array
          [0] array
             [0] array
                [0] <> = 1
                [1] <> = 2
          [1] array
             [0] array
                [0] <> = 3
                [1] <> = 4
       [1] array
          [0] array
             [0] array
                [0] <> = 5
                [1] <> = 6
          [1] array
             [0] array
                [0] <> = 7
                [1] <> = 8
       [2] array
          [0] array
             [0] array
                [0] <> = 9
                [1] <> = 10
          [1] array
             [0] array
                [0] <> = 11
                [1] <> = 12
     square_array = 
       [0] array
          [0] = string(0x7f481900dbc0, 2, [12])
       [1] array
          [0] = string(0x7f481900dbc3, 2, [34])
       [2] array
          [0] = string(0x7f481900dbc6, 2, [56])
     i = 3
     j = 2
     k = 1
     z = 2
     r = 12
     copy_crash_state = --
     fam = --
     fam_int = --

Here are the results comparing glider, gdb, lldb and bt in time spent. Frame variables were not enumerated for lldb or gdb in this scenario. Both glider and bt walk the stack and crawl memory up to a depth of 20. glider currently doesn’t have a succinct format. The bt-nv column represents bt with only line-number information utilized (similar to default lldb backtrace behavior) by passing the --no-variables option.

Threadsglidergdblldbbt-nvbt
323.450.140.180.120.15
645.590.260.180.120.20
1289.880.510.420.130.29
25618.490.980.370.150.46
51235.781.971.350.190.82
102470.793.942.740.291.58
2048142.698.175.150.503.06


{% img center http://backtrace.io/images/comparison.png 640 480 ‘compare’ ‘compare’ %}

Our core libraries are fast and this demonstrates some of the spare cycles our advanced tracer will be taking advantage of to provide cool features. Our platform provides rich support for DWARF to extract many compound expressions from registers, memory and more. Our stack is built on very fast parsers, efficient memory management, a custom libunwind (expect patches upstream as soon as we expand platform support) and fast data structures (some from the Concurrency Kit library).

Usage

The bt program is fairly barebones. Usage is demonstrated through examples. Output has been truncated for brevity in some cases.

$ bt --help
sbahra@black:~$ bt --help
Usage: bt [--help] [-p] [-f <max frames>] [-m <max depth>]
          [--no-symbols] [--no-lines] [--no-variables]
          [--sizes] [--memory-map] [--assembly] [--types]
          [--thread <pid>,<pid>,...] [--map <file>] [<pid>]

Trace a target process

Simply specify the target process ID.

sbahra@black:~$ bt 3157
PID: 3157
--------------------------------------------------------------------------------
Thread 3157
  [  0] libc-2.13.so             __GI_tcsetattr (../sysdeps/unix/sysv/linux/tcsetattr.c:88)
     fd = --
     optional_actions = --
     termios_p = (parameter) reference(0, 0x7fffb552aa90)
       -><> = 
          .c_iflag = 25606
          .c_oflag = 1
          .c_cflag = 191
          .c_lflag = 2592
          .c_line = 0
          .c_cc = 
             [0] <> = 3
             [1] <> = 28
             [2] <> = 127
             [3] <> = 21

Omit variable information.

Specify the --no-variables option.

sbahra@black:~/projects/crash/src$ bt `pgrep soffice` --no-variables
PID: 18963
--------------------------------------------------------------------------------
Thread 18964
  [  0] libpthread-2.13.so       pthread_cond_timedwait (../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215)
  [  1] libuno_sal.so.3          rtl_cache_wsupdate_all(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/sal/rtl/source/alloc_cache.cxx:1411)
  [  2] libpthread-2.13.so       start_thread

Thread 18966
  [  0] libc-2.13.so             accept (../sysdeps/unix/syscall-template.S:82)
  [  1] libuno_sal.so.3          osl_acceptPipe (pipe.c:430)

Trace specific threads

Our tracers have robust support for non-stop tracing of multithreaded software. In this mode only a subset of threads are stopped for tracing. This allows for minimal intrusion and is part of what allows our advanced tracer to perform temporal analysis for certain classes of bugs. Simply pass a comma-separated list of thread identifiers to the --thread option. The --no-variables option was passed in for brevity.

sbahra@black:~$ bt `pgrep soffice` --thread 18964,18966 --no-variables
PID: 18963
--------------------------------------------------------------------------------
Thread 18964
  [  0] libpthread-2.13.so       pthread_cond_timedwait (../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215)
  [  1] libuno_sal.so.3          rtl_cache_wsupdate_all(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/sal/rtl/source/alloc_cache.cxx:1411)
  [  2] libpthread-2.13.so       start_thread

Thread 18966
  [  0] libc-2.13.so             accept (../sysdeps/unix/syscall-template.S:82)
  [  1] libuno_sal.so.3          osl_acceptPipe (pipe.c:430)
  [  2] libsofficeapp.so         osl::Pipe::accept(osl::StreamPipe&) (/tmp/buildd/libreoffice-3.5.4+dfsg2/solver/unxlngx6.pro/inc/osl/pipe.hxx:141)
  [  3] libsofficeapp.so         desktop::OfficeIPCThread::run() (/tmp/buildd/libreoffice-3.5.4+dfsg2/desktop/source/app/officeipcthread.cxx:655)
  [  4] libsofficeapp.so         threadFunc (/tmp/buildd/libreoffice-3.5.4+dfsg2/solver/unxlngx6.pro/inc/osl/thread.hxx:190)
  [  5] libuno_sal.so.3          osl_thread_start_Impl (thread.c:277)
  [  6] libpthread-2.13.so       start_thread

Output current instruction

The --assembly option outputs current assembly instruction. Advanced tracer will support blobs. The --no-variables and --thread options were used for brevity.

sbahra@black:~$ bt `pgrep soffice` --thread 18967 --assembly --no-variables
PID: 18963
--------------------------------------------------------------------------------
Thread 18967
  [  0] libc-2.13.so             poll (../sysdeps/unix/sysv/linux/poll.c:87)
        => mov %rax, %rdx
  [  1] libvclplug_genlo.so      x11::SelectionManager::dispatchEvent(int) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3737)
        => test %eax, %eax
  [  2] libvclplug_genlo.so      x11::SelectionManager::run(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3775)
        => lea -0x40(%rbp), %rax
  [  3] libvclplug_genlo.so      call_SelectionManager_run (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:108)
        => leave 
  [  4] libuno_sal.so.3          osl_thread_start_Impl (thread.c:277)
        => jmp 0xffffffffffffffdf
  [  5] libpthread-2.13.so       start_thread

Output register values

If sufficient debug information is present to unwind register values then register values can be very useful. Use the --registers option to enable this. The usefulness of register values is dependent on the ABI of your platform. Output truncated for brevity.

sbahra@black:~$ bt `pgrep soffice` --thread 18967 --assembly --no-variables --registers
PID: 18963
--------------------------------------------------------------------------------
Thread 18967
  [  0] libc-2.13.so             poll (../sysdeps/unix/sysv/linux/poll.c:87)
        => mov %rax, %rdx
--------------------------------------------------------------------------------
  %rax=0xfffffffffffffdfc               %rdx=0x3e8  %rcx=0xffffffffffffffff
           %rbx=0x155c120                 %rsi=0x1      %rdi=0x7f4d8cccea40
      %rbp=0x7f4d8ccceb40      %rsp=0x7f4d8cccea00                    %r8=0
               %r9=0x4a17                   %r10=0               %r11=0x293
      %r12=0x7f4d91afb306      %r13=0x7f4d8cccf9c0      %r14=0x7f4da595e040
                 %r15=0x3      %rip=0x7f4da47f7223
--------------------------------------------------------------------------------
  [  1] libvclplug_genlo.so      x11::SelectionManager::dispatchEvent(int) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3737)
        => test %eax, %eax
--------------------------------------------------------------------------------
  %rax=0xfffffffffffffdfc               %rdx=0x3e8  %rcx=0xffffffffffffffff
           %rbx=0x155c120                 %rsi=0x1      %rdi=0x7f4d8cccea40
      %rbp=0x7f4d8ccceb40      %rsp=0x7f4d8cccea30                    %r8=0
               %r9=0x4a17                   %r10=0               %r11=0x293
      %r12=0x7f4d91afb306      %r13=0x7f4d8cccf9c0      %r14=0x7f4da595e040
                 %r15=0x3      %rip=0x7f4d9548d716
--------------------------------------------------------------------------------
  [  2] libvclplug_genlo.so      x11::SelectionManager::run(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3775)
        => lea -0x40(%rbp), %rax
--------------------------------------------------------------------------------
  %rax=0xfffffffffffffdfc               %rdx=0x3e8  %rcx=0xffffffffffffffff
           %rbx=0x155c120                 %rsi=0x1      %rdi=0x7f4d8cccea40
      %rbp=0x7f4d8cccec10      %rsp=0x7f4d8ccceb50                    %r8=0
               %r9=0x4a17                   %r10=0               %r11=0x293
      %r12=0x7f4d91afb306      %r13=0x7f4d8cccf9c0      %r14=0x7f4da595e040
                 %r15=0x3      %rip=0x7f4d9548d97f

Limit memory crawl depth

The -m option limits the maximum crawl depth. An dereference or enumeration of an aggregate type’s members is considered a crawl operation.

sbahra@black:~$ bt `pgrep soffice` --thread 18967 -m 1
PID: 18963
--------------------------------------------------------------------------------
Thread 18967
  [  0] libc-2.13.so             poll (../sysdeps/unix/sysv/linux/poll.c:87)
     fds = --
     nfds = --
     timeout = (parameter) 1000
     oldtype = 0
     result = --

  [  1] libvclplug_genlo.so      x11::SelectionManager::dispatchEvent(int) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3737)
     this = (parameter) reference(0x7f4d8cccea38, 0x18694c0)
       -><> = 
     millisec = (parameter) 1000
     aGuard = 
       .pResetT = reference(0x7f4d8cccea58, 0x1869830)

  [  2] libvclplug_genlo.so      x11::SelectionManager::run(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3775)
     pThis = (parameter) reference(0x7f4d8ccceb58, 0x18694c0)
       -><> = void
     This = reference(0x7f4d8cccebb0, 0x18694c0)
       -><> = 
     aLast = 
       .tv_sec = 1410895868
       .tv_usec = 31562
     xFact = 

  [  3] libvclplug_genlo.so      call_SelectionManager_run (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:108)
     pMgr = (parameter) reference(0x7f4d8cccec28, 0x18694c0)
       -><> = void

  [  4] libuno_sal.so.3          osl_thread_start_Impl (thread.c:277)
     pData = (parameter) reference(0x7f4d8cccec48, 0x187a6a0)
       -><> = void
     terminate = --
     pImpl = reference(0x7f4d8cccec48, 0x187a6a0)
       -><> = 

  [  5] libpthread-2.13.so       start_thread

Output types

Types are output in storage order with the --types option. Crawl depth is limited for brevity here.

PID: 18963
--------------------------------------------------------------------------------
Thread 18967
  [  0] libc-2.13.so             poll
     {pointer(struct(pollfd))} fds = --
     {typedef(nfds_t, long unsigned int)} nfds = --
     {int} timeout = (parameter) 1000
     {int} oldtype = 0
     {int} result = --

  [  1] libvclplug_genlo.so      x11::SelectionManager::dispatchEvent(int)
     {const pointer(class(SelectionManager))} this = (parameter) reference(0x7f4d8cccea38, 0x18694c0)
       {class(SelectionManager)} -><> = 
     {int} millisec = (parameter) 1000
     {typedef(ResettableMutexGuard, class(ResettableGuard<osl::Mutex>))} aGuard = 
       {pointer(class(Mutex))} .pResetT = reference(0x7f4d8cccea58, 0x1869830)

  [  2] libvclplug_genlo.so      x11::SelectionManager::run(void*)
     {pointer(void)} pThis = (parameter) reference(0x7f4d8ccceb58, 0x18694c0)
       {void} -><> = void
     {pointer(class(SelectionManager))} This = reference(0x7f4d8cccebb0, 0x18694c0)
       {class(SelectionManager)} -><> = 
     {struct(timeval)} aLast = 
       {typedef(__time_t, long int)} .tv_sec = 1410896070
       {typedef(__suseconds_t, long int)} .tv_usec = 266926
     {class(Reference<com::sun::star::lang::XMultiServiceFactory>)} xFact = 

  [  3] libvclplug_genlo.so      call_SelectionManager_run
     {pointer(void)} pMgr = (parameter) reference(0x7f4d8cccec28, 0x18694c0)
       {void} -><> = void

  [  4] libuno_sal.so.3          osl_thread_start_Impl
     {pointer(void)} pData = (parameter) reference(0x7f4d8cccec48, 0x187a6a0)
       {void} -><> = void
     {int} terminate = --
     {pointer(typedef(Thread_Impl, struct(osl_thread_impl_st)))} pImpl = reference(0x7f4d8cccec48, 0x187a6a0)
       {typedef(Thread_Impl, struct(osl_thread_impl_st))} -><> = 

  [  5] libpthread-2.13.so       start_thread

Output memory map

The --memory-map option outputs the target process memory map.

sbahra@black:~$ bt `pgrep vim` --no-variables --memory-map |head
PID: 20166
--------------------------------------------------------------------------------
Memory map
--------------------------------------------------------------------------------
          400000 - 5b5000           r-xp /usr/bin/vim.basic
          7b4000 - 7b5000           r--p /usr/bin/vim.basic
          7b5000 - 7ca000           rw-p /usr/bin/vim.basic
          7ca000 - 7d6000           rw-p <anonymous>
         24f7000 - 268e000          rw-p [heap]
    7f0ddab46000 - 7f0ddab51000     r-xp /lib/x86_64-linux-gnu/libnss_files-2.13.so

Use cached maps file

Massive processes may have a very large number of map entries. In some cases, we have observed that reading /proc/pid/maps becomes a significant bottleneck. For this reason, users may cache map dumps (among other assets) and re-use them for future tracing sessions. To use this option, specify the --map option.

sbahra@black:~$ cat /proc/`pgrep vim`/maps > maps
sbahra@black:~$ bt --no-variables --map maps `pgrep vim`
warning: map cache [maps] in use.
PID: 20166
--------------------------------------------------------------------------------
Thread 20166
  [  0] libc-2.13.so             __select (../sysdeps/unix/syscall-template.S:82)
  [  1] vim.basic                RealWaitForChar (os_unix.c:5197)
  [  2] vim.basic                WaitForChar (os_unix.c:4868)
  [  3] vim.basic                mch_inchar (os_unix.c:436)
  [  4] vim.basic                ui_inchar (ui.c:193)
  [  5] vim.basic                inchar (getchar.c:3025)
  [  6] vim.basic                vgetorpeek (getchar.c:2800)
  [  7] vim.basic                vgetc (getchar.c:1582)
  [  8] vim.basic                safe_vgetc (getchar.c:1787)
  [  9] vim.basic                edit (edit.c:735)
  [ 10] vim.basic                invoke_edit.isra.9 (normal.c:9143)
  [ 11] vim.basic                nv_open (normal.c:8457)
  [ 12] vim.basic                normal_cmd (normal.c:1193)
  [ 13] vim.basic                main_loop (main.c:1297)
  [ 14] vim.basic                main (main.c:1001)
  [ 15] libc-2.13.so             __libc_start_main

And more…

There are more options available. The advanced tracer is where things get really interesting and we look forward to sharing some of those features with the public soon.

Availability

If you’re interested in our blazingly fast platform and rich debugging features, please request access at signup@backtrace.io. The bt tool is available for download at our Shareware page. This simple tool is just the tip of the iceberg. All our software is available in commercial-friendly licenses.