Backtrace I/O is building a turn-key infrastructure platform to detect, aggregate, analyze and collaboratively fix software bugs of all types for even the most demanding software applications. We are taking a unique approach to the problem, from how backtraces are generated to how crashes are stored and analyzed. As engineers working on enterprise software, backtraces are exceptionally useful to us. In production, backtraces can provide key insights to real-world performance. In bug reports, backtraces and basic environmental data are usually the only thing engineers have to go on for determining and fixing the root cause of a crash. Backtraces with insufficient information quickly become useless and lead engineers to draw incorrect conclusions that lead to much wasted time and effort. Today’s tools can fail to perform for demanding applications and we would like to fix that.
In order to build the best platform for crash analysis we know we must begin with the best tracers and crash data formats. In this post we will explore some of the performance advantages of our tracers compared to other tools and announce some of our first shareware.
A major component of our platform is the advanced tracer and tracing framework, which provide features such as temporal analysis, modern crash deduplication (in cooperation with our database) and more.
The advanced tracer relies on our core backtrace library and today we’re happy
to provide public access to prerelease binaries of the lightweight core library
tracer bt, the same library our advanced tracer builds on. We do not expect
this version of the lightweight tracer to be bug-free. You can find the
lightweight tracer at our Shareware page.
Please contact us at signup@backtrace.io to request access to our platform
and advanced features as we make pieces available. If you have any feature
requests or bug reports, please contact us at team@backtrace.io so we can
make things happen.
The lightweight tracer is a barebones modern replacement for the traditional
pstack and gstack system utilities. The purpose of the tool is to exercise
our core client-side debugging libraries without any advanced features. These
core libraries include fast DWARF, executable format and backtrace generation
support including a framework for non-blocking server-side debugging. In this
version, we are providing support for FreeBSD and Linux on x86-64 (plenty of
other platforms are on the way). Read on for performance results and usage.
Performance
Let’s begin by demonstrating performance on several complex real-world
applications. The metrics we primarily care about are max resident memory usage
and time to completion. We compare gstack (a GDB
wrapper on RHEL), lldb (of the LLVM project),
glider (of GIMLI) and the Backtrace I/O bt tool.
The significant performance differential is due to the specialized tracer
functionality in our core libraries.
These tests were completed on Debian 7.5.0. The Linux 3.2.0 kernel is installed and the processor is a 4-core Intel Core i7-2600K box at 3.40GHz. The machine had 16GB of RAM and we made sure relevant caches were warmed up. GDB 7.4.1 was used. It has been verified that GDB 7.7 on the same box exhibits nearly identical performance. The trunk (217927) version of LLDB was used and compiled with optimizations enabled.
Even though the bt tool supports DWARF2, DWARF3 and DWARF4, the usage of
DWARF3 is enforced to allow glider to run for comparison. GCC has
switched to DWARF4 since version 4.8, which we support.
The memory usage column refers to peak resident memory in megabytes and total
time refers to time to generate backtrace in seconds. Both glider and bt will
access all stack reachable variables and crawl through them by default.
Only a summary backtrace (versus a full detailed backtrace) is requested from
gstack (gdb) and thread backtrace all (lldb) in these tests unless
otherwise stated. Results are from several runs, after things have warmed up.
Both bt and gdb are on parity with line number mapping information while
glider is inaccurate. I found that LLDB failed to reliably unwind past signal frames
on Linux and that some variable forms were unsupported.
The performance numbers are also without any of the fancier optimizations our advanced
tracer utilizes.
### IceWeasel (Firefox)
The following are relevant statistics for Mozilla IceWeasel 24.8.0 with 670 mapped segments and 37 threads. There is approximately 1.3GB worth of DWARF3 debug data across these objects.
| iceweasel | Memory Usage | Total Time |
|---|---|---|
| gdb | 1313.50MB | 15.41s |
| glider | 1128.08MB | 2.82s |
| lldb | 1950.71MB | 54.45s |
| bt | 217.62MB | 00.29s |
### LibreOffice
The following are relevant statistics for LibreOffice 3.5 with 726 mapped segments and 4 threads. There is approximately 2.1GB worth of DWARF3 debug data across these objects (with only a small subset immediately relevant to the trace). LLDB failed to load several debug files in this scenario and so the data has been omitted. I may revisit lldb performance with LibreOffice at a future date.
| libreoffice | Memory Usage | Total Time |
|---|---|---|
| gdb | 1355.69MB | 19.13s |
| glider | 360.37MB | 05.90s |
| bt | 91.08MB | 00.14s |
### Chromium
The following are relevant statistics for Chromium 35.0.1916.114 with 466 mapped segments and 1 thread. There is approximately 2.6GB worth of debug data in a single executable here.
| chromium | Memory Usage | Total Time |
|---|---|---|
| gdb | 2634.71MB | 54.00s |
| glider | 3017.08MB | 05.63s |
| lldb | 3810.00MB | 02:10.65s |
| bt | 461.15MB | 00.61s |
### Impact of Thread Count
Modern large-scale processes can scale in the thousands if not tens of
thousands of threads, hundreds of gigabytes of memory usage and tens of
thousands of memory mappings. The following is from an artificial crash
scenario across a variable number of threads at 60000 mappings and a stack
frame depth of 10 for every thread. The program was compiled with DWARF3 for
comparison to glider. GCC 4.7.2 was used for compilation at -O2 optimization
level. Some variables were optimized away completely. Unfortunately, we did
not have time to scale the test up further but users have reported 3 second
trace times on a 30K+ thread 500GB+ RSS process with almost 200K map entries.
Both glider and bt emit more information than GDB bt full output as they
actually crawl the stack and chase pointers.
Every frame looks similar to the following (bt output):
[ 5] crash recurse (crash.c:224)
state = (parameter) reference(0, 0x143a280)
-><> =
.e =
.n_thread = 4
.delay = 1000000
.fault_depth = 419714448
.stack_depth = 0
.fault_tid = 0
.state =
.threads = reference(0x143a298, 0x143a250)
-><> = 139947633866496
.function = reference(0x143a2a0, 0x4011b0, scenario_align_fault)
-><> = function(0x4011b0, scenario_align_fault)
.state = reference(0x143a2a8, 0)
.id = 0
counter_pointer = reference(0x60259c, 0x60259c, global_counter)
-><> = 44
tls_local = --
pointer_to_array_of_8 = reference(0x602540, 0x602540, global_8)
-><> =
[0] <> = 121
[1] <> = 122
[2] <> = 123
[3] <> = 124
[4] <> = 125
[5] <> = 126
[6] <> = 127
[7] <> = 128
array_of_pointer = --
mesohard =
[0] array
[0] <> = reference(0x7f481900dcc0, 0x4141414141414141)
-><> = --
[1] <> = reference(0x7f481900dcc8, 0x4141414141414141)
-><> = --
[2] <> = reference(0x7f481900dcd0, 0x4141414141414141)
-><> = --
[3] <> = reference(0x7f481900dcd8, 0x4141414141414141)
-><> = --
[4] <> = reference(0x7f481900dce0, 0x4141414141414141)
-><> = --
[5] <> = reference(0x7f481900dce8, 0x4141414141414141)
-><> = --
[6] <> = reference(0x7f481900dcf0, 0x4141414141414141)
-><> = --
[7] <> = reference(0x7f481900dcf8, 0x4141414141414141)
-><> = --
extreme_pack =
.potato = 1
example_enum = 1 (CRASH_ENUM_ORANGES)
example_variable = 1
example_register = --
example_const = --
example_long_double = --
example_double = --
pack =
.apples = 1
.oranges = 1
.pineapples = 2
.grapes = 0
example_union =
.u8 = 254
.u16 = 65534
.u32 = 4294967294
.u64 = 18446744073709551614
.junk =
.a = 18446744073709551614
.b = 0
enum_array =
[0] <> = 0 (CRASH_ENUM_APPLES)
[1] <> = 2 (CRASH_ENUM_PINEAPPLES)
[2] <> = 5 (!)
[3] <> = 1 (CRASH_ENUM_ORANGES)
md_array =
[0] array
[0] <> = 1
[1] <> = 1
[2] <> = 0
[1] array
[0] <> = 2
[1] <> = 2
[2] <> = 4
example_deref_me = reference(0x602560, 0x602560, example_deref_base)
-><> = 917992.869600
string_array = --
simple_array = --
example_complex = --
complex_result = 4.000000 + i4.000000
float_complex = 1.000000 + i3.000000
long_complex = 1.000000 + i3.000000
negative_double = -1.000000
md_array_static =
[0] array
[0] <> = 1
[1] <> = 4
[1] array
[0] <> = 2
[1] <> = 5
[2] array
[0] <> = 3
[1] <> = 6
md_3d_array =
[0] array
[0] array
[0] <> = 1
[1] <> = 2
[1] array
[0] <> = 3
[1] <> = 4
[2] array
[0] <> = 5
[1] <> = 6
[1] array
[0] array
[0] <> = 7
[1] <> = 8
[1] array
[0] <> = 9
[1] <> = 10
[2] array
[0] <> = 11
[1] <> = 12
[2] array
[0] array
[0] <> = 13
[1] <> = 14
[1] array
[0] <> = 15
[1] <> = 16
[2] array
[0] <> = 17
[1] <> = 18
md_3d_1d_array =
[0] array
[0] array
[0] <> = 1
[1] <> = 2
[2] <> = 3
md_4d_array =
[0] array
[0] array
[0] array
[0] <> = 1
[1] <> = 2
[1] array
[0] array
[0] <> = 3
[1] <> = 4
[1] array
[0] array
[0] array
[0] <> = 5
[1] <> = 6
[1] array
[0] array
[0] <> = 7
[1] <> = 8
[2] array
[0] array
[0] array
[0] <> = 9
[1] <> = 10
[1] array
[0] array
[0] <> = 11
[1] <> = 12
square_array =
[0] array
[0] = string(0x7f481900dbc0, 2, [12])
[1] array
[0] = string(0x7f481900dbc3, 2, [34])
[2] array
[0] = string(0x7f481900dbc6, 2, [56])
i = 3
j = 2
k = 1
z = 2
r = 12
copy_crash_state = --
fam = --
fam_int = --
Here are the results comparing glider, gdb, lldb and bt in time spent.
Frame variables were not enumerated for lldb or gdb in this scenario. Both
glider and bt walk the stack and crawl memory up to a depth of 20. glider
currently doesn’t have a succinct format. The bt-nv column represents bt
with only line-number information utilized (similar to default lldb backtrace
behavior) by passing the --no-variables option.
| Threads | glider | gdb | lldb | bt-nv | bt |
|---|---|---|---|---|---|
| 32 | 3.45 | 0.14 | 0.18 | 0.12 | 0.15 |
| 64 | 5.59 | 0.26 | 0.18 | 0.12 | 0.20 |
| 128 | 9.88 | 0.51 | 0.42 | 0.13 | 0.29 |
| 256 | 18.49 | 0.98 | 0.37 | 0.15 | 0.46 |
| 512 | 35.78 | 1.97 | 1.35 | 0.19 | 0.82 |
| 1024 | 70.79 | 3.94 | 2.74 | 0.29 | 1.58 |
| 2048 | 142.69 | 8.17 | 5.15 | 0.50 | 3.06 |
{% img center http://backtrace.io/images/comparison.png 640 480 ‘compare’ ‘compare’ %}
Our core libraries are fast and this demonstrates some of the spare cycles our advanced tracer will be taking advantage of to provide cool features. Our platform provides rich support for DWARF to extract many compound expressions from registers, memory and more. Our stack is built on very fast parsers, efficient memory management, a custom libunwind (expect patches upstream as soon as we expand platform support) and fast data structures (some from the Concurrency Kit library).
Usage
The bt program is fairly barebones. Usage is demonstrated through examples.
Output has been truncated for brevity in some cases.
Print help
$ bt --help
sbahra@black:~$ bt --help
Usage: bt [--help] [-p] [-f <max frames>] [-m <max depth>]
[--no-symbols] [--no-lines] [--no-variables]
[--sizes] [--memory-map] [--assembly] [--types]
[--thread <pid>,<pid>,...] [--map <file>] [<pid>]
Trace a target process
Simply specify the target process ID.
sbahra@black:~$ bt 3157
PID: 3157
--------------------------------------------------------------------------------
Thread 3157
[ 0] libc-2.13.so __GI_tcsetattr (../sysdeps/unix/sysv/linux/tcsetattr.c:88)
fd = --
optional_actions = --
termios_p = (parameter) reference(0, 0x7fffb552aa90)
-><> =
.c_iflag = 25606
.c_oflag = 1
.c_cflag = 191
.c_lflag = 2592
.c_line = 0
.c_cc =
[0] <> = 3
[1] <> = 28
[2] <> = 127
[3] <> = 21
Omit variable information.
Specify the --no-variables option.
sbahra@black:~/projects/crash/src$ bt `pgrep soffice` --no-variables
PID: 18963
--------------------------------------------------------------------------------
Thread 18964
[ 0] libpthread-2.13.so pthread_cond_timedwait (../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215)
[ 1] libuno_sal.so.3 rtl_cache_wsupdate_all(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/sal/rtl/source/alloc_cache.cxx:1411)
[ 2] libpthread-2.13.so start_thread
Thread 18966
[ 0] libc-2.13.so accept (../sysdeps/unix/syscall-template.S:82)
[ 1] libuno_sal.so.3 osl_acceptPipe (pipe.c:430)
Trace specific threads
Our tracers have robust support for non-stop tracing of multithreaded software. In
this mode only a subset of threads are stopped for tracing. This allows for
minimal intrusion and is part of what allows our advanced tracer to perform
temporal analysis for certain classes of bugs. Simply pass a comma-separated
list of thread identifiers to the --thread option. The --no-variables
option was passed in for brevity.
sbahra@black:~$ bt `pgrep soffice` --thread 18964,18966 --no-variables
PID: 18963
--------------------------------------------------------------------------------
Thread 18964
[ 0] libpthread-2.13.so pthread_cond_timedwait (../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215)
[ 1] libuno_sal.so.3 rtl_cache_wsupdate_all(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/sal/rtl/source/alloc_cache.cxx:1411)
[ 2] libpthread-2.13.so start_thread
Thread 18966
[ 0] libc-2.13.so accept (../sysdeps/unix/syscall-template.S:82)
[ 1] libuno_sal.so.3 osl_acceptPipe (pipe.c:430)
[ 2] libsofficeapp.so osl::Pipe::accept(osl::StreamPipe&) (/tmp/buildd/libreoffice-3.5.4+dfsg2/solver/unxlngx6.pro/inc/osl/pipe.hxx:141)
[ 3] libsofficeapp.so desktop::OfficeIPCThread::run() (/tmp/buildd/libreoffice-3.5.4+dfsg2/desktop/source/app/officeipcthread.cxx:655)
[ 4] libsofficeapp.so threadFunc (/tmp/buildd/libreoffice-3.5.4+dfsg2/solver/unxlngx6.pro/inc/osl/thread.hxx:190)
[ 5] libuno_sal.so.3 osl_thread_start_Impl (thread.c:277)
[ 6] libpthread-2.13.so start_thread
Output current instruction
The --assembly option outputs current assembly instruction. Advanced tracer
will support blobs. The --no-variables and --thread options were used for
brevity.
sbahra@black:~$ bt `pgrep soffice` --thread 18967 --assembly --no-variables
PID: 18963
--------------------------------------------------------------------------------
Thread 18967
[ 0] libc-2.13.so poll (../sysdeps/unix/sysv/linux/poll.c:87)
=> mov %rax, %rdx
[ 1] libvclplug_genlo.so x11::SelectionManager::dispatchEvent(int) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3737)
=> test %eax, %eax
[ 2] libvclplug_genlo.so x11::SelectionManager::run(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3775)
=> lea -0x40(%rbp), %rax
[ 3] libvclplug_genlo.so call_SelectionManager_run (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:108)
=> leave
[ 4] libuno_sal.so.3 osl_thread_start_Impl (thread.c:277)
=> jmp 0xffffffffffffffdf
[ 5] libpthread-2.13.so start_thread
Output register values
If sufficient debug information is present to unwind register values then register
values can be very useful. Use the --registers option to enable this.
The usefulness of register values is dependent on the ABI of your platform. Output
truncated for brevity.
sbahra@black:~$ bt `pgrep soffice` --thread 18967 --assembly --no-variables --registers
PID: 18963
--------------------------------------------------------------------------------
Thread 18967
[ 0] libc-2.13.so poll (../sysdeps/unix/sysv/linux/poll.c:87)
=> mov %rax, %rdx
--------------------------------------------------------------------------------
%rax=0xfffffffffffffdfc %rdx=0x3e8 %rcx=0xffffffffffffffff
%rbx=0x155c120 %rsi=0x1 %rdi=0x7f4d8cccea40
%rbp=0x7f4d8ccceb40 %rsp=0x7f4d8cccea00 %r8=0
%r9=0x4a17 %r10=0 %r11=0x293
%r12=0x7f4d91afb306 %r13=0x7f4d8cccf9c0 %r14=0x7f4da595e040
%r15=0x3 %rip=0x7f4da47f7223
--------------------------------------------------------------------------------
[ 1] libvclplug_genlo.so x11::SelectionManager::dispatchEvent(int) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3737)
=> test %eax, %eax
--------------------------------------------------------------------------------
%rax=0xfffffffffffffdfc %rdx=0x3e8 %rcx=0xffffffffffffffff
%rbx=0x155c120 %rsi=0x1 %rdi=0x7f4d8cccea40
%rbp=0x7f4d8ccceb40 %rsp=0x7f4d8cccea30 %r8=0
%r9=0x4a17 %r10=0 %r11=0x293
%r12=0x7f4d91afb306 %r13=0x7f4d8cccf9c0 %r14=0x7f4da595e040
%r15=0x3 %rip=0x7f4d9548d716
--------------------------------------------------------------------------------
[ 2] libvclplug_genlo.so x11::SelectionManager::run(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3775)
=> lea -0x40(%rbp), %rax
--------------------------------------------------------------------------------
%rax=0xfffffffffffffdfc %rdx=0x3e8 %rcx=0xffffffffffffffff
%rbx=0x155c120 %rsi=0x1 %rdi=0x7f4d8cccea40
%rbp=0x7f4d8cccec10 %rsp=0x7f4d8ccceb50 %r8=0
%r9=0x4a17 %r10=0 %r11=0x293
%r12=0x7f4d91afb306 %r13=0x7f4d8cccf9c0 %r14=0x7f4da595e040
%r15=0x3 %rip=0x7f4d9548d97f
Limit memory crawl depth
The -m option limits the maximum crawl depth. An dereference or enumeration of an aggregate
type’s members is considered a crawl operation.
sbahra@black:~$ bt `pgrep soffice` --thread 18967 -m 1
PID: 18963
--------------------------------------------------------------------------------
Thread 18967
[ 0] libc-2.13.so poll (../sysdeps/unix/sysv/linux/poll.c:87)
fds = --
nfds = --
timeout = (parameter) 1000
oldtype = 0
result = --
[ 1] libvclplug_genlo.so x11::SelectionManager::dispatchEvent(int) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3737)
this = (parameter) reference(0x7f4d8cccea38, 0x18694c0)
-><> =
millisec = (parameter) 1000
aGuard =
.pResetT = reference(0x7f4d8cccea58, 0x1869830)
[ 2] libvclplug_genlo.so x11::SelectionManager::run(void*) (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:3775)
pThis = (parameter) reference(0x7f4d8ccceb58, 0x18694c0)
-><> = void
This = reference(0x7f4d8cccebb0, 0x18694c0)
-><> =
aLast =
.tv_sec = 1410895868
.tv_usec = 31562
xFact =
[ 3] libvclplug_genlo.so call_SelectionManager_run (/tmp/buildd/libreoffice-3.5.4+dfsg2/vcl/unx/generic/dtrans/X11_selection.cxx:108)
pMgr = (parameter) reference(0x7f4d8cccec28, 0x18694c0)
-><> = void
[ 4] libuno_sal.so.3 osl_thread_start_Impl (thread.c:277)
pData = (parameter) reference(0x7f4d8cccec48, 0x187a6a0)
-><> = void
terminate = --
pImpl = reference(0x7f4d8cccec48, 0x187a6a0)
-><> =
[ 5] libpthread-2.13.so start_thread
Output types
Types are output in storage order with the --types option. Crawl
depth is limited for brevity here.
PID: 18963
--------------------------------------------------------------------------------
Thread 18967
[ 0] libc-2.13.so poll
{pointer(struct(pollfd))} fds = --
{typedef(nfds_t, long unsigned int)} nfds = --
{int} timeout = (parameter) 1000
{int} oldtype = 0
{int} result = --
[ 1] libvclplug_genlo.so x11::SelectionManager::dispatchEvent(int)
{const pointer(class(SelectionManager))} this = (parameter) reference(0x7f4d8cccea38, 0x18694c0)
{class(SelectionManager)} -><> =
{int} millisec = (parameter) 1000
{typedef(ResettableMutexGuard, class(ResettableGuard<osl::Mutex>))} aGuard =
{pointer(class(Mutex))} .pResetT = reference(0x7f4d8cccea58, 0x1869830)
[ 2] libvclplug_genlo.so x11::SelectionManager::run(void*)
{pointer(void)} pThis = (parameter) reference(0x7f4d8ccceb58, 0x18694c0)
{void} -><> = void
{pointer(class(SelectionManager))} This = reference(0x7f4d8cccebb0, 0x18694c0)
{class(SelectionManager)} -><> =
{struct(timeval)} aLast =
{typedef(__time_t, long int)} .tv_sec = 1410896070
{typedef(__suseconds_t, long int)} .tv_usec = 266926
{class(Reference<com::sun::star::lang::XMultiServiceFactory>)} xFact =
[ 3] libvclplug_genlo.so call_SelectionManager_run
{pointer(void)} pMgr = (parameter) reference(0x7f4d8cccec28, 0x18694c0)
{void} -><> = void
[ 4] libuno_sal.so.3 osl_thread_start_Impl
{pointer(void)} pData = (parameter) reference(0x7f4d8cccec48, 0x187a6a0)
{void} -><> = void
{int} terminate = --
{pointer(typedef(Thread_Impl, struct(osl_thread_impl_st)))} pImpl = reference(0x7f4d8cccec48, 0x187a6a0)
{typedef(Thread_Impl, struct(osl_thread_impl_st))} -><> =
[ 5] libpthread-2.13.so start_thread
Output memory map
The --memory-map option outputs the target process memory map.
sbahra@black:~$ bt `pgrep vim` --no-variables --memory-map |head
PID: 20166
--------------------------------------------------------------------------------
Memory map
--------------------------------------------------------------------------------
400000 - 5b5000 r-xp /usr/bin/vim.basic
7b4000 - 7b5000 r--p /usr/bin/vim.basic
7b5000 - 7ca000 rw-p /usr/bin/vim.basic
7ca000 - 7d6000 rw-p <anonymous>
24f7000 - 268e000 rw-p [heap]
7f0ddab46000 - 7f0ddab51000 r-xp /lib/x86_64-linux-gnu/libnss_files-2.13.so
Use cached maps file
Massive processes may have a very large number of map entries. In some cases,
we have observed that reading /proc/pid/maps becomes a significant
bottleneck. For this reason, users may cache map dumps (among other
assets) and re-use them for future tracing sessions. To use this option,
specify the --map option.
sbahra@black:~$ cat /proc/`pgrep vim`/maps > maps
sbahra@black:~$ bt --no-variables --map maps `pgrep vim`
warning: map cache [maps] in use.
PID: 20166
--------------------------------------------------------------------------------
Thread 20166
[ 0] libc-2.13.so __select (../sysdeps/unix/syscall-template.S:82)
[ 1] vim.basic RealWaitForChar (os_unix.c:5197)
[ 2] vim.basic WaitForChar (os_unix.c:4868)
[ 3] vim.basic mch_inchar (os_unix.c:436)
[ 4] vim.basic ui_inchar (ui.c:193)
[ 5] vim.basic inchar (getchar.c:3025)
[ 6] vim.basic vgetorpeek (getchar.c:2800)
[ 7] vim.basic vgetc (getchar.c:1582)
[ 8] vim.basic safe_vgetc (getchar.c:1787)
[ 9] vim.basic edit (edit.c:735)
[ 10] vim.basic invoke_edit.isra.9 (normal.c:9143)
[ 11] vim.basic nv_open (normal.c:8457)
[ 12] vim.basic normal_cmd (normal.c:1193)
[ 13] vim.basic main_loop (main.c:1297)
[ 14] vim.basic main (main.c:1001)
[ 15] libc-2.13.so __libc_start_main
And more…
There are more options available. The advanced tracer is where things get really interesting and we look forward to sharing some of those features with the public soon.
Availability
If you’re interested in our blazingly fast platform and rich debugging
features, please request access at signup@backtrace.io. The bt tool is
available for download at our Shareware page.
This simple tool is just the tip of the iceberg. All our software is available
in commercial-friendly licenses.