Symbolification

If you are submitting minidump files then you will need to ensure that debug symbols have been uploaded to Backtrace in order to have accurate classification, deduplication and callstack rendering.

Generating Symbols

Windows

In Visual Studio, your DEBUG build will generate .pdb symbol files. If you wish to generate symbol files in your RELEASE build or in a custom configuration, make sure that "Generate Debug Info" is turned on in your project's Configuration Properties.

Linux

Backtrace supports .pdb and .sym file formats. If you're using .elf binary symbols in Linux, you'll need to convert these to .sym file format using the Breakpad dump_syms utility. See Breakpad Linux Starter Guide for more information.

Symbol Formats and Upload Methods

Regardless of your upload method, Backtrace provides a great of observability into symbol state. Navigate to your project configuration page and click on Symbols to see a record of all symbols.

Backtrace supports .pdb files, .sym files (Breakpad symbol files) and symbol archives (compressed or uncompressed archives containing either .pdb or .sym files). These symbol files can be uploaded manually or hooked up into your build system so they are automatically uploaded.

Symbol Formats Web Browser morgue curl http Notes
Symbol archive (.tar of symbol files) Yes Yes Yes Yes
Compressed archive (.tar.gz or .zip) Yes Yes Yes Yes
Breakpad Symbol Yes Yes Yes Yes
Compressed Breakpad Symbol No Yes (--compression=gzip Yes Yes Requires the HTTP header Content-Encoding: gzip
PDB Yes No Yes Yes Requires the query string parameter upload_file=<filename.pdb>
DLL Yes No Yes Yes Requires the query string parameter upload_file=<filename.dll> It is recommended to include executable code for 64-bit.
EXE Yes No Yes Yes Requires the query string parameter upload_file=<filename.exe> It is recommended to include executable code for 64-bit.
Compressed PDB File No No Yes Yes Requires the HTTP header Content-Encoding: gzipand the query string parameter upload_file=<filename.pdb>
Compressed EXE File No No Yes Yes Requires the HTTP header Content-Encoding: gzipand the query string parameter upload_file=<filename.exe>
Compressed DLL File No No Yes Yes Requires the HTTP header Content-Encoding: gzipand the query string parameter upload_file=<filename.dll>

You may also specify a tag query string parameter to any of your symbol uploads in order to group symbols for ease of management. A tag is simply a group of symbols, like a folder on your filesystem. You may want to have different tags for different versions of your application or different platforms. If no tag is specified, symbols are placed into the anon tag.


The recommended symbol upload format is a symbol archive. This is a .tar.gz or .zip file containing one or more .sym or .pdb files. There are no restrictions on the layout of the files but you must ensure no relative paths are used and that files have the correct basename. For example, debug information for Editor.exe must be in a file called Editor.pdb or Editor.sym. Socorro symbol archives work with Backtrace without further configuration.


Symbol Management

Breakpad and Socorro

Backtrace is completely compatible with existing Breakpad and Socorro users. Simply upload the .sym files through an HTTP POST or the sym_upload tool. Ensure that you have an Access Token in order to upload to your instance. You must provide a token (referring to your symbol access token) and a format query string parameter with the value symbols.

Below is an example invocation of a symbol upload using the sym_upload tool.

sym_upload null_read_av.sym 'http://yourcompany.sp.backtrace.io:6097/post?format=symbols&token=57f2126dcef18bb0d2af35ec1d813f3775ee8228d1d886de522b2aedceff8b87'

HTTP API

In order to build automation around symbol upload, such as integration into a build and release process, you'll want to interface directly with the HTTP API provided by Backtrace.

Simply issue an HTTP POST to <your instance>/post?format=symbols&token=<access token>. An access token can be created by navigating to Projects > Your Project > Symbols > Manage Access Tokens and clicking on the Create a new access token button.

Below is an example of a curl invocation to submit a symbol archive.

curl --data-binary @symbols.tar 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=5ae2136d4ef181b0d2afa5ef1d81ff377eea8228d1d883d4552621ed1eff8b87'

Example curl Commands for HTTP Symbol Upload

Type Command
Symbol Archive (.tar) curl --data-binary @symbols.tar 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken'
Compressed archive (.tar.gz or .zip) curl --data-binary @symbols.zip 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken'
Breakpad Symbol curl --data-binary @symbol.sym 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken'
Compressed Breakpad Symbol curl -H 'Content-Encoding: gzip' --data-binary @symbol.sym 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken'
PDB curl --data-binary @symbol.pdb 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken&upload_file=symbol.pdb'
DLL curl --data-binary @symbol.dll 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken&upload_file=symbol.dll'
EXE curl --data-binary @symbol.exe 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken&upload_file=symbol.exe'
Compressed PDB File curl -H 'Content-Encoding: gzip' --data-binary @symbol.pdb.gz 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken&upload_file=symbol.pdb'
Compressed DLL File curl -H 'Content-Encoding: gzip' --data-binary @symbol.dll.gz 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken&upload_file=symbol.dll'
Compressed EXE File curl -H 'Content-Encoding: gzip' --data-binary @symbol.exe.gz 'https://yourcompany.sp.backtrace.io:6098/post?format=symbols&token=yourtoken&upload_file=symbol.exe'

Web Browser

Navigate to your project configuration page and click on Symbols in order to manage symbols through your web browser. You are able to manually upload .pdb, .sym and compressed archives of .sym or .pdb files directly in your web browser. Follow the on-screen instructions for more details.

Troubleshooting

If you are blocked on uploading symbols from the command line, try uploading directly from your web browser. Instructions are available available above.

Invalid Token

If you receive an "invalid token" error in the response when uploading symbols via HTTP, check to make sure that you're using a symbol token and not a project token. You can create a symbol token on the Project Settings page by clicking the Symbols section, then the Manage Access Tokens tab in the middle of the page.

Missing Symbols

Sometimes, symbols can get missed during the symbol upload process. A list of missing symbols for a dump is available in the web debugger annotations pane. For more information about how to acquire this list, refer to the Web Debugger guide.

After uploading missing symbols, you can Reprocess Objects to have the dumps with missing symbols reprocessed. Note that the grouping of dumps may change after missing symbols are uploaded. To reprocess objects, click on the Reprocess objects button in the project configuration page (either by clicking on Configure Project in the top right menu or clicking on the relevant project in the Configure Organization screen). The Reprocess objects button is in the top right corner of the configuration page.

Locating Missing Symbols

The first step in locating missing symbol files is to determine the name of the symbol file and its debug identifier. There are two ways to do this via the Debugger UI:

  1. When viewing an error that's missing symbols, Backtrace will add a Missing Symbols branch to the Annotations section at the lower-right part of the screen. When you expand an entry number, you can see the path to the corresponding executable, the name of the symbol file, the debug identifier, and version information:

    Annotations

  2. If an error is missing symbols, you can also see this in the callstack. In an affected frame, you will see a triangular warning symbol along with a memory address where the function signature would normally be. If you mouse over one of these frames, you'll see a yellow warning missing symbols pop-up. The debug identifier and the name of the symbol file are in the heading:

    !Callstack Missing Info

You now have the information you need to check your in-house symbol archives for the missing symbol file. How to identify the symbol file depends on what kind of symbol files you're using:

.sym files - If you're using Breakpad .sym files, this is quite simple: The first line of Breakpad sym file is the MODULE record which lists the pdb name and debug identifier. For example MODULE windows x86_64 D0489F894E07424AAB5E626FF8C943DD1 advapi32.pdb

.pdb files - (More Likely) If you're using .pdb files, you need to extract the debug identifier from .pdb files using one of the following methods, and find the matching one:

> dumpbin.exe /headers bcrypt.dll

  Debug Directories

        Time Type       Size      RVA  Pointer
    -------- ------ -------- -------- --------
    584A7C7E cv           23 00021A80    20C80    Format: RSDS, {5C82DF99-0DA0-4C46-A2B2-2ABB82D6B66A}, 1, bcrypt.pdb


Once you locate and upload missing symbols, make sure to Reprocess Objects, as described in the previous section: Missing Symbols

Inaccurate callstack

Windows

On Windows, 64-bit applications store some unwinding information exclusively in the executable object (.exe or .dll file). For this reason, we advise to include the executable code of your application and library during symbol upload. These files can be uploaded as stand-alone files, but you must ensure that the file base name matches the base name of the .pdb file. For example, the debug information for Editor.exe is expected to be in Editor.pdb. It is important that the name of the executable is Editor.exe in this context to pair with Editor.pdb. If the executable was uploaded as Word.exe, then you are unable to pair with Editor.pdb. It is recommended that a symbol archive is used.

Visual Studio

If you are using Visual Studio, you'll need to ensure that symbols are being generated correctly. Additional details are available in the Crashpad Visual Studio document.