Coresnap

coresnap consists of two primary components. The archive tool and coresnapd. archive is responsible for writing out core files to disk and communicating with the coresnapd journaling mechanism. archive is only relevant to Linux targets and is used to handle writes and invocations from the core_pattern mechanism.

coresnap invokes the Backtrace snapshot tool on core files automatically and then sends the resulting snapshot to the object store using the object store client. Configuration of coresnap consists of installing the package, supplying a coroner configuration for coresnapto pass to the object store client and then enabling the service.

Please note that some of the command snippets require administrator privileges.


Installation

These steps require a license number from Backtrace to install the necessary packages. Please contact us to receive a license number before proceeding.

RHEL

$ yum install backtrace-coresnap

Debian

$ apt-get install backtrace-coresnap

Ubuntu

$ apt-get install backtrace-coresnap


Configuration

Please refer to the coroner documentation for coroner-specific configuration options. Your site administrator or team lead probably already have one for your environment. All you have to do is place your coroner configuration file in /usr/local/etc/coroner.cf. This path may be changed by modifying coresnap.conf.

If you wish to modify the Coresnap configuration file, copy /opt/backtrace/etc/coresnap/coresnap.conf to /usr/local/etc/coresnap/coresnap.conf or /etc/coresnap/coresnap.conf and then modify your new configuration file. Refer to the configuration file for configuration options.


Remove Conflicts

Ubuntu

In order to disable the apport crash reporting service, the following commands can be used.

$ service apport stop
$ echo "enabled=0" > /etc/default/apport

It is also possible simply uninstall the apport service with the following commands.

$ apt-get purge apport

RHEL

$ systemctl disable abrtd.service


Enable

The final step is to enable coresnap. You want to disable any crash-reporting solution that is currently enabled on your platform.

init.d

$ /etc/init.d/coresnapd start

The status of coresnapd should report that it is up:

/etc/init.d/coresnapd status

On Ubuntu systems that are using Upstart, the following line enables Coresnap on boot.

$ echo "enabled=1" > /etc/default/coresnapd

On RHEL systems that are still using init.d then the following sequence of commands can be used to enable Coresnap on boot.

/sbin/chkconfig --add coresnapd
/sbin/chkconfig --list coresnapd
/sbin/chkconfig coresnapd on

systemd

$ systemctl start coresnapd

The status of coresnapd should report that it is up:

$ systemctl status coresnapd
● coresnapd.service - Backtrace coredump aggregation service
   Loaded: loaded (/lib/systemd/system/coresnapd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-04-10 17:02:47 EDT; 34min ago
  Process: 15863 ExecStart=/opt/backtrace/sbin/coresnapd $CORESNAPD_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 15864 (coresnapd)
   CGroup: /system.slice/coresnapd.service
           └─15864 /opt/backtrace/sbin/coresnapd

Apr 10 17:02:47 broadwell systemd[1]: Starting Backtrace coredump aggregation service...

At this point, enable coresnapd with the following command.

$ systemctl enable coresnapd


FreeBSD

Currently, coresnapd can be run on a FreeBSD host and handle its kernel core files, but not userland core files. To configure a FreeBSD host, install the coresnap package, then run:

# sysrc coresnapd_enable=YES
# sysrc local_startup="$(sysrc -n local_startup) /opt/backtrace/etc/rc.d"

Copy /opt/backtrace/etc/coresnap/coresnap.conf to /usr/local/etc/coresnap/coresnap.conf and change snapshot.command to read:

snapshot.command = /opt/backtrace/bin/ptrace -o%S --kv=coresnap.object:%B --resource=%a --load= --map-path=%a --core %c %a/boot/kernel/kernel

Now start coresnapd:

# service coresnapd start

Check its status:

$ service coresnapd status
coresnapd is running as pid 613.

Companion scripts are used for FreeBSD kernel coredump processing. To process kernel core archives, the server currently requires kernel.sh to be run as a cron job, which can be done in /etc/crontab:

* * * * * coresnap /opt/backtrace/coresnap/sbin/kernel.sh >/dev/null 2>&1

Note this script must be run as the same user as coresnapd.

To configure FreeBSD hosts to package up kernel core archives and send them to another system running coresnapd, install backtrace_kernel.sh, then:

* * * * * root /usr/local/etc/rc.d/backtrace_kernel.sh onestart > /dev/null 2>&1

To test the setup, simply force a crash: # sysctl debug.kdb.panic=1 Prior to executing this command, the machine should have already been configured to save kernel core files, by configuring a swap device large enough to store one (8GB should do), and running: # sysrc dumpdev=AUTO

Advanced Configuration

Configure coresnap to submit to your projects

Coresnap comes configured to submit all snapshots to the blackhole project by default. In the likely event that you have configured Coroner for other project names, you can modify your coresnap configuration to use these instead.

First, copy out the example coresnap.conf:

$ cp /opt/backtrace/etc/coresnap/coresnap.conf /etc/coresnap/coresnap.conf

Find the line format.1 = blackhole. This specifies that blackhole is the default destination project. We will add a line above this to route crashes to a particular project:

format.1 = testapp %e /testapp$
format.1 = blackhole

The effect of this is that coresnap will submit a crash to the project named testapp using the token name testapp from coroner.cf, if the executable name ends in testapp.

This simple pattern should cover most cases. The default coresnap.conf has inline documentation explaining the various options for the format lines, if you need to pass values to ptrace and coroner in different ways.


Frequently Asked Questions

How do I see what coresnapd is doing?

Coresnap logs error messages in syslog. Refer to syslog to diagnose any potential issues. If there are problems in executing sub-processes, you may run coresnapd by hand with /opt/backtrace/sbin/coresnapd -f. This specifies foreground mode and forces command output mode to inherit. Command output is displayed on your terminal output in this case.

Which project are snapshots submitted to?

By default, all dumps are submitted to a "blackhole" project using the "blackhole" authentication token. Please ensure that these exist in coronerd and your coroner configuration file if this default is used. Otherwise, format rules are typically used for dynamic selection of projects.