Coresnap
coresnap consists of two primary components. The archive
tool and
coresnapd
. archive
is responsible for writing out core files to disk and
communicating with the coresnapd
journaling mechanism. archive
is only
relevant to Linux targets and is used to handle writes and invocations from the
core_pattern
mechanism.
coresnap invokes the Backtrace snapshot tool on core files automatically
and then sends the resulting snapshot to the object store using the object store
client. Configuration of coresnap consists of installing the package, supplying
a coroner
configuration for coresnap
to pass to the object store client and
then enabling the service.
Please note that some of the command snippets require administrator privileges.
Installation
These steps require a license number from Backtrace to install the necessary packages. Please contact us to receive a license number before proceeding.
RHEL
$ yum install backtrace-coresnap
Debian
$ apt-get install backtrace-coresnap
Ubuntu
$ apt-get install backtrace-coresnap
Configuration
Please refer to the coroner
documentation for coroner-specific configuration
options. Your site administrator or team lead probably already have one
for your environment. All you have to do is place your coroner configuration
file in /usr/local/etc/coroner.cf
. This path may be changed by modifying
coresnap.conf
.
If you wish to modify the Coresnap configuration file, copy
/opt/backtrace/etc/coresnap/coresnap.conf
to
/usr/local/etc/coresnap/coresnap.conf
or /etc/coresnap/coresnap.conf
and
then modify your new configuration file. Refer to the configuration file for
configuration options.
Remove Conflicts
Ubuntu
In order to disable the apport
crash reporting service, the following
commands can be used.
$ service apport stop
$ echo "enabled=0" > /etc/default/apport
It is also possible simply uninstall the apport
service with the following
commands.
$ apt-get purge apport
RHEL
$ systemctl disable abrtd.service
Enable
The final step is to enable coresnap. You want to disable any crash-reporting solution that is currently enabled on your platform.
init.d
$ /etc/init.d/coresnapd start
The status of coresnapd
should report that it is up:
/etc/init.d/coresnapd status
On Ubuntu systems that are using Upstart
, the following line
enables Coresnap on boot.
$ echo "enabled=1" > /etc/default/coresnapd
On RHEL systems that are still using init.d
then the following sequence
of commands can be used to enable Coresnap on boot.
/sbin/chkconfig --add coresnapd
/sbin/chkconfig --list coresnapd
/sbin/chkconfig coresnapd on
systemd
$ systemctl start coresnapd
The status of coresnapd
should report that it is up:
$ systemctl status coresnapd
● coresnapd.service - Backtrace coredump aggregation service
Loaded: loaded (/lib/systemd/system/coresnapd.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2016-04-10 17:02:47 EDT; 34min ago
Process: 15863 ExecStart=/opt/backtrace/sbin/coresnapd $CORESNAPD_OPTS (code=exited, status=0/SUCCESS)
Main PID: 15864 (coresnapd)
CGroup: /system.slice/coresnapd.service
└─15864 /opt/backtrace/sbin/coresnapd
Apr 10 17:02:47 broadwell systemd[1]: Starting Backtrace coredump aggregation service...
At this point, enable coresnapd
with the following command.
$ systemctl enable coresnapd
FreeBSD
Currently, coresnapd can be run on a FreeBSD host and handle its kernel core files, but not userland core files. To configure a FreeBSD host, install the coresnap package, then run:
# sysrc coresnapd_enable=YES
# sysrc local_startup="$(sysrc -n local_startup) /opt/backtrace/etc/rc.d"
Copy /opt/backtrace/etc/coresnap/coresnap.conf
to /usr/local/etc/coresnap/coresnap.conf
and change snapshot.command
to read:
snapshot.command = /opt/backtrace/bin/ptrace -o%S --kv=coresnap.object:%B --resource=%a --load= --map-path=%a --core %c %a/boot/kernel/kernel
Now start coresnapd:
# service coresnapd start
Check its status:
$ service coresnapd status
coresnapd is running as pid 613.
Companion scripts are used for FreeBSD kernel coredump processing.
To process kernel core archives, the server currently requires kernel.sh
to be run as a cron job, which can be done in /etc/crontab
:
* * * * * coresnap /opt/backtrace/coresnap/sbin/kernel.sh >/dev/null 2>&1
Note this script must be run as the same user as coresnapd.
To configure FreeBSD hosts to package up kernel core archives and send them
to another system running coresnapd
, install backtrace_kernel.sh
, then:
- Configure the target host for this script:
# sysrc backtrace_kernel_target=coresnap@localhost
- NB: The username must be specified, and must be the same as the user used to run coresnapd on the target.
- Either set
backtrace_kernel.sh
to: - Run on startup:
# sysrc backtrace_kernel_enable=YES
- Or run it as a cron job:
* * * * * root /usr/local/etc/rc.d/backtrace_kernel.sh onestart > /dev/null 2>&1
To test the setup, simply force a crash: # sysctl debug.kdb.panic=1
Prior to executing this command, the machine should have already been
configured to save kernel core files, by configuring a swap device large
enough to store one (8GB should do), and running: # sysrc dumpdev=AUTO
Advanced Configuration
Configure coresnap to submit to your projects
Coresnap comes configured to submit all snapshots to the blackhole
project by
default. In the likely event that you have configured Coroner for other project
names, you can modify your coresnap configuration to use these instead.
First, copy out the example coresnap.conf
:
$ cp /opt/backtrace/etc/coresnap/coresnap.conf /etc/coresnap/coresnap.conf
Find the line format.1 = blackhole
. This specifies that blackhole
is the
default destination project. We will add a line above this to route crashes to
a particular project:
format.1 = testapp %e /testapp$
format.1 = blackhole
The effect of this is that coresnap will submit a crash to the project named
testapp
using the token name testapp
from coroner.cf
, if the executable
name ends in testapp
.
This simple pattern should cover most cases. The default coresnap.conf
has
inline documentation explaining the various options for the format
lines, if
you need to pass values to ptrace
and coroner
in different ways.
Frequently Asked Questions
How do I see what coresnapd is doing?
Coresnap logs error messages in syslog
. Refer to syslog
to diagnose
any potential issues. If there are problems in executing sub-processes,
you may run coresnapd
by hand with /opt/backtrace/sbin/coresnapd -f
.
This specifies foreground mode and forces command output mode to inherit.
Command output is displayed on your terminal output in this case.
Which project are snapshots submitted to?
By default, all dumps are submitted to a "blackhole" project using the
"blackhole" authentication token. Please ensure that these exist in
coronerd
and your coroner
configuration file if this default is used.
Otherwise, format rules are typically used for dynamic selection of projects.