CORESNAP

coresnap consists of two primary components. The archive tool and coresnapd. archive is responsible for writing out core files to disk and communicating with the coresnapd journaling mechanism. archive is only relevant to Linux targets and is used to handle writes and invocations from the core_pattern mechanism.

QUICK START GUIDE

coresnap invokes the Backtrace snapshot tool on core files automatically and then sends the resulting snapshot to the object store using the object store client. Configuration of coresnap consists of installing the package, supplying a coroner configuration for coresnapto pass to the object store client and then enabling the service.

Please note that some of the command snippets require administrator privileges.

INSTALLATION

Make sure that the Backtrace package repository is installed in order to refer to the Backtrace packages. For more information, refer to the documentation portal.

RHEL

$ yum install backtrace-coresnap

Debian

$ apt-get install backtrace-coresnap

Ubuntu

$ apt-get install backtrace-coresnap

CONFIGURATION

Please refer to the coroner documentation for coroner-specific configuration options. Your site administrator or team lead probably already have one for your environment. All you have to do is place your coroner configuration file in /usr/local/etc/coroner.cf. This path may be changed by modifying coresnap.conf.

If you wish to modify the Coresnap configuration file, copy /opt/backtrace/etc/coresnap/coresnap.conf to /usr/local/etc/coresnap/coresnap.conf or /etc/coresnap/coresnap.conf and then modify your new configuration file. Refer to the configuration file for configuration options.

REMOVE CONFLICTS

Ubuntu

In order to disable the apport crash reporting service, the following commands can be used.

$ service apport stop
$ echo "enabled=0" > /etc/default/apport

It is also possible simply uninstall the apport service with the following commands.

$ apt-get purge apport

RHEL

$ systemctl disable abrtd.service

ENABLE CORESNAP

The final step is to enable coresnap. You want to disable any crash-reporting solution that is currently enabled on your platform.

init.d

$ /etc/init.d/coresnapd start

The status of coresnapd should report that it is up:

/etc/init.d/coresnapd status

On Ubuntu systems that are using Upstart, the following line enables Coresnap on boot.

$ echo "enabled=1" > /etc/default/coresnapd

On RHEL systems that are still using init.d then the following sequence of commands can be used to enable Coresnap on boot.

/sbin/chkconfig --add coresnapd
/sbin/chkconfig --list coresnapd
/sbin/chkconfig coresnapd on

systemd

$ systemctl start coresnapd

The status of coresnapd should report that it is up:

$ systemctl status coresnapd
● coresnapd.service - Backtrace coredump aggregation service
   Loaded: loaded (/lib/systemd/system/coresnapd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-04-10 17:02:47 EDT; 34min ago
  Process: 15863 ExecStart=/opt/backtrace/sbin/coresnapd $CORESNAPD_OPTS (code=exited, status=0/SUCCESS)
 Main PID: 15864 (coresnapd)
   CGroup: /system.slice/coresnapd.service
           └─15864 /opt/backtrace/sbin/coresnapd

Apr 10 17:02:47 broadwell systemd[1]: Starting Backtrace coredump aggregation service...

At this point, enable coresnapd with the following command.

$ systemctl enable coresnapd

FreeBSD

Currently, coresnapd can be run on a FreeBSD host and handle its kernel core files, but not userland core files. To configure a FreeBSD host, install the coresnap package, then run:

# sysrc coresnapd_enable=YES
# sysrc local_startup="$(sysrc -n local_startup) /opt/backtrace/etc/rc.d"

Copy /opt/backtrace/etc/coresnap/coresnap.conf to /usr/local/etc/coresnap/coresnap.conf and change snapshot.command to read:

snapshot.command = /opt/backtrace/bin/ptrace -o%S --kv=coresnap.object:%B --resource=%a --load= --map-path=%a --core %c %a/boot/kernel/kernel

Now start coresnapd:

# service coresnapd start

Check its status:

$ service coresnapd status
coresnapd is running as pid 613.

Companion scripts are used for FreeBSD kernel coredump processing. To process kernel core archives, the server currently requires kernel.sh to be run as a cron job, which can be done in /etc/crontab:

* * * * * coresnap /opt/backtrace/coresnap/sbin/kernel.sh >/dev/null 2>&1

Note this script must be run as the same user as coresnapd.

To configure FreeBSD hosts to package up kernel core archives and send them to another system running coresnapd, install backtrace_kernel.sh, then:

* * * * * root /usr/local/etc/rc.d/backtrace_kernel.sh onestart > /dev/null 2>&1

To test the setup, simply force a crash: # sysctl debug.kdb.panic=1 Prior to executing this command, the machine should have already been configured to save kernel core files, by configuring a swap device large enough to store one (8GB should do), and running: # sysrc dumpdev=AUTO

Frequently Asked Questions

How do I see what coresnapd is doing?

Coresnap logs error messages in syslog. Refer to syslog to diagnose any potential issues. If there are problems in executing sub-processes, you may run coresnapd by hand with /opt/backtrace/sbin/coresnapd -f. This specifies foreground mode and forces command output mode to inherit. Command output is displayed on your terminal output in this case.

Which project are snapshots submitted to?

By default, all dumps are submitted to a "blackhole" project using the "blackhole" authentication token. Please ensure that these exist in coronerd and your coroner configuration file if this default is used. Otherwise, format rules are typically used for dynamic selection of projects.

COMPONENTS

CORESNAPD

coresnapd is a daemon that is responsible for reliable archival and processing of generated core dumps. It is a process management tool that invokes executables and monitors the resultant processes. These processes can be passed format strings that expand to the set of files being monitored. A file belongs to one of five stages: staging, pending, sending, committed and failed. A file can also have an additional asset file. The state of these files is observable by the state of the filesystem at the path specified by the dump.root directive in the coresnap configuration file.

OBJECT LIFECYCLE

coresnap relies on rename(2) semantics for journaling state of processing for a dump. This process consists of the following stages.

Below is an example of the contents of the default coresnap archive directory.

/var/coresnap
/var/coresnap/archive
/var/coresnap/archive/committed
/var/coresnap/archive/committed/coredump
/var/coresnap/archive/committed/coredump/6ee38e56-1de2-4bb0-99af-d842b6d521c1
/var/coresnap/archive/committed/snapshot
/var/coresnap/archive/committed/snapshot/6ee38e56-1de2-4bb0-99af-d842b6d521c1
/var/coresnap/archive/pending
/var/coresnap/archive/failed
/var/coresnap/archive/failed/pending
/var/coresnap/archive/failed/sending
/var/coresnap/archive/sending
/var/coresnap/archive/staging
/var/coresnap/archive/assets
/var/coresnap/archive/assets/6ee38e56-1de2-4bb0-99af-d842b6d521c1

The asset file is persisted until both the snapshot and coredump are unlinked.

ARCHIVE

The archive tool routes standard input to a uniquely-allocated file in the archive specified by the dump.root directive. Files have the permissions specified by dump.mode and are owned by the user and group specified by dump.user and dump.group respectively. If for some reason, the archive fails to generate a file then any intermediate files are unlinked. The archive tool also generated a TAR archive in the asset directory that contains environment assets of interest such as /proc/meminfo and more. The archive tool attempts to write out standard input as a sparse file. Many process dumps contain plenty of zero-filled pages so this may yield significant space-savings. The archive tool aborts if it detects that an invocation causes the mount point of dump.root to violate the free block percentage specified by dump.quota. This protection mechanism is oblivious to concurrent invocations of the archive tool.

CONFIGURATION

The default coresnap configuration file is located in /opt/backtrace/etc/coresnap/coresnap.conf. coresnap tools search for the following configuration files in this order: /etc/coresnap/coresnap.conf, /usr/local/etc/coresnap/coresnap.conf and then /opt/backtrace/etc/coresnap/coresnap.conf. After installation of the package, a user may wish to copy the default configuration file to a higher-priority path. It is not recommended that a user replaces or modifies the default coresnap configuration file.

The various configuration options are documented inline the default configuration file in /opt/backtrace/etc/coresnap/coresnap.conf.