Object Store Installation

This guide goes over the installation and bring-up of coronerd, Backtrace's object-store for post-mortem assets.

Please skip to our guide on object store configuration if you are using Backtrace's hosted service.

Installation

Ubuntu/Debian

$ apt-get install libhwloc4
$ apt-get install libhwloc5
$ apt-get install backtrace-coronerd backtrace-gimli

RHEL

$ yum install backtrace-coronerd backtrace-gimli

System Configuration

The rest of this guide are preliminary system configuration needed before we start coronerd.

Edit /etc/security/ulimits.d/all.conf, and add the following:

* hard nofile 131072
* soft nofile 131072
 root hard nofile 131072
 root soft nofile 131072
echo "vm.max_map_count=16000000" >> /etc/sysctl.conf
sysctl -p

SSL

Please see disable SSL if you'd like to disable SSL on coronerd.

CA-trusted certificates

First, copy your certificate chain file (or certificate file) and private key file to /etc/coronerd/ssl/.

For safety, make the private key have strict permissions:

$ chmod 600 /etc/coronerd/ssl/key.pem
$ ls -lptr /etc/coronerd/ssl/key.pem
-rw------- 1 root root 1704 Nov 24 11:43 /etc/coronerd/ssl/key.pem

Point coronerd to the certificate and key files in /etc/coronerd/coronerd.conf. Please note that all SSL objects must point to your certificate and key files.

Example:

{
    //...
    "console" : {
        "path" : "/var/run/coronerd/coronerd.socket",
        "bind" : {
            "hostname" : "0.0.0.0",
            "service" : "9040"
        },
        "backlog" : 16,
        "ssl" : {
            "certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
            "key" : "/etc/coronerd/ssl/key.pem"
        }
    },
    //...
    "listener" : {
        "write" : {
            "http_bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "6097",
                    "concurrency" : 2000
                }
            ],
            "https_bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "6098",
                    "concurrency" : 2000
                }
            ],
            "threads" : 1,
            "ssl" : {
                "certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
                "key" : "/etc/coronerd/ssl/key.pem"
            }
        },
        "read" : {          
            "bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "4097"
                }
            ],
            "threads" : 1,          
            "ssl" : {
                "certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
                "key" : "/etc/coronerd/ssl/key.pem"
            }
        },
        "http-console" : {
            "bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "443",
                    "concurrency" : 1000
                }
            ],
            "columns" : {
        //...
            },
            "threads" : 1,
            "ssl" : {
                "certificate_chain_file" : "/etc/coronerd/ssl/chain.pem",
                "key" : "/etc/coronerd/ssl/key.pem"
            }
        }
    }
}

If using a certificate file, add the certificate setting alongside the key setting (see example below in Self-Signed Certificates)

Self-signed certificates

A self-signed certificate does not exist in the web of trust which descends from a trusted root certificate authority (CA). Nonetheless, it can be used for the purposes of a secure channel if the self-signed certificate is manually set up to be considered a trusted certificate authority. Using self-signed certificates in this way will only affect the secure channels initiated by coroner and will not affect the secure channels and web-of-trust of the remainder of the system.

A self-signed certificate and key must be generated. Note that in the following command, the user-specified "Common Name" value must later match the host portion of the https://:port entry of the "write" configuration key of the desired universe section in coroner client's configuration. Generate the key (coronerd-key.pem) and certificate (coronerd-cert.pem) via the following command:

$ sudo mkdir /etc/coronerd/ssl
$ openssl req -nodes -new -x509  -keyout /etc/coronerd/ssl/coronerd-key.pem -out /etc/coronerd/ssl/coronerd-cert.pem
Generating a 2048 bit RSA private key
......................+++
......+++
writing new private key to '/etc/coronerd/ssl/coronerd-key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:New York
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Backtrace I/O, LLC
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:127.0.0.1
Email Address []:support@backtrace.io
$ ls -lptr /etc/coronerd/ssl/coronerd-key.pem /etc/coronerd/ssl/coronerd-cert.pem
-rw-rw-r-- 1 root root 1704 Nov 24 11:43 /etc/coronerd/ssl/coronerd-key.pem
-rw-rw-r-- 1 root root 1415 Nov 24 11:43 /etc/coronerd/ssl/coronerd-cert.pem

For safety, make the private key have strict permissions:

$ chmod 600 /etc/coronerd/ssl/coronerd-key.pem
$ ls -lptr /etc/coronerd/ssl/coronerd-key.pem
-rw------- 1 root root 1704 Nov 24 11:43 /etc/coronerd/ssl/coronerd-key.pem

The coroner-key.pem file should remain private and only accessible on the host(s) running coronerd. The coroner-cert.pem file is public and must be accessible on all hosts running the coroner client.

Step 2

coronerd must be configured to use the certificate and key files. Make sure that both files are accessible on the host which coronerd runs on. Then, in your coronerd configuration file, make sure the console, and write, read, and http-console sections under listener have SSL stanzas.

Example:

{
    //...
    "console" : {
        "path" : "/var/run/coronerd/coronerd.socket",
        "bind" : {
            "hostname" : "0.0.0.0",
            "service" : "9040"
        },
        "backlog" : 16,
        "ssl" : {
            "certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
            "key" : "/etc/coronerd/ssl/coronerd-key.pem"
        }
    },
    //...
    "listener" : {
        "write" : {
            "http_bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "6097",
                    "concurrency" : 2000
                }
            ],
            "https_bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "6098",
                    "concurrency" : 2000
                }
            ],
            "threads" : 1,
            "ssl" : {
                "certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
                "key" : "/etc/coronerd/ssl/coronerd-key.pem"
            }
        },
        "read" : {          
            "bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "4097"
                }
            ],
            "threads" : 1,          
            "ssl" : {
                "certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
                "key" : "/etc/coronerd/ssl/coronerd-key.pem"
            }
        },
        "http-console" : {
            "bind" : [
                {
                    "hostname" : "0.0.0.0",
                    "service" : "443",
                    "concurrency" : 1000
                }
            ],
            "columns" : {
        //...
            },
            "threads" : 1,
            "ssl" : {
                "certificate" : "/etc/coronerd/ssl/coronerd-cert.pem",
                "key" : "/etc/coronerd/ssl/coronerd-key.pem"
            }
        }
    }
}
Step 3

Set the coroner client configuration file (coroner.cf) as noted in Client Installation | Self-Signed Certificates

Troubleshooting

"error: cURL failure (SSL peer certificate or SSH remote key was not OK): SSL: certificate subject name 'XXX' does not match target host name 'YYY'"

This means that the certificate was generated with a "Common Name" field set to XXX, but that the client knows the server as YYY (i.e., uses a configuration "write" value of "https://YYY:port").

Solution 1:

Make sure that the client accesses the server via the name which is used as the "Common Name" in the certificate. This may require correcting DNS entries, IP addresses, and/or routes.

Solution 2:

Regenerate the certificate and key used by the coronerd server and have it use the new YYY hostname of the coronerd for the certificate's "Common Name".

Start coronerd

$ /etc/init.d/coronerd start

To verify that coronerd is running properly:

$ /etc/init.d/coronerd status

Create Organization and Admin User

Before accessing the coronerd object store via Web UI, we need to create the organization object and an initial admin user via the command-line morgue utility.

morgue is installed by the NodeJS NPM utility. To install, simply run npm install backtrace-morgue -g. If you need NodeJS, please see The Node.JS website

Launch `morgue setup, pointing to the URL of your coronerd instance. Follow the prompts to create the organization and initial admin user, then you'll be ready to continue setting up via the Web UI.

If using self-signed SSL certificates, pass the -k flag to morgue setup.

$ morgue setup https://coronerd.mydomain.com                                                                                                                                                                                    
Determining system state...unconfigured

Create an organization
We must first configure the organization that is using the object store.
Please provide a one word name for the organization using the object store.
For example, if your company name is "Appleseed Systems I/O", you could
use the name "appleseed". The name must be lowercase.

Organization name: testing-bt

Create an administrator
We must create an administrator user. This user will be used to configure
the server as well as perform system-wide administrative tasks.

Username: jdoe
E-mail address: jdoe@mydomain.com
Password: ************

Confirm password: ************

Next: Continue on to coronerd setup