Regardless of whether you want to design a test environment that is as close to reality as possible, or want to (better) secure Checkmk in the intranet, or to set up an internal network environment using TLD without a registrar (for example, a Windows network with Samba Active Directory), there is only one realistic solution: A standalone Certificate Authority which combines security with flexibility. By following such a procedure, a basic infrastructure can be set up in under an hour.

The procedure for issuing SSL/TLS certificates using an internal CA is especially useful for test environments or insular networks where Checkmk servers have a manageable number of users. It is also the only way to obtain certificates if you use one of the five reserved TLDs internally:  .example, .invalid, .local, .localhost, or .test. This is often the default case in Windows (Samba) environments and popular everywhere else to prevent both name resolution on the internet and routing to the internet. There are no registrars for these domains, so no ownership can be confirmed.

In this article we will show you how to build your own CA. This can be done with Linux on-board resources and requires little prior knowledge. Our example uses a domain in the .test TLD, however using your own CA you can issue certificates for any (even outside!) domains.

Even though a single person on a computer can perform all of the required actions, to illustrate the separation of roles in the certification process, we will have Carla as the Certificate Authority administrator, Bob as the Intermediate Key maintainer, and finally Alice, the administrator who gets the Server Certificates signed. You can increase the number of Intermediate levels as needed, depending on your requirements.

Security notes

  • This guide does not use a Certificate Revocation List. However, in larger production environments, you should use one to be able to quickly withdraw trust from any potentially compromised keys!
  • Even in test environments, strong passwords must be used and private keys must be well secured, because the CA key used can be used to create certificates for any domain. Systems that trust the certificate of a ‘lost’ key are thus vulnerable to ‘(wo)man-in-the-middle attacks’!

Creating keys for the CA and root certificate

For security reasons, Carla uses a notebook without an Internet connection to work with her CA. By storing the notebook in a safe, she also precludes the possibility of a third person installing a (hardware) keylogger on it.

Since in the course of working with the CA one creates some files with differing applications, it is advisable to use a uniquely named directory structure for this task. Carla creates this under /home/carla/ca:

carla@nb:~$ for d in certs newcerts crl private ; do mkdir -p ~/ca/$d ; done
carla@nb:~$ for f in index.txt serial ; do touch ~/ca/$f ; done

In the ~/ca/ folder, she places a ca.cnf file, an example of which we have prepared and attached at the end of this article. Carla then creates the CA's private key, which she stores in AES256 encrypted form and provides with a long passphrase:

carla@nb:~$ cd ca
carla@nb:~/ca$ openssl genrsa -aes256 -out private/ca.key.pem 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
.............++++
..........++++
e is 65537 (0x010001)
Enter pass phrase for private/ca.key.pem:
Verifying - Enter pass phrase for private/ca.key.pem:

The root certificate follows, here with a validity of twelve years and two weeks - more on the choice of validity periods below. This takes the relevant parameters for location and organization from the ca.cnf configuration file. Common Name and Email Address should be meaningful:

carla@nb:~/ca$ openssl req -config ca.cnf  -new \
    -key private/ca.key.pem -x509 -days 4398 \
    -sha256 -extensions v3_ca -out certs/ca.cert.pem
Enter pass phrase for private/ca.key.pem:

You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields, but you can leave some blank. For some fields there will be a default value, if you enter '.', the field will be left blank.

---
Country Name (2 letter code) [DE]:
State or Province Name [Bavaria]:
Locality Name [Munich]:
Organization Name [Stark Industries Ltd.]:
Organizational Unit Name []:
Common Name []: Stark Industries Root Certificate
Email Address []: carla@starkindustries.test

Carla next verifies the root certificate with the following command:

carla@nb:~/ca$ openssl x509 -noout -text -in certs/ca.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            25:a0:77:02:e7:29:07:f7:b5:d5:ba:b5:a3:4e:2b:eb:b3:e7:a6:c0
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = DE, ST = Bavaria, L = Munich, O = Stark Industries Ltd., CN = Stark Industries Root Certificate, emailAddress = carla@starkindustries.test
        Validity
            Not Before: Feb 15 14:52:12 2022 GMT
            Not After : Mar 2 14:52:12 2034 GMT
        Subject: C = DE, ST = Bavaria, L = Munich, O = Stark Industries Ltd., CN = Stark Industries Root Certificate, emailAddress = carla@starkindustries.test
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    00:d7:a9:89:94:dd:02:66:bf:fd:8e:c2:70:4b:d0:
[…]

Carla now copies the ca.cnf configuration file and provides it to her colleague Bob as a template. This way, Bob has the appropriate parameters as well as the company name and location in the correct notation.

The Intermediate Certificate

Bob is the one who does the day-to-day work with certificates for the local network. His job is to use an intermediate certificate to sign the server keys that administrators send to him. To do this, he first creates a directory structure and uses the folder in (like intermediate) in his home directory for this purpose:

bob@pc:~$ for d in certs newcerts crl private ; do mkdir -p ~/im/$d ; done
bob@pc:~$ for f in index.txt serial ; do touch ~/im/$f ; done

In the im folder, he places the configuration file he received from Carla as im.cnf and edits in it the default file paths to his home directory and role. He then also creates a private key that he will use for the next few years:

bob@pc:~$ cd im
bob@pc:~/im$ openssl genrsa -aes256 -out private/im.key.pem 4096

The next step is to create the Certificate Signing Request (CSR) for this key, the .csr file name  suffix is commonly used here:

bob@pc:~/im$ openssl req -config im.cnf -new -sha256 \
    -key private/im.key.pem -out certs/imbob.csr

[…]
Country Name (2 letter code) [DE]:
State or Province Name [Bavaria]:
Locality Name [Munich]:
Organization Name [Stark Industries Ltd.]:
Organizational Unit Name []:
Common Name []: Stark Industries Bobs Intermediate Certificate
Email Address []: bob@starkindustries.test

The Common Name here must be different from that of the root certificate. And here, too, you should use an accessible e-mail address. Bob takes the imbob.csr file to Carla's office for a cup of coffee. She fetches the notebook with the CA key from the safe, copies Bob's CSR into her certs folder and signs Bob's CSR – with a validity period of four years and two weeks in this example:

carla@nb:~$ cd ca
carla@nb:~/ca$ openssl ca -config ca.cnf \
    -extensions v3_intermediate_ca \
    -days 1476 -rand_serial -notext -md sha256 \
    -in certs/imbob.csr -out certs/imbob.pem

Now the passphrase for the key will be requested, the contents of the Certificate Signing Request are displayed and the Intermediate Certificate is created following confirmation. Carla now gives Bob the two PEM files ca.cert.pem (the PEM-encoded certificate of the Certificate Authority) and imbob.pem (the Intermediate certificate), shuts down the notebook and locks it in the safe. Back at his computer, Bob puts both files in the ~/im/certs folder.

Creating a key for the Checkmk site

Alice is the administrator of the internal Checkmk servers at Stark Industries Ltd. To secure a new Checkmk server, she first creates the server key. Here it is pragmatic not to use a passphrase, i.e. to do without -des3 or -aes256, otherwise you would have to re-enter this passphrase every time you start the server. For the filename, the server's host name is most appropriate.

alice@pc:~$ openssl genrsa -out checkmk.starkindustries.test.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
.................................++++
.......++++
e is 65537 (0x010001)

This is followed by the creation of the CSR. Again the questions as to the organization and department must be answered. Important here is the Common Name, which should correspond to the server's primary host name. To keep track, Alice names the key file and CSR file after the server on which both are used. A configuration file is optional for Alice, here she omits it:

alice@pc:~$ openssl req -new -key checkmk.starkindustries.test.key \
    -out checkmk.starkindustries.test.csr

[...]
Country Name (2 letter code) [DE]:
State or Province Name [Bavaria]:
Locality Name [Munich]:
Organization Name [Stark Industries Ltd.]:
Organizational Unit Name []:
Common Name (e.g. server FQDN or YOUR name) []: checkmk.starkindustries.test
Email Address []: alice@starkindustries.test

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

The challenge password may remain empty. Now Alice creates an X509 V3 certificate extension configuration file, checkmk.starkindustries.test.ext:

/home/alice/checkmk.starkindustries.test.ext

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = checkmk.starkindustries.test
DNS.2 = monitoring.starkindustries.test

Important: Even if you want to use a certificate for only one host name, the extension file is mandatory for all modern browsers. In this case it will contain only a single line under [alt_names].

Signing a certificate

With the two files, checkmk.starkindustries.test.csr and checkmk.starkindustries.test.ext in her possession, Alice invites herself to Bob's office for a cup of macha tea. Bob now uses his intermediate certificate to sign the CSR he brought with him, which will be valid for one year:

bob@pc:~$ cd im
bob@pc:~/im$ openssl x509 -CAcreateserial -req \
    -in certs/checkmk.starkindustries.test.csr \
    -CA certs/imbob.pem -CAkey private/im.key.pem \
    -out certs/checkmk.starkindustries.test.crt -days 365 \
    -sha256 -extfile certs/checkmk.starkindustries.test.ext

Alice receives three files from Bob in return: ca.cert.pem and imbob.pem represent the certificate chain and checkmk.starkindustries.test.crt is the certificate belonging to the server key checkmk.starkindustries.test.key. Alice can now roll out these files on the server:

/etc/apache2/sites-enabled/default-ssl.conf
SSLEngine on
SSLCertificateKeyFile /etc/certs/checkmk.starkindustries.test.key
SSLCertificateChainFile /etc/certs/imbob.pem
SSLCertificateFile /etc/certs/checkmk.starkindustries.test.crt

Importing the CA

The ways to import a CA certificate as trusted differ from browser to browser. Usually it is sufficient to add the ca.cert.pem certificate under Settings > Privacy and Security > Certificates > Import.

So that certificate management does not represent a stumbling block when automatically updating Checkmk agents, in the Agent Bakery we have provided the option to pass a separate CA certificate that is only used for agent updates. The system certificates remain untouched, but agent updates will still be possible.

As an alternative to distribution via an Agent Updater, you can integrate the root certificate into the host's local CA database. To do this, copy the ca.cert.pem file to /usr/local/share/ca-certificates/starkindustries.crt. Then regenerate the cache:

root@linux# update-ca-certificates

Under Windows it is possible to manage the system certificates via the MMC snap-in ‘Certificates’. This is necessary, for example, if you want to use a Microsoft browser to access a Checkmk secured with its own CA. You can read about the exact procedure in the article on Microsofts Knowledge Base Artikel PKI. Alternatively, you can distribute certificates via Intune.

Certificate validity duration and cascades

The entire certificate cascade only functions as long as all certificates are valid. Therefore, you should keep track of validity periods and create a new certificate at each point in the chain in good time. If the term of Carla's root certificate is twelve years and she thus issues intermediate certificates valid for a maximum of four years to Bob, she must issue herself a new root certificate in good time before eight years expire and use this one for signing in the future. Bob, on the other hand, who issues certificates valid for a maximum of two years, has to stop by for a coffee with Carla every two years to get a new certificate valid for a further four years.

Alice knows that Bob can only issue server certificates valid for two years. If Alice needs many server certificates as Stark Industries expands, Bob, as the holder of an intermediate certificate for Alice, can issue another intermediate certificate with a validity of two years. In this way, Alice can then take on the role of a subordinate CA that issues certificates with a validity of one year. In this case, she must check in with Bob annually.

In the configuration example attached below, the SSLCertificateChainFile required is not Bob's intermediate certificate, but a concatenation of Bob's and Alice's intermediate certificates.

What about compromised certificates?

At first glance, Carla's security measures may seem a bit paranoid, but compromised root certificates are difficult to deal with because you ultimately have to remove the certificate on all clients that use it. For this reason, we advise locking away the root certificate even in comparatively small test environments - it doesn't have to be your own notebook, often the USB stick in the safe is secure enough. Regular signing can then be done using the intermediate certificate.

In manageable environments, Bob can keep a list of certificates signed with a particular intermediate certificate and notify affected server admins that they need new certificates when an IM certificate has been compromised.

This is more difficult if Bob himself turns out to be untrustworthy because, for example, he has issued certificates for other people's domains in order to spy on accounting traffic. So in growing environments, keeping a Certificate Revocation List is a good idea. Jamie Nguyen describes the required additional steps very well in his blog.

Appendix: The ca.cnf configuration file

The configuration file /home/carla/ca/ca.cnf copies (and thus overwrites) many sections of the system-wide openssl.cnf. Copy this file as the basis for your own CA configuration (back to key generation). Then first customize the directories (in the [ CA_default ] section) and the company data (in the [ req_distinguished_name ] section).

/home/carla/ca/ca.cnf
# OpenSSL root CA configuration file.
# Copy to `/home/username/ca/ca.cnf`.

[ ca ]
# `man ca`
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = /home/carla/ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.cert.pem

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_strict

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ policy_loose ]
# Allow the intermediate CA to sign a more diverse range of certificates.
# See the POLICY FORMAT section of the `ca` man page.
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

# Optionally, specify some defaults.
countryName_default             = DE
stateOrProvinceName_default     = Bavaria
localityName_default            = Munich
0.organizationName_default      = Stark Industries Ltd.
organizationalUnitName_default  =
emailAddress_default            =

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ usr_cert ]
# Extensions for client certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ server_cert ]
# Extensions for server certificates (`man x509v3_config`).
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ crl_ext ]
# Extension for CRLs (`man x509v3_config`).
authorityKeyIdentifier=keyid:always

[ ocsp ]
# Extension for OCSP signing certificates (`man ocsp`).
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, digitalSignature
extendedKeyUsage = critical, OCSPSigning


Related Articles

Raw vs. Enterprise: When is it advisable to switch?
Raw vs. Enterprise: When is it advisable to switch?
This blog shows the most important differences between the Checkmk Raw Edition and the Checkmk Enterprise Edition and helps you to decide whether it…
Open observability: How to integrate Datadog in Checkmk
Open observability: How to integrate Datadog in Checkmk
Integrate Monitors and events from Datadog in Checkmk to save time and find root causes faster than ever before. This blog shows how it is done.
Sending Checkmk notifications via Telegram
Sending Checkmk notifications via Telegram
In this article you will learn how to use the popular messaging platform Telegram for sending notifications from Checkmk.