SSH access to the Discoverer and Discoverer+ login nodes using PKCS#11 HSM and PIV devices (smart cards)
Important
This document does not provide information about how to use HSM and PIV devices for SSH authentication on Windows systems. This is due to problematic HSM and PIV support on Windows systems - different abstraction layers and frequent driver compatibility issues that can break functionality after system updates. This guide is specifically designed for Linux and macOS users.
Introduction
What HSM and PIV devices are
HSM (Hardware Security Module) is a physical device that provides secure cryptographic operations and key storage. HSM devices are designed to protect sensitive cryptographic material from unauthorized access and tampering. HSM devices are more expensive, but they are more powerful and can be used for other purposes like key management, data encryption, etc. Some HSM devices support FIPS/NIST standartisations. Here we are particularly interested in HSM devices that support PKCS#11 interface and operations with assymetric key pairs (RSA, ECDSA).
PIV (Personal Identity Verification) devices are smart card-based authentication tokens that make use of the FIPS 201 standard, originally developed for U.S. federal government employee identification. Note that some PIV devices do not strictly follow FIPS 201 standard (always check the device specifications). PIV devices are designed as multi-purpose identity tokens for authentication, digital signing, and secure communication. PIV devices are cheaper, but they are less powerful and rarely be used for other purposes than authentication and digital signing. For instance, it is rare to find a PIV device that provides data encryption. Here we are particularly interested in PIV devices that support PKCS#11 interface and operations with assymetric key pairs (RSA, ECDSA).
Warning
Limited or no support for ED25519 curve: Although most HSM and PIV devices currently available in the market support operations with public key algorithms, this support is typically restricted to RSA and NIST-P elliptic curves. The vendors’ attempt to conform to FIPS or other standards is the reason for that limitation. Some devices, like the SmartCard HSM, can support a wider variety of elliptic curves, including Brainpool curves. However, it is exceedingly uncommon to acquire an HSM or PIV device that validates the ED25519 curve. A contributing factor to the absence of support is the fact that X.509v3 certificates are incompatible with the ED25519 keys. Given that post-quantum cryptography has already entered the market, it is improbable that the companies that manufacture HSM and PIV devices will allocate resources to the implementation of ED25519 support in the cryptographic protocol. As they move forward with the adoption of post-quantum algorithms for digital signing and key exchange, they intend to cease improving the implementation of traditional public key algorithms.
Both HSM and PIV devices typically consist of:
Smart card: A credit card-sized or SIM-sized device containing:
Secure microprocessor with tamper-resistant protection
Protected memory for storing private keys and certificates (it may supports different slots for storing private keys and certificates that can be used for different purposes)
Cryptographic engine for performing signing and encryption operations (PIV devices may not support data encryption operations)
USB card reader: Provides the interface between the smart card and the computer system
Note
Many modern HSM and PIV devices combine both components into a single monolithic USB device for convenience and security. Also note that it is very rare to obtain HSM devices that do not support PKCS#11 interface. While some PIV devices may not support PKCS#11 interfaces. Always check the device documentation to ensure it supports assymetric key pair generation and signing operations through PKCS#11 interface.
Key differences between HSM and PIV devices
Aspect |
HSM Devices |
PIV Devices |
|---|---|---|
Primary purpose |
Cryptographic operations and key storage |
Identity verification and authentication |
Security focus |
High-security crypto operations |
Multi-purpose identity management |
Key storage |
Protected memory with tamper-resistant design |
Standard smart card storage with basic protection |
Cryptographic engine |
Dedicated secure processor |
Standard smart card processor |
Use cases |
Document signing, SSH authentication, encryption |
System login, physical access, document signing, SSH auth |
Standards compliance |
PKCS#11, various crypto standards |
FIPS 201, PIV standard |
HSM or PIV backed SSH authentication using PKCS#11 as a key provider
Important
PKCS#11 is a standardized interface that allows applications (like SSH clients and SSH agents) to use the keys stored on the HSM and PIV devices. Therefore PKCS#11 acts as a formal SSH key provider. Its presence is necessary because without it neither the SSH agent nor the SSH client can list and utilize for signing operations the private keys residing in the HSM and PIV devices protected memory.
Keeping the SSH private keys on HSM and PIV devices and using PKCS#11 as a key provider is safer than storing the key files on the local filesystem:
Aspect |
|
HSM/PIV device storage |
|---|---|---|
private key location |
Stored as files on filesystem |
Stored in protected memory, never leaves device |
key access |
Direct file access by applications |
Through PKCS#11 interface only |
security risk |
Vulnerable to file system attacks, cold boot attacks |
Protected by tamper-resistant hardware |
key operations |
Performed in system memory |
Performed inside secure processor |
portability |
Keys can be copied, backed up, moved |
Keys may not be extracted or copied |
compromise risk |
High - keys accessible to malware, unauthorized users |
Low - keys protected protected even on compromised systems |
PKCS#11 interface can be accessed through the OpenSC library, which is supported on Linux, macOS, and other operating systems, or based on vendor-provided drivers.
Important
Only HSM and PIV devices that support public key cryptography (RSA, ECDSA) and provide PKCS#11 interfaces can be employed for SSH authentication on Linux and macOS. These are essential requirements for the SSH client to communicate with and utilize the smart card device for authentication operations. Note that the private keys are stored in protected memory within the device and never leave this secure environment. All cryptographic operations (signing, encryption) are performed by the secure processor inside the device, ensuring that private keys remain protected even when the device is connected to potentially compromised systems. The later does not mean that the use of HSM and PIV devices on a compromised system can proted your data - it just protects the private key stored in the device.
SSH authentication works by having the SSH client perform digital signing operations using the private key from the asymmetric key pair stored in the protected memory of the HSM or PIV device (the SSH agent facilitates the communication between the SSH client and the HSM or PIV device). The SSH server then validates these signatures using the corresponding public key that has been registered in the server-side authorized keys file.
HSM and PIV devices that store X.509v3 certificates, and can perform digital signing operations through their PKCS#11 interface, can also be used for key-based SSH authentication. This is possible because both digital signing and SSH authentication use the same underlying asymmetric key pair - the private key for signing operations and the public key (which is embedded in the X.509v3 certificate) for verification. Therefore, any smart card device that contains a private key corresponding to a public key in its stored X.509v3 certificate, and provides a PKCS#11 interface, can be used for SSH authentication.
Warning
Discoverer and Discoverer+ setup: The SSH servers running on Discoverer and Discoverer+ login nodes are configured to use key-based authentication only. We do not validate X.509v3 certificates on the SSH servers, and therefore we do not intercept, check, or store any X.509v3 metadata (subject, subject alternative names, etc.). This ensures user privacy by not collecting any personal information from the certificates.
Using your own PKCS#11 HSM or PIV devices
Warning
This part of the document covers the “bring your own HSM or PIV device” cases, where users can use their own HSM or PIV devices for SSH authentication. Therefore, these devices are not obtained through a contract with EU-recognized PKI certificate authorities.
Security considerations
Warning
Before purchasing any HSM or PIV device, verify its OpenSC compatibility by checking the OpenSC supported cards list and testing on your target operating systems.
Important
Consider the device’s country of origin and vendor reputation. Avoid devices produced by vendors in jurisdictions that may require implementing backdoors or compromise device security.
Recommended HSM devices and purchasing options
If you do not already have in your possession an HSM or PIV device with PKCS#11 interface, such devices can be purchased online from various vendors.
We recommend purchasing at least two HSM or PIV devices and using the second one as a backup device in case the first one becomes lost. This will prevent you from being locked out of our login nodes.
Here are some HSM and PIV devices we have tested and recommend:
Nitrokey HSM2 - Open-source hardware security module with excellent OpenSC support for Linux and macOS
SmartCard HSM - Professional smart card-based HSM with proven OpenSC compatibility
YubiKey 5 series (PIV) - Personal identity verification device with OpenSC support for Linux and macOS.
Note
For using YubiKey 5 series PIV devices, follow the instructions in SSH access to the Discoverer and Discoverer+ login nodes with YubiKey PIV smartcard for using it for SSH authentication.
YubiHSM2 - Enterprise-grade hardware security module with comprehensive OpenSC support (high-end solution)
Feel free to purchase and use any other HSM or PIV device (at your own risk), but before ordering ensure it meets the following requirements:
Compatible with Linux and macOS systems
Supports public key cryptography (RSA, ECDSA)
Provides PKCS#11 interface for SSH authentication through OpenSC or vendor-provided PKCS#11 drivers
Important
YubiKey 5 series and other PIV (personal identity verification) devices can also be used for SSH authentication, but they are not traditional HSM PKCS#11 devices, even though they provide PKCS#11 interface. The way such devices can be utilized for authentication is presented here SSH access to the Discoverer and Discoverer+ login nodes with YubiKey PIV smartcard.
Installation of the software components
Most probably, you have some or most of the components, which installation is recommended below. Anyway, run the suggested commands to ensure everything needed is already in your system:
Linux systems
# RHEL/Rocky Linux/Fedora
sudo dnf install opensc openssh-clients
# Debian/Ubuntu
sudo apt-get install opensc opensc-pkcs11 libp11-3 openssh-client
macOS systems
# Using Homebrew
brew install opensc openssh
Generate key pair directly on the HSM or PIV device
First, check if the HSM or PIV device supports at least RSA keys (≥2048 bits) and ECDSA (≥384 bits):
# You may need to change the full path to the module to match your OpenSC installation
pkcs11-tool --module /usr/lib64/opensc-pkcs11.so --list-mechanisms
The output that verifies the necessary support is available will include the following:
ECDSA-SHA384, keySize={192,521}, sign, verify
ECDSA-SHA512, keySize={192,521}, sign, verify
...
RSA-X-509, keySize={1024,4096}, hw, decrypt, sign, verify
RSA-PKCS, keySize={1024,4096}, hw, decrypt, sign, verify
If the examined HSM or PIV device does not support those key sizes, we recommend obtaining a more modern (up-to-date) one.
Limit the number of key pairs available in the HSM or PIV device
Warning
The fewer private keys on your HSM or PIV device, the better for SSH authentication. When the SSH agent loads the PKCS#11 provider, it discovers ALL private keys on the device. The SSH client then attempts authentication with each key sequentially until one succeeds.
Here is an example. If your HSM or PIV device contains 4 private keys, but only 1 is registered for your user on the SSH server, the authentication process will fail 3 times before succeeding. Most SSH servers (including our login nodes) have rate limiting that bans IP addresses after a certain number of consecutive failed attempts.
Example bad scenario:
Your HSM or PIV device has 4 private keys (IDs 01, 02, 03, 04)
Only key #04 is registered for your user on the SSH server running on our login nodes
SSH client tries key #01 → FAIL (1st failure)
SSH client tries key #02 → FAIL (2nd failure)
SSH client tries key #03 → FAIL (3rd failure)
SSH client tries key #04 → SUCCESS (but server may have already banned your IP)
Therefore, use HSM or PIV devices with no more than 3 private keys for SSH authentication to avoid triggering rate limits on the SSH servers running on the login nodes.
Key-pair generation process
Warning
At this point, you must know at least the User PIN of the device to unlock its protected memory and utilize the key generation process. It is recommended to also know the SO-PIN in case you lock the device by entering a wrong PIN too many times.
Important
Knowing the SO-PIN does not necessarily mean you can unblock the User PIN if it is blocked. If the HSM or PIV device was initialized with a policy where the SO-PIN can only be used only for device initialization and cannot reset the User PIN, then the only remaining option is to reset the device completely. This would result in the loss of all stored keys and certificates, which could be catastrophic for your SSH access and other services.
First, check if your HSM or PIV device already contains key pairs (you may need to change the full path to the module to match your OpenSC installation):
# List private keys (requires PIN)
pkcs11-tool --module /usr/lib64/opensc-pkcs11.so --list-objects --type privkey --login
This check is necessary to verify you have at least one usable key pair on the examined device. For the same reason, you can list the installed public keys with --type pubkey. However, keep in mind that public keys can be stored in the device’s memory without their corresponding private keys, which can create a false impression. Therefore, to make sure you have the entire list of key pairs that might be appropriate for SSH authentication, always list the private keys. Also mind the presence of --login in the command line. In order to use the private keys, the PIN must be provided.
In case you already have private keys usable for authentication on the device, go to Loading the identities from PKCS#11 provider into the SSH agent. Exporting the public SSH key and learn how to extract the public keys and communicate them with us, so we can register them with your POSIX account you have with us on Discoverer and Discoverer+. Otherwise continue with the instructions given below.
If your HSM or PIV device does not already contain a key pair, or if the existing key pairs use weak algorithms (RSA < 2048 bits, ECDSA < 384 bits), you can generate a new one directly on the device using pkcs11-tool. This ensures the private key is generated within the secure environment of the device and never leaves the device.
Note
The --id parameter must reference a slot that is not already occupied (that is why we recommend listing the private keys first before starting a new key pair generation) - this does not work on SmartCard HSM and Nitrokey HSM 2 (read below):
# Generate ECDSA key pair on HSM or PIV device (384 bits minimum, requires PIN)
pkcs11-tool --module /usr/lib64/opensc-pkcs11.so \
--login --keypairgen --key-type ec:secp384r1 \
--id 01 --label "SSH-Key-ECDSA"
# Generate RSA key pair on HSM or PIV device (2048 bits minimum, requires PIN), only if you cannot generate or use ECDSA keys
pkcs11-tool --module /usr/lib64/opensc-pkcs11.so \
--login --keypairgen --key-type rsa:2048 \
--id 02 --label "SSH-Key-RSA"
Warning
DKEK-provisioned HSM restriction (may affect the use of Smart Card HSM and Nitrokey HSM 2): If your SmartCard HSM is already provisioned with DKEK shares, the keys generated using pkcs11-tool will be stored in the HSM protected memory, but may not be displayed or selectable for use. In such cases, users must rely on the SmartCard HSM documentation to add keys using PKCS#12 files using the saved as files DKEK share. Therefore, before attempting to generate new keys, verify if your SmartCard HSM device has DKEK shares configured.
See Loading the identities from PKCS#11 provider into the SSH agent. Exporting the public SSH key below for understanding how to obtain copy of the public key that should be sent to us for registration.
Bring your existing SSH key to the HSM or PIV device
Warning
As previously mentioned, the majority of HSM and PIV devices do not support the ED25519 curve. Consequently, we will not be discussing importing private keys from ~/.ssh/id_ed25519 here.
Here we consider the situation when you already have your SSH private key, saved as ~/.ssh/id_ecdsa, ~/.ssh/id_rsa, or stored as a part of a PKCS#12 file. For that purpose, be sure your HSM or PIV device supports the import of private keys. Some devices may not provide such a functionality, or you need to use tools different than those provided by OpenSC.
Important
SmartCard HSM and Nitrokey HSM2: Direct import of private keys into HSM devices using pkcs11-tool or pkcs15-init is not supported by SmartCard HSM and Nitrokey HSM2 devices. The recommended approach there is to use DKEK shares for importing private keys stored in PKCS#12 files.
In case you need to import ~/.ssh/id_ecdsa or ~/.ssh/id_rsa keys into such type of HSM or PIV devices, you need first to convert those keys into PKCS#12 files (generating and adding a formal X.509v3 certificate on top of each private key):
# Generate a formal self-signed X.509v3 certificate from the private key (validity is of no significance here)
openssl req -new -x509 -key ~/.ssh/id_ecdsa -out /tmp/my.crt -days 3650 -subj "/CN=SSH-ECDSA-Key"
# Create PKCS#12 container with private key and certificate (use single password!)
openssl pkcs12 -export -in /tmp/my.crt -inkey ~/.ssh/id_ecdsa -out /tmp/my.p12 -name "SSH-ECDSA-Key"
And then follow the instructions provided here:
to import the created PKCS#12 file (/tmp/my.p12) into the HSM or PIV device using DKEK shares. The same applies to the case when you already have PKCS#12 file derived without converting ~/.ssh/id_ecdsa or ~/.ssh/id_rsa.
Important
YubiHSM 2: YubiHSM 2 devices support import of asymmetric private keys using wrapped key material. The device can import RSA and ECDSA private keys that are properly wrapped.
Here we need to convert the PKCS#12 file into YubiHSM 2 wrapped format and import the result into the protected memory:
Extract private key from PKCS#12:
# Extract private key from PKCS#12 file
openssl pkcs12 -in your_key.p12 -nocerts -out private.pem
Create Wrap key (if not already present):
# Create Wrap key file
echo -en '\x00\x11\x22\x33\x44\x55\x66\x77\x88\x99\xaa\xbb\xcc\xdd\xee\xff' > wrap.key
# Import Wrap key into YubiHSM 2
yubihsm-shell -p password -a put-wrap-key -i 20 -c all --delegated all --informat bin --in wrap.key
Wrap the private key:
# For RSA keys
yubihsm-wrap -a rsa2048 -c sign-pkcs -d 1,2,5 --id 30 --label SSH_RSA_Key --in private.pem --wrapkey wrap.key --out private.yhw
# For ECDSA keys
yubihsm-wrap -a ecp256 -c sign-ecdsa -d 1,2,5 --id 31 --label SSH_ECDSA_Key --in private.pem --wrapkey wrap.key --out private.yhw
Import wrapped key:
# Import the wrapped key into YubiHSM 2
yubihsm-shell -p password -a put-wrapped --wrap-id 20 --in private.yhw
Supported asymmetric key types: - RSA keys (various key sizes: rsa2048, rsa3072, rsa4096) - ECDSA (NIST-P) keys (various curves: ecp256, ecp384, ecp521)
Note: The Wrap key must already be present on the YubiHSM 2 device before importing wrapped keys. The .yhw format is the YubiHSM 2 wrapped key format. For detailed procedures, refer to the YubiHSM 2 documentation.
DKEK vs Wrap keys: - DKEK (SmartCard HSM/Nitrokey HSM2): Establishes a logical security domain across multiple devices. Keys are encrypted with DKEK shares rather than passwords. DKEK shares are created using the device’s random number generator and encrypted with custodian passwords. - Wrap keys (YubiHSM 2): Simple 16-byte binary keys used for wrapping/unwrapping individual keys. Each wrapped key is encrypted with a specific Wrap key stored on the device.
Important
Other HSM devices: If your HSM device supports direct private key import (check vendor’s documentation), you may use this command line to import directly a private key stored inside an existing PKCS#12 file (see the example above about how to create that file):
pkcs15-init --auth-id 01 -f pkcs12 -S /tmp/my.p12
See Loading the identities from PKCS#11 provider into the SSH agent. Exporting the public SSH key below for understanding how to obtain copy of the public key that should be sent to us for registration.
SSH client side configuration
Client-side configuration (optional)
Create or modify ~/.ssh/config:
# HSM-based SSH configuration for multiple hosts
Host login.discoverer.bg
HostName login.discoverer.bg
Port 22
User your-discoverer-username
PKCS11Provider /usr/lib64/opensc-pkcs11.so
IdentitiesOnly yes
IdentityFile none
Host login-plus.discoverer.bg
HostName login-plus.discoverer.bg
Port 22
User your-discoverer-username
PKCS11Provider /usr/lib64/opensc-pkcs11.so
IdentitiesOnly yes
IdentityFile none
Host login.bg.discoverer.bg
HostName login.bg.discoverer.bg
Port 2222
User your-discoverer-username
PKCS11Provider /usr/lib64/opensc-pkcs11.so
IdentitiesOnly yes
IdentityFile none
Loading the identities from PKCS#11 provider into the SSH agent. Exporting the public SSH key
# You may replace /usr/lib64/opensc-pkcs11.so with the full path to your PKCS#11 provider library
ssh-add -s /usr/lib64/opensc-pkcs11.so
At this point, entering the PIN will be requested. Afterwards, the identity will be visible when executing:
ssh-add -L
From the list of the keys provided, select those you want us to register for your Discoverer or Discoverer+ account with us. Send the copy of the selected keys following the instructions given in Extracting the public key for registering.
In case you need to list the hashes of each key accessible to the SSH agent, run:
# Test SSH key extraction (this lists the hashes of the keys!). You may replace /usr/lib64/opensc-pkcs11.so with the full path to your PKCS#11 provider library
ssh-keygen -D /usr/lib64/opensc-pkcs11.so -l
Important: Do not unplug the HSM or PIV device until you unload the PKCS#11 provider library (see below how to do that). Otherwise you may not be able to log in.
Unloading the identities from PKCS#11 provider from the SSH agent
Execute this command before unplugging HSM or PIV device:
# You may replace /usr/lib64/opensc-pkcs11.so with the full path to your PKCS#11 provider library
ssh-add -e /usr/lib64/opensc-pkcs11.so
Troubleshooting
Common issues
PKCS#11 module not found
# Check module availability ls -la /usr/lib64/opensc-pkcs11.so
Certificate not recognized
# List certificates in HSM pkcs11-tool --module /usr/lib64/opensc-pkcs11.so \ --list-objects --type cert
SSH connection failures
# Enable verbose SSH debugging ssh -v -o PKCS11Provider=/usr/lib64/opensc-pkcs11.so user@host
HSM token is unplugged until the PKCS#11 provider module is loaded
# You may replace /usr/lib64/opensc-pkcs11.so with the full path to your PKCS#11 provider library ssh-add -e /usr/lib64/opensc-pkcs11.so
Cannot SSH login due to the error “sign_and_send_pubkey: signing failed for XXXXXXX from agent: agent refused operation”
This error occurs when the SSH agent is not configured correctly to access the private key stored on the HSM or PIV device or the connection to the device is lost.
In many cases, the error is caused by detaching the HSM or PIV device from the USB port without prompty unloading the PKCS#11 module from the system, i.e. without executing this command before the detachment:
# Detach the PKCS#11 module for communicating with # the HSM or PIV device from the SSH agent: ssh-add -e /usr/lib64/opensc-pkcs11.so
If that situation occurs, you may need to kill the SSH agent process and start it again. This will reset the connection to the HSM or PIV device and you will be allowed to re-add the PKCS#11 provider to the SSH agent and properly connect to the device:
killall ssh-agent; eval "$(ssh-agent)"
Once the SSH agent is restarted, re-add the PKCS#11 provider to the SSH agent:
ssh-add -s /usr/lib64/opensc-pkcs11.so
and try to perform the SSH login again.
Another aspect of the problem is that sometimes, even if the SSH agent is killed in the way shown above, the newly attached one is available only in the current terminal session (where the killing and starting took place). To make the newly started SSH agent available in all terminal sessions, you need to restart the bash session for the current user, running the window manager. One way to do that is to close the terminal window and open a new one. Then in that newly created terminal, execute:
# Restart the bash session for the current user exec bash
Note
This is a common issue when using PKCS#11 in terminal in GNOME.
I re-programmed the HSM or PIV device and now even if I provide the correct credentials, the OpenSC library cannot perform
In many cases, the OpenSC tools keep specific to the device cache information inside:
~/.cache/opensc/
You may safely remove the content stored inside and try using the device again:
rm -fr ~/.cache/opensc/*
Getting help
For technical support and implementation assistance, contact our support team. See Getting help.