SSH access to the Discoverer and Discoverer+ login nodes with YubiKey PIV smartcard

Introduction

The proposed PIV configuration is intended for advanced HPC users and system administrators who need maximum security for remote access. Here, the implementation hardware-based key storage prevents SSH private keys from being stored in volatile memory, eliminating the risk of key extraction through memory dumps or cold boot attacks.

This guide explains how to set up your YubiKey PIV smartcard for SSH authentication using its PIV application on Linux and macOS systems.

It covers two approaches: importing an existing OpenSSH key pair from the ~/.ssh folder, or generating a new key pair—both of which are imported into slot 9a of the YubiKey PIV smartcard. In either case, a PKCS#12 file container is used to securely store the keys on the file system before importing them into slot 9a.

Important

This documentation assumes that users are familiar with SSH agent usage and management. The private keys stored on the PIV smartcard can only be accessed through the SSH agent using the PKCS#11 interface. The SSH agent acts as a bridge between SSH clients and the hardware-stored keys, allowing secure authentication without direct access to the private key material.

Warning

This guide focuses on importing X.509v3 certificates with their complementary private keys into the PIV slots of YubiKey PIV smartcards. Therefore, only RSA and ECDSA SSH keys are supported. Ed25519 keys are not supported by this type of setup.

Cost of deployment (gross estimate with VAT incl): ~ €140, per user

Prerequisites

• YubiKey PIV smartcard devices (at least two of them - one should be used as a backup)
• Up-to-date Linux or macOS system (see Software Installation below for required tools and installation instructions)
• Administrative access (to install the software, if necessary)

Warning

CRITICAL SECURITY WARNING: YubiKey devices running firmware versions below 5.7 are vulnerable to security attacks. Do not use or purchase YubiKey devices with firmware versions below 5.7 for any security-sensitive operations. If your YubiKey has firmware version below 5.7, return it (if possible) or dispose it, and purchase a new YubiKey with firmware version 5.7+. For detailed information about this vulnerability, see:

https://www.yubico.com/support/security-advisories/ysa-2024-03/

Important

IMPORTANT PURCHASE WARNING: Some resellers may still have in sell YubiKey 5C smartcards with old firmware versions (<5.7). Since the firmware version is not printed on the product packaging (it is Yubico fault!), you must unbox the smartcard and check its firmware version immediately after purchase using the command

yubico-piv-tool -a status | grep ^Version

or

ykman piv info | grep ^"PIV version"

Alternatively you may check the version of the firmware using Yubico Authenticator on Android and iOS.

Warning

If the firmware version shown is below 5.7, return the device immediately and request a replacement with firmware 5.7+. Do not accept any device with older firmware. See “CRITICAL SECURITY WARNING” above.

Software Installation

Linux (Fedora/RHEL/Rocky Linux/Alma Linux):

# Install EPEL repository (required for yubico-piv-tool)
sudo dnf install epel-release

# Install required packages
sudo dnf install yubico-piv-tool yubikey-manager opensc openssh-clients

Linux (Ubuntu/Debian):

# Update package list
sudo apt update

# Install required packages
sudo apt install yubico-piv-tool yubikey-manager opensc openssh-client

macOS:

# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install required packages
brew install yubico-piv-tool yubikey-manager opensc openssh

Check YubiKey Status and Firmware Version

Attach the YubiKey smartcard to an USB port on your system. Then start a terminal and execute

yubico-piv-tool -a status

This command will display the version of the firmware installed, the serial number of the smartcard, as well as some other PIV-specific information:

Example output:

$ yubico-piv-tool -a status
Version:     5.7.1
Serial Number:       81209505
CHUID:
CCC:
PIN tries left:      3

In case you already have certificates and keys installed in the PIV slots, a longer output will be printed out. For example:

Version:     5.7.1
Serial Number:       81209505
CHUID: 3019d4e739da739ced39ce739d836858210842108421c84210c3eb3410d54658aaa9eb
        82eb6f924964ddfef813350832303330303130313e00fe00
CCC:   f015a000000116ff02013b4b8ce1b3962b5bb06650ad6af10121f20121f300f40100f5
        0110f600f700fa00fb00fc00fd00fe00
Slot 9a:
     Algorithm:      ECCP384
     Subject DN:     CN=Test User

Warning

SECURITY CHECK: If the firmware version shown is below 5.7, refer to the Prerequisites section above for the required firmware version number. Do not proceed with this setup on devices with outdated firmware, regardless of any assurances from the vendor about the risk being “minor.” Using a YubiKey with firmware below 5.7 exposes you to serious security vulnerabilities. See “CRITICAL SECURITY WARNING” above.

Smartcard Initialization

Why do we need to initialize the smartcard?

The YubiKey PIV smartcard comes with default PIN, PUK, and Management Key values that are publicly known and should never be used in production environments. Initialization is a critical security requirement that sets unique, secure values for these administrative credentials.

Warning

Initialization will permanently delete all existing data on the PIV smartcard, including any stored certificates, keys, and configuration settings. This process cannot be undone.

Default values (DO NOT USE):

• PIN: 123456
• PUK: 12345678
• Management Key: 010203040506070801020304050607080102030405060708

Recommended secure values:

• PIN: 6-8 digit number (you’ll choose this)
• PUK: 8-digit number (you’ll choose this)
• Management Key: 24-byte AES192 key (automatically generated)

Step 1: Set a new PIN

# Set a new PIN (you'll be prompted for the default PIN, then the new one):
yubico-piv-tool -a change-pin

Important

If you try this on a Yubikey PIV device with PIN already set, and you do not know that PIN (it is not the default one, or does not match any PIN stored by you in your password wallet, or you don’t remember the PUK), then the only way you can proceed with the initialization is after resetting the PIV application on the Yubikey following Reset PIV Smart Card. If you remember the PUK, try Unblock PIV PIN.

Step 2: Set a new PUK

# Set a new PUK (you'll be prompted for the default PUK, then the new one):
yubico-piv-tool -a change-puk

Important

If you try this on a Yubikey PIV device with PUK already set, and you do not know that PUK (it is not the default one, or does not match any PUK stored by you in your password wallet), then the only way you can proceed with the initialization is after resetting the PIV application on the Yubikey following Reset PIV Smart Card

Step 3: Set the new management key using AES192 algorithm

# Generate a new management key:
openssl rand -hex 24

# Set the new management key using AES192 algorithm
# (you'll be prompted for the default management key, then the new one):
yubico-piv-tool -a set-mgm-key --new-key-algo=AES192 -k -

Important

If you try this on a Yubikey PIV device with management already set (different than the default one), and you do not know that management key (it is not the default one, or does not match any such key stored by you in your password wallet), then the only way you can proceed with the initialization is after resetting the PIV application on the Yubikey following Reset PIV Smart Card

Step 4: Verify the initialization

# Verify that the initialization was successful
ykman piv info

Importing Keys into YubiKey PIV Slots

Note: This guide does not recommend generating keys directly on the smartcard for backup and redundancy reasons. Instead, we’ll import externally generated keys.

Note on OpenSSH Private Key Format Compatibility

Modern OpenSSH private keys use a new format that is not compatible with the PIV standard. If you have existing OpenSSH keys in the new format, you’ll need to convert them to the traditional PEM format before importing.

Converting OpenSSH format keys to PEM format:

# Copy your existing key to a new file
cp ~/.ssh/id_ecdsa ~/.ssh/id_ecdsa_pem

# Convert the key to PEM format
ssh-keygen -p -f ~/.ssh/id_ecdsa_pem -m pem

Creating X.509v3 certificate and PKCS#12 container:

# Create a self-signed X.509v3 certificate
openssl req -x509 -new -sha384 \
    -key ~/.ssh/id_ecdsa_pem \
    -out ~/.ssh/id_ecdsa_pem.crt \
    -subj "/CN=SSH Key" \
    -days 3650

# Create PKCS#12 container with the private key and certificate
openssl pkcs12 -export -maciter \
    -inkey ~/.ssh/id_ecdsa_pem \
    -in ~/.ssh/id_ecdsa_pem.crt \
    -name "SSH Key" \
    -out ~/.ssh/id_ecdsa_pem.p12

Importing PKCS#12 into slot 9a:

# Import the PKCS#12 container into slot 9a
# (you'll be prompted for the management key):
yubico-piv-tool -s 9a -K PKCS12 \
    -a import-key -a import-cert \
    -i ~/.ssh/id_ecdsa_pem.p12 -k -

Importing externally generated RSA or ECDSA key from PKCS#12 file into slot 9a

If you have a PKCS#12 file with an RSA or ECDSA key and certificate:

# Generate a new ECDSA key pair (example)
openssl ecparam -genkey -name secp521r1 | \
    openssl ec -aes256 -out private.pem

# Create a self-signed certificate
openssl req -x509 -new -sha384 \
    -key private.pem \
    -out formal.crt \
    -subj "/CN=SSH Key" \
    -days 3650

# Create PKCS#12 container
openssl pkcs12 -export -maciter \
    -inkey private.pem \
    -in formal.crt \
    -name "SSH Key" \
    -out formal.p12

# Import into slot 9a
yubico-piv-tool -s 9a -K PKCS12 \
    -a import-key -a import-cert \
    -i formal.p12 -k -

To extract the OpenSSH public key for registration with the remote system, run ssh-add -L. Copy the public key from the command output and provide it to the system administrator for registration with your account, following the documentation in SSH Public Key Exchange.

SSH Agent Configuration

The SSH agent needs permission to access the private key stored on the PIV smartcard. This is done through the PKCS#11 interface.

Add the PIV smartcard key to the SSH agent:

# Add the PIV smartcard key to the SSH agent
ssh-add -s /usr/lib64/opensc-pkcs11.so

Note: The path to the PKCS#11 library may vary depending on your system:

• Linux: /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so (Ubuntu/Debian) or /usr/lib64/opensc-pkcs11.so (Fedora/RHEL)
• macOS: /usr/local/lib/opensc-pkcs11.so (if installed via Homebrew)

Verify the key is accessible:

# List the keys available to the SSH agent
ssh-add -L

This command will display the public key from the PIV smartcard, confirming that the SSH agent can access it.

Extract the public key for remote system registration:

# Extract the public key for registration with remote systems
ssh-add -L

Copy the output and provide it to the system administrator for registration with your account on the remote system.

Remove the key from SSH agent (if needed):

# Remove the PIV smartcard key from the SSH agent
ssh-add -e /usr/lib64/opensc-pkcs11.so

Troubleshooting

Unblock PIV PIN

If the PIN is forgotten but the PUK is known:

ykman piv access unblock-pin

Note

Provide first the PUK, and then set a new PIN. In case you do not remember the PUK, nothing can be done to obtain access to the keys stored in the PIV card. In that case you may only reset the PIV card (see next section).

Reset PIV Smart Card

Warning

The command shown below resets the PIN, PUK, and management key to their default values. All stored keys and certificates will be erased.

To remove only the information stored in the PIV application on your Yubikey, execute:

ykman piv reset --force

Note

This operation does not remove stored passkeys (FIDO2 credentials) from the YubiKey.

Fixing “sign_and_send_pubkey: signing failed for XXXXXXX from agent: agent refused operation” during SSH login

This error occurs when the SSH agent is not configured correctly to access the private key stored on the YubiKey PIV smart card or the connection to that device is lost.

In many cases, the error is caused by detaching the device from the USB port without prompty unloading the PKCS#11 module from the system, i.e. without executing this command before the detachment:

# Detach the PKCS#11 module for communicating with
# the YubiKey device from the SSH agent:
ssh-add -e /usr/lib64/opensc-pkcs11.so

If that situation occurs, you may need to kill the SSH agent process and start it again. This will reset the connection to the YubiKey PIV smartcard and you will be allowed to re-add the PKCS#11 provider to the SSH agent and properly connect to the device:

killall ssh-agent; eval "$(ssh-agent)"

Once the SSH agent is restarted, re-add the PKCS#11 provider to the SSH agent:

ssh-add -s /usr/lib64/opensc-pkcs11.so

and try to perform the SSH login again.

Another aspect of the problem is that sometimes, even if the SSH agent is killed in the way shown above, the newly attached one is available only in the current terminal session (where the killing and starting took place). To make the newly started SSH agent available in all terminal sessions, you need to restart the bash session for the current user, running the window manager. One way to do that is to close the terminal window and open a new one. Then in that newly created terminal, execute:

# Restart the bash session for the current user
exec bash

Note

This is a common issue when using PKCS#11 in terminal in GNOME.

I re-programmed the YubiKey PIV smart card and now even if I provide the correct credentials, the OpenSC library cannot perform

In many cases, the OpenSC tools keep specific to the device cache information inside:

~/.cache/opensc/

You may safely remove the content stored inside and try using the device again:

rm -fr ~/.cache/opensc/*