SSH access to the Discoverer and Discoverer+ login nodes with YubiKey PIV smartcard =================================================================================== .. contents:: Table of Contents :local: Introduction ------------ The proposed PIV configuration is intended for advanced HPC users and system administrators who need maximum security for remote access. Here, the implementation hardware-based key storage prevents SSH private keys from being stored in volatile memory, eliminating the risk of key extraction through memory dumps or cold boot attacks. This guide explains how to set up your YubiKey PIV smartcard for SSH authentication using its PIV application on Linux and macOS systems. It covers two approaches: importing an existing OpenSSH key pair from the ``~/.ssh`` folder, or generating a new key pair—both of which are imported into slot 9a of the YubiKey PIV smartcard. In either case, a PKCS#12 file container is used to securely store the keys on the file system before importing them into slot 9a. .. important:: This documentation assumes that users are familiar with SSH agent usage and management. The private keys stored on the PIV smartcard can only be accessed through the SSH agent using the PKCS#11 interface. The SSH agent acts as a bridge between SSH clients and the hardware-stored keys, allowing secure authentication without direct access to the private key material. .. warning:: This guide focuses on importing X.509v3 certificates with their complementary private keys into the PIV slots of YubiKey PIV smartcards. Therefore, only RSA and ECDSA SSH keys are supported. **Ed25519 keys are not supported** by this type of setup. Cost of deployment (gross estimate with VAT incl): ~ €140, per user .. _prerequisites: Prerequisites ------------- * `YubiKey PIV smartcard `_ devices (at least two of them - one should be used as a backup) * Up-to-date Linux or macOS system (see :ref:`software-installation` below for required tools and installation instructions) * Administrative access (to install the software, if necessary) .. warning:: **CRITICAL SECURITY WARNING**: YubiKey devices running firmware versions below 5.7 are vulnerable to security attacks. **Do not use or purchase YubiKey devices with firmware versions below 5.7** for any security-sensitive operations. If your YubiKey has firmware version below 5.7, return it (if possible) or dispose it, and purchase a new YubiKey with firmware version 5.7+. **For detailed information about this vulnerability, see:** https://www.yubico.com/support/security-advisories/ysa-2024-03/ .. important:: **IMPORTANT PURCHASE WARNING**: Some resellers may still have in sell YubiKey 5C smartcards with old firmware versions (<5.7). Since the firmware version is not printed on the product packaging (it is Yubico fault!), you must unbox the smartcard and check its firmware version immediately after purchase using the command .. code-block:: bash yubico-piv-tool -a status | grep ^Version or .. code-block:: bash ykman piv info | grep ^"PIV version" Alternatively you may check the version of the firmware using Yubico Authenticator on Android and iOS. .. warning:: **If the firmware version shown is below 5.7, return the device immediately and request a replacement with firmware 5.7+. Do not accept any device with older firmware. See "CRITICAL SECURITY WARNING" above.** .. _software-installation: Software Installation --------------------- Linux (Fedora/RHEL/Rocky Linux/Alma Linux): .. code-block:: bash # Install EPEL repository (required for yubico-piv-tool) sudo dnf install epel-release # Install required packages sudo dnf install yubico-piv-tool yubikey-manager opensc openssh-clients Linux (Ubuntu/Debian): .. code-block:: bash # Update package list sudo apt update # Install required packages sudo apt install yubico-piv-tool yubikey-manager opensc openssh-client macOS: .. code-block:: bash # Install Homebrew if not already installed /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Install required packages brew install yubico-piv-tool yubikey-manager opensc openssh Check YubiKey Status and Firmware Version ----------------------------------------- Attach the YubiKey smartcard to an USB port on your system. Then start a terminal and execute .. code-block:: bash yubico-piv-tool -a status This command will display the version of the firmware installed, the serial number of the smartcard, as well as some other PIV-specific information: **Example output:** .. code-block:: bash $ yubico-piv-tool -a status Version: 5.7.1 Serial Number: 81209505 CHUID: CCC: PIN tries left: 3 In case you already have certificates and keys installed in the PIV slots, a longer output will be printed out. For example: .. code-block:: bash Version: 5.7.1 Serial Number: 81209505 CHUID: 3019d4e739da739ced39ce739d836858210842108421c84210c3eb3410d54658aaa9eb 82eb6f924964ddfef813350832303330303130313e00fe00 CCC: f015a000000116ff02013b4b8ce1b3962b5bb06650ad6af10121f20121f300f40100f5 0110f600f700fa00fb00fc00fd00fe00 Slot 9a: Algorithm: ECCP384 Subject DN: CN=Test User .. warning:: **SECURITY CHECK**: If the firmware version shown is below 5.7, refer to the :ref:`prerequisites` section above for the required firmware version number. **Do not proceed with this setup on devices with outdated firmware, regardless of any assurances from the vendor about the risk being "minor."** Using a YubiKey with firmware below 5.7 exposes you to serious security vulnerabilities. See "CRITICAL SECURITY WARNING" above. Smartcard Initialization ------------------------- **Why do we need to initialize the smartcard?** The YubiKey PIV smartcard comes with default PIN, PUK, and Management Key values that are publicly known and should never be used in production environments. Initialization is a critical security requirement that sets unique, secure values for these administrative credentials. .. warning:: Initialization will permanently delete all existing data on the PIV smartcard, including any stored certificates, keys, and configuration settings. This process cannot be undone. **Default values (DO NOT USE):** * **PIN**: 123456 * **PUK**: 12345678 * **Management Key**: 010203040506070801020304050607080102030405060708 **Recommended secure values:** * **PIN**: 6-8 digit number (you'll choose this) * **PUK**: 8-digit number (you'll choose this) * **Management Key**: 24-byte AES192 key (automatically generated) **Step 1: Set a new PIN** .. code-block:: bash # Set a new PIN (you'll be prompted for the default PIN, then the new one): yubico-piv-tool -a change-pin .. important:: If you try this on a Yubikey PIV device with PIN already set, and you do not know that PIN (it is not the default one, or does not match any PIN stored by you in your password wallet, or you don't remember the PUK), then the only way you can proceed with the initialization is after resetting the PIV application on the Yubikey following :ref:`reset-piv-card`. If you remember the PUK, try :ref:`unblock-piv-pin`. **Step 2: Set a new PUK** .. code-block:: bash # Set a new PUK (you'll be prompted for the default PUK, then the new one): yubico-piv-tool -a change-puk .. important:: If you try this on a Yubikey PIV device with PUK already set, and you do not know that PUK (it is not the default one, or does not match any PUK stored by you in your password wallet), then the only way you can proceed with the initialization is after resetting the PIV application on the Yubikey following :ref:`reset-piv-card` **Step 3: Set the new management key using AES192 algorithm** .. code-block:: bash # Generate a new management key: openssl rand -hex 24 # Set the new management key using AES192 algorithm # (you'll be prompted for the default management key, then the new one): yubico-piv-tool -a set-mgm-key --new-key-algo=AES192 -k - .. important:: If you try this on a Yubikey PIV device with management already set (different than the default one), and you do not know that management key (it is not the default one, or does not match any such key stored by you in your password wallet), then the only way you can proceed with the initialization is after resetting the PIV application on the Yubikey following :ref:`reset-piv-card` **Step 4: Verify the initialization** .. code-block:: bash # Verify that the initialization was successful ykman piv info Importing Keys into YubiKey PIV Slots -------------------------------------- **Note**: This guide does not recommend generating keys directly on the smartcard for backup and redundancy reasons. Instead, we'll import externally generated keys. **Note on OpenSSH Private Key Format Compatibility** Modern OpenSSH private keys use a new format that is not compatible with the PIV standard. If you have existing OpenSSH keys in the new format, you'll need to convert them to the traditional PEM format before importing. **Converting OpenSSH format keys to PEM format:** .. code-block:: bash # Copy your existing key to a new file cp ~/.ssh/id_ecdsa ~/.ssh/id_ecdsa_pem # Convert the key to PEM format ssh-keygen -p -f ~/.ssh/id_ecdsa_pem -m pem **Creating X.509v3 certificate and PKCS#12 container:** .. code-block:: bash # Create a self-signed X.509v3 certificate openssl req -x509 -new -sha384 \ -key ~/.ssh/id_ecdsa_pem \ -out ~/.ssh/id_ecdsa_pem.crt \ -subj "/CN=SSH Key" \ -days 3650 # Create PKCS#12 container with the private key and certificate openssl pkcs12 -export -maciter \ -inkey ~/.ssh/id_ecdsa_pem \ -in ~/.ssh/id_ecdsa_pem.crt \ -name "SSH Key" \ -out ~/.ssh/id_ecdsa_pem.p12 **Importing PKCS#12 into slot 9a:** .. code-block:: bash # Import the PKCS#12 container into slot 9a # (you'll be prompted for the management key): yubico-piv-tool -s 9a -K PKCS12 \ -a import-key -a import-cert \ -i ~/.ssh/id_ecdsa_pem.p12 -k - **Importing externally generated RSA or ECDSA key from PKCS#12 file into slot 9a** If you have a PKCS#12 file with an RSA or ECDSA key and certificate: .. code-block:: bash # Generate a new ECDSA key pair (example) openssl ecparam -genkey -name secp521r1 | \ openssl ec -aes256 -out private.pem # Create a self-signed certificate openssl req -x509 -new -sha384 \ -key private.pem \ -out formal.crt \ -subj "/CN=SSH Key" \ -days 3650 # Create PKCS#12 container openssl pkcs12 -export -maciter \ -inkey private.pem \ -in formal.crt \ -name "SSH Key" \ -out formal.p12 # Import into slot 9a yubico-piv-tool -s 9a -K PKCS12 \ -a import-key -a import-cert \ -i formal.p12 -k - To extract the OpenSSH public key for registration with the remote system, run ``ssh-add -L``. Copy the public key from the command output and provide it to the system administrator for registration with your account, following the documentation in :doc:`ssh_key_exchange`. .. _ssh-agent-access: SSH Agent Configuration ----------------------- The SSH agent needs permission to access the private key stored on the PIV smartcard. This is done through the PKCS#11 interface. **Add the PIV smartcard key to the SSH agent:** .. code-block:: bash # Add the PIV smartcard key to the SSH agent ssh-add -s /usr/lib64/opensc-pkcs11.so **Note**: The path to the PKCS#11 library may vary depending on your system: * **Linux**: ``/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so`` (Ubuntu/Debian) or ``/usr/lib64/opensc-pkcs11.so`` (Fedora/RHEL) * **macOS**: ``/usr/local/lib/opensc-pkcs11.so`` (if installed via Homebrew) **Verify the key is accessible:** .. code-block:: bash # List the keys available to the SSH agent ssh-add -L This command will display the public key from the PIV smartcard, confirming that the SSH agent can access it. **Extract the public key for remote system registration:** .. code-block:: bash # Extract the public key for registration with remote systems ssh-add -L Copy the output and provide it to the system administrator for registration with your account on the remote system. **Remove the key from SSH agent (if needed):** .. code-block:: bash # Remove the PIV smartcard key from the SSH agent ssh-add -e /usr/lib64/opensc-pkcs11.so Troubleshooting --------------- .. _unblock-piv-pin: Unblock PIV PIN ~~~~~~~~~~~~~~~ If the PIN is forgotten but the PUK is known: .. code-block:: bash ykman piv access unblock-pin .. note:: Provide first the PUK, and then set a new PIN. In case you do not remember the PUK, nothing can be done to obtain access to the keys stored in the PIV card. In that case you may only reset the PIV card (see next section). .. _reset-piv-card: Reset PIV Smart Card ~~~~~~~~~~~~~~~~~~~~ .. warning:: The command shown below resets the PIN, PUK, and management key to their default values. All stored keys and certificates will be erased. To remove only the information stored in the PIV application on your Yubikey, execute: .. code-block:: bash ykman piv reset --force .. note:: This operation does not remove stored passkeys (FIDO2 credentials) from the YubiKey. Fixing "sign_and_send_pubkey: signing failed for XXXXXXX from agent: agent refused operation" during SSH login -------------------------------------------------------------------------------------------------------------- This error occurs when the SSH agent is not configured correctly to access the private key stored on the YubiKey PIV smart card or **the connection to that device is lost.** In many cases, the error is caused by detaching the device from the USB port without prompty unloading the PKCS#11 module from the system, i.e. without executing this command before the detachment: .. code-block:: bash # Detach the PKCS#11 module for communicating with # the YubiKey device from the SSH agent: ssh-add -e /usr/lib64/opensc-pkcs11.so If that situation occurs, you may need to kill the SSH agent process and start it again. This will reset the connection to the YubiKey PIV smartcard and you will be allowed to re-add the PKCS#11 provider to the SSH agent and properly connect to the device: .. code-block:: bash killall ssh-agent; eval "$(ssh-agent)" Once the SSH agent is restarted, re-add the PKCS#11 provider to the SSH agent: .. code-block:: bash ssh-add -s /usr/lib64/opensc-pkcs11.so and try to perform the SSH login again. Another aspect of the problem is that sometimes, even if the SSH agent is killed in the way shown above, the newly attached one is available only in the current terminal session (where the killing and starting took place). To make the newly started SSH agent available in all terminal sessions, you need to restart the bash session for the current user, running the window manager. One way to do that is to close the terminal window and open a new one. Then in that newly created terminal, execute: .. code-block:: bash # Restart the bash session for the current user exec bash .. note:: This is a common issue when using PKCS#11 in terminal in GNOME. I re-programmed the YubiKey PIV smart card and now even if I provide the correct credentials, the OpenSC library cannot perform ------------------------------------------------------------------------------------------------------------------------------- In many cases, the OpenSC tools keep specific to the device cache information inside: .. code-block:: bash ~/.cache/opensc/ You may safely remove the content stored inside and try using the device again: .. code-block:: bash rm -fr ~/.cache/opensc/*