Many webservices generate ssh keys to access their service. With the amount of services the number of SSH keys grows. To avoid dealing with the keys in the command line and the ssh_config(5), you can simply add the ssh-key to the local ssh-agent(1) to manage them.
The ssh-agent(1) is a program used to hold ssh private keys used to authenticate to remote systems via ssh public key authentication. ssh clients use it to authenticate to the remote via the keys in the ssh-agent.
When the ssh-agent is started, the output does not show output meant to be read by humans
$ ssh-agent -s SSH_AUTH_SOCK=/tmp/ssh-TsOpH27013/agent.27013; export SSH_AUTH_SOCK; SSH_AGENT_PID=27014; export SSH_AGENT_PID; echo Agent pid 27014;
The output are shell commands (-s for bash shell) that should be executed. To do this in one command, the eval command can be used.
# Start the agent $ eval $(ssh-agent -s)
When the ssh-agent is started, it prints out the environment variables that need to be set. These environment variables are required so the client programs, like ssh, know how to connect to the ssh-agent. To directly set these environment variables, the output of the ssh-agent is passed as parameter to the eval command. The “eval” command will execute the parameters passed to it as if they were directly entered into the shell. In this case, the output of the ssh-agent is directly executed.
With the ssh-agent now running, there are not yet any keys added to the agent. Adding an ssh-key to the ssh-agent is done via the ssh-add(1) command. As a parameter, ssh-add takes the filename of the ssh private-key. If the default location and file name is used for the key (~/.ssh/id_rsa), the filename parameter can be omitted.
# Add the default key $ ssh-add ~/.ssh/id_rsa Enter passphrase for /home/user/.ssh/id_rsa: Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
When the private key is secured by a password (which is suggested), ssh-add will ask for it to add the key to the agent.
With the ssh-agent running and the key added to the agent, any ssh session can now be authenticated with that key. Additional ssh-keys, if needed, can be added using ssh-add as described above. Instead of entering the ssh-key password each time, the agent manages the keys and only asks once for the password of the keys.
To list the ssh-keys currently available in the ssh-agent, the -l option can be used. The fingerprints of the keys in the agent are shown, like in the example below.
$ ssh-add -l 4096 d1:43:75:a7:95:16:93:9b:4d:5e:d8:98:61:7d:92:4c /home/user/.ssh/id_rsa (RSA) 2048 1a:51:51:9a:b6:8f:42:45:e1:f1:d7:74:b5:7b:e4:ff /home/user/.ssh/another_user_key (RSA)
With the -L option, the keys in the ssh-agent can be listed as well. While the -l option shows the fingerprint for each key, the -L option shows the public key part of each key. This can be especially helpful if the ssh-key should be added to a remote server for authentication. The output lines can directly be used to add the key to the authorized_keys file.
$ ssh-add -L ssh-rsa AAAAB4NzaC4Xc2FA2A...Me3IABDICy+WANsg5Mc= /home/user/.ssh/id_rsa ssh-rsa AAAAB4NzaC4Xc2FA2A...VpJaZ5EawNpQaPvqEw== /home/user/.ssh/another_user_key
(The above output has been shortened as the public key sections are quite long and really relevant to this example).
To remove a key from the ssh-agent, the -d option can be used as the example below shows. The parameter given is the file name as it is shown in the list of keys. This identifies the key to be removed.
$ ssh-add -d /home/user/.ssh/id_rsa Identity removed: /home/user/.ssh/id_rsa ( ssh-key-comment) $ ssh-add -l 2048 1a:51:51:9a:b6:8f:42:45:e1:f1:d7:74:b5:7b:e4:ff /home/user/.ssh/another_user_key (RSA)
Removing all private keys from the ssh-agent can be achieved with the -D option as shown below. In this case it is not necessary to add identifiers for the individual keys.
$ ssh-add -D All identities removed.
Keeping the private keys secure is important. It is suggested that the ssh-agent is terminated or locked when it is not needed anymore. To terminate the ssh-agent the following command can be executed.
# kill the ssh-agent $ eval $(ssh-agent -s -k) Agent pid 24329 killed
This command – like starting the agent – uses the eval command. In this case not to set environment variables but to unset them as the agent has been stopped.
Automate the ssh-agent start
With all the above in mind, it seems to be a lot of commands to remember and enter every time. There has to be a way to automate at least some of this, and there is.
The ssh-agent should be started when a Shell session is started and the default key should be added to the ssh-agent, but with this approach, every shell opened and every session started gets its own ssh-agent and the password is asked for adding the key to the ssh-agent. To share the ssh-agent between the sessions, the following can be added to the “~/.bashrc”.
# Check if the ssh-agent is already running
if [[ "$(ps -u $USER | grep ssh-agent | wc -l)" -lt "1" ]]; then
#echo "$(date +%F@%T) - SSH-AGENT: Agent will be started"
# Start the ssh-agent and redirect the environment variables into a file
ssh-agent -s >~/.ssh/ssh-agent
# Load the environment variables from the file
. ~/.ssh/ssh-agent >/dev/null
# Add the default key to the ssh-agent
ssh-add ~/.ssh/id_rsa
else
#echo "$(date +%F@%T) - SSH-AGENT: Agent already running"
. ~/.ssh/ssh-agent >/dev/null
fi
The command in the if statement is constructed in a way to list all ssh-agents running in the context of the user shells/session’s user. If the user has no ssh-agent running yet, the ssh-agent is started. For the purpose of reusing the ssh-agent for any parallel sessions, the output of the ssh-agent is not directly passed to the “eval” command but stored into a file. This file is then loaded in the second step. As final step the keys are loaded via ssh-add. The ssh-add command will cause the session to immediately ask for the ssh-key passphrases.
If the user already has an ssh-agent running (from a parallel session/shell) the else part is executed which simply loads the environment variables needed. Those where stored in a file when the agent was started.
This way, the agent is started with the first login and used for every further session started.
Automate the ssh-agent termination
To secure the ssh-keys after all sessions for a user are ended, the ssh-agent should be stopped when the user exits the sessions.
bash(1) supports a script to be executed on logout from a session. The file that is executed on logout is called ~/.bash_logout.
The same way as the section in the .bashrc, the script in the .bash_logout needs to be aware of the logged in sessions as the ssh-agent is shared between the sessions.
# Check how many sessions are still logged in
if [[ "$(who | grep "$USER" | wc -l)" -le "1" ]]; then
#echo "$(date +%F@%T) - SSH-AGENT: Agent will be stopped"
# Remove all keys from the ssh-agent
ssh-add -D >/dev/null
# Kill the ssh-agent
eval $(ssh-agent -k)
# remove the file with the ssh-agent environment variables
rm -f ~/.ssh/ssh-agent
else
#echo "$(date +%F@%T) - SSH-AGENT: Still sessions logged in"
fi
The script above placed in the .bash_logout file will be executed when the user will logout from a session. The command in the if statement will list all logged in sessions (who(1)) and reduces them to only sessions of the current user (grep). The “wc -l” as last part of this command will count the resulting lines which makes it easier to compare. The comparison checks if there are less or equal to one session (at this point, the current session is still active so “1” session active will be considered the last session. When this condition is met, all ssh-keys are removed from (ssh-add -D) the and the agent is stopped (ssh-agent -s -k). To cleanup properly, the file where the ssh-agent output was stored is deleted as well.
Terminate ssh-agent when no logout was performed
The ssh session might be disrupted by a timeout or when the network connection drops. In such cases the shell does not receive the logout command and the ssh-agent is not terminated by the script in the .bash_logout script. To ensure that the ssh-agent is ended and the ssh-keys are secure, the following can be done.
Create a script which can be executed as a cron job. This script looks similar to the script in the .bash_logout. It differs only by the amount of sessions that should be active before it terminates the ssh-agent. The cron script starts its action when no session is active any more for that user.
# Check how many sessions are still logged in
if [[ "$(who | grep "$USER" | wc -l)" -le "0" ]]; then
# If no sessions, check if the ssh-agent is still running
if [[ -e ~/.ssh/ssh-agent ]]; then
echo "$(date +%F@%T) - SSH-AGENT: Agent will be stopped"
# load the ssh-agent environment variables (used to stop the ssh-agent)
. ~/.ssh/ssh-agent
# Remove all keys from the ssh-agent
ssh-add -D
# Kill the ssh-agent
eval $(ssh-agent -k)
# remove the file with the ssh-agent environment variables
rm -f ~/.ssh/ssh-agent
else
echo "$(date +%F@%T) - SSH-AGENT: Agent not running"
fi
else
echo "$(date +%F@%T) - SSH-AGENT: Still sessions logged in"
fi
The script (in this example “~/.cron_ssh-agent”) can now be added to the cron jobs. To do so, open the crontab in edit mode (-e).
$ crontab -e
The cron job can now be added. Depending on the configuration of your system, the crontab is opened in vim or whatever editor is configured for it. Add the following line to execute the script every 5 minutes.
*/5 * * * * bash ~/.cron_ssh-agent >/dev/null
With the .bash_logout script in place and the cron job configured, the ssh-agent will be immediately stopped when you logout. In case of a connection drop, the ssh-agent will be automatically stopped at the latest 5 minutes after the connection was lost.
Read more of my posts on my blog at https://blog.tinned-software.net/.
