Kerberized SSH with homedirs on Kerberized NFS

Jurjen Bokma

Last modified: "2016-10-10 14:56:36 apprentice"


We have an SSH server with users' home directories on kerberized NFS. So authenticating to SSH using Kerberos is nice. In addition, the client side Kerberos credentials are passed to the server, where they should be used to obtain access to $HOME.

    • Kerberos is configured to work on the ssh server as well as the client. You have a principal, and can get credentials for it:

      apprentice@client:~$ kinit apprentice
      apprentice@MYDOMAIN.COM's Password:
      apprentice@client:~$ klist
      Credentials cache: FILE:/tmp/krb5cc_1000
      Principal: apprentice@MYDOMAIN.COM

      Issued                Expires               Principal
      Apr 14 14:59:19 2016  Apr 15 00:59:16 2016  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM

    • Both server and client are members of a Kerberos realm, i.e. they have host keys:

      apprentice@client:~$ sudo ktutil list

      Vno  Type                     Principal                          Aliases
        3  aes256-cts-hmac-sha1-96$@MYDOMAIN.COM

      The realm in our case is a MS Active Directory domain.


      The principal is not the Kerberos-conventional fqdn. It's an IP number with a dollar sign appended. This happens more often in MS domains, and it has important consequences below.

    • The server holds Kerberos keys for host/<hostname>@<REALM>:

      apprentice@ssh-server:~$ sudo ktutil list|grep host
            111  aes256-cts-hmac-sha1-96  host/

  1. /etc/ssh/sshd_config should look like this:

    # Main settings
    Port 22
    HostKey /etc/ssh/ssh_host_rsa_key
    HostKey /etc/ssh/ssh_host_dsa_key
    HostKey /etc/ssh/ssh_host_ecdsa_key
    # Settings disabled for security
    StrictModes                      yes
    PermitRootLogin                  no
    PermitEmptyPasswords             no
    HostbasedAuthentication          no
    # Kerberos-related stuff
    GSSAPIAuthentication             yes
    GSSAPIKeyExchange                yes 1
    GSSAPIStoreCredentialsOnRekey    yes 2
    GSSAPICleanupCredentials         yes 3
    # Debian/ubuntu defaults which are different from upstream sshd defaults
    Protocol                         2
    ChallengeResponseAuthentication  no
    PrintMotd                        no
    AcceptEnv                        LANG LC_*
    Subsystem sftp                   /usr/lib/openssh/sftp-server
    Banner                           /etc/
    UsePAM                           yes
    X11Forwarding                    yes
    X11UseLocalhost                                  no
    # Authentication mechanisms
    PasswordAuthentication           no
    PubkeyAuthentication             no
    GSSAPIAuthentication             no
    # Other custom settings
    AllowAgentForwarding             yes
    MaxStartups                      10:30:60
    AuthorizedKeysFile               /var/lwp/ssh/%u/authorized_keys %h/.ssh/authorized_keys 4
    # Restricted logins (if any)
    # These hosts are allowed to use Kerberos
    Match Address 5
    GSSAPIAuthentication yes
    PubkeyAuthentication yes
    KerberosAuthentication yes
    AuthenticationMethods publickey,gssapi-keyex publickey,gssapi-with-mic publickey,password 6


    Without GSSAPIKeyExchange, GSSAPIStoreCredentialsOnRekey doesn't seem to work.


    Kerberos by default uses short-lived tickets, but they may be renewed for a much longer time. When such a renewal happens on the client, the new key must also become available to the server. Doing this upon rekeying of the connection is a trick, but one that works.


    When the user logs out from the server, his credentials must not linger there.


    When the user has multiple ssh session to the server, this may delete credentials that are still in use by another session.


    This has little to do with Kerberos, but it's useful. When the user logs in on the server, the ssh daemon, in order to verify the user's ssh keys, will by default want to access ~/.ssh/authorized_keys. But it cannot, because $HOME is on a kerberized NFS4 share, and in our configuration, root doesn't have read access there. So we store the authorized_keys elsewhere.


    Our Kerberos server is not open to the world, so we only allow users to authenticate with Kerberos from where they can reach the Kerberos server and obtain a ticket.


    This setting is tricky, and I don't understand it fully. The meaning according to the docs is to allow either of the three comma-separated, ordered sets of two authentication methods.

    But change the order of publickey and gssapi-keyex, and it doesn't work any more. I suppose that some lookups done for the public key aso work for the keyex, but not the other way around.

    Also, gssapi-keyex authenticates hosts, but not users. It is apparently built-in mandatory that users do get authenticated, because gssapi-mic must also succeed, even if gssapi-keyex worked. And without the gssapi-keyex in this list, key exchange is forbidden, which leads to credentials not being updated, which in turn leads to access to $HOME being soon denied.


    It is also possible to specify:

    AuthenticationMethods publickey,gssapi-keyex publickey,gssapi-with-mic publickey password

    In that case, after succesful public key authentication, both gssapi-keyex and gssapi-with-mic will be attempted, giving the full benefit of Kerberos authentication and credential-passing. Yet if public key authentication fails, password authentication on its own may still succeed.

    Of course you need to reload the ssh server:

    apprentice@server:~$ sudo service ssh reload

    Maybe you even want to run it in the foreground for test:

    apprentice@server:~$ sudo service ssh stop
    apprentice@server:~$ /usr/sbin/sshd -D -d

  2. You want this in ~/.ssh/config:

    ForwardX11 yes
    host ssh-server 1
          HostName 2
          GSSAPIAuthentication yes
          GSSAPIKeyExchange yes
    #     GSSAPIClientIdentity$ 3
    #     GSSAPIServerIdentity 4
          GSSAPIDelegateCredentials yes 5
          GSSAPIRenewalForcesRekey yes 6
    #     GSSAPITrustDns yes 7


    This host specification matches any name by which the host may be called...


    ... but the HostName is very specific: the FQDN. This is used by Kerberos to know for which host to obtain a ticket. And if the ticket is wrong about that, kerberos authentication will fail.


    Kerberos assumes the client has a principal that uses its FQDN. In some MS Active Directory based realms, like ours, it doesn't, and the IP number is used with a '$' attached. Yet it works without this setting.


    Same as ClientIdentity, but now for the server.


    This setting is crucial. It allows the server to use the client's credentials, e.g. to obtain a ticket for NFS server side.


    When the credentials get renewed client side, also notify the server. If you don't set this, credentials will rapidly expire on the server.


    Instead of specifying the hostname manually, you could trust DNS.

  3. apprentice@client:~$ ssh apprentice@ssh-server
    Last login: Thu Apr 14 15:31:45 2016 from
    apprentice@ssh-server:~$ klist
    Credentials cache: FILE:/tmp/krb5cc_1001_6MxnQ5kmIX
    Principal: apprentice@MYDOMAIN.COM

    Issued                Expires               Principal
    Apr 14 15:31:55 2016  Apr 15 00:59:16 2016  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM


    There is no NFS ticket, as I had logged in just before. When I logged out, the previous credentials were destroyed, but the kernel lags behind in discovering that they have gone. In a while, it will find out there are no credentials, but it'll then use my credentials to get a new ticket, and access $HOME for me anyway.

  4. apprentice@client:~$ klist
    Credentials cache: FILE:/tmp/krb5cc_1000
    Principal: apprentice@MYDOMAIN.COM

    Issued                Expires               Principal
    Apr 14 14:59:19 2016  Apr 15 00:59:16 2016  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
    Apr 14 15:31:44 2016  Apr 15 00:59:16 2016  host/
    apprentice@client:~$ krenew -v
    krenew: renewing credentials for apprentice@MYDOMAIN.COM
    apprentice@client:~$ klist
    Credentials cache: FILE:/tmp/krb5cc_1000
    Principal: apprentice@MYDOMAIN.COM

    Issued                Expires               Principal
    Apr 14 15:37:22 2016  Apr 15 01:37:19 2016  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM

    See that the server-side ticket has been updated:

    apprentice@ssh-server:~$ klist
    Credentials cache: FILE:/tmp/krb5cc_1001_6MxnQ5kmIX
    Principal: apprentice@MYDOMAIN.COM

    Issued                Expires               Principal
    Apr 14 15:39:22 2016  Apr 15 01:37:19 2016  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM

  5. Run the SSH server in the foreground, with debugging on:

    sudo /usr/sbin/sshd -D -dd

    Run the client verbosely:

    ssh -vvv apprentice@ssh-server

    Chances are that your server doesn't have the correct host keys. Perhaps you need to add one:

    sudo msktutil --update --enctypes 16 --computer-name$ --realm MYDOMAIN.COM --base OU=PCS,OU=workplaces,OU=MYDOMAIN,DC=mydomain,DC=com -s host/