Configuring RHEL7 for Kerberized NFSv4 mounts

Configuration to enable mounting

These steps are required in all cases.

1. Install packages that are frequently missing (Use Puppet when appropriate, e.g. DAC servers)

   # yum install adcli sssd nfs4-acl-tools

2. Configure /etc/krb5.conf with at least these lines (Use Puppet when appropriate, e.g. DAC servers)

   [libdefaults]
   default_realm = KIEWIT.DARTMOUTH.EDU
   dns_lookup_realm = false
   dns_lookup_kdc = true
   forwardable = true
   rdns = false

3. Edit the Domain line in /etc/idmapd.conf so it matches the NFSv4 Domain setting on the Isilon (KIEWIT.DARTMOUTH.EDU).

   Domain = KIEWIT.DARTMOUTH.EDU

4. Join the system to the KIEWIT domain with adcli.   This requires an AD account with admin privileges in the OU where you are creating the computer account.   Unprivileged NetIDs can create computer accounts but seemingly not with the SPNs we need.   In this command replace adminaccount as appropriate and for Research Computing systems change OU=Servers to OU=Research.

   # adcli join kiewit.dartmouth.edu -S lexington2.dartmouth.edu -U adminaccount -O \
   'OU=Servers,dc=Kiewit,dc=Dartmouth,dc=edu' --service-name=nfs --service-name=host
   

5. Since domain membership is critical, check that /etc/krb5.keytab was created with the right stuff.

   # klist -k -t /etc/krb5.keytab
   Keytab name: FILE:/etc/krb5.keytab
   KVNO Timestamp Principal
   ---- ------------------- ------------------------------------------------------
   2 10/17/2018 14:40:17 MYHOST$@KIEWIT.DARTMOUTH.EDU
   [...]

6. Edit /etc/sysconfig/nfs and add a line so the kernel rechecks TGTs once an hour. The default is to only check when it would have expired (10 hours).

   RPCGSSDARGS="-t 3600"

7. The rpcgssd and rpcidmapd services must be running. They should start automatically at boot time now that /etc/krb5.keytab exists but if we start them manually we should be able to mount things right away.

   # systemctl start rpcidmapd
   # systemctl start rpcgssd

8. Make a mount!

   # mount -t nfs -o sec=krb5,nfsvers=4.0 server:/ifs/path/to/export /mount_point

Configuring the sssd service

Configuring the sssd service enables NetID logins (and the automatic acquisition of a Kerberos TGT) based on group membership defined in /etc/sssd/sssd.conf. Running sssd is not necessary for mounting the Kerberized NFSv4 storage but without that you'll need to manually acquire the TGT for accessing anything (use the kinit command). 

1. This authconfig command adds an auth, account, password and session line to password-auth-ac and system-auth-ac; and an account and session line to fingerprint-auth-ac, and smartcard-auth-ac.  It also adds sss to the end of the passwd, shadow, group, services, netgroup, and autmount lines in /etc/nsswitch.conf

   # authconfig --enablesssd --enablesssdauth --update

2. Create /etc/sssd/sssd.conf with ownership root:root and 0400 permissions. Change the group on the ad_access_filter line as appropriate to allow only those NetIDs to login.  Without that line any NetID can login.  Also change override_homedir as appropriate.

   [sssd]
   debug_level = 0x0270
   services = nss, pam
   domains = KIEWIT.DARTMOUTH.EDU
   
   [nss]
   debug_level = 0x0270
   # exclude root from the sss NSS database
   filter_groups = root
   filter_users = root
   # loginShell not set in AD
   override_shell = /bin/bash
   # May want to override home directories too
   # override_homedir = /path/%u
   
   [domain/KIEWIT.DARTMOUTH.EDU]
   debug_level = 0x0270
   # type of provider
   id_provider = ad
   auth_provider = ad
   access_provider = ad
   # e.g. allows 'getent passwd user' without specifying domain
   use_fully_qualified_names = False
   # same as false, which is required for AD, but preserves case in NSS ops
   case_sensitive = Preserving
   # Explicitly disable sudo from AD
   sudo_provider = none
   # UID/GID lower limit - sss ignores if lower
   min_id = 10000
   # Default domain
   ldap_idmap_default_domain = KIEWIT.DARTMOUTH.EDU
   # Don't map to objectSID, use uidNumber/gidNumber
   ldap_id_mapping = False
   # Allow members of "rc-login" group to login
   ad_access_filter = DOM:KIEWIT.DARTMOUTH.EDU:(memberOf=CN=rc-login,OU=Groups,OU=Research,DC=KIEWIT,DC=DARTMOUTH,DC=EDU)
   # Don't try to update AD DNS server
   dyndns_update = False
   # Don't try to update machine password
   ad_maximum_machine_account_password_age = 0

3. Enable and start the sssd service

   # systemctl enable sssd
   # systemctl start sssd

4. You can check that sssd is doing something by looking up a NetID (that isn't in your passwd file). You should get back a passwd file entry with a home directory that looks like /home/Kiewit/netid (or whatever you specified in /etc/sssd/sssd.conf).

   $ getent passwd dz99918
   dz99918:*:17020:10001:Research Computing:/home/Kiewit/dz99918:

Configuring Duo Multi-factor authentication in PAM

Not required.  Need to flesh this section out

1. Modify /etc/pam.d/sshd

2. Modify /etc/pam.d/system-auth-ac

Troubleshooting

• Date/time must be the same on the client as the server. Too much clock skew breaks everything.
• The Security Type (or “Flavor” in the GUI) of the export on the Isilon must match what the client requests with the “sec=” mount option. krb5 is Kerberos5; krb5i is Kerberos5 Integrity; krb5p is Kerberos5 Privacy.

  $ isi nfs exports view ExportNum --zone ZoneName | grep Security
  Security Type: krb5

• The Isilon virtual server and the client each need nfs and HOST service principals in Active Directory (actually I am not certain about an nfs SPN for the client but we have always made them). The server SPNs are part of the ISILON-EDC and/or ISILON-BMR computer object(s) (i.e. those are CNs to look for in AD). Client has his own computer object with CN equal to hostname.
• The key version number on the client (in /etc/krb5.keytab) must match what's in AD. If they don't match, the easiest thing to do is: remove the system from the domain; delete /etc/krb5.keytab; rejoin the domain. In the example below 2 = 2 so life is good.

  # kinit -k -t /etc/krb5.keytab HOST$
  # kvno HOST$
  HOST$@KIEWIT.DARTMOUTH.EDU: kvno  2
  # klist -k -t /etc/krb5.keytab
  Keytab name: FILE:/etc/krb5.keytab
  KVNO Timestamp Principal
  ---- ------------------- ------------------------------------------------------
  2 12/15/2017 22:44:46 HOST$@KIEWIT.DARTMOUTH.EDU
  [...]

• Issues with joining the server to domain with adcli results in error like:

  adcli: joining domain kiewit.dartmouth.edu failed: Couldn’t create computer account: CN=SASSER1,OU=Servers,dc=Kiewit,dc=Dartmouth,dc=edu: 00002083: AtrErr: DSID-031519D9, #2:
  0: 00002083: DSID-031519D9, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90303 (servicePrincipalName):len 22
  1: 00002083: DSID-031519D9, problem 1006 (ATT_OR_VALUE_EXISTS), data 0, Att 90303 (servicePrincipalName):len 24
  

  This error can be caused if gethost() does not return the FQDN. Try setting using:

  sudo hostnamectl set-hostname servername.dartmouth.edu

• For additional troubleshooing information while joining the domain add --verbose to the adcli command
• If when mounting you get and "invalid mount option" error with correct looking mount options, check that rpc.gssd is running.  If it's not, then "sec=krb5" will cause this error.
• Reboot. There are services that must be running and that may have ordering dependencies. We've never bothered to figure put exactly what they are because the system gets it right at boot time assuming the configuration files are all in place. e.g. krb5.conf, rcpgssd, rpcidmapd, gssproxy