Windows Server 2016: Access Denied for Admin Shares

We recently had an issue that started of with a problem promoting some domain controllers in Azure.  We’d set up the site to site VPN and wanted to stretch the Forest into Azure.  It seemed like a simple plan…  Problems arose when we promoted the domain controller with errors such as

• System log: Can’t process the GPO xxx because access is denied
• Application log: Can’t auto-enrol a certificate because access is denied

We also found that all of the administrative shares came back with Access Denied, no matter which account was used nor where the share was accessed from….

Now we knew we were in trouble since Bing/Google showed pretty much nothing of value.   You know it is bad when Uncle Google and Auntie Bing don’t give you at least a bit of a steer…

This is a brand new Domain Controller, freshly joined to a nearly new Windows Server 2016 domain with policies applied per Center for Internet Security (CIS) and also disabling NetBIOS.

After a couple of days of basic troubleshooting, then diving more into the Enterprise CA to check permissions we found nothing really amiss.  We tried granting ‘Everyone’ rights to enrol/auto-enrol certificates and relaxed DCOM security from its default, but to no avail.  No blocks on any firewalls, nothing seemed untoward, well aside from the DC having issues applying policies and not having a Kerberos certificate of course.

We then moved onto Wireshark on the CA to see what was happening.  Lo and behold the DCOM to the CA was fine, however after the DC has asked the CA for the cert, then CA connects back on IPC$ and guess what…  Access Denied.  Wireshark showed that the SMB negotiation succeeded and the two Windows Server 2016 servers decided on SMB3 with SHA256 and AES256, all good so far, but as the CA asks for a session connection it is told ‘no, no, no, no you’re not allowed’

This was a massive clue though as it showed that actually the problem was talking TO the new Domain Controller, and it was an SMB problem of some sort.  Now if you’ve ever looked at the CIS policies (732 pages for Windows Server base OS alone) you know it is not a trivial task to work out what could be causing the issue here… but I knew it was SMB now!  I explored UNC Hardening, which turned out to not be the problem but then came across a security setting called ‘Server SPN Target Name Validation Level’

The explanation text for this setting is as follows:

Microsoft network server: Server SPN target name validation level

This policy setting controls the level of validation a computer with shared folders or printers (the server) performs on the service principal name (SPN) that is provided by the client computer when it establishes a session using the server message block (SMB) protocol.

The server message block (SMB) protocol provides the basis for file and print sharing and other networking operations, such as remote Windows administration. The SMB protocol supports validating the SMB server service principal name (SPN) within the authentication blob provided by a SMB client to  prevent a class of attacks against SMB servers referred to as SMB relay attacks. This setting will affect both SMB1 and SMB2.

This security setting determines the level of validation a SMB server performs on the service principal name (SPN) provided by the SMB client when trying to establish a session to an SMB server.

The options are:

Off – the SPN  is not required or validated by the SMB server from a SMB client.

Accept if provided by client – the SMB server will accept and validate the SPN provided by the SMB client and allow a session to be established if it matches the SMB server’s list of SPN’s for itself. If the SPN does NOT match, the session request for that SMB client will be denied.

Required from client - the SMB client MUST send a SPN name in session setup, and the SPN name provided MUST match the SMB server that is being requested to establish a connection.  If no SPN is provided by client, or the SPN provided does not match, the session is denied.

Default: Off

All Windows operating systems support both a client-side SMB component and a server-side SMB component. This setting affects the server SMB behavior, and its implementation should be carefully evaluated and tested to prevent disruptions to file and print serving capabilities. Additional information on implementing and using this to secure your SMB servers can be found at the Microsoft website  (http://go.microsoft.com/fwlink/?LinkId=144505).

CIS recommends setting it to “Accept if provided by client” and so the server would check the SPN that the client asked to connect to and fail if it wasn’t an SPN for the server.  I turned on Object Access: File Share failure audit logging and confirmed that it was indeed 5168: SPN check for SMB/SMB2 failed.

Ok so I know it’s using CIFS and I can see that the computer object in AD had the HOST SPNs registered ok which should work.  Even explicitly adding CIFS SPNs still wouldn’t resolve the issue, however now I knew the setting that was causing the problem since setting it back to the default of Off made everything work.  A quick Google returned a number of results now, including Windows Server UserVoice which suggests that the combination of turning on this setting and turning off NetBIOS produces the symptoms we saw.

Hopefully this description of the symptoms, the troubleshooting process and the solution help someone in the future and saves them the 3 days of work it too to get to a conclusion

BONUS for reading to the end: disabling of NetBIOS is done with a neat one liner of PowerShell

Get-WMIObject win32_networkadapterconfiguration | ? { $_.IpEnabled -eq $true } | % { $_.settcpipnetbios(2) }