Hi folks,
I thought I’d depart from our usual format on this change reminder.
The ‘NTLMv1 is turned off’ change is happening in 13 days. We’ve been sending user notifications based on a really small slice of all the possible sources of NTLMv1 logons. Based on that really small slice of the overall picture, usage is down, but we don’t have a comprehensive picture of what the impact will be. Put simply, the NTLMv1 logon events are logged on your servers, not on the NETID DCs.
If you’ve already taken action and applied the workarounds, we thank you immensely.
But if that isn’t the case, then you really should take a look because on August 12th some set of your users will likely have problems. I don’t want to be the guy telling you on 8/12 that you should have paid attention to the warnings we sent, and that now in a very short period of time, you’ll need to become an expert on NTLMv1 workarounds and apply them to all the various computers you support that are having problems.
If you aren’t sure, send in an email to help@uw.edu and we’ll help you reach more certainty. If you want help looking at your server logs, we can do that—we’ll even send user notifications, if you want that.
If you don’t want help, and haven’t yet done anything, here are the things I’d recommend now:
- Read https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds. It’s your key to fixing things up either proactively or reactively. It links to all the resources we know about or have created.
- If you run your own Windows servers, you really need to run the powershell script in workaround J on those servers. It’ll give you a list of which client computers are misconfigured and the users (which is helpful if you don’t recognize the computer name). It’s really quite easy to use, and even if you don’t know powershell, we can help you run this with very little effort.
- If you run a Windows domain that trusts NETID, then a highly effective, low cost action to take is to set a group policy setting in your domain root that sets the LMCompatabilityLevel to 3 at least (5 would be ideal). See workaround C for more on that. This (level 3) will allow your domain-joined Windows computers to send NTLMv2 (level 5 will require NTLMv2).
- If you are in the situation where you run an IIS web server with Windows Integrated authentication, then you really need to consult the workarounds associated with problem #9 and take action. Almost all non-Windows web clients are not able to do NTLMv2, so you will be in a very awkward position without proactive action.
The workarounds page has changed quite a bit in the last 6 weeks, and we thank folks like Armand Bularoro for sharing workarounds they discovered. If you have something we haven’t covered there, we’d really like to capture it to help folks who are caught unprepared on 8/12.
We plan to remind folks again next week and the day before the change. There are also a couple more sets of user notifications we’ll send out. If you share the output of the powershell script from your servers with us, you can leverage our user notification process. That user notification process has been *really* effective (in 5 weeks we went from ~800 users to ~100), and most of the thanks on that goes the excellent folks in the UW-IT Service Center who help users walk through what they need to do.
The change will be on Tuesday, 8/12 at 10am.
Brian Arkills
UW-IT, Identity and Access Management
UWWI Service Manager
From: Brian Arkills Sent: Tuesday, July 8, 2014 12:19 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers
This is an update for the high-impact service change.
The date for this change has been moved.
What and When:
On Tuesday, August 12 (8/12/2014) at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.
More Info:
- We have moved the change date back to give customers operating web servers that rely on Windows Integrated authentication and the NETID domain more time to make changes to address the non-Windows browser issues we’ve noted in previous weeks.
- The Known Problems/Workarounds document has had several modifications over the last couple weeks, most notably adding addition options for web servers currently using Windows Integrated.
- We apologize for falling behind in sharing our log details and user notification lists. The Known NTLMv1 Logons page should be up to date with all the log analysis and user notification lists, and will remain up to date.
- We are modifying our user notification schedule to reflect the new change date. The new user notification dates are: 7/8 (already went out), 7/15, 7/22, 7/29, 8/5, and 8/11.
- Week to week comparisons based on our logs indicate the user notifications are effective:
|
events |
users |
computers |
| 6/17-6/23 |
420944 |
857 |
928 |
| 6/24-7/1 |
35444 |
251 |
275 |
| 7/1-7/7 |
19647 |
203 |
198 |
From: Brian Arkills Sent: Tuesday, July 1, 2014 2:36 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers
This is an update for the high-impact service change in 2 weeks.
More Info:
- NTLMv1 use is significantly down in the server log files available to the UWWI service team. Last week’s logs suggest that as many as 75% of the misconfigured computers we were seeing a month ago have now been fixed. The UWWI service does not have access to your log files. Only you can check whether your users will be affected. See the original announcement below for resources to help you do that.
- An update on the non-Windows browser known problem we mentioned last week: Making a change to an IIS web server which is configured to use Windows Integrated authentication may be a workaround to consider. We’ve added removing Integrated Windows authentication and adding Basic authentication (with SSL required) as a workaround to our documentation. The Dynamics AX service plans to apply this workaround. If you do have a IIS web server with Windows Integrated enabled, you should check your logs for NTLMv1 use.
- We continue to email user notifications to those users that are in the log files we have access to. We sent a round of notifications today to 250 users. We plan to send additional user notifications on: 7/8, 7/14, and 7/15.
From: Brian Arkills Sent: Tuesday, June 24, 2014 2:16 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers
This is an update for the high-impact service change in 3 weeks.
More Info:
- We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
- The 1st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
- We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.
From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers
This is an update for this high-impact service change.
A date for the change has been set:
What and When:
On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.
More Info:
Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.
We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.
We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.
From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers
A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.
What and When:
This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.
Why:
A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.
What you need to do:
On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.
The good news is that we’ve done a lot of work to help assist you in getting things fixed up.
There are several things we’d like IT support staff to do:
- Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
- Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
- Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
- Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
-
Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons
Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds
PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1
PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1
IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance
Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299
Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss
If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J
Brian Arkills
UW-IT, Identity and Access Management
UWWI Service Manager
