Archives: Service News

This is an update for the high-impact service change.

 

The date for this change has been moved.

 

What and When:

On Tuesday, August 12 (8/12/2014) at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

• We have moved the change date back to give customers operating web servers that rely on Windows Integrated authentication and the NETID domain more time to make changes to address the non-Windows browser issues we’ve noted in previous weeks.
• The Known Problems/Workarounds document has had several modifications over the last couple weeks, most notably adding addition options for web servers currently using Windows Integrated.
• We apologize for falling behind in sharing our log details and user notification lists. The Known NTLMv1 Logons page should be up to date with all the log analysis and user notification lists, and will remain up to date.
• We are modifying our user notification schedule to reflect the new change date. The new user notification dates are: 7/8 (already went out), 7/15, 7/22, 7/29, 8/5, and 8/11.
• Week to week comparisons based on our logs indicate the user notifications are effective:

 

	events	users	computers
6/17-6/23	420944	857	928
6/24-7/1	35444	251	275
7/1-7/7	19647	203	198

 

 

From: Brian Arkills Sent: Tuesday, July 1, 2014 2:36 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for the high-impact service change in 2 weeks.

 

More Info:

• NTLMv1 use is significantly down in the server log files available to the UWWI service team. Last week’s logs suggest that as many as 75% of the misconfigured computers we were seeing a month ago have now been fixed. The UWWI service does not have access to your log files. Only you can check whether your users will be affected. See the original announcement below for resources to help you do that.
• An update on the non-Windows browser known problem we mentioned last week: Making a change to an IIS web server which is configured to use Windows Integrated authentication may be a workaround to consider. We’ve added removing Integrated Windows authentication and adding Basic authentication (with SSL required) as a workaround to our documentation. The Dynamics AX service plans to apply this workaround. If you do have a IIS web server with Windows Integrated enabled, you should check your logs for NTLMv1 use.
• We continue to email user notifications to those users that are in the log files we have access to. We sent a round of notifications today to 250 users. We plan to send additional user notifications on: 7/8, 7/14, and 7/15.

 

 

From: Brian Arkills Sent: Tuesday, June 24, 2014 2:16 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for the high-impact service change in 3 weeks.

 

More Info:

• We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
• The 1^st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
• We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

 

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

This is an update for the high-impact service change in 2 weeks.

 

More Info:

• NTLMv1 use is significantly down in the server log files available to the UWWI service team. Last week’s logs suggest that as many as 75% of the misconfigured computers we were seeing a month ago have now been fixed. The UWWI service does not have access to your log files. Only you can check whether your users will be affected. See the original announcement below for resources to help you do that.
• An update on the non-Windows browser known problem we mentioned last week: Making a change to an IIS web server which is configured to use Windows Integrated authentication may be a workaround to consider. We’ve added removing Integrated Windows authentication and adding Basic authentication (with SSL required) as a workaround to our documentation. The Dynamics AX service plans to apply this workaround. If you do have a IIS web server with Windows Integrated enabled, you should check your logs for NTLMv1 use.
• We continue to email user notifications to those users that are in the log files we have access to. We sent a round of notifications today to 250 users. We plan to send additional user notifications on: 7/8, 7/14, and 7/15.

 

 

From: Brian Arkills Sent: Tuesday, June 24, 2014 2:16 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for the high-impact service change in 3 weeks.

 

More Info:

• We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
• The 1^st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
• We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

 

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

This is an update for the high-impact service change in 3 weeks.

 

More Info:

• We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
• The 1^st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
• We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

 

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

A service change is planned for the UWWI NETID domain service. You may see variations of this notice from other services that are dependent on UWWI for directory information, e.g. perhaps a notice from UW Exchange about the impact on the global address list.

 

What and When:

On Thursday, May 29^th, we’ll release a new version of our UWWI Person Directory Agent. This will result in quite a few changes to the naming attributes on UWWI user objects. The naming attributes affected include: displayName, givenName, sn, and initials. See More Info below for details.

 

Why:

This release fixes three known problems, gives us some flexibility we haven’t had previously, sets a default value so all UWWI users have a displayName, and gets UWWI out of a risky practice it shouldn’t have been in. See More Info below for details.

 

What you need to do:

Nothing: unfortunately, except in specific cases that do not cover the entire UW NetID population, there currently is very little a user can do modify what name data is displayed in directories and applications at the UW today. This fact remains true with or without this release.

 

To be clear, the following populations have some control of their enterprise naming data:

• Employees. Go to the Employee Self-Service (ESS) application and choose to publish your name.
• Students. The Registrar has publishable name data. This page (https://sdb.admin.washington.edu/students/uwnetid/address.asp) in the Student Personal Services app controls whether the name data the Registrar has is publishable or not.
• Shared UW NetIDs. Talk with the UW-IT Service Center (help@uw.edu) to get the “AVF displayName” value changed.All other populations have no publishable name data.UWWI encourages customers to ask UW-IT to prioritize the Preferred Name project if they’d like to see an enterprise solution to this problem.More Info:
•  

The effects of this release are:

1. “Dead” accounts no longer interfere with active users getting name data (see prior announcements for more about “dead” accounts). This may not sound important, but this issue affects thousands of accounts.
2. A bug that could result in a leading space in the displayName is fixed. This bug prevented affected users from getting provisioned to Azure Active Directory, inclusion in the Exchange global address list, and mailbox provisioning to Exchange Online.
3. The work we did 3 years ago to allow the name value from the naming data source for Shared UW NetIDs to get provisioned upon UW NetID password change will flow automatically without any need for a password change. This means that any Shared UW NetID that hasn’t had a password change in the last 3 years will benefit. See http://blogs.uw.edu/barkills/2011/06/08/uwwi-user-displayname-update-3/ for my description of that prior work.
4. We increase our flexibility by allowing one of the new custom attributes we added a couple months ago to be an overriding source of data. This means that when there aren’t any other enterprise options, UWWI can still satisfy its customers. This is costly from a support perspective, so we’d prefer that customers influence an enterprise solution here, but we know that sometimes things must happen now. J
5. All UWWI user accounts provisioned by UWWI have a default displayName and sn value. That default value is the UW NetID. If there is no usable naming data source, a UWWI user will end up with this value. This is the case for a large set of personal UW NetIDs.
6. We stop using a naming data source (the so-called “Official Name”) which has no clear privacy or publish guidance. When we launched the UWWI Person Directory Agent 6 years ago, we made a decision to use this data source. At that time, we recognized there was a privacy issue and chose to partially mitigate this by truncating the givenName portion of the name data. But over time, we’ve come to recognize that re-using this data source is not appropriate. Almost all UWWI user accounts that have a displayName value that looks like “B. Arkills” get their name data from this naming data source. The accounts where that is the case are predominantly personal UW NetIDs who are not employees or students.

 

There is quite a bit more detail about all of those things, and if you want to dig in deeper, look at http://www.netid.washington.edu/documentation/pdsToUwwiMapping.aspx. It’s been updated to reflect this release. It documents the new behavior in the section entitled “Complex Attribute Mappings.”

 

So in summary, we’ve fixed some bugs, added a default value, made name provisioning more consistent for Shared UW NetIDs, given ourselves an override, and stopped using a problematic data source.

 

If you have questions about this planned work, please send email to help@uw.edu with “UWWI Person Directory Agent” in the subject line. If you’d like to ask UW-IT to prioritize an enterprise solution, send email to help@uw.edu with “Preferred Name project” in the subject line.

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

Some minor service changes are planned to the UWWI NETID domain service.

 

What and When:

Today at 11am, we will add three group policy settings to the Default Domain Policy and add one setting to the Default Domain Controllers Policy. These settings are:

 

Default Domain Policy

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Allow Local System to use computer identity for NTLM = Enabled

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Outgoing NTLM traffic to remote servers = Audit All

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Audit Incoming NTLM Traffic = Enable auditing for all accounts

 

Default Domain Controllers Policy

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Audit NTLM authentication in this domain = Enable all

 

After we’ve turned off NTLMv1 on the domain controllers, we expect to remove the three settings with “Restrict NTLM” in their name.

 

What you need to do:

Nothing. This is a low impact change. All computers in the domain might result in an additional 1MB of log data.

 

Domain admins in domains that trust NETID are strongly encouraged to set these group policy settings in their Windows domain. See the More Info section for why.

 

More info:

First, please note that the three settings with “Restrict NTLM” in their name do not prevent NTLM—they are so named because they are part of a package of settings designed to help organizations eliminate NTLM.

 

http://technet.microsoft.com/en-us/library/jj852275.aspx and http://blogs.technet.com/b/askds/archive/2009/10/08/ntlm-blocking-and-you-application-analysis-and-auditing-methodologies-in-windows-7.aspx describe these settings in more detail.

 

We are setting ‘Network security: Allow Local System to use computer identity for NTLM’ to allow Vista or newer computers to leverage their computer identity (instead of anonymous) when performing so-called “null session” interactions over the network. Besides being a general improvement, this will have a side benefit to our efforts to eliminate NTLMv1 authentications in the NETID domain. It’ll more clearly identify in logs which computers are the source of NTLMv1.

 

We are setting the three settings with “Restrict NTLM” in their name to gain more detailed information about NTLMv1 use. Note that these settings generate additional log entries for NTLM traffic, separate from those already generated in the Windows Security log.

 

We believe all of these changes will allow us to more specifically target responsible individuals whose computers are misconfigured to use NTLMv1. We also believe this extra logging will later aid reactive troubleshooting efforts on the day that we turn NTLMv1 off on the NETID domain controllers. Domains that trust NETID are likely to be put in the position of needing to help identify the specific cause, and the extra information generated by these settings will be valuable for that. So we encourage all domains that trust NETID to also apply these settings.

 

If you have questions about this work, please send email to help@uw.edu with “UWWI NTLMv1 logging work” in the subject line.

 

-B