Archives: Service News

Several changes are planned for the NETID domain service.

 

What and When:

The week of July 21, 2014 (7/21/2014), each NETID domain controller will be demoted, upgraded to Windows Server 2012 R2, then promoted. Only one domain controller will be affected any given day.

 

7/21: bane.netid.washington.edu

7/22: vader.netid.washington.edu

7/23: maul.netid.washington.edu

7/24: sidious.netid.washington.edu

7/25: tyranus.netid.washington.edu

 

What you need to do:

If you’ve hard-coded specific domain controller names in applications or code, you will need to adjust that configuration, otherwise you don’t need to do anything.

 

More info:

No networks are changing—the domain controllers will remain at the same IP addresses.

 

The domain controller with the FSMO roles will move from vader to tyranus from 7/22 to 7/25. On 7/25, all FSMO roles will return to vader. The active UWWI kiwi client will also transition with exactly the same details.

 

If you have questions or concerns, please contact us by sending email to help@uw.edu with “UWWI” somewhere in the subject line.

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

 

This is an update for the high-impact service change.

 

The date for this change has been moved.

 

What and When:

On Tuesday, August 12 (8/12/2014) at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

• We have moved the change date back to give customers operating web servers that rely on Windows Integrated authentication and the NETID domain more time to make changes to address the non-Windows browser issues we’ve noted in previous weeks.
• The Known Problems/Workarounds document has had several modifications over the last couple weeks, most notably adding addition options for web servers currently using Windows Integrated.
• We apologize for falling behind in sharing our log details and user notification lists. The Known NTLMv1 Logons page should be up to date with all the log analysis and user notification lists, and will remain up to date.
• We are modifying our user notification schedule to reflect the new change date. The new user notification dates are: 7/8 (already went out), 7/15, 7/22, 7/29, 8/5, and 8/11.
• Week to week comparisons based on our logs indicate the user notifications are effective:

 

	events	users	computers
6/17-6/23	420944	857	928
6/24-7/1	35444	251	275
7/1-7/7	19647	203	198

 

 

From: Brian Arkills Sent: Tuesday, July 1, 2014 2:36 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for the high-impact service change in 2 weeks.

 

More Info:

• NTLMv1 use is significantly down in the server log files available to the UWWI service team. Last week’s logs suggest that as many as 75% of the misconfigured computers we were seeing a month ago have now been fixed. The UWWI service does not have access to your log files. Only you can check whether your users will be affected. See the original announcement below for resources to help you do that.
• An update on the non-Windows browser known problem we mentioned last week: Making a change to an IIS web server which is configured to use Windows Integrated authentication may be a workaround to consider. We’ve added removing Integrated Windows authentication and adding Basic authentication (with SSL required) as a workaround to our documentation. The Dynamics AX service plans to apply this workaround. If you do have a IIS web server with Windows Integrated enabled, you should check your logs for NTLMv1 use.
• We continue to email user notifications to those users that are in the log files we have access to. We sent a round of notifications today to 250 users. We plan to send additional user notifications on: 7/8, 7/14, and 7/15.

 

 

From: Brian Arkills Sent: Tuesday, June 24, 2014 2:16 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for the high-impact service change in 3 weeks.

 

More Info:

• We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
• The 1^st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
• We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

 

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

This is an update for the high-impact service change in 2 weeks.

 

More Info:

• NTLMv1 use is significantly down in the server log files available to the UWWI service team. Last week’s logs suggest that as many as 75% of the misconfigured computers we were seeing a month ago have now been fixed. The UWWI service does not have access to your log files. Only you can check whether your users will be affected. See the original announcement below for resources to help you do that.
• An update on the non-Windows browser known problem we mentioned last week: Making a change to an IIS web server which is configured to use Windows Integrated authentication may be a workaround to consider. We’ve added removing Integrated Windows authentication and adding Basic authentication (with SSL required) as a workaround to our documentation. The Dynamics AX service plans to apply this workaround. If you do have a IIS web server with Windows Integrated enabled, you should check your logs for NTLMv1 use.
• We continue to email user notifications to those users that are in the log files we have access to. We sent a round of notifications today to 250 users. We plan to send additional user notifications on: 7/8, 7/14, and 7/15.

 

 

From: Brian Arkills Sent: Tuesday, June 24, 2014 2:16 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for the high-impact service change in 3 weeks.

 

More Info:

• We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
• The 1^st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
• We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

 

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

This is an update for the high-impact service change in 3 weeks.

 

More Info:

• We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
• The 1^st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
• We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

 

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

A service change is planned for the UWWI NETID domain service. You may see variations of this notice from other services that are dependent on UWWI for directory information, e.g. perhaps a notice from UW Exchange about the impact on the global address list.

 

What and When:

On Thursday, May 29^th, we’ll release a new version of our UWWI Person Directory Agent. This will result in quite a few changes to the naming attributes on UWWI user objects. The naming attributes affected include: displayName, givenName, sn, and initials. See More Info below for details.

 

Why:

This release fixes three known problems, gives us some flexibility we haven’t had previously, sets a default value so all UWWI users have a displayName, and gets UWWI out of a risky practice it shouldn’t have been in. See More Info below for details.

 

What you need to do:

Nothing: unfortunately, except in specific cases that do not cover the entire UW NetID population, there currently is very little a user can do modify what name data is displayed in directories and applications at the UW today. This fact remains true with or without this release.

 

To be clear, the following populations have some control of their enterprise naming data:

• Employees. Go to the Employee Self-Service (ESS) application and choose to publish your name.
• Students. The Registrar has publishable name data. This page (https://sdb.admin.washington.edu/students/uwnetid/address.asp) in the Student Personal Services app controls whether the name data the Registrar has is publishable or not.
• Shared UW NetIDs. Talk with the UW-IT Service Center (help@uw.edu) to get the “AVF displayName” value changed.All other populations have no publishable name data.UWWI encourages customers to ask UW-IT to prioritize the Preferred Name project if they’d like to see an enterprise solution to this problem.More Info:
•  

The effects of this release are:

1. “Dead” accounts no longer interfere with active users getting name data (see prior announcements for more about “dead” accounts). This may not sound important, but this issue affects thousands of accounts.
2. A bug that could result in a leading space in the displayName is fixed. This bug prevented affected users from getting provisioned to Azure Active Directory, inclusion in the Exchange global address list, and mailbox provisioning to Exchange Online.
3. The work we did 3 years ago to allow the name value from the naming data source for Shared UW NetIDs to get provisioned upon UW NetID password change will flow automatically without any need for a password change. This means that any Shared UW NetID that hasn’t had a password change in the last 3 years will benefit. See http://blogs.uw.edu/barkills/2011/06/08/uwwi-user-displayname-update-3/ for my description of that prior work.
4. We increase our flexibility by allowing one of the new custom attributes we added a couple months ago to be an overriding source of data. This means that when there aren’t any other enterprise options, UWWI can still satisfy its customers. This is costly from a support perspective, so we’d prefer that customers influence an enterprise solution here, but we know that sometimes things must happen now. J
5. All UWWI user accounts provisioned by UWWI have a default displayName and sn value. That default value is the UW NetID. If there is no usable naming data source, a UWWI user will end up with this value. This is the case for a large set of personal UW NetIDs.
6. We stop using a naming data source (the so-called “Official Name”) which has no clear privacy or publish guidance. When we launched the UWWI Person Directory Agent 6 years ago, we made a decision to use this data source. At that time, we recognized there was a privacy issue and chose to partially mitigate this by truncating the givenName portion of the name data. But over time, we’ve come to recognize that re-using this data source is not appropriate. Almost all UWWI user accounts that have a displayName value that looks like “B. Arkills” get their name data from this naming data source. The accounts where that is the case are predominantly personal UW NetIDs who are not employees or students.

 

There is quite a bit more detail about all of those things, and if you want to dig in deeper, look at http://www.netid.washington.edu/documentation/pdsToUwwiMapping.aspx. It’s been updated to reflect this release. It documents the new behavior in the section entitled “Complex Attribute Mappings.”

 

So in summary, we’ve fixed some bugs, added a default value, made name provisioning more consistent for Shared UW NetIDs, given ourselves an override, and stopped using a problematic data source.

 

If you have questions about this planned work, please send email to help@uw.edu with “UWWI Person Directory Agent” in the subject line. If you’d like to ask UW-IT to prioritize an enterprise solution, send email to help@uw.edu with “Preferred Name project” in the subject line.

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager