Archives: Service News

This is an update for this high-impact service change.

A date for the change has been set:

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

A service change is planned for the UWWI NETID domain service. You may see variations of this notice from other services that are dependent on UWWI for directory information, e.g. perhaps a notice from UW Exchange about the impact on the global address list.

What and When:

On Thursday, May 29^th, we’ll release a new version of our UWWI Person Directory Agent. This will result in quite a few changes to the naming attributes on UWWI user objects. The naming attributes affected include: displayName, givenName, sn, and initials. See More Info below for details.

Why:

This release fixes three known problems, gives us some flexibility we haven’t had previously, sets a default value so all UWWI users have a displayName, and gets UWWI out of a risky practice it shouldn’t have been in. See More Info below for details.

What you need to do:

Nothing: unfortunately, except in specific cases that do not cover the entire UW NetID population, there currently is very little a user can do modify what name data is displayed in directories and applications at the UW today. This fact remains true with or without this release.

To be clear, the following populations have some control of their enterprise naming data:

• Employees. Go to the Employee Self-Service (ESS) application and choose to publish your name.
• Students. The Registrar has publishable name data. This page (https://sdb.admin.washington.edu/students/uwnetid/address.asp) in the Student Personal Services app controls whether the name data the Registrar has is publishable or not.
• Shared UW NetIDs. Talk with the UW-IT Service Center (help@uw.edu) to get the “AVF displayName” value changed.All other populations have no publishable name data.UWWI encourages customers to ask UW-IT to prioritize the Preferred Name project if they’d like to see an enterprise solution to this problem.More Info:
•  

The effects of this release are:

1. “Dead” accounts no longer interfere with active users getting name data (see prior announcements for more about “dead” accounts). This may not sound important, but this issue affects thousands of accounts.
2. A bug that could result in a leading space in the displayName is fixed. This bug prevented affected users from getting provisioned to Azure Active Directory, inclusion in the Exchange global address list, and mailbox provisioning to Exchange Online.
3. The work we did 3 years ago to allow the name value from the naming data source for Shared UW NetIDs to get provisioned upon UW NetID password change will flow automatically without any need for a password change. This means that any Shared UW NetID that hasn’t had a password change in the last 3 years will benefit. See http://blogs.uw.edu/barkills/2011/06/08/uwwi-user-displayname-update-3/ for my description of that prior work.
4. We increase our flexibility by allowing one of the new custom attributes we added a couple months ago to be an overriding source of data. This means that when there aren’t any other enterprise options, UWWI can still satisfy its customers. This is costly from a support perspective, so we’d prefer that customers influence an enterprise solution here, but we know that sometimes things must happen now. J
5. All UWWI user accounts provisioned by UWWI have a default displayName and sn value. That default value is the UW NetID. If there is no usable naming data source, a UWWI user will end up with this value. This is the case for a large set of personal UW NetIDs.
6. We stop using a naming data source (the so-called “Official Name”) which has no clear privacy or publish guidance. When we launched the UWWI Person Directory Agent 6 years ago, we made a decision to use this data source. At that time, we recognized there was a privacy issue and chose to partially mitigate this by truncating the givenName portion of the name data. But over time, we’ve come to recognize that re-using this data source is not appropriate. Almost all UWWI user accounts that have a displayName value that looks like “B. Arkills” get their name data from this naming data source. The accounts where that is the case are predominantly personal UW NetIDs who are not employees or students.

There is quite a bit more detail about all of those things, and if you want to dig in deeper, look at http://www.netid.washington.edu/documentation/pdsToUwwiMapping.aspx. It’s been updated to reflect this release. It documents the new behavior in the section entitled “Complex Attribute Mappings.”

So in summary, we’ve fixed some bugs, added a default value, made name provisioning more consistent for Shared UW NetIDs, given ourselves an override, and stopped using a problematic data source.

If you have questions about this planned work, please send email to help@uw.edu with “UWWI Person Directory Agent” in the subject line. If you’d like to ask UW-IT to prioritize an enterprise solution, send email to help@uw.edu with “Preferred Name project” in the subject line.

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

There are several things we’d like IT support staff to do:

1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

Some minor service changes are planned to the UWWI NETID domain service.

What and When:

Today at 11am, we will add three group policy settings to the Default Domain Policy and add one setting to the Default Domain Controllers Policy. These settings are:

Default Domain Policy

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Allow Local System to use computer identity for NTLM = Enabled

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Outgoing NTLM traffic to remote servers = Audit All

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Audit Incoming NTLM Traffic = Enable auditing for all accounts

Default Domain Controllers Policy

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Audit NTLM authentication in this domain = Enable all

After we’ve turned off NTLMv1 on the domain controllers, we expect to remove the three settings with “Restrict NTLM” in their name.

What you need to do:

Nothing. This is a low impact change. All computers in the domain might result in an additional 1MB of log data.

Domain admins in domains that trust NETID are strongly encouraged to set these group policy settings in their Windows domain. See the More Info section for why.

More info:

First, please note that the three settings with “Restrict NTLM” in their name do not prevent NTLM—they are so named because they are part of a package of settings designed to help organizations eliminate NTLM.

http://technet.microsoft.com/en-us/library/jj852275.aspx and http://blogs.technet.com/b/askds/archive/2009/10/08/ntlm-blocking-and-you-application-analysis-and-auditing-methodologies-in-windows-7.aspx describe these settings in more detail.

We are setting ‘Network security: Allow Local System to use computer identity for NTLM’ to allow Vista or newer computers to leverage their computer identity (instead of anonymous) when performing so-called “null session” interactions over the network. Besides being a general improvement, this will have a side benefit to our efforts to eliminate NTLMv1 authentications in the NETID domain. It’ll more clearly identify in logs which computers are the source of NTLMv1.

We are setting the three settings with “Restrict NTLM” in their name to gain more detailed information about NTLMv1 use. Note that these settings generate additional log entries for NTLM traffic, separate from those already generated in the Windows Security log.

We believe all of these changes will allow us to more specifically target responsible individuals whose computers are misconfigured to use NTLMv1. We also believe this extra logging will later aid reactive troubleshooting efforts on the day that we turn NTLMv1 off on the NETID domain controllers. Domains that trust NETID are likely to be put in the position of needing to help identify the specific cause, and the extra information generated by these settings will be valuable for that. So we encourage all domains that trust NETID to also apply these settings.

If you have questions about this work, please send email to help@uw.edu with “UWWI NTLMv1 logging work” in the subject line.

-B

A service change is planned for the UWWI NETID domain service.

What and When:

Three sets of schema changes will be made on Wednesday, April 2. The schema changes are:

• Windows Server 2012 R2 schema
• System Center Configuration Manager schema
• 2014 Custom schema changesWhat you need to do: Schema changes either extend or change the capabilities of Active Directory by defining new attributes and/or object classes. These changes add new attributes, add new classes, and in some cases add the new attributes to existing classes. In the case of the custom schema change, we are modifying the definition of a couple existing attributes, but these changes are either to give it a more appropriate description or improve the indexing or retention upon deletion behavior. Existing values in those attributes are not affected by any of those changes.
• More info:
• No impact is expected.

We are applying the WS2012R2 schema at this time (instead of waiting until summer when we’ll upgrade the domain controllers) to address a known issue with the ActiveDirectory powershell module.

We are applying the SCCM schema at this time in preparation for a UW-IT project to deploy SCCM in what will likely become a new service.

We are applying the 2014 custom schema at this time to:

• Retain UW NetID type information for various UWWI operational process
• Retain UW Entity type information for various UWWI operational process, including the Kerberos Delegation Sensitivity setting
• To have a location to store requested exceptions to the Kerberos Delegation Sensitivity policy (see http://www.netid.washington.edu/documentation/policy.aspx#delegation)
• To have a method of overriding the UWWI Person Data Agent behavior for UWWI user displayName population.
• To update our eduPerson class implementation to reflect changes to the specification since we deployed it in 2006
• To tweak the indexing and retention upon deletion behavior of previously added custom attributes

If you have questions about this planned work, please send email to help@uw.edu with “UWWI 2014 schema work” in the subject line.

-B

A new capability is available. Some minor changes (e.g. OU permissions) have happened to enable this.

What and When:

Delegated OU customers are now able to create, delete, and manage Group Managed Service Accounts (gMSAs). This provides a self-service, higher-security option for non-interactive applications, services, and scheduled tasks that run automatically but need a security credential.

What you need to do:

Nothing.

You may want to re-evaluate the credentials used by any existing Windows-based applications, services, or scheduled tasks and consider using a gMSA instead. In particular, gMSAs provide an excellent solution for these non-interactive processes for the scenario where an IT administrator leaves. You don’t need to manually change all the passwords known by that IT administrator or accept a risk by not changing those passwords.

More info:

See http://www.netid.washington.edu/documentation/groupManagedServiceAccounts.aspx for how to get started.

Note that gMSAs are more like AD computer accounts than like AD user accounts, and gMSAs are not UW NetIDs. gMSAs can be members of UW Groups.

If you have questions about this new capability, either respond to this email over on the uwwi-discuss@uw.edu mailing list, or send email to help@uw.edu with “UWWI gMSAs” in the subject line to get someone from the UWWI service team.

-B

This email is to inform you that there is another mailing list you might be interested in joining.

uwwi-discuss@uw.edu is a mailman list with the following two intentions:

a) Enable more detailed information about the UWWI service to be communicated than the uwwi-announce list

b) Enable two-way conversation, i.e. anyone on the list can post to it

With respect to a), the UWWI service plans to send change notifications that we generally wouldn’t send to uwwi-announce. There are quite a few changes that happen to the UWWI service which are not judged to be impactful, but whose details may be of interest, so we don’t bother everyone on uwwi-announce with them. As an example, we made a fairly big architectural change to the UWWI Group Sync Agent in January that we didn’t send to the uwwi-announce mailing list, but likely would be interesting to some number of customers. If they are notable, they make it into our semi-annual newsletters, but not much detail is shared even then. Near the end of each month, I’m also planning on sending a monthly operational update to the uwwi-discuss list which will give you a sense of what’s going on, and what planned changes we think are coming up in the next month.

With respect to b), there currently isn’t a good way for the UWWI service to do customer outreach, where we capture your input about the service. By having 2-way conversations, we’d like to enable you to have a smaller feedback loop (and hear what other customers think/want). There also isn’t a good way for you to share solutions with each other. If I had a nickel for how many times I’ve sent one of you to another because I knew you had done some work in a particular area, I’d be rich. 🙂 But that targeted direction isn’t really fair, and I think a mailing list allows folks to choose to contribute as time allows instead of being put on the spot.

Go to https://mailman1.u.washington.edu/mailman/listinfo/uwwi-discuss to join the uwwi-discuss mailing list.

And again, it’s not required, but it may be something you’d like to join.

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager