Archives: Service News

Some minor service changes are planned to the UWWI NETID domain service.

What and When:

Today at 11am, we will add three group policy settings to the Default Domain Policy and add one setting to the Default Domain Controllers Policy. These settings are:

Default Domain Policy

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Allow Local System to use computer identity for NTLM = Enabled

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Outgoing NTLM traffic to remote servers = Audit All

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Audit Incoming NTLM Traffic = Enable auditing for all accounts

Default Domain Controllers Policy

Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Network security: Restrict NTLM: Audit NTLM authentication in this domain = Enable all

After we’ve turned off NTLMv1 on the domain controllers, we expect to remove the three settings with “Restrict NTLM” in their name.

What you need to do:

Nothing. This is a low impact change. All computers in the domain might result in an additional 1MB of log data.

Domain admins in domains that trust NETID are strongly encouraged to set these group policy settings in their Windows domain. See the More Info section for why.

More info:

First, please note that the three settings with “Restrict NTLM” in their name do not prevent NTLM—they are so named because they are part of a package of settings designed to help organizations eliminate NTLM.

http://technet.microsoft.com/en-us/library/jj852275.aspx and http://blogs.technet.com/b/askds/archive/2009/10/08/ntlm-blocking-and-you-application-analysis-and-auditing-methodologies-in-windows-7.aspx describe these settings in more detail.

We are setting ‘Network security: Allow Local System to use computer identity for NTLM’ to allow Vista or newer computers to leverage their computer identity (instead of anonymous) when performing so-called “null session” interactions over the network. Besides being a general improvement, this will have a side benefit to our efforts to eliminate NTLMv1 authentications in the NETID domain. It’ll more clearly identify in logs which computers are the source of NTLMv1.

We are setting the three settings with “Restrict NTLM” in their name to gain more detailed information about NTLMv1 use. Note that these settings generate additional log entries for NTLM traffic, separate from those already generated in the Windows Security log.

We believe all of these changes will allow us to more specifically target responsible individuals whose computers are misconfigured to use NTLMv1. We also believe this extra logging will later aid reactive troubleshooting efforts on the day that we turn NTLMv1 off on the NETID domain controllers. Domains that trust NETID are likely to be put in the position of needing to help identify the specific cause, and the extra information generated by these settings will be valuable for that. So we encourage all domains that trust NETID to also apply these settings.

If you have questions about this work, please send email to help@uw.edu with “UWWI NTLMv1 logging work” in the subject line.

-B

A service change is planned for the UWWI NETID domain service.

What and When:

Three sets of schema changes will be made on Wednesday, April 2. The schema changes are:

• Windows Server 2012 R2 schema
• System Center Configuration Manager schema
• 2014 Custom schema changesWhat you need to do: Schema changes either extend or change the capabilities of Active Directory by defining new attributes and/or object classes. These changes add new attributes, add new classes, and in some cases add the new attributes to existing classes. In the case of the custom schema change, we are modifying the definition of a couple existing attributes, but these changes are either to give it a more appropriate description or improve the indexing or retention upon deletion behavior. Existing values in those attributes are not affected by any of those changes.
• More info:
• No impact is expected.

We are applying the WS2012R2 schema at this time (instead of waiting until summer when we’ll upgrade the domain controllers) to address a known issue with the ActiveDirectory powershell module.

We are applying the SCCM schema at this time in preparation for a UW-IT project to deploy SCCM in what will likely become a new service.

We are applying the 2014 custom schema at this time to:

• Retain UW NetID type information for various UWWI operational process
• Retain UW Entity type information for various UWWI operational process, including the Kerberos Delegation Sensitivity setting
• To have a location to store requested exceptions to the Kerberos Delegation Sensitivity policy (see http://www.netid.washington.edu/documentation/policy.aspx#delegation)
• To have a method of overriding the UWWI Person Data Agent behavior for UWWI user displayName population.
• To update our eduPerson class implementation to reflect changes to the specification since we deployed it in 2006
• To tweak the indexing and retention upon deletion behavior of previously added custom attributes

If you have questions about this planned work, please send email to help@uw.edu with “UWWI 2014 schema work” in the subject line.

-B

A new capability is available. Some minor changes (e.g. OU permissions) have happened to enable this.

What and When:

Delegated OU customers are now able to create, delete, and manage Group Managed Service Accounts (gMSAs). This provides a self-service, higher-security option for non-interactive applications, services, and scheduled tasks that run automatically but need a security credential.

What you need to do:

Nothing.

You may want to re-evaluate the credentials used by any existing Windows-based applications, services, or scheduled tasks and consider using a gMSA instead. In particular, gMSAs provide an excellent solution for these non-interactive processes for the scenario where an IT administrator leaves. You don’t need to manually change all the passwords known by that IT administrator or accept a risk by not changing those passwords.

More info:

See http://www.netid.washington.edu/documentation/groupManagedServiceAccounts.aspx for how to get started.

Note that gMSAs are more like AD computer accounts than like AD user accounts, and gMSAs are not UW NetIDs. gMSAs can be members of UW Groups.

If you have questions about this new capability, either respond to this email over on the uwwi-discuss@uw.edu mailing list, or send email to help@uw.edu with “UWWI gMSAs” in the subject line to get someone from the UWWI service team.

-B

This email is to inform you that there is another mailing list you might be interested in joining.

uwwi-discuss@uw.edu is a mailman list with the following two intentions:

a) Enable more detailed information about the UWWI service to be communicated than the uwwi-announce list

b) Enable two-way conversation, i.e. anyone on the list can post to it

With respect to a), the UWWI service plans to send change notifications that we generally wouldn’t send to uwwi-announce. There are quite a few changes that happen to the UWWI service which are not judged to be impactful, but whose details may be of interest, so we don’t bother everyone on uwwi-announce with them. As an example, we made a fairly big architectural change to the UWWI Group Sync Agent in January that we didn’t send to the uwwi-announce mailing list, but likely would be interesting to some number of customers. If they are notable, they make it into our semi-annual newsletters, but not much detail is shared even then. Near the end of each month, I’m also planning on sending a monthly operational update to the uwwi-discuss list which will give you a sense of what’s going on, and what planned changes we think are coming up in the next month.

With respect to b), there currently isn’t a good way for the UWWI service to do customer outreach, where we capture your input about the service. By having 2-way conversations, we’d like to enable you to have a smaller feedback loop (and hear what other customers think/want). There also isn’t a good way for you to share solutions with each other. If I had a nickel for how many times I’ve sent one of you to another because I knew you had done some work in a particular area, I’d be rich. 🙂 But that targeted direction isn’t really fair, and I think a mailing list allows folks to choose to contribute as time allows instead of being put on the spot.

Go to https://mailman1.u.washington.edu/mailman/listinfo/uwwi-discuss to join the uwwi-discuss mailing list.

And again, it’s not required, but it may be something you’d like to join.

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

Two service changes are planned for the UWWI NETID domain service.

What and When:

Over the next 3 weeks we will be making two changes that affect the NETID domain controllers. The timing of these changes is not exact, due to various dependency issues.

The planned changes are:

1. We will turn on Windows Firewall on the NETID domain controllers. Access to the NETID domain controllers will remain unchanged–all campus networks will be permitted by the Windows Firewall configuration. We expect to make this change tomorrow.
2. The private view of the netid.washington.edu DNS zone will move from campus DNS to AD-integrated DNS. The public view of this zone will remain as is. DNS records in the zone will remain identical. This change is dependent on other services (and their design) so there is a possibility this won’t happen in the next 3 weeks.What you need to do:
3. More info:
4. That said, both of these changes have the potential to cause significant impact should something not go as expected. So while we don’t expect impact, we do want you to be aware that this work is planned. Should you encounter a wide-spread issue, please contact UW-IT via help@uw.edu or via the UW-IT Operations phone number (206-221-5000).
5. You should *not* change the DNS settings on your computers–continuing to use the campus DNS servers for DNS resolution is the right configuration.
6. No impact is expected. Both of these changes have already been tested and implemented in our evaluation environment.

We are implementing Windows Firewall on our domain controllers for a number of reasons. Those reasons include:

• Due diligence in security configuration to meet various policies
• Having the future ability to easily block specific IP addresses on UW networks across all the domain controllers, to limit exposure to an identified threat

We are moving our DNS zone for a variety of reasons. Those reasons include:

• Support for self-service DNS registration
• Rapid server provisioning. We’ll be able to promote/demote a domain controller without involvement from others, and in far less time (~2 hours vs. 12-18 hours)
• Improved control of who can get a record in this DNS zone

This change helps us to support a couple important outcomes:

• Better support for the UW Geo-Redundancy scenarios (e.g. allowing us to quickly bring up replacement domain controllers without waiting on other services that might be down)
• Paves the way for putting a domain controller in Azure, possibly in 2014
• Improves the experience of adding support for multiple Active Directory Sites (in 2014, we anticipate adding a site for our Tierpoint data center and probably Azure)

If you have questions about this planned work, please send email to help@uw.edu with “UWWI firewall and DNS work” in the subject line.

-B

A service change is planned for the UWWI NETID domain service.

What and When:

Next week we will mark approximately 50,000 NETID user accounts as expired, to bring these accounts into closer alignment with their current UW NetID status. All 50,000 of these NETID user accounts no longer have a valid UW NetID.

What you need to do:

No impact is expected. All of these NETID user accounts no longer have an associated UW NetID. None of these accounts have logged in over the course of the past 20 months.

More info:

In early 2012, the NETID domain experienced a significant incident as an outcome of a change made in late 2011. As a workaround to that incident, we unexpired user accounts that were expired at that time. Reconciling which accounts actually should be expired has been outstanding work that until now hasn’t been prioritized.

The NETID domain service tries to align as closely with the UW NetID service as it can, but there are notable differences due to the technology constraints and business rules required. When a UW NetID loses its password, the NETID domain service doesn’t delete the user account, but instead marks the account as expired and resets the password to a long random string unknown to anyone. This is primarily to retain the objectSID in case access to resources via user account is needed again, but there are also auditing reasons to retain the objectSID.

There is other work scheduled later this quarter that will adjust our practices around UW NetIDs that lose their password (i.e. UW NetIDs that are no longer valid). This work will result in a couple changes.

1. NETID users which go into this state will:

• be expired
• get random password
• get renamed cn/samAccountName values slightly to values that clearly indicate the user account is dead, but still retain the original UW NetID string

• be moved to a new OU set aside for “dead” NETID users
• be disabled
• have all optional attribute values removed

2. Users which have been in this “dead” state for 7 years will be deleted
3. Users which have been in this “dead” state, which are determined to be part of a transitory population, will be deleted after 1 year. An example of a transitory user would be an applicant.

We don’t plan on additionally announcing when that work happens, because that work is judged to be very low impact due to the fact that it doesn’t affect active users. However, we do think there is value in being transparent about the kinds of lifecycle practices we are employing.

Questions and Answers:

• I thought UW NetIDs were good for ever. Did something change?No, nothing changed. It’s just that amongst the large number of UW NetIDs there are always exceptions to the rule. There are several populations of UW NetIDs that can “lose” their UW NetID, where “lose” means the UW NetID still exists (i.e. can’t be issued to someone else) but the password is removed, and the associated accounts are no longer active. The populations that this can happen to include (but are not limited to):

• Former Applicants (i.e. students that applied but never enrolled)
• Former Shared UW NetIDs
• Former Clinicians
• Former Cascadia
• Former Admin UW NetIDs

Further questions about UW NetIDs and their lifecycle should be directed to the UW NetID service via help@uw.edu with “UW NetID lifecycle question” in the subject line.

• Why don’t you just delete the NETID user account tied to these dead UW NetIDs?

Because once we delete the user account, the objectSID is effectively gone. That objectSID is distributed to some unknown number of resource objects throughout the UW environment in access controls. If any part of the UW environment goes through an audit and has a resource which references that objectSID, they will be left baffled as to who *had* access to that resource. Unfortunately, this distributed nature of Windows access control is such that there is no easy way to “clean up” objectSIDs which are no longer meaningful.

• Why 7 years?

Because this is the longest regulatory period we are aware of where auditing logs need to be kept. If you are aware of a longer period, let us know.

• Why only 1 year for transitory users?

Two reasons: Because there are a lot of them. And it’s pretty unlikely that there are many resources they were granted access to which are significant in nature.

• Why are there so many NETID user accounts without an associated UW NetID?

Because the populations noted above have a lot of turnover and churn. Applicants in particular contribute quite a bit to the overall number. The NETID domain service is so far out of alignment because back in early 2012 as a workaround to a significant incident, we removed ~3 year’s worth of expirations. That’s where the 50k comes from, but additionally there are other expired NETID users (all the users which have expired in the past 20 months). So the total number after this work will be ~90k. That number will get cut down when we run the first 1 year deletion, but in general it will continue to grow.

Because multiple dependent services have encountered significant numbers of problems with this change, this service change has been partially rolled back. In specific, we have reverted the NETID domain controllers to permit NTLMv1 (as previously configured). We have *not* reverted the domain level configuration.

Dependent services with servers in the NETID domain that experienced problems supporting NTLMv2 only will need to implement a group policy that overrides the domain level setting in order to return to the configuration in effect prior to this change.

We are purposely not rolling back the change at the domain level because:

a) this sets an expectation of the right configuration, and those that can’t support it are exceptions

b) this allows each affected service to separately configure their LMCompatibilityLevel settings and move to compliance when ready

There will be future communication about next steps, known problems and workarounds, and when we’ll try to make this service change.

A service change is planned for the UWWI NETID domain service.

What and When:

On Friday July 26 2013 after business hours, we will migrate the NETID domain SYSVOL share from FRS to DFSR.

What you need to do:

Users may experience brief errors or logon failures if they logon during a small window of time after business hours on 7/26. Users will just need to logon again a minute later if affected. Otherwise no impact is expected.

More info:

During the transition period, each NETID domain controller will stop sharing SYSVOL and then start sharing it again. This brief period of time between can cause logon problems.

The DFSR technology has been available with SYSVOL for some time, but we haven’t prioritized making this transition previously. DFSR is more resilient than FRS and plays better across sites. FRS is also deprecated with the next release of Windows Server (expected this year), so we must move off this technology before we introduce any domain controllers with the next release.

We have tested this transition with an offline copy of the NETID domain, so expect no problems.

-B