Archives: Service News

This email is to inform you that there is another mailing list you might be interested in joining.

 

uwwi-discuss@uw.edu is a mailman list with the following two intentions:

a) Enable more detailed information about the UWWI service to be communicated than the uwwi-announce list

b) Enable two-way conversation, i.e. anyone on the list can post to it

 

With respect to a), the UWWI service plans to send change notifications that we generally wouldn’t send to uwwi-announce. There are quite a few changes that happen to the UWWI service which are not judged to be impactful, but whose details may be of interest, so we don’t bother everyone on uwwi-announce with them. As an example, we made a fairly big architectural change to the UWWI Group Sync Agent in January that we didn’t send to the uwwi-announce mailing list, but likely would be interesting to some number of customers. If they are notable, they make it into our semi-annual newsletters, but not much detail is shared even then. Near the end of each month, I’m also planning on sending a monthly operational update to the uwwi-discuss list which will give you a sense of what’s going on, and what planned changes we think are coming up in the next month.

 

With respect to b), there currently isn’t a good way for the UWWI service to do customer outreach, where we capture your input about the service. By having 2-way conversations, we’d like to enable you to have a smaller feedback loop (and hear what other customers think/want). There also isn’t a good way for you to share solutions with each other. If I had a nickel for how many times I’ve sent one of you to another because I knew you had done some work in a particular area, I’d be rich. 🙂 But that targeted direction isn’t really fair, and I think a mailing list allows folks to choose to contribute as time allows instead of being put on the spot.

 

Go to https://mailman1.u.washington.edu/mailman/listinfo/uwwi-discuss to join the uwwi-discuss mailing list.

 

And again, it’s not required, but it may be something you’d like to join.

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

 

 

Two service changes are planned for the UWWI NETID domain service.

 

What and When:

Over the next 3 weeks we will be making two changes that affect the NETID domain controllers. The timing of these changes is not exact, due to various dependency issues.

 

The planned changes are:

1. We will turn on Windows Firewall on the NETID domain controllers. Access to the NETID domain controllers will remain unchanged–all campus networks will be permitted by the Windows Firewall configuration. We expect to make this change tomorrow.
2. The private view of the netid.washington.edu DNS zone will move from campus DNS to AD-integrated DNS. The public view of this zone will remain as is. DNS records in the zone will remain identical. This change is dependent on other services (and their design) so there is a possibility this won’t happen in the next 3 weeks.What you need to do:
3. More info:
4. That said, both of these changes have the potential to cause significant impact should something not go as expected. So while we don’t expect impact, we do want you to be aware that this work is planned. Should you encounter a wide-spread issue, please contact UW-IT via help@uw.edu or via the UW-IT Operations phone number (206-221-5000).
5. You should *not* change the DNS settings on your computers–continuing to use the campus DNS servers for DNS resolution is the right configuration.
6. No impact is expected. Both of these changes have already been tested and implemented in our evaluation environment.

We are implementing Windows Firewall on our domain controllers for a number of reasons. Those reasons include:

• Due diligence in security configuration to meet various policies
• Having the future ability to easily block specific IP addresses on UW networks across all the domain controllers, to limit exposure to an identified threat

 

We are moving our DNS zone for a variety of reasons. Those reasons include:

• Support for self-service DNS registration
• Rapid server provisioning. We’ll be able to promote/demote a domain controller without involvement from others, and in far less time (~2 hours vs. 12-18 hours)
• Improved control of who can get a record in this DNS zone

This change helps us to support a couple important outcomes:

• Better support for the UW Geo-Redundancy scenarios (e.g. allowing us to quickly bring up replacement domain controllers without waiting on other services that might be down)
• Paves the way for putting a domain controller in Azure, possibly in 2014
• Improves the experience of adding support for multiple Active Directory Sites (in 2014, we anticipate adding a site for our Tierpoint data center and probably Azure)

 

If you have questions about this planned work, please send email to help@uw.edu with “UWWI firewall and DNS work” in the subject line.

 

-B

A service change is planned for the UWWI NETID domain service.

 

What and When:

Next week we will mark approximately 50,000 NETID user accounts as expired, to bring these accounts into closer alignment with their current UW NetID status. All 50,000 of these NETID user accounts no longer have a valid UW NetID.

 

What you need to do:

No impact is expected. All of these NETID user accounts no longer have an associated UW NetID. None of these accounts have logged in over the course of the past 20 months.

 

More info:

In early 2012, the NETID domain experienced a significant incident as an outcome of a change made in late 2011. As a workaround to that incident, we unexpired user accounts that were expired at that time. Reconciling which accounts actually should be expired has been outstanding work that until now hasn’t been prioritized.

 

The NETID domain service tries to align as closely with the UW NetID service as it can, but there are notable differences due to the technology constraints and business rules required. When a UW NetID loses its password, the NETID domain service doesn’t delete the user account, but instead marks the account as expired and resets the password to a long random string unknown to anyone. This is primarily to retain the objectSID in case access to resources via user account is needed again, but there are also auditing reasons to retain the objectSID.

 

There is other work scheduled later this quarter that will adjust our practices around UW NetIDs that lose their password (i.e. UW NetIDs that are no longer valid). This work will result in a couple changes.

1. NETID users which go into this state will:

• be expired
• get random password
• get renamed cn/samAccountName values slightly to values that clearly indicate the user account is dead, but still retain the original UW NetID string

• be moved to a new OU set aside for “dead” NETID users
• be disabled
• have all optional attribute values removed

2. Users which have been in this “dead” state for 7 years will be deleted
3. Users which have been in this “dead” state, which are determined to be part of a transitory population, will be deleted after 1 year. An example of a transitory user would be an applicant.

 

We don’t plan on additionally announcing when that work happens, because that work is judged to be very low impact due to the fact that it doesn’t affect active users. However, we do think there is value in being transparent about the kinds of lifecycle practices we are employing.

 

Questions and Answers:

• I thought UW NetIDs were good for ever. Did something change?No, nothing changed. It’s just that amongst the large number of UW NetIDs there are always exceptions to the rule. There are several populations of UW NetIDs that can “lose” their UW NetID, where “lose” means the UW NetID still exists (i.e. can’t be issued to someone else) but the password is removed, and the associated accounts are no longer active. The populations that this can happen to include (but are not limited to):

• Former Applicants (i.e. students that applied but never enrolled)
• Former Shared UW NetIDs
• Former Clinicians
• Former Cascadia
• Former Admin UW NetIDs

 

Further questions about UW NetIDs and their lifecycle should be directed to the UW NetID service via help@uw.edu with “UW NetID lifecycle question” in the subject line.

 

• Why don’t you just delete the NETID user account tied to these dead UW NetIDs?

 

Because once we delete the user account, the objectSID is effectively gone. That objectSID is distributed to some unknown number of resource objects throughout the UW environment in access controls. If any part of the UW environment goes through an audit and has a resource which references that objectSID, they will be left baffled as to who *had* access to that resource. Unfortunately, this distributed nature of Windows access control is such that there is no easy way to “clean up” objectSIDs which are no longer meaningful.

 

• Why 7 years?

 

Because this is the longest regulatory period we are aware of where auditing logs need to be kept. If you are aware of a longer period, let us know.

 

• Why only 1 year for transitory users?

 

Two reasons: Because there are a lot of them. And it’s pretty unlikely that there are many resources they were granted access to which are significant in nature.

• Why are there so many NETID user accounts without an associated UW NetID?

 

Because the populations noted above have a lot of turnover and churn. Applicants in particular contribute quite a bit to the overall number. The NETID domain service is so far out of alignment because back in early 2012 as a workaround to a significant incident, we removed ~3 year’s worth of expirations. That’s where the 50k comes from, but additionally there are other expired NETID users (all the users which have expired in the past 20 months). So the total number after this work will be ~90k. That number will get cut down when we run the first 1 year deletion, but in general it will continue to grow.

Because multiple dependent services have encountered significant numbers of problems with this change, this service change has been partially rolled back. In specific, we have reverted the NETID domain controllers to permit NTLMv1 (as previously configured). We have *not* reverted the domain level configuration.

 

Dependent services with servers in the NETID domain that experienced problems supporting NTLMv2 only will need to implement a group policy that overrides the domain level setting in order to return to the configuration in effect prior to this change.

 

We are purposely not rolling back the change at the domain level because:

a) this sets an expectation of the right configuration, and those that can’t support it are exceptions

b) this allows each affected service to separately configure their LMCompatibilityLevel settings and move to compliance when ready

 

There will be future communication about next steps, known problems and workarounds, and when we’ll try to make this service change.

 

A service change is planned for the UWWI NETID domain service.

 

What and When:

On Friday July 26 2013 after business hours, we will migrate the NETID domain SYSVOL share from FRS to DFSR.

 

What you need to do:

Users may experience brief errors or logon failures if they logon during a small window of time after business hours on 7/26. Users will just need to logon again a minute later if affected. Otherwise no impact is expected.

 

More info:

During the transition period, each NETID domain controller will stop sharing SYSVOL and then start sharing it again. This brief period of time between can cause logon problems.

 

The DFSR technology has been available with SYSVOL for some time, but we haven’t prioritized making this transition previously. DFSR is more resilient than FRS and plays better across sites. FRS is also deprecated with the next release of Windows Server (expected this year), so we must move off this technology before we introduce any domain controllers with the next release.

 

We have tested this transition with an offline copy of the NETID domain, so expect no problems.

 

-B

 

 

A service change is planned for the UWWI NETID domain service.

 

What and When:

On August 1 2013, we’ll remove support for the NTLMv1 authentication protocol from both the NETID domain controllers and at the domain level.

 

What you need to do:

If you have a service that is unable to support Kerberos or NTLMv2 authentication and uses user accounts that aren’t NETID domain user accounts, then you can turn on support for NTLMv1 at your OU level as a workaround. If you do this, we strongly recommend that you begin exploring alternatives as the threat profile for NTLMv1 has greatly increased.

 

If your service requires NETID domain authentication and NLTMv1, we’d be happy to explore alternative solutions with you. Based on an audit over a 24 hour period, we are unaware of any such services.

 

More info:

At service inception in 2006, the NETID domain did not support NTLMv1 authentication. Due to customer requests, in 2007 NTLMv1 support was added after obtaining an exception to UW policies via the UW Privacy Assurance and System Security (PASS) Council. Growing pressures due to UW identity assurance initiatives and a greatly increased threat profile based on cloud-based NTLMv1 cracking tools mean it is time for NTLMv1 to be retired.

 

As implied above, we’ve audited NTLMv1 use via NETID domain user accounts for a 24 hour period and found no use of concern. So we believe this change to be of minimal impact.

 

We are unable to verify the configuration in domains trusting the NETID domain, and a configuration mismatch between trusting domains and the NETID domain is the most likely source of potential issues, but default settings make the possibility of problems unlikely. If you’d like to audit your own Windows domain for NTLMv1 use, if all of your DCs are WS2008 or better, we can share a PowerShell script with you that will extract relevant events over a 24 hour period. If your DCs are at a prior OS level, it is not possible to differentiate NTLMv1 use from NTLMv2 use without doing network captures.

 

Another possible source of issues are non-domain joined computers running Windows XP or previous in a default configuration. In that case, the configuration settings will be incompatible. Windows XP is no longer in mainstream support from Microsoft, and extended support ends in a year, so this potential source of problem should also be limited.

 

More background information about the LMCompatibilityLevel setting is available at http://technet.microsoft.com/en-us/magazine/2006.08.securitywatch.aspx.

 

-B

 

All previously announced DC refresh work has been completed, and the UWWI firewall document has been updated to reflect this. A temporary DC change is planned for the UWWI NETID domain service.

 

What and When:

A temporary domain controller will be promoted and demoted late next week. This temporary DC will be on a network not previously on the UWWI firewall document.

 

What you need to do:

If you have a firewall or network filters, you may need to adjust them (see link below).

 

More info:

Another network has been added to the UWWI firewall list:

172.2.1.0/24 (172.22.1.0-172.22.1.255).

 

Three networks previously on the UWWI firewall list have been removed to reflect completion of prior DC refresh work. For reference, the networks removed are:

-172.22.15.0/27 (172.22.15.0-172.22.15.31)

-172.22.16.64/27 (172.22.16.64-172.22.16.95)

-172.22.238.128/25 (172.22.238.128-172.22.238.255)

 

You can find the list of networks that correspond to NETID DCs to configure in your firewalls at http://www.netid.washington.edu/documentation/trustWithFirewall.aspx.

 

The temporary DC promotion/demotion is to facilitate an offline recovery test of the NETID domain via the Microsoft Active Directory Recovery Execution Services program that will happen the 2nd week of July. We need to have a temporary domain controller promoted and demoted in preparation for this exercise, and due to the nature of how this opportunity came about, we were unable to provide you more advanced warning of this change. We are still exploring ways to “hide” this temporary DC from customers to minimize the impact of not giving folks the customary 2 week warning about a change to the UWWI firewall list, but those explorations are still in process and need to be tempered by maintaining operational stability.

 

From: Brian Arkills Sent: Monday, November 05, 2012 12:02 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: NETID DC promotions and demotions

 

Several changes are planned for the UWWI NETID domain service. These changes will close the existing temporary service capacity gap, which was noted in the email below sent 2 weeks ago.

 

What and When:

Several new domain controllers on a network not previously in the UWWI firewall documentation will be promoted beginning this Thursday, 11/8. One DC will be promoted on 11/8, with 2 additional DCs following over the following week for a total of 3 new NETID domain controllers.

 

After these 3 new DCs have been added, 2 of the existing domain controllers will be demoted as they have reached end of life.

 

What you need to do:

If you have a firewall or network filters, you may need to adjust them (see link below).

 

If you’ve hard-coded specific domain controller names in applications or code, you will need to adjust that configuration. If you have hard-coded either mace.netid.washington.edu or yoda.netid.washington.edu, please change that configuration.

 

If neither of these situations apply to you, then you don’t need to do anything.

 

More info:

Another network has been added to the UWWI firewall list: 172.16.31.0/24 (172.16.31.0-172.16.31.255).

 

The network that leia.netid.washington.edu was on has been removed from the UWWI firewall list. For reference that network was: 172.22.14.0/27 (172.22.14.0-172.22.14.31).

 

You can find the list of networks that correspond to NETID DCs to configure in your firewalls at http://www.netid.washington.edu/documentation/trustWithFirewall.aspx.

 

When all DC demotions are complete, we will remove two of the networks listed in that document, so if you do manage network filters you may want to check back in 3-4 weeks to remove unnecessary networks in your filters in the future.

 

> —–Original Message—–

> From: Brian Arkills

> Sent: Tuesday, October 23, 2012 12:38 PM

> To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu)

> Subject: Netid domain controller (leia) forcible demotion planned ~1pm

> today

>

> Due to internal AD database corruption on Leia.netid.washington.edu and

> replication problems it was having, we have determined that we need to

> demote leia.netid.washington.edu, one of 5 existing domain controllers

> providing the NETID domain service in the UWWI service line. Leia has been

> offline since yesterday evening and won’t be coming back online. You may

> have experienced odd problems because of Leia’s AD corruption. If you think

> you’ve got a lingering issue caused by this, please do open a help request via

> help@uw.edu with UWWI somewhere in the subject line, and we’ll try to

> assist you.

>

> We have already replaced leia as the DNS primary for clients.uw.edu, the

> DDNS zone provided for delegated OU customers, and there has been no

> impact to customers of that service.

>

> This work represents a minor degradation in service capacity, but this is

> expected to be the case only for a short period, as leia was planned to be

> replaced in the coming months.

>

> If, for some reason, you’ve hard-coded something to

> leia.netid.washington.edu, you will want to change that.

>

> Brian Arkills

> UW-IT, Identity and Access Management

> UW Windows Infrastructure technical lead

 

What and When:

On Wednesday, May 8th, the NETID domain schema will be modified. Schema changes associated with Exchange 2010 SP3 will be applied.

 

What you need to do:

Nothing, this is purely informational. There is no expected impact to customers.

 

More info:

Please see http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=5401 for more information about the schema changes.

 

Please send email to help@uw.edu with “UWWI schema work” in the subject if you have any questions about this work.

 

Brian Arkills

UW-IT Identity and Access Management

UW Windows Infrastructure Service Manager