Skip to content

Archives: Service News

News content type for use by UW-IT Services on IT Connect

2013 July

Here’s an update on recent happenings with the UW Windows Infrastructure.

 

==== New Capabilities and Improvements ====

 

* Work to refresh aging NETID domain controllers is complete. All NETID domain controllers are running Windows Server 2012, and the domain and forest functional levels are at Windows Server 2012.

 

* Work to add item-level recovery (via the Active Directory Recycle Bin feature) is complete. If you accidentally delete objects in your delegated OU, we can now recover them. See http://www.netid.washington.edu/documentation/netidItemLevelRestore.aspx for details.

 

* Work to provide OU admins access to operational attributes on all NETID user accounts is complete. This provides more information to OU admins to support troubleshooting efforts. A separate email was sent to OU admins about this.

 

* The UWWI trust policy was changed. You can review this at http://www.netid.washington.edu/documentation/policy.aspx.

 

* A recovery test plan for the NETID domain was created as part of the university’s Business Continuity initiative.

 

* Unix integration capabilities were added to UWWI services. See the spotlight for more info.

 

* The NETID domain has been extended to Microsoft’s Azure Active Directory. See the spotlight for more info.

 

* For web-based applications that only support authentication via Active Directory Federation Services (ADFS), UW-IT is offering limited support for ADFS 2.0 to enable the standard “weblogin” user experience. Contact iam-support@uw.edu for additional information.

 

====Spotlights====

 

* Customers such as Kris Shaw are eagerly taking advantage of new Unix integration capabilities. These new capabilities make it much easier to join Unix/linux computers to the NETID domain. Work included: extending uidNumber assignment to a larger user population, assigning GIDs to all groups in the Groups Service and syncing that data to UWWI, and assigning gidNumber values to all UWWI user accounts. We appreciate the partnership and patience that Kris and others have provided while we’ve worked on this work over several quarters.

 

* UWWI has deployed the Azure Active Directory Sync tool. While this capability was deployed in support of ongoing Office 365 efforts, it has broader usefulness. You can leverage Azure Active Directory (AAD) identities from on-premise applications that are written to take advantage of AAD and from cloud-based applications. This greatly extends the usefulness of the NETID domain services beyond the confines of the p172 network space. Microsoft is investing heavily in this area, with an Azure Active Authentication (preview) capability that permits your smartphone to be used as an additional authentication factor announced recently. We expect this new Azure Active Directory aspect of the UW-IT portfolio to be an active area of growth.

 

* Delegated OU customers that have misconfigured their computer’s primary DNS suffix will see individualized notification efforts to fix this beginning August 1. Related to this, in partnership with efforts at the Library and ISchool, we’ve identified a potential issue that can occur in a limited scenario that can result in the error message “The security database on the server does not have a computer account for this workstation trust relationship.” As a mitigation for this issue, we’ve decided to amend the permissions granted to all computerjoiners groups, granting your computerjoiners group full control permissions to computer objects within your OU. We’ll be making that change in the near future.

 

* Support for the NTLMv1 authentication protocol will be turned off on August 1. UW identity assurance initiatives and improved capabilities in breaking NTLMv1, plus the fact that the exception to UW policies via the UW Privacy Assurance and System Security (PASS) Council was granted 6 years ago, mean it’s time for NTLMv1 to go. A separate announcement will include more details.

 

==== Trends ====

 

* Since January, UWWI has added: 11 delegated OUs (73 total), 3 trusts (57 total), ~1000 computers (6600 total), ~43k users (622k total).

* UWWI support requests have grown by 30%. 151 UWWI support tickets resolved since January (vs. 119 in prior period).

 

You can see metrics about UWWI at http://www.netid.washington.edu/dirinfo/stats.

 

==== What’s Next ====

 

Our objectives for the months ahead include:

 

* During July, we will test an offline recovery of the NETID domain via the Microsoft Active Directory Recovery Execution Services program, as well as getting a health assessment via Microsoft’s Active Directory Risk Assessment Program. You may see a temporary domain controller promoted and demoted in preparation for this exercise–we’re still working out the details. 🙂

* Operational improvements to improve our business continuity stance

* Investigation of replacement for our aging ILM component that provides “white page” data to UWWI.

* Investigation of improved audit log retention and reporting

* Investigation of providing Group Managed Service Account (gMSA) capability, which provides service accounts with passwords that no human ever sees, with automatic password updates built-in.

* Continued support of the Office 365 projects as they integrate the UWWI NETID domain services with Office 365 application deployments.

* Support for a project internal to UW-IT, helping to consolidate the UCSADMIN domain to the NETID domain

 

==== Your Feedback ====

 

Supporting your needs for UWWI capabilities offered via the Basic Services Bundle is our priority, so we welcome feedback on how we can make the UWWI service more valuable to you.

 

The UWWI service has a backlog visible to customers at https://jira.cac.washington.edu/browse/UWWI where you can get more details about possible improvements, current prioritization of that work, and even what we’ve been doing. You can “vote” for items in the backlog to help us rank priorities, or you can contact us via iam-support@uw.edu.

 

 

Removal of support for NTLMv1 authentication in the NETID domain

A service change is planned for the UWWI NETID domain service.

 

What and When:

On August 1 2013, we’ll remove support for the NTLMv1 authentication protocol from both the NETID domain controllers and at the domain level.

 

What you need to do:

If you have a service that is unable to support Kerberos or NTLMv2 authentication and uses user accounts that aren’t NETID domain user accounts, then you can turn on support for NTLMv1 at your OU level as a workaround. If you do this, we strongly recommend that you begin exploring alternatives as the threat profile for NTLMv1 has greatly increased.

 

If your service requires NETID domain authentication and NLTMv1, we’d be happy to explore alternative solutions with you. Based on an audit over a 24 hour period, we are unaware of any such services.

 

More info:

At service inception in 2006, the NETID domain did not support NTLMv1 authentication. Due to customer requests, in 2007 NTLMv1 support was added after obtaining an exception to UW policies via the UW Privacy Assurance and System Security (PASS) Council. Growing pressures due to UW identity assurance initiatives and a greatly increased threat profile based on cloud-based NTLMv1 cracking tools mean it is time for NTLMv1 to be retired.

 

As implied above, we’ve audited NTLMv1 use via NETID domain user accounts for a 24 hour period and found no use of concern. So we believe this change to be of minimal impact.

 

We are unable to verify the configuration in domains trusting the NETID domain, and a configuration mismatch between trusting domains and the NETID domain is the most likely source of potential issues, but default settings make the possibility of problems unlikely. If you’d like to audit your own Windows domain for NTLMv1 use, if all of your DCs are WS2008 or better, we can share a PowerShell script with you that will extract relevant events over a 24 hour period. If your DCs are at a prior OS level, it is not possible to differentiate NTLMv1 use from NTLMv2 use without doing network captures.

 

Another possible source of issues are non-domain joined computers running Windows XP or previous in a default configuration. In that case, the configuration settings will be incompatible. Windows XP is no longer in mainstream support from Microsoft, and extended support ends in a year, so this potential source of problem should also be limited.

 

More background information about the LMCompatibilityLevel setting is available at http://technet.microsoft.com/en-us/magazine/2006.08.securitywatch.aspx.

 

-B

 

RE: NETID DC promotions and demotions

All previously announced DC refresh work has been completed, and the UWWI firewall document has been updated to reflect this. A temporary DC change is planned for the UWWI NETID domain service.

 

What and When:

A temporary domain controller will be promoted and demoted late next week. This temporary DC will be on a network not previously on the UWWI firewall document.

 

What you need to do:

If you have a firewall or network filters, you may need to adjust them (see link below).

 

More info:

Another network has been added to the UWWI firewall list:

172.2.1.0/24 (172.22.1.0-172.22.1.255).

 

Three networks previously on the UWWI firewall list have been removed to reflect completion of prior DC refresh work. For reference, the networks removed are:

-172.22.15.0/27 (172.22.15.0-172.22.15.31)

-172.22.16.64/27 (172.22.16.64-172.22.16.95)

-172.22.238.128/25 (172.22.238.128-172.22.238.255)

 

You can find the list of networks that correspond to NETID DCs to configure in your firewalls at http://www.netid.washington.edu/documentation/trustWithFirewall.aspx.

 

The temporary DC promotion/demotion is to facilitate an offline recovery test of the NETID domain via the Microsoft Active Directory Recovery Execution Services program that will happen the 2nd week of July. We need to have a temporary domain controller promoted and demoted in preparation for this exercise, and due to the nature of how this opportunity came about, we were unable to provide you more advanced warning of this change. We are still exploring ways to “hide” this temporary DC from customers to minimize the impact of not giving folks the customary 2 week warning about a change to the UWWI firewall list, but those explorations are still in process and need to be tempered by maintaining operational stability.

 

From: Brian Arkills Sent: Monday, November 05, 2012 12:02 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: NETID DC promotions and demotions

 

Several changes are planned for the UWWI NETID domain service. These changes will close the existing temporary service capacity gap, which was noted in the email below sent 2 weeks ago.

 

What and When:

Several new domain controllers on a network not previously in the UWWI firewall documentation will be promoted beginning this Thursday, 11/8. One DC will be promoted on 11/8, with 2 additional DCs following over the following week for a total of 3 new NETID domain controllers.

 

After these 3 new DCs have been added, 2 of the existing domain controllers will be demoted as they have reached end of life.

 

What you need to do:

If you have a firewall or network filters, you may need to adjust them (see link below).

 

If you’ve hard-coded specific domain controller names in applications or code, you will need to adjust that configuration. If you have hard-coded either mace.netid.washington.edu or yoda.netid.washington.edu, please change that configuration.

 

If neither of these situations apply to you, then you don’t need to do anything.

 

More info:

Another network has been added to the UWWI firewall list: 172.16.31.0/24 (172.16.31.0-172.16.31.255).

 

The network that leia.netid.washington.edu was on has been removed from the UWWI firewall list. For reference that network was: 172.22.14.0/27 (172.22.14.0-172.22.14.31).

 

You can find the list of networks that correspond to NETID DCs to configure in your firewalls at http://www.netid.washington.edu/documentation/trustWithFirewall.aspx.

 

When all DC demotions are complete, we will remove two of the networks listed in that document, so if you do manage network filters you may want to check back in 3-4 weeks to remove unnecessary networks in your filters in the future.

 

> —–Original Message—–

> From: Brian Arkills

> Sent: Tuesday, October 23, 2012 12:38 PM

> To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu)

> Subject: Netid domain controller (leia) forcible demotion planned ~1pm

> today

>

> Due to internal AD database corruption on Leia.netid.washington.edu and

> replication problems it was having, we have determined that we need to

> demote leia.netid.washington.edu, one of 5 existing domain controllers

> providing the NETID domain service in the UWWI service line. Leia has been

> offline since yesterday evening and won’t be coming back online. You may

> have experienced odd problems because of Leia’s AD corruption. If you think

> you’ve got a lingering issue caused by this, please do open a help request via

> help@uw.edu with UWWI somewhere in the subject line, and we’ll try to

> assist you.

>

> We have already replaced leia as the DNS primary for clients.uw.edu, the

> DDNS zone provided for delegated OU customers, and there has been no

> impact to customers of that service.

>

> This work represents a minor degradation in service capacity, but this is

> expected to be the case only for a short period, as leia was planned to be

> replaced in the coming months.

>

> If, for some reason, you’ve hard-coded something to

> leia.netid.washington.edu, you will want to change that.

>

> Brian Arkills

> UW-IT, Identity and Access Management

> UW Windows Infrastructure technical lead

 

NETID domain schema changes 5/8/2013: Exchange 2010 SP3

What and When:

On Wednesday, May 8th, the NETID domain schema will be modified. Schema changes associated with Exchange 2010 SP3 will be applied.

 

What you need to do:

Nothing, this is purely informational. There is no expected impact to customers.

 

More info:

Please see http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=5401 for more information about the schema changes.

 

Please send email to help@uw.edu with “UWWI schema work” in the subject if you have any questions about this work.

 

Brian Arkills

UW-IT Identity and Access Management

UW Windows Infrastructure Service Manager

 

 

Spring quarter NETID domain controller work

Several changes are planned for the UWWI NETID domain service.

 

What and When:

Domain controller demotions and promotion work will span the Spring quarter. This work is intended to bring the NETID domain to WS2012 functional level.

 

Two existing domain controllers will be demoted, rebuilt with Windows Server 2012, then promoted at a later time.

 

The first demotion will happen next Friday, 4/19. The timing of the other events is unknown, but are expected at some point during Spring quarter.

 

After both of these existing domain controllers are demoted, a third domain controller will be demoted and retired.

 

At the conclusion of this work, we’ll change the domain/forest functional level to WS2012.

 

We do not plan to send other notices about the DC demotions and promotions associated with this work.

 

What you need to do:

If you have a firewall or network filters, you may need to adjust them if they only include specific IP addresses as opposed to the subnet guidance (see link below). All DC promotions are expected to be on networks already in the UWWI firewall guidance documentation.

 

If you’ve hard-coded specific domain controller names in applications or code, you will need to adjust that configuration. If you have hard-coded these domain controllers:

chewie.netid.washington.edu

han.netid.washington.edu,

yoda.netid.washington.edu

 

please change that configuration.

 

If neither of these situations apply to you, then you don’t need to do anything.

 

More info:

Chewie.netid.washington.edu will be demoted on 4/19/2013.

Han.netid.washington.edu will be demoted at some point after a rebuilt chewie is promoted.

Yoda.netid.washington.edu will be demoted at some point after a rebuilt han is promoted.

 

You can find the list of networks that correspond to NETID DCs to configure in your firewalls at http://www.netid.washington.edu/documentation/trustWithFirewall.aspx.

 

Please send email to help@uw.edu with “UWWI DC work” in the subject if you have any questions about this work.

 

Brian Arkills

UW-IT Identity and Access Management

UW Windows Infrastructure Service Manager

 

UWWI NETID domain service change and policy change: domain trusts

A significant change has happened to the NETID domain service.

 

What and When:

 

The NETID domain service has changed its service design and policy. Under specific conditions, the NETID domain will trust another Windows domain. The conditions required are significant.

 

What you need to do:

 

Nothing. One of the key conditions required is that selective authentication is employed on the NETID side of the trust. This means that unless you explicitly permit users/groups from a trusted domain the ‘allowed to authenticate’ permissions on your computer objects, you are completely unaffected.

 

If you have concerns or questions, please send an email to help@uw.edu with “UWWI trust policy change” in the subject.

 

More details:

 

This change is being made for multiple reasons:

  • to enable UW-IT to consolidate its Windows domains into the NETID domain without adversely affecting some of the most critical IT assets of the University
  • to permit a key initiative of the University’s enterprise data warehouse to move forward in providing data visualization capabilities
  • to allow other UW organizations in similar positions an easier way forward, without adversely affecting existing NETID domain customers

 

You can read about the conditions required in the trust section of the revised UWWI policy: http://www.netid.washington.edu/documentation/policy.aspx#trusts

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Technical Lead

2013 January

Here’s an update on recent happenings with the UW Windows Infrastructure.

 

==== New Capabilities and Improvements ====

 

* Work to replace aging NETID domain controllers has resulted in 3 new DCs. This work included applying the WS2012 schema and also partially addressed some geo-redundant disaster recovery goals by locating a domain controller out of the Puget Sound region. Work to refresh existing WS2008R2 DCs to WS2012 continues.

 

* Windows 8, Windows Server 2012, and Office 2013 license activation capabilities were added by replacing the campus KMS server.

 

* The mail attribute value for all UWWI user accounts was changed to <uwnetid>@uw.edu to facilitate Office 365 integration, eliminate user errors, and prevent multiple users from having the same email value.

 

* Work to refactor the UWWI Group Sync Agent to provide near real-time sync for all UW group changes has been completed and deployed. Notable improvements include:

                * Group Service latency to UWWI is significantly reduced

                * UWWI groups are reconciled with the Groups Service now, which self-corrects any errors on UWWI groups that might creep in

                * Course group changes are provisioned to UWWI in near-real time

 

====Spotlights====

 

* A majority of delegated OU customers have misconfigured their computers primary DNS suffix–with greater than 90% of all computers misconfigured. This problem subtly affects functionality, most notably reducing negotiated security levels. A separate announcement will include more details on this issue and plans to address it.

 

* A project to decommission the UW Forest by mid-February 2013 continues. All remaining domains are in the process of domain migrations either to a delegated OU or to a new Windows forest, and all are making good progress.

 

==== Trends ====

 

* Since June, UWWI has added: 10 delegated OUs (62 total), 1 trusts (54 total), ~1100 computers (5600 total), ~17k users (579k total).

* UWWI support requests remain steady. 119 UWWI support tickets resolved since June (vs. 122 in prior period).

* UWWI supports all the new types of institutional groups being piloted in the Groups Service: by degree level, class standing, curriculum, etc.

 

You can see metrics about UWWI at http://www.netid.washington.edu/dirinfo/stats.

 

==== What’s Next ====

 

Our objectives for the months ahead include:

 

* Continued support of the university-wide Business Continuity Initiative by creating geo-redundancy continuity plans for UWWI NETID domain services.

* Continued support of the Office 365 project and the UW Exchange service as it integrates the UWWI NETID domain services with an Office 365 deployment.

* Continue to investigate how Active Directory Federation Services (ADFS) integrates into our overall authentication architecture for customers.

* Invest in changes needed for Unix integration

* Support UW-IT effort to investigate SCCM 2012 delegation features to enable OU customers to deploy SCCM for computer management within the NETID domain.

 

==== Your Feedback ====

 

Supporting your needs for UWWI capabilities offered via the Basic Services Bundle is our priority, so we welcome feedback on how we can make the UWWI service more valuable to you.

 

The UWWI service has a backlog visible to customers at https://jira.cac.washington.edu/browse/UWWI where you can get more details about possible improvements, current prioritization of that work, and even what we’ve been doing. You can “vote” for items in the backlog to help us rank priorities, or you can contact us via iam-support@uw.edu.

 

RE: UWWI user mail attribute change

 

This change is beginning. We expect the change to take more than 24 hours, and possibly as much as 72 hours.

 

From: Brian Arkills Sent: Friday, November 09, 2012 12:28 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: UWWI user mail attribute change

 

A significant change is planned to the NETID domain service.

 

What and When:

 

During the break between fall quarter and winter quarter, UW-IT will be changing the mail attribute value to <uwnetid>@uw.edu for *all* UWWI users. For example, for NETID\barkills, the mail value will change from barkills@washington.edu to barkills@uw.edu.

 

The mail attribute provides a reliable email address for a UWWI user. For example, Exchange uses it in forming its off-line address book.

 

This change will likely span several days. After this change, the UW Exchange service will manage the mail value.

 

What you need to do:

 

Nothing, unless you have a service that leverages the UWWI user mail attribute. If you do, you may need to adjust your service accordingly. Note that users can still direct where email is delivered by changing their UW email forwarding settings, via the UW NetID Manage page.

 

If you have concerns or questions, please send an email to help@uw.edu with “UWWI mail attribute” in the subject.

 

More details:

 

This change is being made for multiple reasons:

  • to address lack of user input validation
  • to reduce the complexity around how the mail value is provisioned
  • to enforce uniqueness, preventing multiple users from having the same email address
  • to enable Office 365 integration with the UWWI NETID domain services; Office 365 has several rules that necessitate this change

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Technical Lead

UWWI user mail attribute change

A significant change is planned to the NETID domain service.

 

What and When:

 

During the break between fall quarter and winter quarter, UW-IT will be changing the mail attribute value to <uwnetid>@uw.edu for *all* UWWI users. For example, for NETID\barkills, the mail value will change from barkills@washington.edu to barkills@uw.edu.

 

The mail attribute provides a reliable email address for a UWWI user. For example, Exchange uses it in forming its off-line address book.

 

This change will likely span several days. After this change, the UW Exchange service will manage the mail value.

 

What you need to do:

 

Nothing, unless you have a service that leverages the UWWI user mail attribute. If you do, you may need to adjust your service accordingly. Note that users can still direct where email is delivered by changing their UW email forwarding settings, via the UW NetID Manage page.

 

If you have concerns or questions, please send an email to help@uw.edu with “UWWI mail attribute” in the subject.

 

More details:

 

This change is being made for multiple reasons:

  • to address lack of user input validation
  • to reduce the complexity around how the mail value is provisioned
  • to enforce uniqueness, preventing multiple users from having the same email address
  • to enable Office 365 integration with the UWWI NETID domain services; Office 365 has several rules that necessitate this change

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Technical Lead

NETID DC promotions and demotions

Several changes are planned for the UWWI NETID domain service. These changes will close the existing temporary service capacity gap, which was noted in the email below sent 2 weeks ago.

 

What and When:

Several new domain controllers on a network not previously in the UWWI firewall documentation will be promoted beginning this Thursday, 11/8. One DC will be promoted on 11/8, with 2 additional DCs following over the following week for a total of 3 new NETID domain controllers.

 

After these 3 new DCs have been added, 2 of the existing domain controllers will be demoted as they have reached end of life.

 

What you need to do:

If you have a firewall or network filters, you may need to adjust them (see link below).

 

If you’ve hard-coded specific domain controller names in applications or code, you will need to adjust that configuration. If you have hard-coded either mace.netid.washington.edu or yoda.netid.washington.edu, please change that configuration.

 

If neither of these situations apply to you, then you don’t need to do anything.

 

More info:

Another network has been added to the UWWI firewall list: 172.16.31.0/24 (172.16.31.0-172.16.31.255).

 

The network that leia.netid.washington.edu was on has been removed from the UWWI firewall list. For reference that network was: 172.22.14.0/27 (172.22.14.0-172.22.14.31).

 

You can find the list of networks that correspond to NETID DCs to configure in your firewalls at http://www.netid.washington.edu/documentation/trustWithFirewall.aspx.

 

When all DC demotions are complete, we will remove two of the networks listed in that document, so if you do manage network filters you may want to check back in 3-4 weeks to remove unnecessary networks in your filters in the future.

 

> —–Original Message—–

> From: Brian Arkills

> Sent: Tuesday, October 23, 2012 12:38 PM

> To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu)

> Subject: Netid domain controller (leia) forcible demotion planned ~1pm

> today

>

> Due to internal AD database corruption on Leia.netid.washington.edu and

> replication problems it was having, we have determined that we need to

> demote leia.netid.washington.edu, one of 5 existing domain controllers

> providing the NETID domain service in the UWWI service line. Leia has been

> offline since yesterday evening and won’t be coming back online. You may

> have experienced odd problems because of Leia’s AD corruption. If you think

> you’ve got a lingering issue caused by this, please do open a help request via

> help@uw.edu with UWWI somewhere in the subject line, and we’ll try to

> assist you.

>

> We have already replaced leia as the DNS primary for clients.uw.edu, the

> DDNS zone provided for delegated OU customers, and there has been no

> impact to customers of that service.

>

> This work represents a minor degradation in service capacity, but this is

> expected to be the case only for a short period, as leia was planned to be

> replaced in the coming months.

>

> If, for some reason, you’ve hard-coded something to

> leia.netid.washington.edu, you will want to change that.

>

> Brian Arkills

> UW-IT, Identity and Access Management

> UW Windows Infrastructure technical lead