Nebula Newsletter March 2015

Welcome to our first semi-annual Nebula service newsletter, which brings you valuable updates and information to help you make the most of our services. Because this is our initial newsletter, we are sending it to all Nebula users. We hope you’ll opt into receiving future issues by signing up for the nebula-announce mailing list, a low-volume mailing list we use to send customers notices about service interruptions and notable design changes, in addition to this newsletter. Go to https://itconnect-test.uw.edu/wares/nebula/contact-us/ for info on how to join that list.

==== New Capabilities and Improvements ====

* Nebula-discuss: A new two-way communication channel was created to encourage discussion of Nebula services: You can send email to it and so can the service team. The primary goal is to enable interaction about the service so that we can hear from you, and you can hear from each other. This will surface business needs, may expose recurring problems we have missed, and provides a way to ask questions about the service. This isn’t intended as a way to make requests.

To join the nebula-discuss mailing list, please see https://itconnect-test.uw.edu/wares/nebula/contact-us/

* Auto-email-responses: In January, we eliminated a confusing email response that said, “Your request has been resolved and will automatically close in 3 business days” when your request for help led to a consulting engagement.

* Billing changes: We changed our process for billing charges for Nebula desktops to simplify it and to make it more accurate. Nebula desktops moved from the recurring portion of your bill to the non-recurring portion of your January and February bill, as an unfortunate side-effect, but we expect to move Nebula desktops back to the recurring portion of your bill in the near future–likely the bill covering March.

* Customer Portal: We updated the MyNebula customer portal to be the MyIT customer portal. This portal provides reports and information about the computers and IT services you use, and provides departmental contacts more information about their department’s use. Several capability improvements have been added in the last six months, which are detailed in its change log.

====Spotlights====

* Customer meeting: A Nebula customer meeting is scheduled for Thursday, March 5, from noon until 1pm in the UW Tower auditorium. The agenda is to review the material in this newsletter in more detail and take any questions. We look forward to seeing you there.

* Self-help: Our support specialist Tobin Wood documented some common workarounds for when your Outlook profile gets corrupted. We plan to publish more of the workarounds in our internal documentation so you will be able to help yourself. We expect to increase the amount of self-help documentation to our Nebula customers.

* NETID user account conversion: We prioritized converting the Nebula2 user accounts to NETID user accounts for customers from UW-IT. This has helped improve our experience and highlight outstanding issues that need workarounds. We expect to leverage these lessons as we prioritize getting other customers converted.

Converting from Nebula2 user accounts to NETID user accounts reduces how often you have to log in, reduces our service costs, simplifies the infrastructure needed, and will enable Nebula to leverage investments made in the service providing the NETID user accounts.

In the next six months, we would like everyone to self-elect to change to NETID user accounts. At the end of that time frame, we’ll be phasing out Nebula2 user accounts. That means we will have to make the switch for you, if you haven’t done so already. We believe if you self-elect, the impact to you will be less.

If you’d like to volunteer your department (or just a single user) for conversion, please send us an email with “Nebula2 to Netid user conversion” in the subject line. There are self-service or assisted options (and we won’t charge extra for basic assistance). The self-service directions are at http:// www.uw.edu/itconnect/wares/nebula/news/netid-logins/.

* Core bundle changes: There are two anticipated changes to the Nebula Core bundle for FY 2016. These changes aren’t set yet, but this is the direction we are exploring at this time:

-We are in discussions to reduce the rate by a small but appropriate amount to reflect the cost of running the service. Our financial forecast suggests we can sustain a reduction, but we’re still running down some outstanding costs and preparing the financial model to support it.

-We plan to remove file services from the Nebula Core bundle for FY 2016 and instead charge for Nebula file services separately.

This would mean the Nebula Core desktop rate would be reduced by the average cost per desktop that Nebula pays to cover Nebula use of UW-IT provided file services. We’d then charge for file service use, i.e., the cost of how much file storage is used.

Today, some of you are subsidizing others’ file service use, while others are using a lot more. In the future, you’ll be paying for what you use.

We anticipate that there will be impacts and consequences of this change. Examples we imagine include:

-You may pay closer attention to which user accounts have Nebula service and file storage. A side benefit is that you will inform us more promptly when users should be removed from Nebula, as we have no good way of knowing this kind of information today.

-You may have a higher interest in no-cost file services such as Google Docs and OneDrive for Business, which encourages users to move their files in a strategically beneficial direction for the UW.

If you’d like to explore Google Docs or OneDrive for Business but need help getting started, let us know.

We will let you know more about these anticipated changes when final decisions have been made. If you’d like a rough idea of your expected monthly costs for file services, you can review this report in the MyIT portal: https://support.nebula.washington.edu/myIT/fileServices.aspx. If you aren’t listed as a departmental contact, this report won’t show you anything useful. If you’d like to get added as a departmental contact, let us know.

==== Trends ====

Below are statistics across the Nebula service. For information specific to you or your department, the MyIT portal has more data: https://support.nebula.washington.edu/myIT/Default.aspx.

* Usage stats. Since August 2014, Nebula has:

-Basic stats: -50 computers (~3400 total today), +150 users (~4900 total today), +150 groups (~3000 total today)

-IE browser: +250 IE11 (~2750 total today) and -100 IE10 (~300 total today)

-OS: +0 Windows 7 (~2900 total today), +125 Windows 8.1 (~350 total today), +0 MacOS (~22 total today)

-Nebula VPN use: +18 sessions on average (~18 sessions total average with a peak of 35)

Notes:

-VPN stats reflect unusual increase due to stats from one VPN server not previously recorded.

* Operational assistance stats

-Support requests have grown by 16.7%; 2451 Nebula support tickets resolved since 8/20/2014 (vs. 2100 in prior period).

Incidents have grown by 223%; 58 Nebula incidents resolved since 8/20/2014 (vs. 26 in prior period).

Notes:

-Prior to the past six-month period, incident reporting was optional, so there was likely quite a few more actual incidents than were recorded.

-We’ve changed operational tools during the last year, so the request and incident comparison for the prior period is suspect and we’ve had to make some data compromises. Next time this data and the comparisons should have more validity.

==== What’s Next ====

Our objectives for the next six months include:

* Activities related to FY 2016 core bundle change, as noted above.

* Activities related to the Nebula2 user transitions, as noted above.

* Make some design changes related to the Mac VPN so it isn’t a blocker for letting go of the Nebula2 user account.

* Publish more self-service documentation to enable you to help yourself. We hope this will help drive down our costs, so we can increase our improvement investments and/or further reduce the service cost.

* Support a UW-IT project team in enabling UW Connect to submit billing data for consulting requests. We anticipate that there may be some changes to your experience of budgets and billing for consulting requests.

* Replace the servers behind our aging software deployment infrastructure (System Center Configuration Manager or SCCM). We also will explore moving Nebula’s software deployment capabilities to the UW Windows Infrastructure service so a broader set of the UW can leverage this capability and contribute packages Nebula customers might use.

* Explore the unreleased Windows 10 operating system. In particular, in tandem with the above software deployment infrastructure refresh, explore how it might enable us to provide a self-service, in-place OS upgrade experience, and other options that would lower our delivery costs. We expect this will enable you to trigger your existing computer to get automatically upgraded without losing your user customizations. In general, our goal is to be prepared to support this new OS shortly after it is released.

* Via a pilot with some higher risk departments, explore a solution that provides data encryption capabilities regardless of where the data is stored, has broad cross-platform support and advanced tracking capabilities (Azure RMS). We suspect this is a strategically important technology for risk mitigation, but we need to verify.

==== Your Feedback ====

Supporting your needs for Managed Workstation capabilities offered via the Nebula service is our priority, so we welcome feedback on how we can make the Nebula service more valuable to you.

You can voice your support for future objectives to help us rank priorities, ask for things that aren’t yet on our radar, or simply contact us via help@uw.edu.

Brian Arkills

UW-IT, Nebula Service Manager

Nebula billing change

A small change is planned to how Nebula does billing.

What and When:

Beginning with the bill you receive in February, Nebula desktops will show up in the non-recurring portion of your bill.

What you need to do:

Nothing. J

Why we are making this change

We’ve made some changes to the way we internally report billing charges to both simplify the process and improve the accuracy. Moving where the Nebula desktop charges show up in your bill is an unfortunate side-effect that we anticipate we’ll be able to change back in the near future.

More info:

It’s possible that you’ll see a Nebula desktop (that has been a Nebula desktop) show up on your bill for the first time, because our previous process allowed a few to go uncharged. The good news is that those prior missed charges are our fault–you won’t be back billed for them.

If you have questions about this planned work, please send email to help@uw.edu with “Nebula billing change” in the subject line.

2015 January

Here’s our semi-annual newsletter update on recent happenings with the UW Windows Infrastructure.

==== New Capabilities and Improvements ====

* Unix, Linux, and Mac Integration with UWWI Active Directory. Many customers already join their Macs, and some join their Unix computers to the NETID domain. We removed obstacles to using PowerBroker Enterprise or Open in the NETID domain, and put documentation together to help guide customers who would like these benefits but don’t know how. See https://wiki.cac.washington.edu/x/nCwJB for more. Customers with tips are encouraged to share them via the community suggestion wiki page: https://wiki.cac.washington.edu/x/-jAJB.

* Domain based DFS capability is now available. This provides redundant distributed file redirection services, allowing you to easily add and remove file servers without impacting your customers. Several customers are already leveraging this capability. See https://wiki.cac.washington.edu/x/obv5Aw for more info. Note: we recently partnered with a customer to get DFS-R working. Our documentation will be updated to reflect this new possibility in the coming months.

* Reduced latency for Entra ID directory synchronization from 3 hours to 1 hour. This primarily benefits customers of the MSCA service, but also benefits those integrating applications with AAD and with the future release of Windows 10 should provide other benefits.

* Self-service SPNs for application UW NetIDs. This permits owners of an application UW NetID to register service principal name values on their own without assistance from the UWWI service. See https://wiki.cac.washington.edu/x/5CwJB for details. This new capability means that customers can manage SPNs on:

-Computers in their delegated OU

-Group Managed Service Accounts (gMSAs) in their delegated OU

-Application UW NetIDs they own

* Major upgrades and refactors:

– Geographic redundancy achieved for all business critical systems in the UWWI service.

– UWWI Group Sync Agent redundancy. We deployed a 2nd passive server with the UWWI Group Sync agent on an Azure VM via the UW-IT Standard Managed Server service. If you’d like to hear more about our experience with Azure VMs, let us know.

– All NETID DCs upgraded Windows Server 2012 R2. Forest and Domain functional level moved to Windows Server 2012 R2.

– UWWI Kiwi Agent version release pending. Admin and Application UW NetID behavior changes.

– WINS server replaced

====Spotlights====

* UWWI service staff had a significantly higher operational load over the past 6 months—historically, about double our usual number of requests in the same period of time.

* The ‘Bring Your Own Zone for DDNS’ work was cancelled, due to lack of customer interest given the constraints we inherit from the UW network design. Customers are highly encouraged to talk to the campus DNS service for needs they have which aren’t currently being met.

* Over the last several months we evaluated two new security capabilities Microsoft provided with Windows Server 2012 R2, Protected Users and Authentication Policies, for use at UW. Our evaluation showed they aren’t effective for the most common scenarios, especially for the most pressing need–protection against the Pass the Hash style attacks behind most of the credit card breach news stories over the past year. For our analysis, see https://wiki.cac.washington.edu/x/8zAJB. Instead, we plan to make the following security investments:

– For privileged user accounts, experiment within UW-IT with some alternate protections and share more broadly if these are effective with some kind of self-service opt-in mechanism,

– Reduce use of NTLMv2,

– Continue active work on reducing and mitigating existing LDAP simple bind logons (passwords sent in clear over the wire),

– We also believe Microsoft will bring some more significant protection capabilities in 2015, so we will watch developments closely

* NTLMv1. Brian Arkills has presented on our experience to other universities on a couple occasions. The latest presentation, given via a webcast that Internet2/InCommon provides, was recorded and can be viewed at: http://internet2.adobeconnect.com/p9kl8urgl67/. This requires installation of the Adobe Connect add-in.

* James Morris is an invaluable part of the UWWI service team. While the UWWI service only has a very small fraction of his time, we put that time to high use by leveraging his excellent design skills in the early parts of our planning and relying on him to provide backup coverage when one or more of the service team are out. James often foresees problems in design and architecture before anyone else, which enables us to improve the design before you see it. We appreciate his contribution and the deep engineering background he brings to our service team.

==== Trends ====

* Since January, UWWI has: +3 delegated OUs (94 total), -1 trusts (56 total), +~1000 computers (9694 total), +~16k users (704k total), -8k groups (89k total).

* UWWI support requests have grown by 85%!!! 347 UWWI support records resolved since July (vs. 188 in prior period).

You can see metrics about UWWI at http://www.netid.washington.edu/dirinfo/stats.

==== What’s Next ====

Our objectives for the 6 months ahead include:

* Continue exploration of deploying an AD-integrated Certificate Authority to enable a variety of multi-factor scenarios and easy internal website certificate renewal.

* Simple Bind Reporting/Notification to improve the security of UW NetIDs.

* Internal documentation refactor to improve our operational effectiveness.

* Minor Group Sync code fixes/improvements

* ADMT 3.2 upgrade

* AD snapshots to improve our ability to recover from unexpected AD incidents including possible AD corruption

* Internal HyperV upgrade with several VM migrations to reduce our operational costs

* UW firewall GPO template to provide customers with a simple way to leverage Windows Firewall

* UWWI security improvements, NTLMv2 explorations and privileged user risk mitigation

* Preferred Name (assuming this work moves forward as part of the HR/P project)

* Partner with Nebula to support new Software Deployment Service via SCCM deployment in NETID

* Support Authentication service in exploring Multi-factor Authentication solutions for Windows

* Support emerging Enterprise Monitoring Service by sharing Windows expertise

* Support the future Microsoft Campus Agreement goals by contributing to a 3-5 year Microsoft technology roadmap

Of the 8 forecasted objectives we listed in the last UWWI News, here’s a review on how they turned out:

7 were successfully completed
1 was started and continues: AD-integrated CA explorations

Note: Of the top 7 incomplete items from last summer’s UWWI customer survey, http://ontheroa.uservoice.com/forums/258239-uwwi, 6 are represented above (4 of the survey items have been marked complete and are no longer visible at the URL). Many of these require other services to prioritize work, and given their competing priorities, some of this work may not be able to move forward. For these initiatives that depend on others, our investment will reflect the priorities you’ve indicated to the extent we aren’t blocked. Should a dependency blockage extend too far or we don’t have confidence that there will be timely progress, we will consider the possibility of moving away from a dependency on a strategically positioned service to a tactical solution we deploy to meet your needs, but that’s an option we don’t yet need to exercise.

==== Your Feedback ====

Supporting your needs for UWWI capabilities offered via the Basic Services Bundle is our priority, so we welcome feedback on how we can make the UWWI service more valuable to you.

The UWWI service has a backlog or roadmap visible to customers at https://wiki.cac.washington.edu/display/UWWI/UWWI+Roadmap where you can see more details about current and some future work items.

You can voice your support for future objectives to help us rank priorities by voting in the survey, ask for things that aren’t yet on our radar, or simply contact us via help@uw.edu.

Brian Arkills

UW-IT, UWWI Service Manager

New mailing list & communication practices for Nebula

We’ll be making a couple changes to the way the Nebula service communicates with you.

This mailing list will continue to function as a 1 way communication channel for important announcements. This will continue to include things like changes to the service design, and other broadly useful communication. My goal is to keep communication to this mailing list to the minimum you might need to be aware of what’s going on with Nebula.

In the past, there have been a number of other Nebula related mailing lists. Some you may have heard about and others you probably didn’t, but at this point all those lists are dead. I’ve co-opted the membership of one of those prior lists (neb-tech) to initially seed the membership of a new mailing list called nebula-discuss (nebula-discuss@uw.edu). This new mailing list serves two functions:

It provides a 2 way communication channel. You can send email to it, and so can the service team. The primary goal here is enable interaction about the service so that we can hear from you, and you can hear from each other. This will surface business needs, it may expose recurring problems we have missed prioritizing, and it should provide you a way to ask questions about the service. This doesn’t mean you send email to this mailing list to get support or make requests.
It provides a second list for us to send much more detailed information about the service. Folks who are on this new nebula-discuss list will see a larger volume of emails than nebula-announce, and because of this expectation, we will send more detail about what we’re doing to the nebula-discuss list. I have a regular practice of sending a monthly operational update about the services I manage. That update includes details about operational trends we are seeing, incidents, a detailed list of planned changes in the upcoming month (higher impact changes would also go to nebula-announce—this is just a more complete list), and some information about what kinds of improvements we’re hoping to work on. I haven’t yet been able to send a monthly update for Nebula because we needed to get this mailing list in place, but I have one drafted that’s been ready to go for a week. J

The idea of a mailing list where anyone can send email can be a little jarring. The 100+ people on this list will see your emails to this list, and any of them might choose to respond. This mailing list is intended to help promote discussion about the service, but if the traffic gets out of hand, I may moderate traffic or decide this mailing list experiment failed. I don’t think that’ll happen, but it’s worth noting that things don’t always turn out how you hope. J

If you are interested in joining this nebula-discuss mailing list, you can find information about how to get on that list at the bottom of this email. And again, if you were on the neb-tech list, you are already on the nebula-discuss list, and you may want to get off it. I’ll be sending this same email to the nebula-discuss mailing list shortly, so you’ll know if you are already on it.

Finally, I’ll be sending a semi-annual newsletter about the Nebula service. This newsletter will recap where we’ve been in the past 6 months by highlighting new capabilities and improvements, spotlighting items of interest, and reporting on trends, but it’ll also forecast what we think is ahead in the next 6 months. The first newsletter will likely go out in late February, and likely will be sent to the nebula-announce mailing list (although I may change my mind and send it to all Nebula users).

Brian Arkills

Nebula Service Manager

Nebula Fall Brown Bag

Thanks to those who joined us for the Nebula Fall Brown Bag! Brian Arkills was there; Brian is the new Service Manager, responsible for managing the service and its quality (Rebecca still manages the team that delivers the service). Brian talked about:

Better communications to customers for more timely information, including semi-annual newsletters.
Migration to NETID logins — this is needed to deploy better back-end tools. It will also allow for multi-factor authentication for those clients that need more security. We are happy to help, no charge; let us know when you are ready.
Windows 10 is coming – we are participating in early adopter program .

Tobin Wood talked about the latest “ransomware” incidents, where a malicious program encrypts files and then demands money for the encryption key. This has affected H: and I: drive files and prevents anyone from accessing them. You can get infected by clicking on attachments, or by going to an infected web site, aka “drive-by”.

Contact us asap if you suspect an infection.
If your area has an infection, copy your files to your desktop before working on them.
The fix: recover and restore encrypted folders from snapshots — this overwrites everything in the existing folder.

File service quotas are scheduled for July 1, 2015. We’re contacting H: drive users first; moving files to Google Drive or the U: Drive are good options. Please examine what you’re storing on H: and I: and review the guidelines at the bottom of the “Using Your Computer” page. Backups, non-work related media files, programs and archived files are all good candidates for removal.

Questions — our favorite part of the meeting!

What happened to Integratum, the web access tool?
It is old technology and incompatible with newer security protocols. Alternatives include Google Drive, Microsoft One Drive, and connecting to a work computer via Remote Desktop.
When should I use the VPN?
When you’re on a public network like a coffeeshop, library or airport, we recommend you connect via the VPN to encrypt your network traffic.
How can I make my login default to NETID?
We can do that; just send us a note.
Why isn’t Adobe Reader on the new images?
Adobe’s latest licensing does not allow us to distribute it via an image. The link you see in a newly imaged computer is to download the Reader software.
The old Nebula Support Request icon keeps coming back. (And why do we have a new NSR?)
We are working on that; send us a note if you keep seeing it. The NSR is handy when you’re having email problems; it sends us a trouble report.
Can deletions from my I: drive be restored?
Only from snapshots.
How do we ensure that emails are deleted?
Look into the Recover Deleted Items information — there’s another Delete option there.

RE: Turning off NTLMv1 on the NETID domain controllers

This change is complete.

Turning off NTLMv1 on the NETID domain controllers

A high-impact service change is planned for the UWWI NETID domain service. We will send an email tomorrow after this change is complete.

What and When:

On Tuesday, August 12 (8/12/2014) at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

What you need to do:

On 8/1/2013, we made this change and rolled it back because of a large unexpected impact. We’ve done a lot of work to help everyone be ready this year, but we still expect this change to not be smooth. It’ll likely be worst the first 4 hours after the change as folks who didn’t prepare discover how to apply the known workarounds, but we also expect that there will be isolated users who discover problems perhaps as much as months later when they finally try to access that service they only infrequently need.

We don’t plan to roll back this change. The cause of problems is primarily outside of our hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting that doesn’t allow NTLMv2.

If you find yourself in need of help tomorrow, don’t email me. Seriously—I’ll be at ground zero and will have little attention for emails in my inbox. Send email to help@uw.edu with “NTLMv1” somewhere in the subject line. Including info about what service the client is trying to connect to will be really useful. There are many folks in the UW-IT Service Center who are familiar with the known problems and workarounds, and know all the resources I’ve sent all of you over the past many months.

If you want to avoid getting help as much as possible, here’s what I’d do if I it was me:

Determine the client and service involved in the problem.
Review the NTLMv1 Known Problems and Workarounds to see if the details from #1 lead to a known workaround: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds
1. If Windows client, refer them to https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299. If domain joined, then adjust the group policy setting: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. Level 3 (“Send NTLMv2 response only”) is the minimum needed to continue to interact with the NETID DCs. We recommend level 5 (“Send NTLMv2 response only. Refuse LM & NTLM.”).
2. If web-based, then:
  1. Have the client connect to https://rivan.netid.washington.edu to see if it can do NTLMv2 or Kerberos. Rivan has been configured to only allow NTLMv1 or Kerberos. After the change, they will be able to use NETID\<theirUWNetIDhere> to test this (before the change, NTLMv1 is still allowed, so it isn’t useful until after the change).
  2. If they can do NTLMv2/Kerberos, then you know the problem is with the web service’s configuration. Contact the web service owners.
  3. If they can’t, then refer to the known problems/workarounds for a client workaround. If there isn’t a client workaround, then contact the web service owners to let them know you’d like them to apply one of the service-side workarounds.
3. If no known problem/workaround is listed, then you will need to contact UW-IT at help@uw.edu (with “NTLMv1”).

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

RE: Turning off NTLMv1 on the NETID domain controllers

Hi folks,

I thought I’d depart from our usual format on this change reminder.

The ‘NTLMv1 is turned off’ change is happening in 13 days. We’ve been sending user notifications based on a really small slice of all the possible sources of NTLMv1 logons. Based on that really small slice of the overall picture, usage is down, but we don’t have a comprehensive picture of what the impact will be. Put simply, the NTLMv1 logon events are logged on your servers, not on the NETID DCs.

If you’ve already taken action and applied the workarounds, we thank you immensely.

But if that isn’t the case, then you really should take a look because on August 12^th some set of your users will likely have problems. I don’t want to be the guy telling you on 8/12 that you should have paid attention to the warnings we sent, and that now in a very short period of time, you’ll need to become an expert on NTLMv1 workarounds and apply them to all the various computers you support that are having problems.

If you aren’t sure, send in an email to help@uw.edu and we’ll help you reach more certainty. If you want help looking at your server logs, we can do that—we’ll even send user notifications, if you want that.

If you don’t want help, and haven’t yet done anything, here are the things I’d recommend now:

Read https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds. It’s your key to fixing things up either proactively or reactively. It links to all the resources we know about or have created.
If you run your own Windows servers, you really need to run the powershell script in workaround J on those servers. It’ll give you a list of which client computers are misconfigured and the users (which is helpful if you don’t recognize the computer name). It’s really quite easy to use, and even if you don’t know powershell, we can help you run this with very little effort.
If you run a Windows domain that trusts NETID, then a highly effective, low cost action to take is to set a group policy setting in your domain root that sets the LMCompatabilityLevel to 3 at least (5 would be ideal). See workaround C for more on that. This (level 3) will allow your domain-joined Windows computers to send NTLMv2 (level 5 will require NTLMv2).
If you are in the situation where you run an IIS web server with Windows Integrated authentication, then you really need to consult the workarounds associated with problem #9 and take action. Almost all non-Windows web clients are not able to do NTLMv2, so you will be in a very awkward position without proactive action.

The workarounds page has changed quite a bit in the last 6 weeks, and we thank folks like Armand Bularoro for sharing workarounds they discovered. If you have something we haven’t covered there, we’d really like to capture it to help folks who are caught unprepared on 8/12.

We plan to remind folks again next week and the day before the change. There are also a couple more sets of user notifications we’ll send out. If you share the output of the powershell script from your servers with us, you can leverage our user notification process. That user notification process has been *really* effective (in 5 weeks we went from ~800 users to ~100), and most of the thanks on that goes the excellent folks in the UW-IT Service Center who help users walk through what they need to do.

The change will be on Tuesday, 8/12 at 10am.

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

From: Brian Arkills Sent: Tuesday, July 8, 2014 12:19 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

This is an update for the high-impact service change.

The date for this change has been moved.

What and When:

On Tuesday, August 12 (8/12/2014) at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

More Info:

We have moved the change date back to give customers operating web servers that rely on Windows Integrated authentication and the NETID domain more time to make changes to address the non-Windows browser issues we’ve noted in previous weeks.
The Known Problems/Workarounds document has had several modifications over the last couple weeks, most notably adding addition options for web servers currently using Windows Integrated.
We apologize for falling behind in sharing our log details and user notification lists. The Known NTLMv1 Logons page should be up to date with all the log analysis and user notification lists, and will remain up to date.
We are modifying our user notification schedule to reflect the new change date. The new user notification dates are: 7/8 (already went out), 7/15, 7/22, 7/29, 8/5, and 8/11.
Week to week comparisons based on our logs indicate the user notifications are effective:

	events	users	computers
6/17-6/23	420944	857	928
6/24-7/1	35444	251	275
7/1-7/7	19647	203	198

From: Brian Arkills Sent: Tuesday, July 1, 2014 2:36 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

This is an update for the high-impact service change in 2 weeks.

More Info:

NTLMv1 use is significantly down in the server log files available to the UWWI service team. Last week’s logs suggest that as many as 75% of the misconfigured computers we were seeing a month ago have now been fixed. The UWWI service does not have access to your log files. Only you can check whether your users will be affected. See the original announcement below for resources to help you do that.
An update on the non-Windows browser known problem we mentioned last week: Making a change to an IIS web server which is configured to use Windows Integrated authentication may be a workaround to consider. We’ve added removing Integrated Windows authentication and adding Basic authentication (with SSL required) as a workaround to our documentation. The Dynamics AX service plans to apply this workaround. If you do have a IIS web server with Windows Integrated enabled, you should check your logs for NTLMv1 use.
We continue to email user notifications to those users that are in the log files we have access to. We sent a round of notifications today to 250 users. We plan to send additional user notifications on: 7/8, 7/14, and 7/15.

From: Brian Arkills Sent: Tuesday, June 24, 2014 2:16 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

This is an update for the high-impact service change in 3 weeks.

More Info:

We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
The 1^st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

This is an update for this high-impact service change.

A date for the change has been set:

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

There are several things we’d like IT support staff to do:

Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager

2014 July

Here’s our semi-annual newsletter update on recent happenings with the UW Windows Infrastructure.

==== New Capabilities and Improvements ====

* Group Managed Service Accounts are available to Delegated OU customers. This provides a self-service, higher-security option for non-interactive applications, services, and scheduled tasks that run automatically but need a security credential. See http://www.netid.washington.edu/documentation/groupManagedServiceAccounts.aspx.

* Kerberos delegation sensitivity enforced. Protections for certain types of UW NetIDs from applications that use this “logon on behalf of” capability. You can waive those protections for a given NETID user if you need to—just contact us.

* Major integration component refactors:

-UWWI Person Data Agent refactor. Upgraded from Microsoft Identity Lifecycle Manager to Microsoft Forefront Identity Manager. Revised data sources. Added name override source. Simplified.

-UWWI Group Sync Agent refactor. Upgraded from unsupported ActiveMQ technology to Amazon Message Bus.

-UWWI Kiwi Agent version release. NETID user deletion behavior revised.

====Spotlights====

* We’d like to ask all customers to provide input on what you’d like to see us invest our continual service improvement time in. Toward that end, we’ve created a survey in UserVoice where you have 5 votes to cast on topics which you would like us to prioritize. We’ve seeded the topic list with 17 ideas, but you can also create new topic ideas. We’ll keep the survey open until the end of August. https://ontheroa.uservoice.com/forums/258239-uwwi

* NTLMv1 efforts. During Winter quarter, we analyzed NTLMv1 authentication afresh with the benefit of knowing which applications had problems during last summer’s failed attempt. During Spring quarter, we generated a comprehensive set of resources to help others identify and turn off their dependency on NTLMv1. This summer, we plan to turn off NTLMv1. We’ve been publishing our log data, and directly contacting users we know to be using NTLMv1. Because of the way NTLM works, removing NTLMv1 is a community effort. Many of you have worked hard on this, and you all deserve the university’s thanks for helping do your part to help clean up this old, insecure authentication protocol. Thanks!! J

* Early in Winter quarter, we transitioned the “private” view of the netid.washington.edu DNS zone from campus DNS to the NETID domain controllers. We did this primarily to improve our operational and business continuity stance: with this change we can demote/promote domain controllers without external assistance, with vastly reduced latency. We believe this will be invaluable for changes such as the upcoming domain controller upgrades. Some non-Windows LDAP clients experienced unexpected problems due to this change and there is a known workaround.

* Eric Kool-Brown joined us two years ago, and has become an invaluable part of the UWWI service team. He’s responsible for all the work on two of the major refactors noted above, and has provided leadership with ADFS. Anyone who has interacted with Eric knows he will leave no stone unturned in his quest to provide a quality outcome. We appreciate his contribution and the deep development and engineering background he brings to our service team.

==== Trends ====

* Since January, UWWI has added: 2 delegated OUs (91 total), 0 trusts (57 total), ~1000 computers (8703 total), ~50k users (688k total), -5k groups (97k total).

* UWWI support requests have grown by 7%. 188 UWWI support tickets resolved since January (vs. 176 in prior period).

You can see metrics about UWWI at http://www.netid.washington.edu/dirinfo/stats.

==== What’s Next ====

NOTE: This time around, we’re only forecasting for the summer quarter, instead of the next 6 months. Your input on the survey will change what we prioritize Fall quarter.

Our objectives for the 3 months ahead include:

* Turn off NTLMv1 on NETID domain controllers. Assist anyone that needs it.

* Upgrade NETID DCs to Windows Server 2012 R2. Announcements about timing coming soon.

* Move the UWWI Group Sync Agent to an active-active architecture, deploying a second agent on an Azure VM, to improve our business continuity availability characteristic.

* Replace Secondary WINS server

* Analyze survey results, summarize, and make future backlog prioritization based on results.

* Evaluate the new Protected Users group and Authentication Policy Silo capabilities for their appropriateness to university use cases and known security gaps.

* Continue exploration of the feasibility of deploying an AD-integrated Certificate Authority.

* Internal operational improvements: SCVMM refresh, some performance counter collection and other metrics improvements, additional server capacity

Of the 10 forecasted objectives we listed in the last UWWI News, here’s a review on how they turned out:

3 were successfully completed: UWWI Group Sync Agent refactor, gMSA release, and ILM replacement.
3 were started and continue: UWWI Group Sync Agent has active-active architecture, Protected Users/Authentication Policy Silo, AD-integrated CA explorations
2 were deferred: Enable dynamic access control (customer interest?), audit log retention/reporting (waiting to align with emerging monitoring service)
1 was externally blocked: Azure project team partnership. This project came to an end, not making as much progress as we hoped. However as an example of the success of that project, 2 weeks ago, UWWI requested the first hybrid VM via the Standard Managed Server service. At this time, UWWI doesn’t have plans to have an Azure VM NETID domain controller, but that may change in the future.
1 is ‘will not pursue’: Add new AD site in Spokane (UW network design made pursuing this prohibitive).

==== Your Feedback ====

Supporting your needs for UWWI capabilities offered via the Basic Services Bundle is our priority, so we welcome feedback on how we can make the UWWI service more valuable to you.

The UWWI service has a backlog or roadmap visible to customers at https://wiki.cac.washington.edu/display/UWWI/UWWI+Roadmap where you can see more details about current and some future work items.

You can voice your support for future objectives to help us rank priorities by voting in the survey, ask for things that aren’t yet on our radar, or simply contact us via help@uw.edu.

Brian Arkills

UW-IT, UWWI Service Manager

NETID DC Demotions, Upgrades, and Promotions

Several changes are planned for the NETID domain service.

What and When:

The week of July 21, 2014 (7/21/2014), each NETID domain controller will be demoted, upgraded to Windows Server 2012 R2, then promoted. Only one domain controller will be affected any given day.

7/21: bane.netid.washington.edu

7/22: vader.netid.washington.edu

7/23: maul.netid.washington.edu

7/24: sidious.netid.washington.edu

7/25: tyranus.netid.washington.edu

What you need to do:

If you’ve hard-coded specific domain controller names in applications or code, you will need to adjust that configuration, otherwise you don’t need to do anything.

More info:

No networks are changing—the domain controllers will remain at the same IP addresses.

The domain controller with the FSMO roles will move from vader to tyranus from 7/22 to 7/25. On 7/25, all FSMO roles will return to vader. The active UWWI kiwi client will also transition with exactly the same details.

If you have questions or concerns, please contact us by sending email to help@uw.edu with “UWWI” somewhere in the subject line.

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager