MWS trends for October 2017

Below are graphical metrics since the last newsletter. A takeaway statement follows each graph to summarize the critical points of data.

Client Operating System Counts

Takeaways: -200 Total, +700 Windows 10 (2240 today), -260 Windows 8.1 (0 today), -800 Windows 7 (870 today). Overall losses were due to cleanup activities that happened during domain migration where many unused systems were retired. A big shift to Windows 10 continues. In future newsletters, we’ll differentiate between versions of Windows 10.

Support Requests and Incidents

Takeaways: Support requests and incidents are generally down about 15%. Spikes in both requests and incidents correspond to the significant workstation migration change activity.

Managed Workstation Newsletter (October 2017)

What’s new?

Customer survey – In the next month, Managed Workstation will send out a survey for feedback on our service. We are looking for ways to improve our service and would like to know what experiences you have had with our team.

Keeping current – Microsoft continues to march forward with its product support lifecycle, and we continue to innovate to provide great management capabilities. We keep you current with Windows and Office with a minimum of hassle. Ideally, you don’t need to keep track of what version you are running and whether it is supported or insecure. If you do like to know those details, we’ve updated our OS lifecycle support document to better reflect what can be supported. Here are a few key upcoming dates:

Windows 10 1607 will no longer be supported in 3/2018.
Windows 7 SP1 will no longer be supported in 5/2018.

We’ll automatically upgrade any managed workstation still running an unsupported version to the current baseline Windows OS. Alternatively, you can control when an upgrade happens by using an icon on your desktop to upgrade. We recommend starting any such upgrade at the end of the day.

Security improvements – Security improvements don’t get a lot of attention because often they silently protect you, but every once in awhile it’s worth calling them out.

New FY18 Rates and service boundary changes –

The Managed Workstation service boundaries have changed. We still provide the same core capabilities, but we are providing a few more things under the monthly workstation rate.

Requesting help streamlined – A new document outlines how to get help, and includes links to a variety of forms for common requests. These forms collect the minimum information needed to fulfill your request, so save everyone time.

Spotlights

Chris Fairfield

What does a day in your life at work look like? My day consists of helping Managed Workstation customers with their computer issues and needs. Most days I work from my desk, but frequently I’ll need to head out across campus to help people in-person.

What particular skills or talents are most essential to be effective in your job? The two main skills required to be effective in my job are communication and problem solving. The former is how we get the information we need to utilize the latter, and problem solving is the largest part of our job.

What do you like about working at the University of Washington? I like working for a large, diverse organization that offers plenty of opportunities for growth and advancement—both personally and professionally.

What are some of your hobbies? I’m currently running a couple of Dungeons and Dragons campaigns. I also enjoy film-making and escape rooms

What is a quote or saying you live by? “This above all, to thine own self be true.”

What is your favorite TV show? It’s impossible to pick just one, but let’s go with the criminally underrated Crazy Ex-Girlfriend

What are some of your favorite foods? The biscuits at Morsel. Sushi. A good Chicago Dog.

Staffing Changes – Tobin Wood’s last day with Managed Workstation was in September 2017.

Rebecca Galloway’s last day with Managed Workstation was in October 2017.

We are sad to see them move on. We will attempt to replace these invaluable staff, and in the meantime we appreciate your understanding as we try to do the same amount of work with fewer people.

Computers moving between domains – We recently moved all MWS workstations from NEBULA2 to the NETID domain. Our original plan didn’t work out, with several unforeseeable complications due to configuration drift in managed workstations. We are extremely grateful for your partnership in getting workstations migrated to the NETID domain. This work reduces the overall cost of providing the service and enables us to leverage improvements that are funded centrally.

Manage Workstation Training Room – The training room has recently undergone an equipment refresh. In addition to the equipment refresh the training room will be available to any UW department, and the rate is $60 per hour. You can learn more about the training room changes at https://itconnect-test.uw.edu/wares/mws/training-room/

What’s next

Our objectives for the next six months include:

Password Manager: We have purchased LastPass Enterprise licensing for Managed Workstation users. Using this software can improve your password management practices. We’ll share more when we’re ready to roll it out.
Note: email communication to every user will be part of the implementation.
Local administrator management: Our approach to local administrator privilege management will shift to leverage the Group’s Service to provide:
- A single place to determine local administrator privs for managed workstations
- More transparency
- Self-service delegation; you’ll be able to add local admins
Windows File Services: We’ll update the server providing these services, transitioning to an offering that can handle confidential data with the ability to encrypt data at-rest by default.
Hiring new staff to fill our existing large gap in staffing. This includes a computer specialist and a software engineer.

Trends

You can review key metrics since the last newsletter.

Your Feedback

Supporting your needs for Managed Workstation capabilities is our priority, so we welcome feedback on how we can make the Managed Workstation service more valuable to you. The mws-announce and mws-discuss mailing lists are good sources of information. We recommend that each customer have at least one individual join the mws-announce mailing list.

You can voice your support for future objectives to help us rank priorities, ask for things that aren’t yet on our radar, or simply contact us via help@uw.edu.

2017 October: Ask the MI team

Here’s a slide deck for an event hosted by the Microsoft Technology community.

askTheMiTeam

Managed Workstation service boundary changes

Service boundaries have changed slightly. See below for details.

What and When

Beginning with adoption of the new FY18 rates (10/1/2017), the Managed Workstation service boundaries have changed. We still provide the same core capabilities, but down in the details, there are a few changes.

MWS rate changes:

Now includes 1 hour of complimentary hardware recommendation advice for each customer account. We’ll help find an appropriate computer model that meets your needs.
Now includes 1 hour of complimentary business needs IT consulting for each customer account. In other words, we’ll sit down with you to listen to the kinds of unmet needs you experience within your department. Based on listening to you, we’ll provide some recommendations about IT solutions that may help. We will use these engagements to discover additional capabilities this service needs to provide.
Now includes re-imaging a managed workstation that has been compromised with the current standard image.
Note: If data transfer, additional application installation, special configuration, or travel to your site is required, that continues to be billable via the MWS Consulting rate.
Application packaging in most scenarios is now included.
- If your application is appropriate for distribution to everyone at the UW, we’ll create the package for you at no additional cost.
- In all other cases, we’ll provide the 1^st hour of package creation at no additional cost. For most applications, this is sufficient.

Note: all application packages are subject to our documented application package support practices. If a released application package falls out of support, it will need to be refreshed or removed. Refreshing a previous application package is treated like a new application package.

MWS Consulting rate changes:

This rate can now be used to get assistance with computers which are not Managed Workstations. To qualify, these computers must run an operating system within our support boundaries (i.e. emerging, baseline, or containment). This assistance has a minimum charge of 30 minutes.
We’ve clarified our printer support boundaries. We will assist customers who need assistance setting up printer drivers or making a connection to a working printer. We do not provide assistance setting up printers or troubleshooting printers; we recommend UW Print Services for that.
We do not provide group management, nor are the subject matter experts on that topic, so do not provide this assistance; we recommend you work with the Groups Service.
Note: this isn’t a change, but given past practices we believe it is worth restating.
We do not provide computer hardware repair, but we are willing to broker and facilitate repair services with your hardware vendor on your behalf.
We do not provide request fulfillment for IT services that are outside the Managed Workstation service or other services in the UW-IT portfolio. For example, we will not help you manage email services that are not provided by UW-IT. Other UW-IT consulting services may provide this type of assistance.

MWS File services rate changes:

Assistance troubleshooting access issues to your shared files within MWS file services is now included. Previously, this required the MWS Consulting rate.
Providing recommendations on permissions to your shared directory within MWS file services

More Info

You can review a detailed description of service boundaries at https://it.uw.edu/wares/mws/design/what-does-the-managed-workstation-rate-include/.

Brian Arkills

Managed Workstation service owner

MWS training room

As of 10/1/2018, the Managed Workstation training room in Roosevelt Commons is no longer available.

How we provide the Managed Workstation service (MWS) training room has changed. See below for details.

What and When

During the first half of 2017, we refreshed the technology in the training room. There are photos of the space on the training room page. It’s a great option for any session that has 19 or fewer participants which each need a computer.

Using the MWS training room still requires a reservation—there is now a form which captures the needed details to streamline the reservation process.

Use of the room is open to any UW department and costs $60/hour. In the past, we bundled the costs associated with the training room into the MWS rate, but University practices have mandated we separate out this cost to ensure fair access and use. We do include ½ hour of our assistance per reservation; if more assistance is needed, we can provide that at the MWS consulting rate.

There are a variety of reservation practices which we have documented at https://it.uw.edu/wares/mws/training-room/#scheduling.

More info

You can find out more info about the training room at https://it.uw.edu/wares/mws/training-room/. There is also a link there to other technology spaces at the UW.

Brian Arkills

Managed Workstation service owner

SMBv1 disabled on NETID domain controllers

We disabled SMBv1 on all domain controllers for the NETID Windows domain.

Over the past couple months, we reached out to customers with resolvable client hostnames which were using SMBv1 to connect to the NETID domain controllers to let them know about this planned change. For somewhat obvious reasons, we didn’t want to publicize that we were still supporting a vulnerable protocol, which is why this change notification is happening after the fact.

What and When

The SMBv1 protocol was disabled on all NETID domain controllers on September 15, 2017.

As you are hopefully aware, the SMBv1 protocol has numerous security issues and vulnerabilities that have been exploited, making news headlines around the world. Microsoft and others have been recommending that SMBv1 be turned off, as it cannot be adequately patched or protected. For more info see below.

What you need to do

We were not able to contact less than a dozen customers which had unresolvable hostnames such as those handed out via DHCP. Those customers may need to update or reconfigure their computers to stop using SMBv1, and to use SMBv2 or SMBv3. How that is done will vary based on the operating system, application, etc., so you may need to contact your vendor(s) for assistance.

For Microsoft Windows clients, https://support.microsoft.com/en-gb/help/2696547/how-to-enable-and-disable-smbv1-smbv2-and-smbv3-in-windows-and-windows provides assistance on disabling and enabling SMB version support. There is a list of known non-Microsoft products which require SMBv1 here: https://blogs.technet.microsoft.com/filecab/2017/06/01/smb1-product-clearinghouse/.

More Info

Microsoft stopped supporting SMBv1 with the demise of Windows Server 2003 in 2015.

Significant security patches for SMBv1 were released in September 2016. And since September 2016, Microsoft has publicly been encouraging everyone to turn off SMBv1. As explained in that post, SMBv1 has serious design flaws that are 20 years old. These flaws most significantly allow person-in-the-middle attacks, but also permit data inspection, and significant performance degradation leading to denial of service attacks.

Ransomware that circulated earlier this year in high volume leveraged SMBv1 vulnerabilities. Those vulnerabilities were patched in March 2017 and were applied to domain controllers.

More recently, a significant denial of service attack, called SMBLoris, has been identified and affects all computer supporting the SMBv1 protocol. Microsoft has declined to patch current OSes for this attack.

In summary, SMBv1 is insecure and as the Microsoft PM responsible for SMB says “SMB1 isn’t good.” We could not in good conscience continue to run it on the NETID domain controllers. We encourage anyone at the UW who is still running it to prioritize retiring it.

Please note that to protect yourself from server operators who have not yet chosen to disable it, you should disable it on your clients.

Brian Arkills
Microsoft Infrastructure service manager
UW-IT

MWS security improvements

Security improvements don’t get a lot of attention because often they silently protect you. And in many cases we silently implement security improvements. But every once in a while, it is worth recognizing these security improvements, even if you don’t see them.

Here’s a recap of some of the recent security improvements MWS has made:

Microsoft’s Local Administrator Password Solution (LAPS) has been implemented for Managed Workstations. This solution provides and manages a complex unique password for the built-in administrator account on each managed workstation to reduce the exposure from a single managed workstation being compromised. Many years ago, the Managed Workstation service had a compromise that affected hundreds (Coreflood). This solution would have prevented that.
Based on a data-driven request from the Office of Chief Information Security Officer, we recently added a firewall rule to protect managed workstations from off-campus access of Remote Desktop. Customers should first connect to the VPN before using Remote Desktop to their managed workstation. This improvement protects all UW accounts from compromise and protects your managed workstation.
SMBv1, an insecure protocol, was disabled across all managed workstations in April, and disabled on domain controllers last month. This helps to protect interception of your data.
Upgrades of the MWS file servers this year were prompted by a variety of vulnerabilities in their software. This also helps to protect interception of your data.
Annually, 1-2% of managed workstations have some kind of compromise, and we’ll continue to invest in security improvements to drive that down further. But when it does happen, we’ve eliminated reimaging costs to help get you back to working on a safe computer.

Managed Workstation FY18 rates

The preliminary FY18 rates for Managed Workstation services are now available. The annual rate change for UW-IT services was postponed this year to be effective 9/1/2017. Management Accounting & Analysis (MAA) provides final approval of rates for all cost-recovery centers at the UW, and these preliminary rates are still pending their approval.

All rates associated with the Managed Workstation service are documented at: https://it.uw.edu/service/managed-workstation-services/#Price.

Rates

The new FY18 rates are:

Managed Workstation rate: $30.00/workstation/month
Managed Workstation file storage: $.25/GB/month
Consulting Services: $104.43/hour
Managed Workstation training room: $60/hour

Changes

In conjunction with these rate changes, there are some changes to the services provided. These changes include:

Managed Workstation training room use is a separate rate, not included in the Managed Workstation rate. This change was mandated by University practices to ensure fair access and use.
There is a single file service rate, regardless of the underlying platform used.
Support assistance for MWS file services is now included in that rate.
There are a few previously defined consulting activities that we will now provide as part of the Managed Workstation rate. Notable examples include:
- 1 hour of complimentary business needs IT consulting per year per customer account
- 1 hour of complimentary computer hardware recommendation consulting per year per customer account
- OS imaging for compromised managed workstation

Requests for the complimentary consulting will result in an in-person visit to discuss your needs.

More details about these changes are available at https://it.uw.edu/wares/mws/design/what-does-the-managed-workstation-rate-include/.

If you have any questions or concerns, please contact Dawn Cullerton or myself via help@uw.edu with a subject line of “FY18 MWS rate questions”.

Brian Arkills
Managed Workstation service owner
UW-IT

Migration update

99.9% of all managed workstations have been migrated to the NETID domain, and today we’ll be removing the 5 remaining workstations still in the Nebula2 domain. We greatly appreciate your partnership in both the user and computer migrations over the past couple years. Your patience while we completed this large undertaking has been amazing.

I’m also happy to report that new computers can be enabled for Managed Workstation services without the Nebula2 domain. https://it.uw.edu/wares/mws/hardware/adding-computer/ documents the steps needed to do that—it’s available today. In a nutshell, you now can “claim” your computer before joining it to the domain.

If you join a computer to the NETID domain without first completing that step, you’ll end up with a workstation that isn’t usable until someone on the NETID domain service team has intervened (send an email to help@uw.edu if you end up in this situation). This behavior is part of the design of the NETID domain service, and is a consequence of sharing the NETID domain with hundreds of other UW organizations. We are working on a more streamlined way to enable the workflow, leveraging the Managed Workstation imaging process. For resource constraint reasons, that work has had to wait while we focused on workstation migrations, but I’m hopeful we’ll have an even better option in the near future.

Now that the users and computers are in the NETID domain, there will be some minor planned outages over the next month to complete migration of remaining infrastructure. We’ll also start adopting the optional capabilities the NETID domain service provides, which was part of the value proposition behind undertaking this migration. An early candidate is the Local Admin Password Solution (LAPS) feature (https://it.uw.edu/wares/msinf/ous/laps/), which will reduce the risk to all Managed Workstation customers from a single computer being compromised. We’ll share more about that when we’re ready to release it.

Brian Arkills
Managed Workstation service owner

NETID domain controller upgrades: 8/2 – 8/28/2017

Several changes are planned for the NETID domain service.

What:
All NETID domain controllers (DCs) will be replaced with new servers running Windows Server 2016. An additional design change will happen during this process of relying on InCommon CA issued certificates for LDAPS access, replacing the existing design that leverages UW CA issued certificates.

When:
8/2/2017: First new WS2016 DC promoted
8/7 – 8/9/2017: 4 new WS2016 DCs promoted and 4 existing WS2012R2 DCs demoted, handled in a +1 new DC, -1 old DC fashion
8/28/2017: Last WS2012R2 DC demoted

What you need to do:
If you have an application or code which relies on the NETID domain service, you may need to adjust its configuration.

Known problems which your application may have include:
-if it does not automatically use the Microsoft DC locator process, but instead hard-codes domain controller names or caches domain controller names for an inordinate period of time
-your system does not trust the InCommon CA

It’s also worth noting that if you have system firewalls that do not follow the published NETID domain service firewall guidance, https://it.uw.edu/wares/msinf/authn/firewalls-with-netid-domain/, you may need to adjust your firewalls.

We have purposely delayed the last WS2012R2 DC demotion for several weeks to allow customers to discover and address unknown problems with their applications.

More info:
All Windows computers joined to the NETID domain are configured via domain group policy to trust the InCommon CA, which accounts for ~99% of all systems which perform LDAPS operations with the NETID domain service. However, non-Windows systems and Windows systems in other domains which trust the NETID domain may not be configured to trust the InCommon CA. Whether they are or not is subject to the platform, vendor defaults, and system operator configuration. If your system does not trust the InCommon CA, you’ll need to configure it to do so. More information about the InCommon CA and UW’s use of it is at: https://it.uw.edu/service/certificate-services/.

All Windows computers use the Microsoft DC locator process. Non-Windows computers generally do not, although there are exceptions. If your system does not automatically locate domain controllers, you may need to manually configure it and/or take actions that clear any cached information.

After the last WS2012R2 DC is demoted, we also plan to raise the domain and forest functional level to WS2016.

If you have questions, concerns, or encounter problems during these changes, please contact us by sending email to help@uw.edu with “MI DC changes” somewhere in the subject line.

Brian Arkills
Microsoft Infrastructure service manager
UW-IT