The majority of data breaches occur due to inadequate patching. Fact. Data from The Ponemon Institute found that 60 percent of breach victims said they were breached due to an unpatched known vulnerability where the patch was not applied.
This is even more the case in the hybrid workplace, which is here to stay according to Gartner, where teams are disparately working across a blend of personal and work devices. New vulnerabilities are being exploited every day. The latest MS Exchange reported that vulnerabilities increased tenfold in just four days in March this year.
For the majority of IT professionals, patching is part of the day-to-day role and responsibility, though is not always positively anticipated. From planning to execution, leading into clean up or recovery – depending on the outcome of the cycle- patching will always be a key task and requirement for the IT function. Ultimately, it’s hard to avoid, so needs to be embraced, planned for, and even enjoyed.
Make patching part of the company culture
The earlier that a regular patching routine is adopted into any company’s culture, the easier it becomes to execute. Think of patching as the month end. A lot of effort and execution goes into the company’s end-of-month performance and activities and patching should be viewed in the same vein. If you can condense patching into a single monthly activity, you will already be way ahead of 1000’s of companies worldwide which have yet to achieve this. Patches provide value, albeit in different forms, but ultimately patching will lead to:
- Improved productivity – Fixing bugs, resolving ongoing issues and providing new functionality will allow users and systems to be optimal, as well as pioneer new ways of working and enhance business processes.
- Improved security – One of the biggest reasons to patch is to tackle the ever-building list of vulnerabilities. Vulnerabilities can be discovered, even after years of a system being released.
Unfortunately, it seems that the older the system, the more vulnerable it becomes, often due to the vendor marking them as EOL (End of life).
On average, a calendar month contains 730.5 hours. In theory, there should be a window in this time where downtime, which can be referred to as a reboot or extended reboot, can be performed. We use the term extended, because after installing updates, the node (the redistribution point or communication endpoint) can often take longer than normal to become available again. If this concept of downtime can be adopted into the planning for a system or service deployment, then in the long term there is an agreed maintenance window. If nodes cannot be taken down, clustering nodes can be utilized to provide resilience with the bonus of performing maintenance while the other node remains active.
There is often a perception that if the latest monthly updates are applied, systems will have no vulnerabilities present on them. This is often not the case, tools such as Tenable can be used to detect vulnerabilities across an entire fleet of systems which can often remain undetected. These tools are becoming more and more widely used, as it’s not just a case of being up to date anymore, it’s a combined process of being up to date and plugging your weaknesses. The term ‘you are only as strong as your weakest link’ comes to mind here, as you can have a fully updated Windows 2016 server, but you may have a Windows 2008 R2 server which has not had any updates since it went EOL. Using additional tooling, you can detect and mitigate where possible, and when mitigation is not possible, you can plan to adopt a strategy to work around or remove the vulnerability altogether.
In recent months, we have found that Microsoft release updates on patch Tuesday, but then recall the patches shortly after to be revised. With the best will in the world, Microsoft cannot test every application of their software, which is where the user base comes in, and specifically their test environments. Most of the patching we undertake (unless critical) is applied a month in lag from release. This gives plenty of time for the patches to come out, be revised, applied to a test platform, required smoke tests carried out and finally applied to production. Again, this process can be built into planning for system deployments, so maintenance windows are applied in accordance with the business requirements and usage of systems.
Emergency patching
There are eventualities where vendor release updates or patches will be found which combat major vulnerabilities, the cases of late being Solarwinds and Microsoft Exchange. These are discovered vulnerabilities which are important to get fixed. They are often flagged as actively being exploited by vulnerability management tools.
Planning for emergency patching is not easy, as this will often disrupt a business’s flow during a given period. However, it needs to have a plan associated. If we lean into the ITIL workflow for change management, you can prepare the business to have an emergency change process which can be called into action as and when required. Even for emergency patching, it is still worthwhile, where possible, to perform this in a test rig. After all, you want a functional system at the end of this, albeit a secure one. Communication is key here and explaining the gravity of the situation is important. Often IT departments will state that something needs to be done, and downtime is occurring, but often the reasoning behind why is left out. Avoid baffling with IT tech talk, but be honest, as vulnerabilities can have a serious impact on a business, which can lead to businesses suffering losses if quick action is not taken.
Patching is also a way for developers to bring in improvements to their solutions. These improvements can be minor or major, depending on what the intended solution does. Adobe often adds new features to its free PDF reader and Windows 10, and often repairs minor issues monthly. It also provides several yearly major feature updates which bring new, useful features that really give the system value for money. This is echoed with Microsoft’s other offerings like Office suites and alternative operating systems. Third-party solutions will generally also be updated with new features, bug fixes, improvements, and stability updates throughout its life cycle. These updates may provide new efficiencies in the way your business works. All vendors will provide release notes for their updates which can be reviewed in life with your business objections to weigh out if the downtime or impact of the updating cycle warrants the downtime. This is often a heavy consideration for remote end user updates.
Patching is a stressful time, often it is a large amount of work in a small period. Patching windows are often small and the volume of activities will be high. Preparation and communication are key, if the business is aware this is happening, you are less likely to get a disgruntled user or system owner which was caught off guard by the cycle.
Don’t let patching hold you back
Preparation, in the form of taking backups/snapshots/checkpoints are invaluable- if the update cycle goes wrong, you have a quick roll back plan to get you out of trouble in a stitch. If your estate consists of virtual or physical machine, consoles access can put your mind at rest. At the point you reboot a node, you are in the dark until it begins to boot back up. Console access can provide reassurance that it is doing the right thing. Ultimately, the option to outsource this process can save the stress and headaches for many IT departments, especially if your change advisory board can be a tedious process.
MSP’s can provide reassurance and an SLA backed service to take care of this task, and ultimately free up precious time in house, to focus on more value adding activities and projects. It makes sense to not let patching hold you or your business back when someone else could efficiently manage this process and be accountable for it.
Daniel Robinson, Technical Support Engineer, Foundation IT
