Dirty COW Linux Kernel Vulnerability Fixed

Last week a very serious vulnerability in the Linux kernel, the so called Dirty COW, was reported. Our dedicated Linux kernel team immediately addressed the issues and were able to patch it in less than 24 hours on the majority of our servers. What is more, we managed to do this without server reboot and we avoided the downtime that normally results from such kernel update activities. To learn more about the vulnerability and how we addressed it read below.

What is that Dirty COW about?

The Dirty COW vulnerability allows attackers to gain root access to servers and take control over the whole system. The security hole was detected by researcher Phil Oester, who found out a race condition in the way the Linux kernel’s memory subsystem handles copy-on-write (COW) breakages of private read-only memory mappings. Attackers can use this to gain write access to otherwise read-only mappings and this way take control over whole systems. For more technical information you may check the official vulnerability page and this site which is dedicated to the vulnerability.

How widespread is the issue?

The issue most probably affects hundreds of thousands, if not millions, of Linux based machines. If you are not running the latest version of the Linux kernel you should be worried. In fact, even if you are running the latest kernel you should still be worried, as at the time this post is written not all vendors have patched their respective kernels yet. If you want to try and hack your own system you can visit this Github page and use the PoCs provided on it. According to the reports, the following Linux distro versions are vulnerable (please note that this is not a complete list but rather a list of the most popular Linux distros):

1. Red Hat Enterprise Linux 7.x
2. Red Hat Enterprise Linux 6.x
3. Red Hat Enterprise Linux 5.x
4. CentOS Linux 7.x
5. CentOS Linux 6.x
6. CentOS Linux 5.x
7. Debian Linux wheezy
8. Debian Linux jessie
9. Debian Linux stretch
10. Debian Linux sid
11. Ubuntu Linux precise (LTS 12.04)
12. Ubuntu Linux trusty
13. Ubuntu Linux xenial (LTS 16.04)
14. Ubuntu Linux yakkety
15. Ubuntu Linux vivid/ubuntu-core
16. SUSE Linux Enterprise 11 and 12.
17. Openwrt

What is the standard issue resolution?

The easiest way to protect your computers running Linux is to update your Linux distro to the latest version. Keep in mind that this action, however, requires a reboot. You can use the following commands to update your Debian/Ubuntu and RHEL systems:

Debian/Ubuntu:
$ sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade

RHEL:
$ sudo yum update
$ sudo reboot

CentOS:
There is still no official update of the CentOS kernel. At this time the only way to patch your CentOS servers is to follow the instructions from this link.

Once you reboot your Linux computers, ensure that they are running the new kernel by executing the following commands:

$ uname -a
$ uname -r
$ uname -mrs

What SiteGround did to resolve this issue?

For quite some time we use our own custom kernel. This means that when issues like this appear we are not dependent on the updates released by the OS developers. Instead we can get creative immediately and we can craft our own solutions. In this particular case our devops team made use of several updates provided by the upstream kernel developers and the kernel security experts from GRSecurity related with the issue. We adapted them in a timely manner and created a patch that worked for our specific kernel.

As we mentioned, upgrading the kernel means that you will have to reboot your servers. Needless to say a hosting company such as SiteGround cannot simply restart thousands of servers and cause downtime to hundreds of thousands of sites.

That is why we decided to patch the kernel without restarting our servers. We used a tool called kpatch, which allowed us to patch the running kernels without rebooting or restarting any processes. The problem with using such tools is that there is always a chance that something may go wrong and a server may crash. We have different servers configured in different ways that run different kernels. That is why we had to test a big number of scenarios before we applied the patch live on so many servers. We knew about the problem before the Dirty COW website was created and we started working on a fix the moment the kernel developers released the patch. That is why we had enough time to perform the needed checks.

All shared, cloud and dedicated SiteGround servers are already patched.