Security of Linux systems and privacy today

In this article, I am not arguing on whether Linux (or Android, or iOS and any other operating system) is the most suitable platform to entrust your data and personal communications today. Look for such dogmatic views elsewhere. I like, use and develop Linux systems, but I have been in business long enough to realize that security (and privacy as part of it) is a lot more complex business than choosing carefully your OS platform. Instead, I shall attempt to walk through some real world examples of certain sysadmin practices and security assumptions that have failed the Linux platform.

During the first week of June 2013 we found that Hetzner, one of the largest web hosting providers in Europe, has had its server security compromised. As a result, server password hashes and other sensitive data went into the hands of third party individuals, who obviously designed the malware vector and stroke gold. The extent of the information breach at Hetzner is still unknown as I write this. Obviously, being compromised is not good, but not knowing the extent of the compromise is worse. While I do not have data for other ISPs and web hosting providers, I strongly suspect that there will be other companies in the same line of business that were affected by this attack. The use of the Plex panel, PHP/Apache/SSH stacks is widespread

The attack vector that affected the Hetzner folks targeted Apache and openssh servers and quoting Martin Hetzner 'the "backdoor" exclusively infects the RAM...the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected. The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code'. If this proves to be true (they have yet to conclude with their investigation at the time of writing), the real eye opener here is not that Hetzner got hacked. The real eye opener is that we have an effective RAM based Linux rootkit that affects essential services (such as http, https and ssh).

Six months ago, Steelcyber Scientific, a company that I consult for was one of the first to detect and mitigate for another Apache based exploit that installed rogue SSH servers and fished for customer usernames and passwords. Admittedly although less sophisticated, the older exploit was still successful in targeting the same services (sshd and httpd). I made various statements then in customers and IT journalists that people need to start taking Linux server security seriously. The success of Linux in the server and web hosting business has been providing an ideal target for malware writers for a number of years now, contrary to the popular (?) belief that exploits are mainly written for Windows (Android and iOS/OSX systems recently).

I am sure that this trend is going to continue, so I shall need to justify now what I mean by taking Linux security seriously.

One of the most frequently encountered pitfalls that make Linux systems vulnerable to various types of attacks is that most sysadmins still turn SELinux off. Yes, even experienced ones, that do keep their software updated. Yes, even after 10 years since its introduction to the mainstream Linux kernels. Many hosting providers allow customers to do that. Many of the attacks (and certainly the two previously described ones) could have been repelled if proper SELinux policies were in place. An SSH rogue server could not run in /tmp if SELinux is on. An i-frame injection could compromise Apache but could not install/execute files in any system directory.  Yes, it is annoying to install/troubleshoot new or upgraded software with SELinux. The Permissive SELinux mode is your friend to find and correct those issues keeping things contained. When you are done, make sure that the Enforcing mode is back on. Don't just turn it off and rely on traditional Unix and filesystem ACLs to contain things. This does not work.

I am not sure what your view is about logging/auditing systems in Unix and Linux, but I consider that the current utilities are not suitable for providing accountability on which process/user account does what in the system. In addition, today's common log/audit utilities do not allow sysadmins, devops and security experts to construct a forensic post-intrusion/breach picture, so that the extent of information leaks/intrusion is better understood and contained fast. I argue in favour of a better logging/auditing approach in this science paper. The end product, LUARM, is a different philosophy of what to log and how you can search it. We have employed LUARM successfully in monitoring and forensically examining a number of systems, and frankly, we have had great success in realizing fast what happened. If you compare that to traditional syslog approaches, well, good luck, you will have a lot more to examine and filter manually and one day you might find it what actually happened in your systems.

The final point I am trying to raise relates to authentication. Most production Linux systems today employ some sort of PAM complying approach to authenticate users. This is likely not a Single Sign On (SSO) solution. It is also likely that it only uses a username and password combination or an (D)/(R)SA key tied to a client machine. Whether you have a local/LDAP/(NIS, NIS+ if you are really oldie)/AD backend implementation is not relevant. The relevant bit concerns two facts:

i)The use of only a username and password or username/client/cryptographic key

ii)The widespread exposure of the front end daemon (Apache, SSH) in the Internet

(Please understand that I am referring to what is common practice, if you are a cautious sysadmin of a large installation, you probably do things differently.) 

A person once told me: "I use a secure login because I always SSH to the system". I replied rather cynically to that statement. The same person (OK, you know who you are :-) ) uses Skype for communicating with his family because the encryption is adequate. On the other hand, the same person is absolutely disgusted that Verizon turned his/her phone records to NSA. So, I cannot stop being cynical here.

I am not going to go through the theory of why a cryptographically enabled endpoint pair is more secure than a plain text endpoint pair, but still is not secure. Encryption aids security. It is not by itself security and if people do not get that point, they should look at what happened to Hetzner (and other providers that entrusted their security solely on an SSH frontend). The point here is that an entire industry lays all their eggs and/or applies the rule 'one size fits all' when it comes to matching the sensitivity of data, the exposure of the authentication front-end and the criteria of authentication. If you go to share hosting/cloud provider today, the main question is not what data you are going to store there, but what size of CPU/RAM and storage you need. The end result of this is that on cloud provider hard drives today, you can find anything: From personal information (pictures, videos), valuable Intellectual Property, to - I suspect - nuclear missile location details. Yet, nobody has checked the strength of the authentication procedure. You want two factor authentication? Good, build it yourself.

To sum up, proper containment policies. forensically enabled auditing and monitoring, as well as an authentication scheme that is suitable to the sensitivity of the data/services deployed in Linux infrastructures is missing today. If these issues are not addressed, perfectly adequate penguins can leak your information to skilled individuals. Make sure you address these issues with your IT team(s). Stay safe!