Catching an undesired guest in the penguin /tmp room

You might have come across LUARM, a prototype I have built to make an insider misuse detection platform. To cut the long story short, I have looked at various audit log engines and for good reasons, I decided to write my own one, as I found that most of them are really inadequate to provide useful logging in order to:

• Perform a good post mortem analysis of a security incident,
• Link events to user entities (accountability),
• Be able to use the logged data to perform event correlation easily.

There are of course many other goals of the prototype that I am hoping to present at the USENIX Security 11 event, if the paper is ready on time. What I wish to show here is how a good structure of an audit log makes life easier, with or without insiders in mind.

Using LUARM, I managed to catch a compromised penguin carrying a Perl IRC bot like this one. It  did get into the system, because a XAMPP server was running with lots of outdated components, especially when it comes to PHP, introducing many exploitable vulnerabilities . OK, the point is that we compile from source, we always update frequently, we take care of he permissions of /tmp dirs, etc,etc, that every professional sysadmin should know...But, sooner or later, you are going to miss something here and there, only to discover it from logs (or a network traffic log). Here is what happens with LUARM.

Before I start describing the incident, I should say a few words about LUARM, so you get a basic idea of how it works...Not that it is complicated, and of course the sources are at Sourceforge, but just for starters.

The above figure displays the module client-server architecture of the LUARM audit engine. On the left, we can see a set of audited computer clients. Every client is running a unique instance of a set of monitoring scripts. Each of the client scripts audits a particular system level aspect of the operating system: 'netactivity.pl' audits the addition and creation of endpoints, 'fileactivity.pl' records various file operations, 'psactivity.pl' provides process execution audit records and 'hwactivity.pl' keeps a log of hardware devices that are connected or disconnected from the system. The right hand side contains the centralized server part of the architecture where audit data are stored, maintained and queried in a MySQL based RDBMS (other RDBMS systems could be used as well). The Perl programming language is used to implement the modules and the communication between client and server is performed via a Perl DBI interface. 

On the server side we store the audit data, in order to safeguard audit integrity and audit volume scalability. Each client authenticates to the LUARM server using separate, unique credentials that do not interfere with the authentication domain of the client.

Starting with the database of the infected host in question, we sit at the basic MySQL console on the LUARM server and we try to get our bearings on the netinfo table, as my NIDS gave an alert about some unusual traffic patterns towards a number of hosts, the most persistent of them were the hosts 'undernet.irc.justedge.net' and 'dana.basefreak.nl', so we ask LUARM to verify activity towards these hosts :

mysql> show tables;

+---------------------+

| Tables_in_panoitpsl |

+---------------------+

| fileinfo            |

| groupinfo           |

| hostinfo            |

| hwinfo              |

| netinfo             |

| netint              |

| netroute            |

| psinfo              |

+---------------------+

8 rows in set (0.00 sec)

mysql> describe netinfo;

+--------------+-------------+------+-----+---------+----------------+

| Field        | Type        | Null | Key | Default | Extra          |

+--------------+-------------+------+-----+---------+----------------+

| endpointinfo | bigint(20)  | NO   | PRI | NULL    | auto_increment |

| cyear        | smallint(6) | NO   |     | NULL    |                |

| cmonth       | tinyint(4)  | NO   |     | NULL    |                |

| cday         | tinyint(4)  | NO   |     | NULL    |                |

| chour        | tinyint(4)  | NO   |     | NULL    |                |

| cmin         | tinyint(4)  | NO   |     | NULL    |                |

| csec         | tinyint(4)  | NO   |     | NULL    |                |

| transport    | tinytext    | NO   |     | NULL    |                |

| sourceip     | tinytext    | NO   |     | NULL    |                |

| sourcefqdn   | tinytext    | YES  |     | NULL    |                |

| sourceport   | smallint(6) | NO   |     | NULL    |                |

| destip       | tinytext    | NO   |     | NULL    |                |

| destfqdn     | tinytext    | YES  |     | NULL    |                |

| destport     | smallint(6) | NO   |     | NULL    |                |

| ipversion    | tinyint(4)  | NO   |     | NULL    |                |

| username     | tinytext    | NO   |     | NULL    |                |

| pid          | smallint(6) | NO   |     | NULL    |                |

| application  | tinytext    | NO   |     | NULL    |                |

| dyear        | smallint(6) | YES  |     | NULL    |                |

| dmonth       | tinyint(4)  | YES  |     | NULL    |                |

| dday         | tinyint(4)  | YES  |     | NULL    |                |

| dhour        | tinyint(4)  | YES  |     | NULL    |                |

| dmin         | tinyint(4)  | YES  |     | NULL    |                |

| dsec         | tinyint(4)  | YES  |     | NULL    |                |

| md5sum       | text        | NO   |     | NULL    |                |

+--------------+-------------+------+-----+---------+----------------+

25 rows in set (0.00 sec)

mysql> select COUNT(*) from netinfo where destfqdn='undernet.irc.justedge.net' ; 

+----------+

| COUNT(*) |

+----------+

|      963 |

+----------+

1 row in set (0.13 sec)

mysql> select COUNT(*) from netinfo where destfqdn='dana.basefreak.nl ' ;

+----------+

| COUNT(*) |

+----------+

|     1408 |

+----------+

1 row in set (0.12 sec)

Indeed, LUARM verifies a lot of hits towards these hosts, so given the fishy names, we need to dig out more from the LUARM endpoint information. In particular, we need to see which username and application created these endpoints, so we issue the following SQL queries, limiting the number of results to 5 (to prevent flooding our console with endpoint info):

mysql> select application,username,pid from netinfo where destfqdn='dana.basefreak.nl' LIMIT 5;

+-------------+----------+------+

| application | username | pid  |

+-------------+----------+------+

| crond       | nobody   | 7371 |

| crond       | nobody   | 7371 |

| crond       | nobody   | 7371 |

| crond       | nobody   | 7371 |

| crond       | nobody   | 7371 |

+-------------+----------+------+

5 rows in set (0.11 sec)

mysql> select application,username,pid from netinfo where destfqdn='undernet.irc.justedge.net' LIMIT 10;

+-------------+----------+------+

| application | username | pid  |

+-------------+----------+------+

| crond       | nobody   | 21995 |

| crond       | nobody   | 21995 |

| crond       | nobody   | 21995 |

| crond       | nobody   | 21995 |

| crond       | nobody   | 21995 |

+-------------+----------+-------+

5 rows in set (0.12 sec)

So, we have now a username of 'nobody' a crond application which appear to relate to these endpoints via two distinct PIDs, 7371 and 21995. We have to find out what did these two processes do, as this is something but not very helpful. Thus, we need to jump to the psinfo table:

mysql> describe psinfo;

+-----------+--------------+------+-----+---------+----------------+

| Field     | Type         | Null | Key | Default | Extra          |

+-----------+--------------+------+-----+---------+----------------+

| psentity  | bigint(20)   | NO   | PRI | NULL    | auto_increment |

| md5sum    | text         | NO   |     | NULL    |                |

| username  | tinytext     | NO   |     | NULL    |                |

| pid       | smallint(6)  | NO   |     | NULL    |                |

| ppid      | smallint(6)  | NO   |     | NULL    |                |

| pcpu      | decimal(3,1) | NO   |     | NULL    |                |

| pmem      | decimal(3,1) | NO   |     | NULL    |                |

| command   | text         | NO   |     | NULL    |                |

| arguments | mediumtext   | NO   |     | NULL    |                |

| cyear     | smallint(6)  | NO   |     | NULL    |                |

| cmonth    | tinyint(4)   | NO   |     | NULL    |                |

| cday      | tinyint(4)   | NO   |     | NULL    |                |

| cmin      | tinyint(4)   | NO   |     | NULL    |                |

| chour     | tinyint(4)   | NO   |     | NULL    |                |

| csec      | tinyint(4)   | NO   |     | NULL    |                |

| dyear     | smallint(6)  | YES  |     | NULL    |                |

| dmonth    | tinyint(4)   | YES  |     | NULL    |                |

| dday      | tinyint(4)   | YES  |     | NULL    |                |

| dhour     | tinyint(4)   | YES  |     | NULL    |                |

| dmin      | tinytext     | YES  |     | NULL    |                |

| dsec      | tinyint(4)   | YES  |     | NULL    |                |

+-----------+--------------+------+-----+---------+----------------+

21 rows in set (0.00 sec)

mysql> select COUNT(*) from psinfo where username='nobody' ;

+----------+

| COUNT(*) |

+----------+

|       57 |

+----------+

1 row in set (0.05 sec)

Well, we have had 57 intercepted executions of processes with that network credentials, which means that some of these will give us more info. So, we can expand the info on these 57 processes by issuing a:

mysql> select command,username,arguments,pid,ppid from psinfo where username='nobody' \G;

.... and amongst the many results from httpd apache we, we see the following:

*************************** 56. row ***************************

  command: /usr/bin/perl

 username: nobody

arguments: ./x 79.125.109.171 0 0

      pid: 9253

     ppid: 8343

*************************** 57. row ***************************

  command: /usr/bin/perl

 username: nobody

arguments: ./x 85.121.125.74 0 0

      pid: 29312

     ppid: 8343

57 rows in set (0.05 sec)

Clearly, we have traced down the offending application now, when it tried to communicate with the command servers (85.121.125.74 and 79.125.109.171). A quick navigation to the /tmp dir shows the culprit and the way to clean the perl bot infested penguin. Note how the psinfo table gives a pid and a Parent PID (ppid). Using that information in conjunction with the username and endpoint info, we can make very powerful an fast correlations on host events. As an example, let's take the pid 23913 and check to see the source, destination and exact time information of the endpoints created by that process:

mysql> select sourceport,destport,cday,chour,cmin  from netinfo where username='nobody' AND pid='29312' \G;

...........

*************************** 523. row ***************************

sourceport: 32767

  destport: 6670

      cday: 5

     chour: 7

      cmin: 12

*************************** 524. row ***************************

sourceport: 32767

  destport: 6670

      cday: 5

     chour: 7

      cmin: 13

*************************** 525. row ***************************

sourceport: 0

  destport: 0

      cday: 5

     chour: 19

      cmin: 0

525 rows in set (0.13 sec)

.......

I am sure some of you will say that a competent sysadmin could find that by simply looking at looking at a couple of files such as the Apache Logs for example. Sure, a competent sysadmin is likely to have faced a perl IRC bot before. However, what if you had to deal with something that you were not familiar with on a very busy box that was compromised and the logs were erased or logrotated. Are many tools able to offer you that level of detail and correlation? Think again and whether you look at legitimate users or general system auditing issues, give LUARM a go!