Visualizing aggregates of events

Elastic Stack Kibana
1

Hello folks,

First of all, let me apologize if I've posted this to the wrong forum; I wasn't certain whether it is a Kibana- or a Logstash-related issue.

I am running a honeypot (Cowrie) that generates a JSON log of events. There are all kinds of events - login attempt, successful login, command input, file download, logout, etc. I can use Kibana to visualize on a per-event basis just fine - for instance, by monitoring the "successful login" event I can map the IPs connecting to the honeypot, or I can monitor how many different files have been uploaded, etc.

However, I would like to visualize something that doesn't fit in a single event but encompasses several events. When different attacking bots connect to the honeypot, they issue different commands that can be used to identify them. In the log, each command is a single event; they can be grouped together by the session ID, which is also present in the log.

For instance, the attack by a typical Mirai-infected device (i.e., the bot; not the server that spreads the infection) would issue the following commands:

enable
system
shell
sh
/bin/busybox MIRAI

The entries in the log corresponding to a complete Mirai attack session would look like this:

{"eventid": "cowrie.session.connect", "timestamp": "2017-01-24T21:44:57.304654Z", "session": "1035fa4e", "message": "New connection: 112.162.137.143:39677 (192.168.0.103:23) [session: TT517]", "src_port": 39677, "system": "cowrie.telnet.transport.HoneyPotTelnetFactory", "isError": 0, "src_ip": "112.162.137.143", "dst_port": 23, "dst_ip": "192.168.0.103", "sensor": "vesselin-pc"}
{"eventid": "cowrie.login.success", "username": "root", "timestamp": "2017-01-24T21:44:58.876736Z", "message": "login attempt [root/pass] succeeded", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "password": "pass", "sensor": "vesselin-pc"}
{"eventid": "cowrie.log.open", "timestamp": "2017-01-24T21:44:59.544447Z", "message": "Opening TTY Log: log/tty/20170124-234459-None-517i.log", "ttylog": "log/tty/20170124-234459-None-517i.log", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.input", "timestamp": "2017-01-24T21:45:00.258912Z", "message": "CMD: enable", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "enable", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.success", "timestamp": "2017-01-24T21:45:00.262274Z", "message": "Command found: enable ", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "enable ", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.input", "timestamp": "2017-01-24T21:45:01.113115Z", "message": "CMD: system", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "system", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.failed", "timestamp": "2017-01-24T21:45:01.114400Z", "message": "Command not found: system", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "system", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.input", "timestamp": "2017-01-24T21:45:01.116013Z", "message": "CMD: shell", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "shell", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.failed", "timestamp": "2017-01-24T21:45:01.117142Z", "message": "Command not found: shell", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "shell", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.input", "timestamp": "2017-01-24T21:45:01.871043Z", "message": "CMD: sh", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "sh", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.success", "timestamp": "2017-01-24T21:45:01.872692Z", "message": "Command found: sh ", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "sh ", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.input", "timestamp": "2017-01-24T21:45:02.590821Z", "message": "CMD: /bin/busybox MIRAI", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "/bin/busybox MIRAI", "sensor": "vesselin-pc"}
{"eventid": "cowrie.command.success", "timestamp": "2017-01-24T21:45:02.591760Z", "message": "Command found: /bin/busybox MIRAI", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "session": "1035fa4e", "input": "/bin/busybox MIRAI", "sensor": "vesselin-pc"}
{"eventid": "cowrie.log.closed", "timestamp": "2017-01-24T21:45:03.186712Z", "message": "Closing TTY Log: log/tty/20170124-234459-None-517i.log after 3 seconds", "ttylog": "log/tty/20170124-234459-None-517i.log", "system": "CowrieTelnetTransport,517,112.162.137.143", "src_ip": "112.162.137.143", "session": "1035fa4e", "duration": 3.6424460411071777, "sensor": "vesselin-pc", "isError": 0, "size": 1271}
{"eventid": "cowrie.session.closed", "timestamp": "2017-01-24T21:45:03.195397Z", "message": "Connection lost after 5 seconds", "system": "CowrieTelnetTransport,517,112.162.137.143", "isError": 0, "src_ip": "112.162.137.143", "duration": 5.890791893005371, "session": "1035fa4e", "sensor": "vesselin-pc"}

I have written a stand-alone Python script that can parse the complete log and, using a database of attack patterns, determine which particular bots have attacked the honeypot. The output of it would look something like this:

24-Jan-2017 16:26:13 Mirai bot (MEMES) (IP: 175.203.80.139)
24-Jan-2017 16:27:13 Hajime (refused) (IP: 5.22.85.33)
24-Jan-2017 16:33:04 Mirai bot (OBJPRN) (IP: 181.23.50.121)
24-Jan-2017 16:33:12 Mirai server (NETLPRNN) (truncated) (IP: 109.236.86.63)
...

My question is - can I somehow use Kibana to visualize the different bots that are attacking the honeypot? (For instance, the most active bots, total number of different bots, number of different kinds of bots, etc.)

Perhaps it could be implemented as a Logstash filter (I'm just guessing here) that adds an additional filed (bot name) to the log entries? My problem is that this information cannot be obtained by examining just a single entry; it is the result of analyzing all entries in a group that has the same session ID field.

2

How does that work exactly?

Yes.
You can ingest the logs as it, but that doesn't contain what sort of bot it is.
Or you can process the post-python logs.
Or you can build the python into a filter in LS and then process the events and add the bot name as a field, that way you get everything.

3

I'm not sure I understand the question. It's a Python script. It reads a (text) database containing sets of regular expressions that match particular bot behavior. It then parses the JSON log entry by entry, picking the commands issued during each login session and tries to match them with the sets in the database. Are you asking about how it is invoked from the command line, or the format of the database, or about something else?

That's probably the best solution, but I can't figure out how to do it. I found some Logstash filter called "aggregate" which seems to be made for such cases (processing info from more than one line in the log) but the given examples are way too trivial and I can't figure out how to do what I want done. It doesn't help that I don't know Ruby, either.

4

Nope, exactly that :slight_smile:

Can you express the patterns in that script as groks perhaps?

5

Not sure; I'm not sufficiently familiar with grok. It's a proprietary language (meaning that I made it up) that consists mostly of regular expressions. If a line is preceded by <caracter>:, then it is a special line, possibly containing regular expressions. Otherwise it's just a string that has to be matched exactly. Here are examples of special lines:

+: foo

The above line is present one or more times.

*:(foo|bar)

The above line is present zero or more times.

r:foo.+

The above line is a regular expression that must be present exactly once.

{: some lines }:

The group of lines

some
lines

is present one or more times. Any of them could be preceded by one of the special prefixes that indicate optional regular expressions.

Here is an example entry, defining the attack pattern of a Mirai server (i.e., not the bot that runs on devices):

[name]
Mirai server (original)
[commands]
enable
shell
sh
/bin/busybox ECCHI
/bin/busybox ps; /bin/busybox ECCHI
/bin/busybox cat /proc/mounts; /bin/busybox ECCHI
+:/bin/busybox echo -e \'\\x6b\\x61\\x6d\\x69.*\' > .+\.nippon; /bin/busybox cat .+\.nippon; /bin/busybox rm .+\.nippon
/bin/busybox ECCHI
+:rm .+\.t; rm .+\.sh; rm .+\.human
r:cd .+
/bin/busybox cp /bin/echo dvrHelper; >dvrHelper; /bin/busybox chmod 777 dvrHelper; /bin/busybox ECCHI
/bin/busybox cat /bin/echo
/bin/busybox ECCHI
*:cat /proc/cpuinfo; /bin/busybox ECCHI
/bin/busybox wget; /bin/busybox tftp; /bin/busybox ECCHI
r:/bin/busybox wget http://.+/dlr/dlr\..+ -O dvrHelper; /bin/busybox chmod 777 dvrHelper; /bin/busybox ECCHI
r:\./dvrHelper telnet\..+; /bin/busybox IHCCE
*:/bin/busybox wget; /bin/busybox tftp; /bin/busybox ECCHI
*:/bin/busybox wget http://.+/dlr/dlr\..+ -O dvrHelper; /bin/busybox chmod 777 dvrHelper; /bin/busybox ECCHI
*:\./dvrHelper telnet\..+; /bin/busybox IHCCE
/bin/busybox ECCHI
6

So for that, you're saying that Mirai is identifiable by any of those commands?

7

They are the commands used by an attacking Mirai server, yes. Not the bot (the program that works on the vulnerable cameras) but the server (the thing that uploads the bot to the cameras).

The above lines describe a full attack pattern. The "+:" lines can occur more than once (depending on how many writable directories exist on the vulnerable device) and the "*:" lines could be missing (they are present only in attacks against ARM platforms).

So, back to the subject, I guess what I want done is not possible?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.