Auditbeat sending path through has hex

Elastic Stack Beats
1

Having a painful problem. Using auditbeat to monitor a directory. It would be nice if it gave the full path to whatever was happening rather than the relative path (unless there is a setting for ensuring the full path always, but I keep getting paths such as

706D2F395F505443205465616D2F4E69636B422F6D6176656E2D7368617265642D7574696C732D302E342F736861646F772D6D6F73742D6D6176656E2D7368617265642D7574696C732D302E342D7363616E726573756C74732F626173652F6D6176656E2D7368617265642D7574696C732D302E342D736F757263652D72656C656173652D302F6D6176656E2D7368617265642D7574696C732D302E342F7372632F6D61696E2F6A6176612F6F72672F6170616368652F6D6176656E2F7368617265642F7574696C732F696F2F5363616E436F6E647563746F722E6A617661

Sometimes it is part text and part hex. In this case it translates to

pm/9_PTC Team/NickB/maven-shared-utils-0.4/shadow-most-maven-shared-utils-0.4-scanresults/base/maven-shared-utils-0.4-source-release-0/maven-shared-utils-0.4/src/main/java/org/apache/maven/shared/utils/io/ScanConductor.java

Wasn't sure it was hex until I dumped it in a converter.

It is the same in:

auditd.paths
auditd.summary.object.primary
file.path

This is not in all messages, just some of them.

Cheers, N

2

The hex decoding will be fixed in 6.2.4 which will be released very soon. See https://github.com/elastic/beats/pull/6687.

Auditbeat should be passing the path through exactly as they are received from the kernel. So probably the kernel is sending relative paths for these events. Could you share the full event in JSON format?

3

@andrewkroh Thanks for the quick reply! Hope I have pasted it correctly below.

{
  "_index": "auditbeat-6.2.3-2018.04.17",
  "_type": "doc",
  "_id": "QUeK02IBMgsju5oVqjTK",
  "_version": 1,
  "_score": null,
  "_source": {
    "event": {
      "module": "auditd",
      "category": "audit-rule",
      "action": "opened-file",
      "type": "syscall"
    },
    "@version": "1",
    "host": "ptc38501-01.ptc.com",
    "file": {
      "mode": "0664",
      "uid": "568",
      "group": "twc",
      "gid": "595",
      "inode": "258379478",
      "owner": "n3burd",
      "device": "00:00",
      "path": "ScanConductor.java"
    },
    "tags": [
      "ptc_dir_access",
      "beats_input_raw_event"
    ],
    "user": {
      "egid": "501",
      "uid": "0",
      "sgid": "501",
      "name_map": {
        "egid": "ptc",
        "uid": "root",
        "sgid": "ptc",
        "fsgid": "ptc",
        "fsuid": "n3burd",
        "auid": "root",
        "gid": "root",
        "euid": "n3burd",
        "suid": "n3burd"
      },
      "fsgid": "501",
      "fsuid": "568",
      "gid": "0",
      "auid": "0",
      "euid": "568",
      "suid": "568"
    },
    "beat": {
      "version": "6.2.3",
      "hostname": "ptc38501-01.ptc.com",
      "name": "ptc-desk"
    },
    "process": {
      "cwd": "/PTC/twc/pm/9_PTC Team/NickB/maven-shared-utils-0.4/shadow-most-maven-shared-utils-0.4-scanresults/base/maven-shared-utils-0.4-source-release-0/maven-shared-utils-0.4/src/main/java/org/apache/maven/shared/utils/io",
      "pid": "27682",
      "name": "smbd",
      "exe": "/usr/sbin/smbd",
      "ppid": "28912"
    },
    "auditd": {
      "sequence": 1165041009,
      "summary": {
        "how": "/usr/sbin/smbd",
        "object": {
          "primary": "ScanConductor.java",
          "type": "file"
        },
        "actor": {
          "primary": "root",
          "secondary": "root"
        }
      },
      "session": "16220",
      "result": "success",
      "paths": [
        {
          "item": "0",
          "mode": "042775",
          "name": "2F5054432F7477632F706D2F395F505443205465616D2F4E69636B422F6D6176656E2D7368617265642D7574696C732D302E342F736861646F772D6D6F73742D6D6176656E2D7368617265642D7574696C732D302E342D7363616E726573756C74732F626173652F6D6176656E2D7368617265642D7574696C732D302E342D736F757263652D72656C656173652D302F6D6176656E2D7368617265642D7574696C732D302E342F7372632F6D61696E2F6A6176612F6F72672F6170616368652F6D6176656E2F7368617265642F7574696C732F696F",
          "dev": "fd:03",
          "nametype": "PARENT",
          "ogid": "595",
          "rdev": "00:00",
          "ouid": "568",
          "inode": "141116713"
        },
        {
          "item": "1",
          "mode": "0100664",
          "name": "ScanConductor.java",
          "rdev": "00:00",
          "ogid": "595",
          "nametype": "CREATE",
          "dev": "fd:03",
          "ouid": "568",
          "inode": "258379478"
        }
      ],
      "data": {
        "a3": "0",
        "tty": "(none)",
        "arch": "x86_64",
        "a1": "20042",
        "syscall": "open",
        "exit": "86",
        "a2": "1b4",
        "a0": "7fb6818d01e0"
      }
    },
    "@timestamp": "2018-04-17T12:18:20.997Z"
  },
  "fields": {
    "@timestamp": [
      "2018-04-17T12:18:20.997Z"
    ]
  },
  "highlight": {
    "tags": [
      "@kibana-highlighted-field@ptc_dir_access@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1523967500997
  ]
}
4

@andrewkroh Sorry, just realised I pasted the hex one in, give me a mo while I try to find a 'normal' one!

5

Perhaps in this case Auditbeat could join the path from the PARENT (paths.item.0) and the child (paths.item.1). I'll have to do some more research to see if it would be safe to apply this in all cases. What is the associated audit rule that triggered this event?

In the meantime you could use Logstash to join the paths.

6

@andrewkroh Hi Andrew, that was the sort of thing I was hoping for, but cannot see that in the current messages. I had wondered about file.path and something else, but couldn't figure out what the something else.

Had thought about process.cwd with file.path, but realised that running an ls command in that directory would give that to you, but running ls against a different directory would not. I think the following is what you are after.

auditbeat.modules:

# The kernel metricset collects events from the audit framework in the Linux
# kernel. You need to specify audit rules for the events that you want to audit.
- module: auditd
  resolve_ids: true
  failure_mode: log
  backlog_limit: 8196
  rate_limit: 0
  include_raw_message: false
  include_warnings: false
  audit_rules: |
    -w /PTC/twc/ -p wr -k ptc_dir_access

Don't see path.items in the list of fields? Thanks, N

7

Sorry, I wasn't very clear that I was talking about the fields inside of the event that you posted.

What I was referring to was that Auditbeat sends a list of objects inside of auditd.paths. Item 0 in the list contains the parent directory. And item 1 in the list contains the file name. So with a bit Logstash config magic you could join the the name fields to create an absolute path.

8

@andrewkroh OK makes sense, not that you weren't clear, I'm still learning!

Not 100% sure that will work as just looked at one of the messages and it just has

{
  "rdev": "00:00",
  "item": "0",
  "ouid": "568",
  "inode": "145173566",
  "name": "pm/9_PTC Team/NickB/maven-resources-plugin-2.5",
  "ogid": "595",
  "nametype": "NORMAL",
  "dev": "fd:03",
  "mode": "042775"
}

Not item 1. I would have expected /PTC/projecta/ in front of this path.

I had wondered how to address those fields in auditd.paths. Is it the same in Logstash such as [auditd][paths][ at this point I have no idea for the rest!

Thanks for your time. N

9

OK think I've figured this one out. The person has mapped the directory /PTC/twc to a Windows Network drive using Samba (Grrrr), it is showing the process.cwd as /PTC/twc so in the case of Samba (smbd) processes, it would be a case of adding the process.cwd to item 0 name.

The good news is that I'm not interested in smdb calls, which is an event action of checked-metadata-of, I'm interested in the event actions of (i think) is open file.

I've been searching the net and I cannot find an obvious list of event actions. Do you know if there is one?

Thanks, N

PS. As much as I have been challenged in getting this going, I'm glad I've stuck with it as I believe that it is going to get me there in the end and I will definitely have you to thank! Just need to get rid of all the noise!

10

The event.action value is derived from either the syscall name or the event.type field. It's driven by this table from which you can grep action to get a list of all possible actions.

Auditbeat 6.2.4 is now released and it has the hex decoding fix for paths.

11

Thanks. Just tried installing 6.2.4 and get the following:

[nhopper@ptc38501-01 ~]$ sudo rpm -vi auditbeat-6.2.4-x86_64.rpm
warning: auditbeat-6.2.4-x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing packages for installation...
	file /usr/share/auditbeat/bin/auditbeat from install of auditbeat-6.2.4-1.x86_64 conflicts with file from package auditbeat-6.2.3-1.x86_64
	file /usr/share/auditbeat/.build_hash.txt from install of auditbeat-6.2.4-1.x86_64 conflicts with file from package auditbeat-6.2.3-1.x86_64
	file /usr/share/auditbeat/NOTICE.txt from install of auditbeat-6.2.4-1.x86_64 conflicts with file from package auditbeat-6.2.3-1.x86_64
	file /usr/share/auditbeat/README.md from install of auditbeat-6.2.4-1.x86_64 conflicts with file from package auditbeat-6.2.3-1.x86_64
	file /usr/share/auditbeat/kibana/6/index-pattern/auditbeat.json from install of auditbeat-6.2.4-1.x86_64 conflicts with file from package auditbeat-6.2.3-1.x86_64

Is this something I need to overcome? Is it a change to the rpm command?

Thanks, N

12

That's a warning from RPM telling you that you do not have the signing key installed for which that package was signed.

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

https://www.elastic.co/guide/en/beats/auditbeat/current/setup-repositories.html#_yum

13

Thanks, sorted that. Are there system limits that might stop Auditbeat sending messages.

At 00:59:59.992 the ELK system just stopped receiving messages.

I have rebooted systems, restarted server, restarted Auditbeat, removed any filtering and nothing seems to get messages sent again.

Off toipc, sorry

14

I would check the Auditbeat logs for errors and take a look at the "Non-zero metrics in last 30s" to see if there were any failures with the output.

15

Hi.... err yes....... Seem to have lost about 3 billion messages. I it was doing a back up so rsync hit it pretty hard. I'm currently working on removing 'noise' from what is getting past Logtstash filtering. Time to look at what these are telling me and figure out how to fix. Another learning curve!

|2018-04-19T16:39:36.870+0100|INFO|[auditd]|auditd/audit_linux.go:192|audit status from kernel at start|{"audit_status": {"Mask":892548912,"Enabled":1,"Failure":1,"PID":0,"RateLimit":0,"BacklogLimit":8196,"Lost":3446808471,"Backlog":0,"FeatureBitmap":0,"BacklogWaitTime":0}}|
|---|---|---|---|---|---|
|2018-04-19T16:40:04.236+0100|INFO|[monitoring]|log/log.go:124|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"auditd":{"lost":293},"beat":{"cpu":{"system":{"ticks":17180,"time":17188},"total":{"ticks":46220,"time":46228,"value":46220},"user":{"ticks":29040,"time":29040}},"info":{"ephemeral_id":"c853ae42-1c9b-49b6-8416-b06b174d331d","uptime":{"ms":30041}},"memstats":{"gc_next":9708848,"memory_alloc":8230792,"memory_total":762385304,"rss":34197504}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":23511,"batches":26,"total":23511},"read":{"bytes":156},"type":"logstash","write":{"bytes":3366714}},"pipeline":{"clients":1,"events":{"active":559,"published":24070,"retry":864,"total":24070},"queue":{"acked":23511}}},"metricbeat":{"auditd":{"auditd":{"events":24073,"success":24073}}},"system":{"cpu":{"cores":38},"load":{"1":0.84,"15":0.24,"5":0.37,"norm":{"1":0.0221,"15":0.0063,"5":0.0097}}}}}}|
16

The values you posted are for a single 30s window. So having more will give you a better idea of what's happening. I filtered and formated with jq '. | {output: .monitoring.metrics.libbeat.output, pipeline: .monitoring.metrics.libbeat.pipeline}'.

{
  "output": {
    "events": {
      "acked": 23511,
      "batches": 26,
      "total": 23511
    },
    "read": {
      "bytes": 156
    },
    "type": "logstash",
    "write": {
      "bytes": 3366714
    }
  },
  "pipeline": {
    "clients": 1,
    "events": {
      "active": 559,
      "published": 24070,
      "retry": 864,
      "total": 24070
    },
    "queue": {
      "acked": 23511
    }
  }
}

You might be having some back-pressure issues with LS or ES because there was 864 retries. This means that the output had to retry delivering the event to Logstash. I'd check the LS logs and maybe even try out the Pipeline Viewer.

17

@andrewkroh Thanks. I had been looking at the previous 15 minutes or hour. At the moment it is running and has been since last Friday. The only thing that is different was I reinstalled Auditbeat on the Redhat system using a repository and yum install. Anything new happens or I need help with I'll raise a different thread.

I keep having to say thank you and I genuinely mean it.