JSON filter, query

[runtman (Anthony Cleaves)]

Hello, I am receiving alerts from SOPHOS in clear JSON, below is an example:

{"end": "2017-11-15T21:35:18.000Z", "customer_id": "x", "endpoint_type": "computer", "dhost": "x", "type": "Event::Endpoint::WebControlViolation", "name": "'https://static.site24x7rum.com' warned due to category 'Alcohol & Tobacco'", "rt": "2017-11-15T21:35:21.942Z", "severity": "low", "duid": "5771219d2fa3360f8915b74f", "suser": "x", "group": "x", "id": "x", "whitelist_properties": {}, "endpoint_id": "x"}

Reading up I should be able to use the JSON filter, I am using the following:

filter {
  if [type] == "sophos.logs" {
    json {
      source => "message"
    }
  }
}

However when this goes into kibana, nothing is filtered. The entire message is a single field, is this to be expected?

[Christian_Dahlqvist]

@runtman

if [type] == "sophos.logs" {

Is this what the type field is set to on the events that does not get parsed correctly?

[runtman (Anthony Cleaves)]

Yes,

t fields.log_type sophos.logs

Filebeat config for more clarity:

- input_type: log
  paths:
    - /root/repos/Sophos-Central-SIEM-Integration/log/result.txt
  fields:
    source: syslog-server
    log_type: sophos.logs
    environment: x
  tags: ['sophos-logs']

[Christian_Dahlqvist]

Can you show what an event looks like once it is in Kibana (please cut and paste)?

[runtman (Anthony Cleaves)]

The white parts are just text that I don't want to share

Edit:
Is it because for raw data is being escaped (Not sure why)

"message": "{\"suser\": \"x", \"endpoint_type\": \"computer\", \"id\": \"x\", \"severity\": \"low\", \"dhost\": \"x\", \"customer_id\": \"x\", \"rt\": \"2017-11-16T10:30:49.666Z\", \"whitelist_properties\": {}, \"duid\": \"5x\", \"endpoint_id\": \"x\", \"name\": \"User trusted low reputation download from x\", \"group\": \"DOWNLOAD_REPUTATION\", \"end\": \"2017-11-16T10:30:49.666Z\", \"type\": \"Event::Endpoint::DownloadReputationUserAuthorised\"}",

[magnusbaeck (Magnus Bäck)]

type is set to "log" in your case so your conditional is wrong. It's [fields][log_type] that you're setting to "sophos.logs" so you either have to change your conditional or rename the field. Perhaps you should consider setting Filebeat's fields_under_root option to true.

[runtman (Anthony Cleaves)]

Oh my, no idea how I didn't spot that. I will re-test, thanks.

[runtman (Anthony Cleaves)]

What a moron, thank you so much.