Logstash Config File Error2

Elastic Stack Logstash
1

Can someone please review this file for errors? I have been over it and over it and I keep getting this error:
[2019-03-01T14:46:07,935][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 3076, column 20 (byte 99857) after filter {\n if [syslog-host_from] =~ "fireeye-nx" {\n json {\n source => "message"\n \n mutate ",

I have removed every mutate statement down to the last one and I still get the same error. Here is the original file. Does anyone see an issue?

filter {
if [syslog-host_from] =~ "fireeye-nx" {
json {
source => "message"

        mutate {
           remove_field => [ "version", "appliance-id", "msg", "product", "occurred", "vlan", "class", "name", "interface", "ack" ]
        }
        mutate {
           rename => { "appliance" => "server_name" }
           rename => [ "[alert][dst][ip]", "destination_ip" ]
           rename => [ "[alert][dst][port]", "destination_port" ]
           rename => [ "[alert][dst][mac]", "destination_mac" ]
           rename => { "alert-url" => "signature_info" }
           rename => { "name" => "rule_type" }
           #action
           rename => [ "[action][id]", "sid" ]
           rename => [ "[action][severity]", "priority" ]
           rename => [ "[action][uuid]", "uid" ]
           rename => [ "[src][ip]", "source_ip" ]
           rename => [ "[src][port]", "source_port" ]
           rename => [ "[src][mac]", "source_mac" ]
           rename => [ "[explanation][ips-detected][cve-id]", "cve" ]
           rename => [ "[explanation][ips-detected][match-count]", "match-count" ]
           rename => [ "[explanation][ips-detected][attack-mode]", "attack-mode" ]
           rename => [ "[explanation][ips-detected][action-taken]", "action" ]
           rename => [ "[explanation][ips-detected][sig-id]", "sid" ]
           rename => [ "[explanation][ips-detected][mvx-status]", "mvx-status" ]
           rename => [ "[explanation][ips-detected][sig-revision]", "rev" ]
           rename => [ "[explanation][ips-detected][sig-name]", "alert" ]
         }
		 
		 mutate {
           replace => { "type" => "fireeye" }
  }         
 }
}

}

2

You have a mutate filter nested inside a json filter. You need to add a } to close the json filter before the mutate filter.

3

I actually added that change after a previous error. I will revert that change and post new error. Thank you for the quick response.

4

New File:

filter {
if [syslog-host_from] =~ "fireeye-nx" {
json {
source => "message"
}
mutate {
remove_field => [ "version", "appliance-id", "msg", "product", "occurred", "vlan", "class", "name", "interface", "ack" ]
}
mutate {
rename => { "appliance" => "server_name" }
rename => [ "[alert][dst][ip]", "destination_ip" ]
rename => [ "[alert][dst][port]", "destination_port" ]
rename => [ "[alert][dst][mac]", "destination_mac" ]
rename => { "alert-url" => "signature_info" }
rename => { "name" => "rule_type" }
#action
rename => [ "[action][id]", "sid" ]
rename => [ "[action][severity]", "priority" ]
rename => [ "[action][uuid]", "uid" ]
rename => [ "[src][ip]", "source_ip" ]
rename => [ "[src][port]", "source_port" ]
rename => [ "[src][mac]", "source_mac" ]
rename => [ "[explanation][ips-detected][cve-id]", "cve" ]
rename => [ "[explanation][ips-detected][match-count]", "match-count" ]
rename => [ "[explanation][ips-detected][attack-mode]", "attack-mode" ]
rename => [ "[explanation][ips-detected][action-taken]", "action" ]
rename => [ "[explanation][ips-detected][sig-id]", "sid" ]
rename => [ "[explanation][ips-detected][mvx-status]", "mvx-status" ]
rename => [ "[explanation][ips-detected][sig-revision]", "rev" ]
rename => [ "[explanation][ips-detected][sig-name]", "alert" ]
}

         mutate {
           replace => { "type" => "fireeye" }
  }
 }
}

Error:
[2019-03-01T15:58:56,463][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"TypeError", :message=>"no implicit conversion of Array into Hash",

5

The first rename is OK, rename expects a hash. The second is not, since it contains an array.

6

How would I go about fixing this? Would I need a separate mutate statement for arrays?

7

To provide context the fields are nested in the log:

[alert][dst][ip] is actually

alert:
dst:
ip
port
mac

8

That should be

 rename => { "[alert][dst][ip]" => "destination_ip" }
9

OK, so that definitely fixed the config file. New issue though, the logs are not showing up in Kibana anymore and when they are ingested Logstash is throwing the following alert:

[2019-03-01T18:04:46,124][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-syslog-2019.03.01", :_type=>"doc", :routing=>nil}, #LogStash::Event:0x64390edd], :response=>{"index"=>{"_index"=>"logstash-syslog-2019.03.01", "_type"=>"doc", "_id"=>"dtBuOmkBH9aLgbCWEjuq", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [alert] of type [text]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:174"}}}}}

My guess is that the "alert" field that it mentions is coming back blank as this is the parent field of many nested fields. See log below:
{
"alert": {
"ack": "no",
"action": "blocked",
"alert-url": "https://hexxxxx-cms-ssh.hex01.helix.apps.xxxxxx.xxx/event_stream/events_for_bot?ev_id=1450064&lms_iden=0025905E4418",
"dst": {
"ip": "x.x.x.x"
},
"explanation": {
"malware-detected": {
"malware": {
"name": "Phish.URL"
}
}
},
"id": "1450064",
"name": "infection-match",
"occurred": "2019-03-01T17:49:09Z",
"severity": "minr",
"src": {
"host": "xxxxxxx",
"ip": "x.x.x.x",
"vlan": "0"
},
"uuid": "95486f3b-f231-4bd1-xxxx-173bxxxxxx19"
},
"appliance": "xxxxxxx",
"appliance-id": "xxxxxxx",
"msg": "concise",
"product": "xxxxxxx",
"version": "xxxxxxxx"
}

10

Is the error message in the elasticsearch log file more informative?

11

Im afraid elastic doesn't generate a log for this event at all.

12

So elasticsearch expects alert to be a text field, not a structured object. It has to be one or the other. What does alert look like on the records already in elasticsearch?

13

It is starting to look more and more like this log source needs a dedicated template. I was trying to avoid that by renaming fields to match the existing template.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.