Fields not appearing in Kibana which are defined in Logstash

Hi all,
I've defined some fields in logstash filter which is stored in /etc/logstash/conf.d/
The filter is as follows:

input {
beats {
port => 5044
}
}

filter {
if [type] == "log" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]

   match => { "message" => "%{LOGLEVEL:loglevel}\s+%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}%{SPACE}%{TIME:time}\,%{POSINT:just_no}\s+(\[%{WORD:module}\.py)\:%{POSINT:line_no}]\s+%{WORD:added-word} : %{WORD:method} : %{UUID:word} : %{WORD:service-type}: %{NUMBER:time}" }
  }

   syslog_pri { }
   date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }

}
}
output {
elasticsearch {
hosts => ["xxx:9200","xxx:9200"] //private-ip
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

There is no error in either logstash error files nor rubydebug is showing any error.
I am able to get the message which is defined but I am not able to get the fields also defined.
Kibana just shows default fields.
Here is the screenshot for the Kibana dashboard

I've tried refreshing, other things. It would be great if anybody could help

You're only applying your filters for events with the type "log" but the type is actually "doc". Make sure your Filebeat configuration is aligned with your Logstash configuration.

Hey thanks for replying, I tried with the 'doc' change but still the same issue.
And i have my filebeat.yml prospector configured as 'type:log' and the filter is as specified.

So what does an example document look like now?

The error log has the following format, i guess that is what you meant by 'doc'

INFO 28-12-2017 09:28:55,133 [views.py:43] ITR : post : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.13

No, what does it look like in Kibana? Copy/paste from the JSON tab.

Hi, the json looks as follows:

{
"_index": "filebeat-2018.03.15",
"_type": "doc",
"_id": "0AvNKGIBCuM7FZHjjvOB",
"_version": 1,
"_score": null,
"_source": {
"offset": 2049300,
"tags": [
"beats_input_codec_plain_applied"
],
"host": "ao-dev-entity-detection-37lf",
"source": "/tmp/log_files/entity_detection.log",
"prospector": {
"type": "log"
},
"@timestamp": "2018-03-15T08:36:21.674Z",
"beat": {
"name": "ao-dev-entity-detection-37lf",
"hostname": "ao-dev-entity-detection-37lf",
"version": "6.2.2"
},
"message": "INFO 15-03-2018 08:36:20,995 [entity_detection.py:186] ED : image_prediction : fbee26e1-96c3-43ac-b578-2f07388090ff : START: 1521102981.0",
"@version": "1"
},
"fields": {
"@timestamp": [
"2018-03-15T08:36:21.674Z"
]
},
"highlight": {
"beat.name": [
"ao-dev-entity-@kibana-highlighted-field@detection@/kibana-highlighted-field@-37lf"
],
"beat.hostname": [
"ao-dev-entity-@kibana-highlighted-field@detection@/kibana-highlighted-field@-37lf"
],
"host": [
"ao-dev-entity-@kibana-highlighted-field@detection@/kibana-highlighted-field@-37lf"
]
},
"sort": [
1521102981674
]
}

Okay. Not sure what's going on here.

I am not getting what is the '_type' here in the json. i.e '_type' : doc. I have set the filebeat to send it as log in the prospectors and in the json also you can see prospector type is 'log'. I guess document_type is set default to 'doc' in newer versions and is my filebeat not able to send it as a type: log?

Update:
I know if the string doesn't parse, it will give parsefailure tag but still, just for checking if the error is getting parsed or not, I tried parsing just one word i.e the log file looked like:
ERROR

So, the same issue exists, the message is getting parsed but still the document_type is I guess 'doc' even after setting it to 'log' in filebeat prospectors and the fields are still not appearing that are defined in the filter plugin of logstash.

I am not getting what is the '_type' here in the json. i.e '_type' : doc.

Here's the reason:

document_type => "%{[@metadata][type]}"

Removed that but still the _type is set to 'doc' and still not able to get the fields in Kibana.

To speed up the debugging I suggest you comment your elasticsearch output and use a stdout { codec => rubydebug } output. Please provide

  • your complete configuration,
  • an example line of input,
  • and the output of the stdout plugin configured as above.

Hi my configuration is as follows:
I am just testing with one node where ELK is installed and filebeat is on another system.
filebeat.yml -> stored in /etc/filebeat/filebeat.yml

filebeat.propectors:

  • type: log
    enabled: true
    paths:
    • /tmp/*.log
    • /var/log/syslog

output.logstash:
hosts: ["x.x.x.x:X", "x.x.x.x:X"]
loadbalance: true

logstash.conf:

input {
beats {
port => 5044
}
}

filter {
if [type] == "log" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]

   match => { "message" => "%{LOGLEVEL:loglevel}\s+%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}%{SPACE}%{TIME:time}\,%{POSINT:just_no}\s+(\[%{WORD:module}\.py)\:%{POSINT:line_no}]\s+%{WORD:added-word} : %{WORD:method} : %{UUID:word} : %{WORD:service-type}: %{NUMBER:time}" }
  }

   syslog_pri { }
   date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }

}
}
output {
elasticsearch {
hosts => ["xxx:9200","xxx:9200"] //private-ip
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}

Now with the stdout => rubydebug:
input: INFO 28-12-2017 09:28:55,133 [views.py:43] ITR : post : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.13

output:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-03-22 11:36:53.420 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-03-22 11:36:53.437 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[FATAL] 2018-03-22 11:36:53.484 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in block in validate_all'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:264:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in'"]}
[ERROR] 2018-03-22 11:36:53.495 [main] Logstash - java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

Even after setting the path settings, I am getting the error, but when i run logstash as service, I am not getting any error.

Either switch the data directory or run Logstash as a user who has write access to /usr/share/logstash/data.

I was running the command in /usr/share/logstash.

Hi, I am able to get the output through command line but the fields are not appearing in my Kibana:

The Logstash filter with stdin input:
input {

stdin{ }

beats {

port => 5044

}

}

filter {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]

   match => { "message" => "%{LOGLEVEL:loglevel}\s+%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}%{SPACE}%{TIME:time}\,%{POSINT:just_no}\s+(\[%{WORD:module}\.py)\:%{POSINT:line_no}]\s+%{WORD:added-word} : %{WORD:method} : %{UUID:word} : %{WORD:service-type}: %{NUMBER:time}" }
  }

   syslog_pri { }
   date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }

}

output {

stdout { codec => rubydebug }
elasticsearch {
hosts => ["hosts"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
}
}

This is the following input:
INFO 28-12-2017 09:28:55,133 [views.py:43] ITR : post : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.13

This is the output:
{
"just_no" => "135",
"line_no" => "36",
"@timestamp" => 2018-03-23T08:17:12.933Z,
"word" => "667075d6-03ca-4a75-b35c-9c718bced9b7",
"received_from" => "i-ao-elkcluster-node1-test",
"syslog_facility" => "user-level",
"@version" => "1",
"message" => "INFO 28-12-2017 09:28:55,135 [image_to_roi_service.py:36] ITR : image_to_roi_service : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.14",
"syslog_severity_code" => 5,
"added-word" => "ITR",
"loglevel" => "INFO",
"service-type" => "START",
"received_at" => "2018-03-23T08:17:12.933Z",
"time" => [
[0] "09:28:55",
[1] "1514453335.14"
],
"syslog_facility_code" => 1,
"method" => "image_to_roi_service",
"module" => "image_to_roi_service",
"syslog_severity" => "notice",
"host" => "i-ao-elkcluster-node1-test"
}

EDIT:
Now it is coming as tag, while i want them as fields.

Now it is coming as tag, while i want them as fields.

What do you mean? This looks perfectly fine to me. What are you expecting to get instead?

This is how it looks in the json tab:

{
"_index": "filebeat-2018.03.23",
"_type": "doc",
"_id": "3HgaUmIBQvRNa7ATylnJ",
"_version": 1,
"_score": null,
"_source": {
"@version": "1",
"syslog_severity": "notice",
"tags": [
"beats_input_codec_plain_applied"
],
"received_at": "2018-03-23T09:05:03.349Z",
"module": "views",
"@timestamp": "2018-03-23T09:05:03.349Z",
"syslog_facility": "user-level",
"offset": 5836579,
"word": "a6fed863-0f1d-4c9e-b842-314b83c9e25f",
"syslog_severity_code": 5,
"host": "i-ao-elkstack-withfilebeat-test",
"just_no": "29",
"service-type": "END",
"message": "INFO 28-12-2017 09:29:45,29 [views.py:83] ITR : post : a6fed863-0f1d-4c9e-b842-314b83c9e25f : END: 1514453385.03",
"loglevel": "INFO",
"method": "post",
"prospector": {
"type": "log"
},
"received_from": "i-ao-elkstack-withfilebeat-test",
"time": [
"09:29:45",
"1514453385.03"
],
"syslog_facility_code": 1,
"source": "/tmp/example4.log",
"beat": {
"hostname": "i-ao-elkstack-withfilebeat-test",
"version": "6.2.2",
"name": "i-ao-elkstack-withfilebeat-test"
},
"line_no": "83",
"added-word": "ITR"
},
"fields": {
"@timestamp": [
"2018-03-23T09:05:03.349Z"
],
"received_at": [
"2018-03-23T09:05:03.349Z"
]
},
"highlight": {
"source": [
"/@kibana-highlighted-field@tmp@/kibana-highlighted-field@/example4.log"
]
},
"sort": [
1521795903349
]
}

The info is coming, but its coming as tags, while I want them as fields.

The info is coming, but its coming as tags, while I want them as fields.

I repeat: I don't understand what you mean by that. What is the expected result? Show concrete examples instead of using ambiguous terms.

Hey, Thank You but that issue was resolved.