# Fields not appearing in Kibana which are defined in Logstash **URL:** \ **Category:** Logstash\ **Created:** [March 19, 2018, 11:14am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537 "2018-03-19T11:14:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 19, 2018, 11:14am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/1 "2018-03-19T11:14:27Z")
Hi all, I've defined some fields in logstash filter which is stored in /etc/logstash/conf.d/ The filter is as follows: input { beats { port =\> 5044 } } filter { if [type] == "log" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } add\_field =\> ["received\_at", "%{@timestamp}"] add\_field =\> ["received\_from", "%{host}"] ``` match => { "message" => "%{LOGLEVEL:loglevel}\s+%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}%{SPACE}%{TIME:time}\,%{POSINT:just_no}\s+(\[%{WORD:module}\.py)\:%{POSINT:line_no}]\s+%{WORD:added-word} : %{WORD:method} : %{UUID:word} : %{WORD:service-type}: %{NUMBER:time}" } } syslog_pri { } date { match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } ``` } } output { elasticsearch { hosts =\> ["xxx:9200","xxx:9200"] //private-ip sniffing =\> true manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } There is no error in either logstash error files nor rubydebug is showing any error. I am able to get the message which is defined but I am not able to get the fields also defined. Kibana just shows default fields. Here is the screenshot for the Kibana dashboard ![kibana](https://us1.discourse-cdn.com/elastic/original/3X/6/b/6b0b12b8d489b2bc67ef114b3c54d0ac81263764.png) I've tried refreshing, other things. It would be great if anybody could help ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 19, 2018, 1:28pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/2 "2018-03-19T13:28:59Z")
You're only applying your filters for events with the type "log" but the type is actually "doc". Make sure your Filebeat configuration is aligned with your Logstash configuration. ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 19, 2018, 2:03pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/3 "2018-03-19T14:03:01Z")
Hey thanks for replying, I tried with the 'doc' change but still the same issue. And i have my filebeat.yml prospector configured as 'type:log' and the filter is as specified. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 19, 2018, 2:21pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/4 "2018-03-19T14:21:14Z")
So what does an example document look like now? ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 20, 2018, 8:49am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/5 "2018-03-20T08:49:54Z")
The error log has the following format, i guess that is what you meant by 'doc' INFO 28-12-2017 09:28:55,133 [views.py:43] ITR : post : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.13 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 20, 2018, 6:14pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/6 "2018-03-20T18:14:56Z")
No, what does it look like in Kibana? Copy/paste from the JSON tab. ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 21, 2018, 4:28am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/7 "2018-03-21T04:28:11Z")
Hi, the json looks as follows: { "\_index": "filebeat-2018.03.15", "\_type": "doc", "\_id": "0AvNKGIBCuM7FZHjjvOB", "\_version": 1, "\_score": null, "\_source": { "offset": 2049300, "tags": [ "beats\_input\_codec\_plain\_applied" ], "host": "ao-dev-entity-detection-37lf", "source": "/tmp/log\_files/entity\_detection.log", "prospector": { "type": "log" }, "@timestamp": "2018-03-15T08:36:21.674Z", "beat": { "name": "ao-dev-entity-detection-37lf", "hostname": "ao-dev-entity-detection-37lf", "version": "6.2.2" }, "message": "INFO 15-03-2018 08:36:20,995 [entity\_detection.py:186] ED : image\_prediction : fbee26e1-96c3-43ac-b578-2f07388090ff : START: 1521102981.0", "@version": "1" }, "fields": { "@timestamp": [ "2018-03-15T08:36:21.674Z" ] }, "highlight": { "beat.name": [ "ao-dev-entity-@kibana-highlighted-field@detection@/kibana-highlighted-field@-37lf" ], "beat.hostname": [ "ao-dev-entity-@kibana-highlighted-field@detection@/kibana-highlighted-field@-37lf" ], "host": [ "ao-dev-entity-@kibana-highlighted-field@detection@/kibana-highlighted-field@-37lf" ] }, "sort": [ 1521102981674 ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 21, 2018, 8:12pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/8 "2018-03-21T20:12:51Z")
Okay. Not sure what's going on here. ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 22, 2018, 4:25am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/9 "2018-03-22T04:25:02Z")
I am not getting what is the '\_type' here in the json. i.e '\_type' : doc. I have set the filebeat to send it as log in the prospectors and in the json also you can see prospector type is 'log'. I guess document\_type is set default to 'doc' in newer versions and is my filebeat not able to send it as a type: log? Update: I know if the string doesn't parse, it will give parsefailure tag but still, just for checking if the error is getting parsed or not, I tried parsing just one word i.e the log file looked like: ERROR So, the same issue exists, the message is getting parsed but still the document\_type is I guess 'doc' even after setting it to 'log' in filebeat prospectors and the fields are still not appearing that are defined in the filter plugin of logstash. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 22, 2018, 6:49am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/10 "2018-03-22T06:49:51Z")
> I am not getting what is the '\_type' here in the json. i.e '\_type' : doc. Here's the reason: ``` document_type => "%{[@metadata][type]}" ``` ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 22, 2018, 6:56am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/11 "2018-03-22T06:56:08Z")
Removed that but still the \_type is set to 'doc' and still not able to get the fields in Kibana. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 22, 2018, 6:58am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/12 "2018-03-22T06:58:33Z")
To speed up the debugging I suggest you comment your elasticsearch output and use a `stdout { codec => rubydebug }` output. Please provide - your complete configuration, - an example line of input, - and the output of the stdout plugin configured as above. ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 22, 2018, 11:58am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/14 "2018-03-22T11:58:30Z")
Hi my configuration is as follows: I am just testing with one node where ELK is installed and filebeat is on another system. filebeat.yml -\> stored in /etc/filebeat/filebeat.yml filebeat.propectors: - type: log enabled: true paths: - /tmp/\*.log - /var/log/syslog output.logstash: hosts: ["x.x.x.x:X", "x.x.x.x:X"] loadbalance: true logstash.conf: input { beats { port =\> 5044 } } filter { if [type] == "log" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } add\_field =\> ["received\_at", "%{@timestamp}"] add\_field =\> ["received\_from", "%{host}"] ``` match => { "message" => "%{LOGLEVEL:loglevel}\s+%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}%{SPACE}%{TIME:time}\,%{POSINT:just_no}\s+(\[%{WORD:module}\.py)\:%{POSINT:line_no}]\s+%{WORD:added-word} : %{WORD:method} : %{UUID:word} : %{WORD:service-type}: %{NUMBER:time}" } } syslog_pri { } date { match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } ``` } } output { elasticsearch { hosts =\> ["xxx:9200","xxx:9200"] //private-ip sniffing =\> true manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" } } Now with the stdout =\> rubydebug: input: INFO 28-12-2017 09:28:55,133 [views.py:43] ITR : post : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.13 output: WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2018-03-22 11:36:53.420 [main] scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} [INFO] 2018-03-22 11:36:53.437 [main] scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [FATAL] 2018-03-22 11:36:53.484 [main] runner - An unexpected error occurred! {:error=\>#\, :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:in`validate\_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:264:in`execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:in`run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in`'"]} [ERROR] 2018-03-22 11:36:53.495 [main] Logstash - java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit Even after setting the path settings, I am getting the error, but when i run logstash as service, I am not getting any error. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 22, 2018, 12:10pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/15 "2018-03-22T12:10:14Z")
Either switch the data directory or run Logstash as a user who has write access to /usr/share/logstash/data. ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 22, 2018, 12:19pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/16 "2018-03-22T12:19:17Z")
I was running the command in /usr/share/logstash. ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 23, 2018, 6:32am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/17 "2018-03-23T06:32:16Z")
Hi, I am able to get the output through command line but the fields are not appearing in my Kibana: The Logstash filter with stdin input: input { stdin{ } # beats { # port =\> 5044 # } } filter { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } add\_field =\> ["received\_at", "%{@timestamp}"] add\_field =\> ["received\_from", "%{host}"] ``` match => { "message" => "%{LOGLEVEL:loglevel}\s+%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}%{SPACE}%{TIME:time}\,%{POSINT:just_no}\s+(\[%{WORD:module}\.py)\:%{POSINT:line_no}]\s+%{WORD:added-word} : %{WORD:method} : %{UUID:word} : %{WORD:service-type}: %{NUMBER:time}" } } syslog_pri { } date { match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } ``` } output { stdout { codec =\> rubydebug } elasticsearch { hosts =\> ["hosts"] sniffing =\> true manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" } } This is the following input: INFO 28-12-2017 09:28:55,133 [views.py:43] ITR : post : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.13 This is the output: { "just\_no" =\> "135", "line\_no" =\> "36", "@timestamp" =\> 2018-03-23T08:17:12.933Z, "word" =\> "667075d6-03ca-4a75-b35c-9c718bced9b7", "received\_from" =\> "i-ao-elkcluster-node1-test", "syslog\_facility" =\> "user-level", "@version" =\> "1", "message" =\> "INFO 28-12-2017 09:28:55,135 [image\_to\_roi\_service.py:36] ITR : image\_to\_roi\_service : 667075d6-03ca-4a75-b35c-9c718bced9b7 : START: 1514453335.14", "syslog\_severity\_code" =\> 5, "added-word" =\> "ITR", "loglevel" =\> "INFO", "service-type" =\> "START", "received\_at" =\> "2018-03-23T08:17:12.933Z", "time" =\> [ [0] "09:28:55", [1] "1514453335.14" ], "syslog\_facility\_code" =\> 1, "method" =\> "image\_to\_roi\_service", "module" =\> "image\_to\_roi\_service", "syslog\_severity" =\> "notice", "host" =\> "i-ao-elkcluster-node1-test" } EDIT: Now it is coming as tag, while i want them as fields. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 24, 2018, 8:12pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/18 "2018-03-24T20:12:29Z")
> Now it is coming as tag, while i want them as fields. What do you mean? This looks perfectly fine to me. What are you expecting to get instead? ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [March 25, 2018, 6:30am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/19 "2018-03-25T06:30:10Z")
This is how it looks in the json tab: { "\_index": "filebeat-2018.03.23", "\_type": "doc", "\_id": "3HgaUmIBQvRNa7ATylnJ", "\_version": 1, "\_score": null, "\_source": { "@version": "1", "syslog\_severity": "notice", "tags": [ "beats\_input\_codec\_plain\_applied" ], "received\_at": "2018-03-23T09:05:03.349Z", "module": "views", "@timestamp": "2018-03-23T09:05:03.349Z", "syslog\_facility": "user-level", "offset": 5836579, "word": "a6fed863-0f1d-4c9e-b842-314b83c9e25f", "syslog\_severity\_code": 5, "host": "i-ao-elkstack-withfilebeat-test", "just\_no": "29", "service-type": "END", "message": "INFO 28-12-2017 09:29:45,29 [views.py:83] ITR : post : a6fed863-0f1d-4c9e-b842-314b83c9e25f : END: 1514453385.03", "loglevel": "INFO", "method": "post", "prospector": { "type": "log" }, "received\_from": "i-ao-elkstack-withfilebeat-test", "time": [ "09:29:45", "1514453385.03" ], "syslog\_facility\_code": 1, "source": "/tmp/example4.log", "beat": { "hostname": "i-ao-elkstack-withfilebeat-test", "version": "6.2.2", "name": "i-ao-elkstack-withfilebeat-test" }, "line\_no": "83", "added-word": "ITR" }, "fields": { "@timestamp": [ "2018-03-23T09:05:03.349Z" ], "received\_at": [ "2018-03-23T09:05:03.349Z" ] }, "highlight": { "source": [ "/@kibana-highlighted-field@tmp@/kibana-highlighted-field@/example4.log" ] }, "sort": [ 1521795903349 ] } The info is coming, but its coming as tags, while I want them as fields. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 25, 2018, 4:26pm UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/20 "2018-03-25T16:26:58Z")
> The info is coming, but its coming as tags, while I want them as fields. I repeat: I don't understand what you mean by that. What is the expected result? Show concrete examples instead of using ambiguous terms. ---
**Author:** ![falsekingpin](https://avatars.discourse-cdn.com/v4/letter/f/aca169/32.png) [@falsekingpin](https://discuss.elastic.co/u/falsekingpin)\ **Post date:** [April 4, 2018, 9:37am UTC](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537/21 "2018-04-04T09:37:11Z")
Hey, Thank You but that issue was resolved. [Next page](https://discuss.elastic.co/t/fields-not-appearing-in-kibana-which-are-defined-in-logstash/124537.md?page=2)