# Fields are not being parsed in new version of kibna **URL:** \ **Category:** Logstash\ **Created:** [April 4, 2018, 1:45pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748 "2018-04-04T13:45:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 4, 2018, 1:45pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/1 "2018-04-04T13:45:43Z")
Hi fileds such as ApplicationPool , ConnectedServerPort , bytesReceivedFromClient , httpResponse , uriPathWithParam etc are not being segregated, in kibana, due to which we are unable to perform visualizations, based on the above fields. ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/4/6425dae8a6bd36972d997c1a8c503e5b8aa19b38.png) i have been able to do the same in the older version of kibana as attached ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/9/f9acb8dc5692a5c6bca488bac38cb96a035ddb39.png) Please can some one guide me Thanks ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 4, 2018, 3:48pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/2 "2018-04-04T15:48:00Z")
If screenshot 1 is from latest version of Kibana, I dont even see the said fields in table. Please check that log message getting parsed or not. Please paste entire json tab data. And if possible do a comparison between Old and New Kibana JSON data ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 5, 2018, 7:46am UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/3 "2018-04-05T07:46:11Z")
old version Json Data { "\_index": "logstash-2018.04.05", "\_type": "ZXTMLogs", "\_id": "AWKUwjnmHZOn2S2gqLu0", "\_score": 1, "\_source": { "clientIpAddress": "10.187.146.191", "totalTimeTakenForClient": 0.044582, "requestTimestamp": "05/Apr/2018:08:41:03 +0100", "timeTakenToConstructResponse": 0.044582, "httpmethod": "POST", "message": "0.044582 10.187.146.191 [05/Apr/2018:08:41:03 +0100] VS\_UKBDR\_Portal,MGD&Filerepository\_10.187.174.33\_80 - "urn:BT.MGD.Runtime.API.IMGDService/IMGDService/GetNext" POST /Service.svc 200 1131 1200 0.044582 10.187.133.0:61057 PL\_UKBDR\_MGDService\_61057 -", "type": "ZXTMLogs", "VirtualServer": "VS\_UKBDR\_Portal,MGD&Filerepository\_10.187.174.33\_80", "path": "F:/UKBlogsstachconfiguration/Logstobestached/ZTMLogpath/05\_Apr\_2018\_08\_41\_DR.log", "ConnectedServerPort": "61057", "@timestamp": "2018-04-05T07:42:59.480Z", "ApplicationPool": "PL\_UKBDR\_MGDService\_61057", "bytesReceivedFromClient": 1131, "@version": "1", "host": "BWD10427003", "bytesSentToClient": 1200, "uriPathWithParam": "/Service.svc", "httpResponse": 200, "SoapAction": ""urn:BT.MGD.Runtime.API.IMGDService/IMGDService/GetNext"", "Connected\_Server": "10.187.133.0" }, "fields": { "@timestamp": [ 1522914179480 ] } } new version { "\_index": "logstash-2018.04.05", "\_type": "doc", "\_id": "o7u7lGIB\_8p3Tx4A4ddl", "\_score": 1, "\_source": { "offset": 370390, "@timestamp": "2018-04-05T07:36:03.197Z", "source": "/software/stm/zxtm/log/VS\_UKB\_Supportapp.log", "prospector": { "type": "log" }, "message": "1.072272 147.151.218.102 [05/Apr/2018:08:35:56 +0100] VS\_UKB\_Supportapp - - GET /Download.aspx?downloadType=ChatLog&fromDate=04-04-2018&toDate=04-04-2018 200 2025 145259 1.068775 10.54.33.104:61016 PL\_Supportchat 10.54.33.104:61016", "tags": [ "beats\_input\_codec\_plain\_applied" ], "@version": "1", "beat": { "hostname": "blp10427011", "name": "blp10427011", "version": "6.2.3" }, "host": "blp10427011" }, "fields": { "@timestamp": [ "2018-04-05T07:36:03.197Z" ] } } and i am using the same logstash config in both the cases ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 5, 2018, 8:11am UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/4 "2018-04-05T08:11:13Z")
I am not sure whats happening. But, your logstash grok is unable to parse your message. Thats the reason in your latest Kibana json you dont see any of those fields. and as thr fields are not available your Kibana analytics is failing. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 5, 2018, 8:23am UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/5 "2018-04-05T08:23:14Z")
It doesn't look like any grok filter is applied. If it was failing I would expect to see a grok parse failure tag on the document. Have you changed your Logstash config? ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 5, 2018, 11:42am UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/6 "2018-04-05T11:42:53Z")
can i know where i can see grok failure logs? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 5, 2018, 11:50am UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/7 "2018-04-05T11:50:37Z")
You should be able to see a tag on the document indicating that it failed. The only tag you have is `beats_input_codec_plain_applied`, which is why I suspect grok may not have been applied at all. If you look at the structure of the documents, it does not contain the fields typically added by filebeat, so it is possible this was collected and processed directly by Logstash. The new record does however seem to have been collected by Filebeat as additional fields are available here. If Filebeat was configured to write directly to Elasticsearch instead of sending data through Logstash, that might explain why no grok processing has taken place. ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 5, 2018, 1:48pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/8 "2018-04-05T13:48:16Z")
ok so will try to redirect to elastic search and try ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 5, 2018, 1:49pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/9 "2018-04-05T13:49:05Z")
hi can you suggest how can i debug grok ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 5, 2018, 1:50pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/10 "2018-04-05T13:50:48Z")
Is the data going through Logstash? It looks like Filebeat might be sending it directly to Elasticsearch, bypassing the grok processing in Logstash. ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 5, 2018, 3:02pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/11 "2018-04-05T15:02:59Z")
Please paste us your Logstash and Filebeat config. Also pleae add stdout{codec=\>rubydebug} tou your output. Also as Chris said, it might be directly sending to ES intance. Please verify ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 6, 2018, 3:45pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/12 "2018-04-06T15:45:22Z")
file beat config as below #=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - type: log #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ ##output.elasticsearch: # Array of hosts to connect to # hosts: ["10.52.33.97:63354"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["10.52.33.97:63354"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key ## #ssl.key: "/etc/pki/client/cert.key" ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 6, 2018, 3:47pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/13 "2018-04-06T15:47:46Z")
Your filebeat is direclty posting to Elasticsearch. It should be logstash if you want to parse your message. And I see your logstash host and elastic host and port numbers are same. Please check that also! ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 6, 2018, 5:53pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/14 "2018-04-06T17:53:08Z")
oh may be i forgot to comment elastic search hosts, will that work if i comment the whole section of elastic search or remove ot completely ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 6, 2018, 11:23pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/15 "2018-04-06T23:23:14Z")
It should work, given that you are giving proper logstash host details. ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 11, 2018, 12:38pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/16 "2018-04-11T12:38:24Z")
###################### Filebeat Configuration Example ######################### # This file is an example configuration file highlighting only the most common # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # [https://www.elastic.co/guide/en/beats/filebeat/index.html](https://www.elastic.co/guide/en/beats/filebeat/index.html) # For more available modules and options, please see the filebeat.reference.yml sample # configuration file. #=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - type: log #============================= Filebeat modules =============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/\*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s #==================== Elasticsearch template setting ========================== setup.template.settings: index.number\_of\_shards: 3 #index.codec: best\_compression #\_source.enabled: false #================================ General ===================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging #============================== Dashboards ===================================== # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here, or by using the `-setup` CLI flag or the `setup` command. #setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the [artifacts.elastic.co](http://artifacts.elastic.co) # website. #setup.dashboards.url: #============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: [http://localhost:5601/path](http://localhost:5601/path) # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" #============================= Elastic Cloud ================================== # These settings simplify using filebeat with the Elastic Cloud ([https://cloud.elastic.co/](https://cloud.elastic.co/)). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `:`. #cloud.auth: #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["10.52.33.97:63354"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: critical, error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["\*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["\*"] i am not trying to put this beaqtconfic, please can you verify if its fine or any corrections required. ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 11, 2018, 12:46pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/17 "2018-04-11T12:46:48Z")
i am now trying to put this beat config, please can you verify if its fine or any corrections required. ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 11, 2018, 12:50pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/18 "2018-04-11T12:50:47Z")
> [@kolli\_dilip](#): > > i am now trying to put this beat config, please can you verify if its fine or any corrections required. Looks good to me, check that Logstash input is pointed to this port 63354 ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 11, 2018, 1:01pm UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/19 "2018-04-11T13:01:29Z")
yea log stash id listening to beats at 63354, thanks for checking, will update once i am able to apply it ---
**Author:** ![kolli\_dilip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kolli_dilip/32/33821_2.png) [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Post date:** [April 12, 2018, 10:02am UTC](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748/20 "2018-04-12T10:02:47Z")
i later corrected and finalised beatconfig as below ###################### Filebeat Configuration Example ######################### ``` # This file is an example configuration file highlighting only the most common # options. The filebeat.reference.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # https://www.elastic.co/guide/en/beats/filebeat/index.html # For more available modules and options, please see the filebeat.reference.yml sample # configuration file. ``` #=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - type: log #============================= Filebeat modules =============================== #filebeat.config.modules: # Glob pattern for configuration loading #path: ${path.config}/modules.d/\*.yml # Set to true to enable config reloading #reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s #==================== Elasticsearch template setting ========================== # setup.template.settings: #index.number\_of\_shards: 3 #index.codec: best\_compression #\_source.enabled: false #================================ General ===================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging #============================== Dashboards ===================================== # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here, or by using the `-setup` CLI flag or the `setup` command. #setup.dashboards.enabled: false # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the [artifacts.elastic.co](http://artifacts.elastic.co) # website. #setup.dashboards.url: #============================== Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. #setup.kibana: # Kibana Host # Scheme and port can be left out and will be set to the default (http and 5601) # In case you specify and additional path, the scheme is required: [http://localhost:5601/path](http://localhost:5601/path) # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 #host: "localhost:5601" #============================= Elastic Cloud ================================== # These settings simplify using filebeat with the Elastic Cloud ([https://cloud.elastic.co/](https://cloud.elastic.co/)). # The cloud.id setting overwrites the `output.elasticsearch.hosts` and # `setup.kibana.host` options. # You can find the `cloud.id` in the Elastic Cloud web UI. #cloud.id: # The cloud.auth setting overwrites the `output.elasticsearch.username` and # `output.elasticsearch.password` settings. The format is `:`. #cloud.auth: #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. #hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["10.52.33.97:63354"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: critical, error, warning, info, debug #logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["\*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["\*"] * * * now the data is back on kibana, but still its not getting parsed as per my expectation [Next page](https://discuss.elastic.co/t/fields-are-not-being-parsed-in-new-version-of-kibna/126748.md?page=2)