# Logstash parsing data incorrectly **URL:** \ **Category:** Logstash\ **Created:** [March 12, 2024, 9:43am UTC](https://discuss.elastic.co/t/logstash-parsing-data-incorrectly/355221 "2024-03-12T09:43:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![gyrao\_72](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gyrao_72/32/108254_2.png) [@gyrao\_72](https://discuss.elastic.co/u/gyrao_72)\ **Post date:** [March 12, 2024, 9:43am UTC](https://discuss.elastic.co/t/logstash-parsing-data-incorrectly/355221/1 "2024-03-12T09:43:32Z")
0 I am using Logstash version **6.5.4** Data is read and parsed using Filebeat -\> Logstash -\> Elasticsearch But as I can see on Kibana for a few cases, I am getting duplicate field data i.e data is converted into an array of duplicate field data separated by a comma . My message sent from filebeat was: ```auto 2024-03-12 13:27:00.126,d85b4ecb-c4b7-4168-bcc1-9b6a3508ce4a,System,NotificationsProcessor,MQ_TO_JOBSERVICE,Tue Mar 12 13:27:00 IST 2024,TENANT,,null,SUCCESSFUL,94,126,'Task completed successfully',TESTING,null,'{"notificationsCount":31}' ``` ```auto { "_version": 3, "_source": { "jobSpecificMetaData": [ "{\"notificationsCount\":31}", "{\"notificationsCount\":31}", "{\"notificationsCount\":31}" ], "current_step_time_ms": 94, "request_id": [ "d85b4ecb-c4b7-4168-bcc1-9b6a3508ce4a", "d85b4ecb-c4b7-4168-bcc1-9b6a3508ce4a", "d85b4ecb-c4b7-4168-bcc1-9b6a3508ce4a" ], "tenant_mode": [ "TESTING", "TESTING", "TESTING" ], "total_time_ms": 126, "created_timestamp": [ "2024-03-12 13:27:00.126", "2024-03-12 13:27:00.126", "2024-03-12 13:27:00.126" ], "@timestamp": "2024-03-12T07:57:02.887Z", "execution_level": [ "TENANT", "TENANT", "TENANT" ], "status": [ "SUCCESSFUL", "SUCCESSFUL", "SUCCESSFUL" ] } } ``` I am unable to debug this issue. Kindly help as this is breaking my dashboard and reporting. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 12, 2024, 11:13am UTC](https://discuss.elastic.co/t/logstash-parsing-data-incorrectly/355221/2 "2024-03-12T11:13:18Z")
You need to share both your `filebeat.yml` and your logstash configuration. ---
**Author:** ![gyrao\_72](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gyrao_72/32/108254_2.png) [@gyrao\_72](https://discuss.elastic.co/u/gyrao_72)\ **Post date:** [March 12, 2024, 11:23am UTC](https://discuss.elastic.co/t/logstash-parsing-data-incorrectly/355221/3 "2024-03-12T11:23:41Z")
Filebeat ```auto filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /usr/local/apache-tomcat/logs/access_log*.txt fields: release_version: 1619 #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["host1","host2"] loadbalance: true ``` Logstash: ```auto input { beats { port => 5044 } } filter{ mutate { remove_field => ["offset","prospector","input","version","tags"] add_field => { "host" => "%{[beat][name]}" } } grok { match => { "message" => "\[%{USER:tenant} (?-|[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)(?:\s+%{DATA:api_user_name})?\] %{IP:client_ip} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} %{NOTSPACE:url} HTTP/%{NUMBER:http_version}\" \[%{GREEDYDATA:OperationName}\] \[%{GREEDYDATA:useragent}\] %{NUMBER:server_response} %{NUMBER:bytes_sent} %{NUMBER:req_process_ms} \[%{GREEDYDATA:androidAppReleaseVersion}\] \[%{GREEDYDATA:androidDeviceUUID}\] \[%{GREEDYDATA:egress_time} %{GREEDYDATA:script_excl_egress_time} %{GREEDYDATA:channel_src_code} %{GREEDYDATA:courier_src_code}\] \[%{DATA:api_status} %{DATA:request_identifier} %{DATA:error_response_code} %{GREEDYDATA:error_response_message}\]" } match => { "message" => "\[%{USER:tenant} (?-|[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)(?:\s+%{DATA:api_user_name})?\] %{IP:client_ip} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} %{NOTSPACE:url} HTTP/%{NUMBER:http_version}\" \[%{GREEDYDATA:OperationName}\] \[%{GREEDYDATA:useragent}\] %{NUMBER:server_response} %{NUMBER:bytes_sent} %{NUMBER:req_process_ms} \[%{GREEDYDATA:androidAppReleaseVersion}\] \[%{GREEDYDATA:androidDeviceUUID}\] \[%{GREEDYDATA:egress_time} %{GREEDYDATA:script_excl_egress_time} %{GREEDYDATA:channel_src_code} %{GREEDYDATA:courier_src_code}\]"} match => { "message" => "\[%{USER:tenant} (?-|[a-zA-Z0-9_.+=:-]+@[0-9A-Za-z][0-9A-Za-z-]{0,62}(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*)\] %{IP:client_ip} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} %{NOTSPACE:url} HTTP/%{NUMBER:http_version}\" \[%{GREEDYDATA:OperationName}\] \[%{GREEDYDATA:useragent}\] %{NUMBER:server_response} %{NUMBER:bytes_sent} %{NUMBER:req_process_ms} \[%{GREEDYDATA:androidAppReleaseVersion}\] \[%{GREEDYDATA:androidDeviceUUID}\]" } } if [useragent] != "-" and [useragent] != "" { useragent { add_tag => ["UA"] source => "useragent" } } if [tenant] == '-'{ drop { } } if "_grokparsefailure" in [tags] { drop { } } ruby { code => " event.set('index-name', event.get('source').split('_log.').last.split('.').first); if event.get('OperationName') != '-' && event.get('url') != nil if event.get('url').include?'/services/soap/?' event.set('url', '/services/soap/' + event.get('OperationName') + event.get('url').split('/services/soap/').last); elsif event.get('url').include?'/services/soap/' event.set('url', '/services/soap/' + event.get('OperationName') + '/' + event.get('url').split('/services/soap/').last); else event.set('url', '/services/soap/' + event.get('OperationName') + event.get('url').split('/services/soap').last); end end if event.get('url').include?'?' event.set('base_url',event.get('url').split('?').first); event.set('query_params',event.get('url').split('?').last); else event.set('base_url',event.get('url')); end event.set('cloud',event.get('host').split('.',2).last.split('.').first); if event.get('cloud').start_with?'cloud' event.set('client_type','seller'); else event.set('client_type','enterprise'); end if event.get('base_url').include?'soap' event.set('api_group','soap'); else event.set('api_group','rest'); end if event.get('egress_time') != nil && event.get('egress_time') == '-' event.set('egress_time','0'); end if event.get('script_excl_egress_time') != nil && event.get('script_excl_egress_time') == '-' event.set('script_excl_egress_time','0'); end if event.get('error_response_code') != nil && event.get('error_response_code') == '-' event.set('error_response_code','0'); end " } mutate { convert => { "req_process_ms" => "integer" "egress_time" => "integer" "script_excl_egress_time" => "integer" } } ruby { code => " event.set('req_prcoess_excl_script_time',event.get('req_process_ms') - event.get('egress_time') - event.get('script_excl_egress_time')); " } } } output { elasticsearch { hosts => ["elastic1","elastic2","elastic3"] index => "access-%{index-name}" } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 9, 2024, 11:24am UTC](https://discuss.elastic.co/t/logstash-parsing-data-incorrectly/355221/4 "2024-04-09T11:24:37Z")
Closed on April 9, 2024, 11:24am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.