# My Logstash is not parsing data properly
**URL:** https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399
**Category:** Logstash
**Created:** [May 6, 2020, 7:51pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399 "2020-05-06T19:51:37Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 6, 2020, 7:51pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/1 "2020-05-06T19:51:38Z")
I have been working on a project to upload VPN logs to a logstash instance that I created, and while I am able to see data being sent to the logstash, I am seeing a "\_dateparsefailure" error showing under [tags] in my output. Also, While some of the data is parsing correctly, some of the data is not showing in the correct fields, which tells me that the data, as a whole, is not parsing correctly, and I'm not sure why. Can someone give any input as to why this might be happening?
---
### Author:  [@ptamba](https://discuss.elastic.co/u/ptamba)
#### Post date: [May 7, 2020, 2:36am UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/2 "2020-05-07T02:36:21Z")
that simply means something is wrong with your logstash configuration. there are hundreds of plugins in logstash, so without an example, it’s hard to imagine. maybe show your config , with your data, then explain what you expect vs what happens.
\_dateparsefailure usually means your using a date filter with a wrong date format
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 7, 2020, 3:24pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/3 "2020-05-07T15:24:11Z")
This is the latest version of my config:
```auto
input {
udp {
port => 42514
tags => ["vpn"]
}
}
filter {
grok {
tag_on_failure => ["parse_failed"]
match => { "message" => '\<%{NUMBER:view}\>%{SYSLOGTIMESTAMP} %{HOSTNAME:fwname} %{GREEDYDATA:raw_message}'}
}
csv {
source => "random"
separator => ","
columns => ["random_number"]
}
csv {
source => "raw_message"
separator => '"'
columns => ["receive", "description", "device"]
}
csv {
source => "receive"
separator => ","
columns => ["receive_time", "serial", "type", "subtype", "FUTURE_USE.1", "time_generated", "vsys", "eventid", "object", "module", "severity"]
}
csv {
source => "description"
separator => ","
columns => ["description"]
}
csv {
source => "device"
separator => ","
columns => ["number", "seqno", "actionflags", "dg_hier_level_1", "dg_hier_level_2", "dg_hier_level_3", "dg_hier_level_4", "vsys.name", "device_name"]
}
mutate {
remove_field => ["random_number", "serial", "FUTURE_USE.1", "vsys", "FUTURE_USE.2", "FUTURE_USE.3", "actionflags", "dg_hier_level_1", "dg_hier_level_2", "dg_hier_level_3", "dg_hier_level_4", "vsys.name"]
}
date {
timezone => "America/Los_Angeles"
match => ["receive_time", "YYYY/MM/dd HH:mm:ss"]
}
}
output {
file {
path => "/opt/logstash/debug-%{+YYYY-MM-dd}.json"
}
stdout { codec => rubydebug }
}
```
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 7, 2020, 3:43pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/4 "2020-05-07T15:43:26Z")
When I look for just the vpn logs that are being sent to this logstash instance, some of the fields that I see being parsed incorrectly are the receive time, subtype, time generated, type, and receive.
There might be other issues, but that is what I see right away that is being parsed incorrectly.
---
### Author:  [@ptamba](https://discuss.elastic.co/u/ptamba)
#### Post date: [May 7, 2020, 5:24pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/5 "2020-05-07T17:24:52Z")
is there a sample data for that config so we can compare the filter config with the data you receive? ? you can omit sensitive information as long as the structure and formatting is intact
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 7, 2020, 6:00pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/6 "2020-05-07T18:00:43Z")
```auto
{
"view": "14",
"fwname": "firewall name",
"column12": "0",
"column15": null,
"receive": "1,2020/05/06 10:59:06,001801026594,SYSTEM,"gateway,0,2020/05/ 06 10:59:06,,"gateway"-config-release,RA-VPN-GW-Site-N,0,0, general,informational,",
"description": "gateway client configuration released. User name : "user's email address", Private IP: "IP_Address", Client version: 5.0.8-4, Device name: "Computer_Name", Client OS version: Microsoft Windows 10 Enterprise , 64-bit, VPN t ype: Device Level VPN.",
"`raw_message`": "1,2020/05/06 10:59:06,001801026594,SYSTEM,"gateway",0,2020 /05/06 10:59:06,,gateway-config-release,RA-VPN-GW-Site-N, 0,0,general,informational,\"gateway client configuration released. User name: "user's email address", Private IP: "IP Address", Client version: 5.0.8-4, De vice name: "Computer Name", Client OS version: Microsoft Windows 10 Enterprise , 64- bit, VPN type: Device Level VPN.\",26694772,0x0,0,0,0,0,,fw_name",
"receive_time": "1",
"device_name": "firewall name",
"severity": "0",
"column14": "informational",
"device": ",26694772,0x0,0,0,0,0,,firewall name",
"number": null,
"eventid": null,
"time_generated": "0",
"module": "RA-VPN-GW-Site-N",
"host": "IP Address",
"message": "<14>May 6 10:59:06 "firewall name" 1,2020/05/06 10:59:06, 001801026594,SYSTEM,"gateway",0,2020/05/06 10:59:06,,gateway-co nfig-release,RA-VPN-GW-"Site"-N,0,0,general,informational,\" gateway client configuration released. User name: "user's email", Private IP: "IP Address", Client version: 5.0.8-4, Device name: "Computer Name", Client OS versio n: Microsoft Windows 10 Enterprise , 64-bit, VPN type: Device Level VPN.\",26694 772,0x0,0,0,0,0,,firewall name",
"tags": [
"vpn",
"_dateparsefailure"
],
"subtype": "SYSTEM",
"seqno": "26694772",
"column13": "general",
"`@version`": "1",
"`@timestamp`": "2020-05-06T17:59:06.209Z",
"type": "001801026594",
"object": "gateway-config-release"
}
```
Here is an example of the output that I see.
---
### Author:  [@ptamba](https://discuss.elastic.co/u/ptamba)
#### Post date: [May 8, 2020, 1:51am UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/7 "2020-05-08T01:51:40Z")
> [@pjepsen](#):
>
> csv { source =\> "raw\_message" separator =\> '"' columns =\> ["receive", "description", "device"] }
your raw\_message contains number "1". it will be parsed to "receive" field and the next parsing
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 8, 2020, 3:16pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/8 "2020-05-08T15:16:07Z")
Yeah, I have been working on getting rid of that "1", but I haven't found a solution to that. I created a csv filter to remove the "1" with `"source => random"`, and send that field to my `"remove_field"` mutate filter, but it didn't remove the "1". Do you have any suggestions?
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 8, 2020, 10:46pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/9 "2020-05-08T22:46:50Z")
I was able to fix the issue the "\_dateparsefailure" error. It was a formatting issue. Now, I'm looking at why some of my fields are named incorrectly, or aren't being removed like they should be.
---
### Author:  [@ptamba](https://discuss.elastic.co/u/ptamba)
#### Post date: [May 9, 2020, 3:12am UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/10 "2020-05-09T03:12:31Z")
if the “1” appears consistently and is decimal, you can remove it using \d in your first grok filter or store it with %{NUMBER:some\_field}
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 11, 2020, 11:24pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/11 "2020-05-11T23:24:40Z")
like this?: match =\> { "message" =\> '\<\d%{`NUMBER:view`}\>%{`SYSLOGTIMESTAMP`} %{`HOSTNAME:fwname`} %{`GREEDYDATA:raw_message`}'}
---
### Author:  [@ptamba](https://discuss.elastic.co/u/ptamba)
#### Post date: [May 12, 2020, 1:10am UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/12 "2020-05-12T01:10:56Z")
> [@pjepsen](#):
>
> "message": "\<14\>May 6 10:59:06 "firewall name" 1,2020/05/06 10:59:06, 001801026594,SYSTEM,"gateway",0,2020/05/06 10:59:06,,gateway-co nfig-release,RA-VPN-GW-"Site"-N,0,0,general,informational," gateway client configuration released. User name: "user's email", Private IP: "IP Address", Client version: 5.0.8-4, Device name: "Computer Name", Client OS versio n: Microsoft Windows 10 Enterprise , 64-bit, VPN type: Device Level VPN.",26694 772,0x0,0,0,0,0,,firewall name",
```auto
match => {"message", "<%{NUMBER:view}>%{SYSLOGTIMESTAMP:timestamp} \"%{DATA:fwname}\" %{NUMBER:uniq},%{GREEDYDATA:message}"
```
that should give you :
```auto
{
"view": "14",
"fwname": "firewall name",
"num": "1",
"message": "2020/05/06 10:59:06, 001801026594,SYSTEM,\"gateway\",0,2020/05/06 10:59:06,,gateway-co nfig-release,RA-VPN-GW-\"Site\"-N,0,0,general,informational,\\\" gateway client configuration released. User name: \"user's email\", Private IP: \"IP Address\", Client version: 5.0.8-4, Device name: \"Computer Name\", Client OS versio n: Microsoft Windows 10 Enterprise , 64-bit, VPN type: Device Level VPN.\\\",26694 772,0x0,0,0,0,0,,firewall name\"",
"timestamp": "May 6 10:59:06"
}
```
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 12, 2020, 4:48pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/13 "2020-05-12T16:48:55Z")
I made that change in my config, but now I'm not seeing a log being generated when I run my logstash.
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 12, 2020, 6:23pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/14 "2020-05-12T18:23:40Z")
I am seeing logs again. I'm not sure what caused the issue, but its good again.
---
### Author:  [@ptamba](https://discuss.elastic.co/u/ptamba)
#### Post date: [May 12, 2020, 7:19pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/15 "2020-05-12T19:19:25Z")
you should then be able to send the message to csv filter and do what you want to do with it
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 12, 2020, 8:58pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/16 "2020-05-12T20:58:47Z")
I am seeing errors in the logs that I wasn't seeing before:
`[2020-05-12T13:50:42,926][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 10, column 36 (byte 164) after filter { \n\tgrok {\n tag_on_failure => [\"parse_failed\"]\n \t match => {\"message\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in `block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in `block in converge_state'"]}`
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 12, 2020, 9:54pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/17 "2020-05-12T21:54:18Z")
I have confirmed that the errors were caused by the match statement that we have been looking at. For some reason, making that change to my match statement in my grok causes my entire logstash instance to crash.
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 12, 2020, 10:29pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/18 "2020-05-12T22:29:15Z")
Now that I am seeing logs being generated again, I am looking at an issue where some of the fields are being incorrectly named: `"column13": "general"`, `"column15": null`, etc. Is there a way to fix this?
---
### Author:  [@ptamba](https://discuss.elastic.co/u/ptamba)
#### Post date: [May 13, 2020, 2:17am UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/19 "2020-05-13T02:17:29Z")
> [@pjepsen](#):
>
> columns =\> ["receive\_time", "serial", "type", "subtype", "FUTURE\_USE.1", "time\_generated", "vsys", "eventid", "object", "module", "severity"]
you will need to adjust the naming of the columns. apologies, there was syntax error in my previous config. after trying your message, I ended up with this result. your message has 28 columns :
```auto
{
"column2" => " 001801026594",
"column11" => "0",
"column16" => " Client version: 5.0.8-4",
"column4" => "gateway",
"@timestamp" => 2020-05-13T02:15:04.680Z,
"column12" => "general",
"column14" => " gateway client configuration released. User name: user's email",
"column7" => nil,
"column5" => "0",
"column13" => "informational",
"column26" => "0",
"column3" => "SYSTEM",
"uniq" => "1",
"column6" => "2020/05/06 10:59:06",
"@version" => "1",
"column9" => "RA-VPN-GW-Site-N",
"column27" => nil,
"column10" => "0",
"column21" => "26694 772",
"column22" => "0x0",
"fwname" => "firewall name",
"column24" => "0",
"column1" => "2020/05/06 10:59:06",
"column18" => " Client OS versio n: Microsoft Windows 10 Enterprise ",
"column17" => " Device name: Computer Name",
"column8" => "gateway-co nfig-release",
"column19" => " 64-bit",
"date" => "May 6 10:59:06",
"raw-message" => "2020/05/06 10:59:06, 001801026594,SYSTEM,gateway,0,2020/05/06 10:59:06,,gateway-co nfig-release,RA-VPN-GW-Site-N,0,0,general,informational, gateway client configuration released. User name: user's email, Private IP: IP Address, Client version: 5.0.8-4, Device name: Computer Name, Client OS versio n: Microsoft Windows 10 Enterprise , 64-bit, VPN type: Device Level VPN.,26694 772,0x0,0,0,0,0,,firewall name",
"column15" => " Private IP: IP Address",
"message" => "<14>May 6 10:59:06 \"firewall name\" 1,2020/05/06 10:59:06, 001801026594,SYSTEM,\"gateway\",0,2020/05/06 10:59:06,,gateway-co nfig-release,RA-VPN-GW-\"Site\"-N,0,0,general,informational,\" gateway client configuration released. User name: \"user's email\", Private IP: \"IP Address\", Client version: 5.0.8-4, Device name: \"Computer Name\", Client OS versio n: Microsoft Windows 10 Enterprise , 64-bit, VPN type: Device Level VPN.\",26694 772,0x0,0,0,0,0,,firewall name",
"column25" => "0",
"column20" => " VPN type: Device Level VPN.",
"column23" => "0",
"column28" => "firewall name"
}
```
here's my config. note that i added the gsub because your sample message contains quotes
```auto
filter {
grok {
match => { 'message' => '%{SYSLOGTIMESTAMP:date} "%{DATA:fwname}" %{NUMBER:uniq},%{GREEDYDATA:raw-message}' }
}
mutate {
gsub => ["raw-message", "\"", ""]
}
csv {
source => "raw-message"
separator => ","
}
}
``
```
---
### Author:  [@pjepsen](https://discuss.elastic.co/u/pjepsen)
#### Post date: [May 13, 2020, 8:27pm UTC](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399/20 "2020-05-13T20:27:22Z")
> [@ptamba](#):
>
> ```auto
> match => { 'message' => '%{SYSLOGTIMESTAMP:date} "%{DATA:fwname}" %{NUMBER:uniq},%{GREEDYDATA:raw-message}' }
>
> ```
````auto
I got an error message when adding this to my grok:
```[2020-05-13T13:19:00,157][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, ,, ] at line 29, column 36 (byte 1161) after filter { \n\tgrok {\n tag_on_failure => [\"parse_failed\"]\n match => { 'message' => '%{SYSLOGTIMESTAMP:date} \"%{DATA:fwname}\" %{NUMBER:uniq},%{GREEDYDATA:raw-message}' }\n\t}\n\tcsv {\n\t\tsource => \"raw_message\"\n\t separator => '\"'\n columns => [\"receive\", \"description\", \"device\"]\n\t}\n\tcsv { \n\t\tsource => \"receive\"\n separator => \",\" \n\t\tcolumns => [\"temp_field\", \"receive_time\", \"serial\", \"type\", \"subtype\", \"FUTURE_USE_1\", \"time_generated\", \"vsys\", \"eventid\", \"object\", \"module\", \"severity\"]\n\t}\n\tcsv {\n source => \"device\"\n separator => \",\"\n columns => [\"number\", \"seqno\", \"actionflags\", \"dg_hier_level_1\", \"dg_hier_level_2\", \"dg_hier_level_3\", \"dg_hier_level_4\", \"vsys_name\", \"device_name\"]\n }\n\t mutate {\n remove_field => [\"temp_field\", \"serial\", \"FUTURE_USE_1\", \"vsys\", \"FUTURE_USE_2\", \"FUTURE_USE_3\", \"actionflags\", \"dg_hier_level_1\", \"dg_hier_level_2\", \"dg_hier_level_3\", \"dg_hier_level_4\", \"vsys_name\"]\n\t\t gsub => [\"raw-message\", \"\\\", \"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in `block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in `exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in `block in converge_state'"]}
[2020-05-13T13:19:00,480][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2020-05-13T13:19:05,354][INFO][logstash.runner] Logstash shut down.
````
It looks like you might have missed some commas. I'm going to add them in, and see what happens.
[Next page](https://discuss.elastic.co/t/my-logstash-is-not-parsing-data-properly/231399.md?page=2)