# Logstash unable to identify tags from filebeat ( grokparsefailure)
**URL:** https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129
**Category:** Logstash
**Created:** [August 7, 2017, 3:04pm UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129 "2017-08-07T15:04:47Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [August 7, 2017, 3:04pm UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129/1 "2017-08-07T15:04:47Z")
Hi, i am sending two different logs files with filebeat agent with different tags on filebeat side to logstash but logstash only able to parse one file and failing for other . Below is my config.
`Can someone help .`
```
input {
beats {
client_inactivity_timeout => 86400
port => 5044
type => "log"
}
}
filter {
if "vus" in [tags] {
mutate {
gsub => [
"message", "\t", " ",
"message", "\n", " "
]
}
grok {
match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}%{WORD:level}%{SPACE}%{JAVACLASS:coidkey}%{SPACE}%
{USER:ident}%{SPACE}%{GREEDYDATA:Url}"}
}
}
else if "download" in [tags] {
mutate {
gsub => [
"message", "\t", " ",
"message", "\n", " "
]
}
grok {
match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}\:\|\:%{SPACE}%{WORD:level}%{SPACE}\:\|\:%{SPA
CE}%{USERNAME:host_name}%{SPACE}\:\|\:%{SPACE}%{DATA:coidkey}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:clientinfo}%{SPACE}\:
\|\:%{SPACE}(%{IP:clientip})?%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:Url}%{SPACE}\:\|\:%{SPACE}%{JAVACLASS:class}%{SPACE}\:\|\:%
{SPACE}%{USER:ident}%{SPACE}%{GREEDYDATA:msg}"}
remove_field => ["ident","offset","name","version","host"]
}
}
}
output {
stdout { codec => rubydebug }
if "_grokparsefailure" in [tags] {
# write events that didn't match to a file
file { "path" => "/tmp/grok_failures.txt" }
}
# if "vus" in [tags] {
# elasticsearch {
# hosts => "dfdevelasticp1.df.jabodo.com:9200"
# user => "elastic"
# password => "fd5dd89c"
# index => "download-%{+YYYY.MM.dd}"
# document_type => "log"
# }
# }
#if [type] == "download"{
else{
elasticsearch {
hosts => "dfsyselastic.df.jabodo.com:9200"
user => "UN"
password => "PW"
index => "vicinio-%{+YYYY.MM.dd}"
document_type => "log"
}
}
}
```
I have tested both my grok they work fine alone.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 8, 2017, 5:30am UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129/2 "2017-08-08T05:30:05Z")
Show what an incorrectly filtered event looks like. Use a `stdout { codec => rubydebug }` output or copy/paste from Kibana's JSON tab.
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [August 8, 2017, 2:17pm UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129/3 "2017-08-08T14:17:23Z")
Hi mangus,
Below is the output from stdout { codec =\> rubydebug }
```
{
"@timestamp" => 2017-08-08T14:07:38.276Z,
"offset" => 138524460,
"@version" => "1",
"input_type" => "log",
"beat" => {
"hostname" => "dfprdsndlbfe2",
"name" => "dfprdsndlbfe2",
"version" => "5.5.1"
},
"host" => "dfprdsndlbfe2",
"source" => "/archives/logs/tomcat7-8080/download.log",
"message" => "[2017-08-08 10:07:38,116] :|: INFO :|: dfprdsndlbfe2.df.jabodo.com :|: a5fa8005dbe049a7b8d9bafa95400b18 :|: [BT:CHROME, BV:59, BL:en, CC:PH] :|:\n\t\t\t\t130.105.229.204 :|: http://www.radiorage.com/index.jhtml?partner=ZXxpt535&s2=-5503188656016333761&s1=604364 :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: DLPInfo, IP: 130.105.229.204]",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}
{
"@timestamp" => 2017-08-08T14:12:38.379Z,
"offset" => 139773176,
"@version" => "1",
"beat" => {
"hostname" => "dfprdsndlbfe2",
"name" => "dfprdsndlbfe2",
"version" => "5.5.1"
},
"input_type" => "log",
"host" => "dfprdsndlbfe2",
"source" => "/archives/logs/tomcat7-8080/download.log",
"message" => "[2017-08-08 10:12:38,321] :|: INFO :|: dfprdsndlbfe2.df.jabodo.com :|: :|: [BT:FF, BV:54, BL:en, CC:US] :|:\n\t\t\t\t65.35.140.148 :|: http://internetspeedtracker.dl.myway.com/blank.jhtml :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: BlankPageServed, IP: 65.35.140.148]",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}
{
```
And on kibana (json)
```
{
"_index": "vicinio-2017.08.08",
"_type": "log",
"_id": "AV3CNGLArR1KM0reUPvn",
"_score": null,
"_source": {
"@timestamp": "2017-08-08T14:16:38.465Z",
"offset": 140810549,
"@version": "1",
"input_type": "log",
"beat": {
"hostname": "dfprdsndlbfe2",
"name": "dfprdsndlbfe2",
"version": "5.5.1"
},
"host": "dfprdsndlbfe2",
"source": "/archives/logs/tomcat7-8080/download.log",
"message": "[2017-08-08 10:16:38,403] :|: INFO :|: dfprdsndlbfe2.df.jabodo.com :|: :|: [BT:CHROME, BV:51, BL:en, CC:US] :|:\n\t\t\t\t66.249.91.151 :|: :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: PageView, IP: 66.249.91.151]",
"type": "log",
"tags": [
"beats_input_codec_plain_applied"
]
},
"fields": {
"@timestamp": [
1502201798465
]
},
"highlight": {
"beat.name": [
"@kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@"
],
"beat.hostname": [
"@kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@"
],
"host": [
"@kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@"
],
"message": [
"[2017-08-08 10:16:38,403] :|: INFO :|: @kibana-highlighted-field@dfprdsndlbfe2@/kibana-highlighted-field@.df.jabodo.com :|: :|: [BT:CHROME, BV:51, BL:en, CC:US] :|:\n\t\t\t\t66.249.91.151 :|: :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: PageView, IP: 66.249.91.151]"
]
},
"sort": [
1502201798465
]
}
```
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 8, 2017, 2:25pm UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129/4 "2017-08-08T14:25:25Z")
You only apply filters to events tagged "download" or "vus" and these events have no such tags.
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [August 8, 2017, 2:34pm UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129/5 "2017-08-08T14:34:49Z")
Hi mangus ,
I checked the filebeat connfig - download tag was missing on one server but the vus tag was present and i see those getting parsed correctly .
I corrected config and restarted filebeat now seeing following
```
{
"@timestamp" => 2017-08-08T14:31:11.724Z,
"offset" => 144374649,
"@version" => "1",
"input_type" => "log",
"beat" => {
"hostname" => "dfprdsndlbfe2",
"name" => "dfprdsndlbfe2",
"version" => "5.5.1"
},
"host" => "dfprdsndlbfe2",
"source" => "/archives/logs/tomcat7-8080/download.log",
"message" => "[2017-08-08 10:31:11,372] :|: INFO :|: dfprdsndlbfe2.df.jabodo.com :|: :|: [BT:CHROME, BV:59, BL:en, CC:IN] :|: 47.247.11.150 :|: :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: PageView, IP: 47.247.11.150]",
"type" => "log",
"tags" => [
[0] "download",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
]
}
{
"@timestamp" => 2017-08-08T14:31:11.724Z,
"offset" => 144374907,
"@version" => "1",
"beat" => {
"hostname" => "dfprdsndlbfe2",
"name" => "dfprdsndlbfe2",
"version" => "5.5.1"
},
"input_type" => "log",
"host" => "dfprdsndlbfe2",
"source" => "/archives/logs/tomcat7-8080/download.log",
"message" => "[2017-08-08 10:31:11,389] :|: INFO :|: dfprdsndlbfe2.df.jabodo.com :|: :|: [BT:CHROME, BV:59, BL:en, CC:IN] :|: 171.79.71.196 :|: :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: PageView, IP: 171.79.71.196]",
"type" => "log",
"tags" => [
[0] "download",
[1] "beats_input_codec_plain_applied",
[2] "_grokparsefailure"
]
}
```
Also i am able to see that tag is coming with input line
```
root@dfsyslogstash2:~# more /tmp/grok_failures.txt
{"@timestamp":"2017-08-08T14:30:42.710Z","offset":144172281,"@version":"1","input_type":"log","beat":{"hostname":"dfprdsndl
bfe2","name":"dfprdsndlbfe2","version":"5.5.1"},"host":"dfprdsndlbfe2","source":"/archives/logs/tomcat7-8080/download.log",
"message":"[2017-08-08 10:30:23,758] :|: INFO :|: dfprdsndlbfe2.df.jabodo.com :|: c30f631536ef4862933590aa4fc58678
:|: [BT:IE, BV:11, BL:en, CC:PT] :|: 195.23.12.27 :|: http://free.fromdoctopdf.com/index.jhtml?partner=Y6xpu065&s1=
1579971&s2=15022026003273067547260080817699725 :|: c.m.w.d.m.UnifiedLoggerWrapper :|: -
[ET: DLPInfo, IP: 195.23.12.27]","type":"log","tags":["download","beats_input_codec_plain_applied","_grokparsefailure"]}
{"@timestamp":"2017-08-08T14:30:42.710Z","offset":144172539,"@version":"1","input_type":"log","beat":{"hostname":"dfprdsndl
bfe2","name":"dfprdsndlbfe2","version":"5.5.1"},"host":"dfprdsndlbfe2","source":"/archives/logs/tomcat7-8080/download.log",
"message":"[2017-08-08 10:30:23,881] :|: INFO :|: dfprdsndlbfe2.df.jabodo.com :|: :|: [BT:FF, BV:49, BL:en, CC:ID
] :|: 180.241.166.108 :|: :|: c.m.w.d.m.UnifiedLoggerWrapper :|: - [ET: backFill
, IP: 180.241.166.108]","type":"log","tags":["download","beats_input_codec_plain_applied","_grokparsefailure"]}
```
---
### Author:  [@Nikhilpawar1985](https://discuss.elastic.co/u/Nikhilpawar1985)
#### Post date: [August 8, 2017, 3:09pm UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129/6 "2017-08-08T15:09:52Z")
Found it. I guess it was a space issue. Not seeing any grokfailures
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 5, 2017, 3:10pm UTC](https://discuss.elastic.co/t/logstash-unable-to-identify-tags-from-filebeat-grokparsefailure/96129/7 "2017-09-05T15:10:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.