# Need help on grok filter
**URL:** https://discuss.elastic.co/t/need-help-on-grok-filter/134575
**Category:** Logstash
**Created:** [June 5, 2018, 9:07am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575 "2018-06-05T09:07:39Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 5, 2018, 9:07am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/1 "2018-06-05T09:07:39Z")
Hi guys,
i would like to ask if what is the correct filter using grok syntax for below output
sample.out
hostname:server1.sgdc.company.net|ipaddress:1XX.XX.XXX.XX|status:ACTIVE
hostname:server2.sgdc.company.net|ipaddress:1YY.YY.YY.YY|status:ACTIVE
I tried this logstash.conf but failing.
input {
file {
type =\> "monitor1"
path =\> "/root/Documents/scripts/sample/sample.out"
start\_position =\> "beginning"
sincedb\_path =\> "/opt/logstash/.sincedb\_sample"
}
}
filter {
if [type] == "monitor1" {
grok {
match =\> { "message" =\> "hostname:%{FQDN:fqdn\_unparsed}|ipaddress:%{IPV4}|status:%{GREEDYDATA:status}" }
}
}
output {
if [type] == "monitor1" {
elasticsearch {
hosts =\> ["1XX.XX.XXX.XX:9200"]
index =\> "monitor1"
}
}
}
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 1:24am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/2 "2018-06-07T01:24:08Z")
any help please.
---
### Author:  [@Krunal\_kalaria](https://discuss.elastic.co/u/Krunal_kalaria)
#### Post date: [June 7, 2018, 4:29am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/3 "2018-06-07T04:29:54Z")
Hey @retxedue,
As you mention above sample.out logs that all the logs pattern are same or different ?
If it is same then no need to write the grok if all the logs pattern are different then grok will works good.
My suggestion to you is use KV filter plugin and seprate the key and value like following example:
filter {
kv {
field\_split =\> ":"
}
kv {
value\_split =\> "|"
}
}
it will parse the all of your logs in key:value pair.
if you want to more elaborate in this then refer this link:
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html)
Thanks & Regards,
Krunal.
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 5:55am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/4 "2018-06-07T05:55:49Z")
all are same. See below sample:
hostname:hou150lnx.hou150.domain.net|ipaddress:1XX.XX.XX.216|status:active
hostname:hou150lnxq1ep1dn1.hou150.domain.net|ipaddress:1XX.XX.XX.215|status:active
hostname:test150lnxq1ep1-secondary.hou150.domain.net|ipaddress:1XX.XX.XX.242|status:standby
hostname:testclnxq1ep2-secondary.domain2.domain.net|ipaddress:1YY.YYY.Y.38|status:standby
hostname:testpwglnxq1fp1.lonpwg.domain.net|ipaddress:1XX.XX.YYY.76|status:active
hostname:testpwglnxq1ep1-primary.domain3.domain.net|ipaddress:1XX.XX.YYY.79|status:active
hostname:testclnxq1dn2.domain2.domain2.net|ipaddress:1YY.YYY.Y.39|status:active
hostname:testlnxq1dn1.domain2.domain.net|ipaddress:1YY.YYY.Y.36|status:active
hostname:testctylnxq1efp1.tcoaty.domain.net|ipaddress:1XX.XX.XXX.30|status:active
hostname:domain3lnxq1dn1.domain3.domain.net|ipaddress:1XX.XX.XXX.81|status:active
I tried using grok constructor below query matched with my logs.
["hostname:[.(?[A-Za-z0-9\_-]+.[A-Za-z0-9\_-]+.net)$]|ipaddress:%{IP:ipaddress}|status:%{GREEDYDATA:status}"]
I tried adding it in my logstash.conf. it is failing can you please help how to add in filter?
this one is failing. What have i missed?
filter {
if [type] == "test" {
grok {
match =\> { "message" =\> "["hostname:[.(?[A-Za-z0-9\_-]+.[A-Za-z0-9\_-]+.net)$]|ipaddress:%{IP:ipaddress}|status:%{GREEDYDATA:status}"]" }
}
}
}
---
### Author:  [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)
#### Post date: [June 7, 2018, 6:13am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/5 "2018-06-07T06:13:56Z")
Hi,
you can use built-in HOSTNAME pattern to get fqdn.
Also "|" character needs escaping.
```
filter {
if [type] == "monitor1" {
grok {
match => { "message" => "hostname:%{HOSTNAME:fqdn_unparsed}\|ipaddress:%{IPV4:ip_addr}\|status:%{GREEDYDATA:status}" }
}
}
}
```
* * *
```
[2018-06-07T09:11:23,163][DEBUG][logstash.pipeline] output received {"event"=>{"status"=>"ACTIVE\r", "@version"=>"1", "message"=>"hostname:server2.sgdc.company.net|ipaddress:100.1.22.34|status:ACTIVE\r", "ip_addr"=>"100.1.22.34", "@timestamp"=>2018-06-07T06:11:22.864Z, "path"=>"C:\\ericsson\\development\\elk\\logstash\\sample.out", "host"=>"TR00200384", "fqdn_unparsed"=>"server2.sgdc.company.net", "type"=>"monitor1"}}
{
"status" => "ACTIVE\r",
"@version" => "1",
"message" => "hostname:server2.sgdc.company.net|ipaddress:100.1.22.34|status:ACTIVE\r",
"ip_addr" => "100.1.22.34",
"@timestamp" => 2018-06-07T06:11:22.864Z,
"path" => "C:\\development\\elk\\logstash\\sample.out",
"host" => "TR00200384",
"fqdn_unparsed" => "server2.sgdc.company.net",
"type" => "monitor1"
}
```
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 6:36am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/6 "2018-06-07T06:36:25Z")
> [@arkady\_renko](#):
>
> filter { if [type] == "monitor1" { grok { match =\> { "message" =\> "hostname:%{HOSTNAME:fqdn\_unparsed}|ipaddress:%{IPV4:ip\_addr}|status:%{GREEDYDATA:status}" } } } }
I tried above pattern but still failing.
input {
file {
type =\> "monitor1"
path =\> "/root/Documents/scripts/monitor/monitor.out"
start\_position =\> "beginning"
}
}
filter {
if [type] == "monitor1" {
grok {
match =\> { "message" =\> "hostname:%{HOSTNAME:fqdn\_unparsed}|ipaddress:%{IPV4:ip\_addr}|status:%{GREEDYDATA:status}" }
}
}
}
output {
if [type] == "monitor1" {
elasticsearch {
hosts =\> ["1XX.XX.XXX.XX:9200"]
index =\> "montor1"
}
}
}
here is the error:
[2018-06-07T14:32:16,733][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}
[2018-06-07T14:32:16,737][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}
[2018-06-07T14:32:17,012][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.2.4"}
[2018-06-07T14:32:17,085][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
[2018-06-07T14:32:17,457][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}
[2018-06-07T14:32:17,644][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://1XX.XXX.XX.XX:9200/](http://1XX.XXX.XX.XX:9200/)]}}
[2018-06-07T14:32:17,645][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://1XX.XXX.XX.XX:9200/](http://1XX.XXX.XX.XX:9200/), :path=\>"/"}
[2018-06-07T14:32:17,704][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://1XX.XXX.XX.XX:9200/](http://1XX.XXX.XX.XX:9200/)"}
[2018-06-07T14:32:17,728][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6}
[2018-06-07T14:32:17,728][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}
[2018-06-07T14:32:17,729][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}
[2018-06-07T14:32:17,731][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}
[2018-06-07T14:32:17,736][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::Elasticsearch", :hosts=\>["[//1XX.XXX.XX.XX:9200](https://1XX.XXX.XX.XX:9200)"]}
[2018-06-07T14:32:17,888][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"}
[2018-06-07T14:32:17,921][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]}
---
### Author:  [@Krunal\_kalaria](https://discuss.elastic.co/u/Krunal_kalaria)
#### Post date: [June 7, 2018, 6:36am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/7 "2018-06-07T06:36:31Z")
Why i am suggesting you to use KV filter plugin its easy to parse the logs and grok is complex to write and all the things so i will suggest you to use kv filter plugin try once its working or not.
Thanks & Regards,
Krunal.
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 6:39am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/8 "2018-06-07T06:39:41Z")
how to write it in kv? for my type of input file ?
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 6:40am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/9 "2018-06-07T06:40:14Z")
may i see your whole logstash.conf here?
---
### Author:  [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)
#### Post date: [June 7, 2018, 6:41am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/10 "2018-06-07T06:41:39Z")
I don't see any error line on the log. All of them INFO.
Escaping still missing in your grok filter.
```
input {
file {
type => "monitor1"
path => "C:\development\elk\logstash\sample.out"
start_position => "beginning"
}
}
filter {
if [type] == "monitor1" {
grok {
match => { "message" => "hostname:%{HOSTNAME:fqdn_unparsed}\|ipaddress:%{IPV4:ip_addr}\|status:%{GREEDYDATA:status}" }
}
}
}
output {
stdout { codec => rubydebug }
}
```
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 6:48am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/11 "2018-06-07T06:48:07Z")
i tried using above config still i am seeing in logs are INFO. I also need to have an output to have index created so i can used for grafana dashboard.
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 7:19am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/12 "2018-06-07T07:19:23Z")
@arkady - can you give sample config using KV?
---
### Author:  [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)
#### Post date: [June 7, 2018, 7:28am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/13 "2018-06-07T07:28:36Z")
Actually your problem is not related with filter plugin.
You should check output configuration in pipeline conf.
I wish i could help with grafana, but i dont have any knowledge about it.
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 7, 2018, 8:12am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/14 "2018-06-07T08:12:11Z")
this is what is in my pipeline.yml
- pipeline.id: main
path.config: "/etc/logstash/conf.d/\*.conf"
---
### Author:  [@arkady\_renko](https://discuss.elastic.co/u/arkady_renko)
#### Post date: [June 7, 2018, 8:59am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/15 "2018-06-07T08:59:47Z")
I meant output configuration below.
Do you see any document in you elasticsearch index?
output {
if [type] == "monitor1" {
{
hosts =\> ["1XX.XX.XXX.XX:9200"]
index =\> "monitor1"
}
}
}
---
### Author:  [@retxedue](https://discuss.elastic.co/u/retxedue)
#### Post date: [June 25, 2018, 7:37am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/16 "2018-06-25T07:37:46Z")
Here is the config. It is not creating any index
input {
file {
type =\> "monitor"
path =\> "/root/Documents/scripts/qradar/monitor.out"
start\_position =\> "beginning"
}
}
filter {
if [type] == "monitor" {
grok {
match =\> { "message" =\> "hostname:%{GREEDYDATA:hostname}|ipaddress:%{IP:ipaddress}|status:%{WORD:status}" }
}
}
}
output {
if [type] == "monitor" {
elasticsearch {
hosts =\> ["1XX.XX.XX.XXX:9200"]
index =\> "monitor"
}
stdout { codec =\> rubydebug }
}
}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 23, 2018, 7:37am UTC](https://discuss.elastic.co/t/need-help-on-grok-filter/134575/17 "2018-07-23T07:37:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.