Hi Here is my logstash config file
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "http://ip:9200"
index => "%{type}-%{+YYYY.MM.dd}"
user => "elastic"
password => "pwd"
}
}
I have logs coming from multiple sources using this pipeline. I want to apply grok on file coming from only one source and no pattern at all for others. The files coming from the source on which i want to grok are coming from multiple paths.
So if the log file path consists of "Apache" i want to apply one grok and if it consists of another string i want to apply one grok. and no grok for others. How can that be done?
I Tried adding tag in filebeat config
- type: log
enabled: true
paths:
- 'D:\3DS-apache\Apache\Apache24\logs\3dx_access.log'
fields:
type: 3dxp_apachetrace
3dxp_tag: 3dxp_apache
fields_under_root: true
- type: log
enabled: true
paths:
- 'path1*'
fields:
type: 3dxp_servicetrace
3dxp_tag: 3dxp_service
fields_under_root: true
In the logs there is a field called 3dxp_tag
But when i try to use the 3dxp_tag in logstash config nothing seems to work, by default the first grok is getting applied
Here is logstash config
input {
beats {
port => 5044
}
}
filter {
if [3dxp_tag] == "3dxp_apache"
{
grok {
match => { "message" => ["\[%{HTTPDATE:timestamp}\] \| %{NUMBER:response} \| (?<duration>%{NUMBER} %{WORD}) \| (?<bytes>%{NUMBER} %{WORD}|%{DATA}) \| %{IP:hostip} \| (%{IP:clientip}|%{DATA:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?<tag1>%{NUMBER} %{WORD}|%{DATA}) \| \"(?<method>%{WORD}) (?<url>%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\""] }
}
}
else if [3dxp_tag] == "3dxp_service"
{
grok {
match => { "message" => ["\[%{HTTPDATE:timestamp}\] \| %{NUMBER:response} \| (?<duration>%{NUMBER} %{WORD}) \| (?<bytes>%{NUMBER} %{WORD}|%{DATA}) \| %{IP:hostip} \| (%{DATA:clientip}|%{IP:clientip}) \| (%{WORD:token}|%{DATA:token}) \| (?<tag1>%{NUMBER} %{WORD}|%{DATA}) \| \"(?<method>%{WORD}) (?<url>%{URIPATHPARAM}) (?:HTTP/%{NUMBER:http_version})\" \| (%{EMAILADDRESS:loginemail}|%{DATA:loginemail})"] }
}
}
}
output {
elasticsearch {
hosts => "http://ip:9200"
index => "%{type}-%{+YYYY.MM.dd}"
user => "elastic"
password => "pwd"
}
}
There is a difference of only one field in both groks, but the second grok isnt getting applied
