Hi All,
I have created two different elastic search clusters.
In first cluster (created newly) ,(v 7.13.2), I have created below config in pipeline.conf under conf.d for logstash and it worked as I see expected fields getting created in index. I verified this from Stack Management --> Index Management --> there is only one index (as data is not getting generated regularly) ---> after clicking on it, in Mappings tab.
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}(?::?%{SECOND})\| %{USERNAME:exchangeId}\| %{DATA:trackingId}\| %{NUMBER:RoundTrip:int}%{SPACE}ms\| %{NUMBER:ProxyRoundTrip:int}%{SPACE}ms\| %{NUMBER:UserInfoRoundTrip:int}%{SPACE}ms\| %{DATA:Resource}\| %{DATA:subject}\| %{DATA:authmech}\| %{DATA:scopes}\| %{IPV4:Client}\| %{WORD:method}\| %{DATA:Request_URI}\| %{INT:response_code}\| %{DATA:failedRuleType}\| %{DATA:failedRuleName}\| %{DATA:APP_Name}\| %{DATA:Resource_Name}\| %{DATA:Path_Prefix}"}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
The second cluster (7.4.0) which was already running, It had below config in
if [log_type] == "access_server" and [app_id] == "as"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{UUID:ConsentID}%{SPACE}%{WORD:TransactionID}%{SPACE}%{WORD:TraceID}%{SPACE}%{GREEDYDATA:messagetext}" } }
mutate {
replace => {
"[type]" => "access_server"
}
}
}
output {
if [log_type] == "access_server" {
elasticsearch {
hosts => ['http://14.1.1.50:9200']
index => "%{type}-%{+YYYY.MM.dd}"
user => elastic
password => NfBxxxxx
}
}
elasticsearch {
hosts => ['http://14.1.1.50:9200']
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => elastic
password => NfBxxxxx
}
}
I modified the config file yesterday (06-July-21) to below to have individual fields as per desired data types, (as per the above first cluster config)
if [log_type] == "access_server" and [app_id] == "as"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}(?::?%{SECOND})\| %{USERNAME:exchangeId}\| %{DATA:trackingId}\| %{NUMBER:RoundTrip:int}%{SPACE}ms\| %{NUMBER:ProxyRoundTrip:int}%{SPACE}ms\| %{NUMBER:UserInfoRoundTrip:int}%{SPACE}ms\| %{DATA:Resource}\| %{DATA:subject}\| %{DATA:authmech}\| %{DATA:scopes}\| %{IPV4:Client}\| %{WORD:method}\| %{DATA:Request_URI}\| %{INT:response_code}\| %{DATA:failedRuleType}\| %{DATA:failedRuleName}\| %{DATA:APP_Name}\| %{DATA:Resource_Name}\| %{DATA:Path_Prefix}" } }
mutate {
replace => {
"[type]" => "access_server"
}
}
}
output {
if [log_type] == "access_server" {
elasticsearch {
hosts => ['http://14.1.1.50:9200']
index => "%{type}-%{+YYYY.MM.dd}"
user => elastic
password => NfBxxxxxx
}
}
elasticsearch {
hosts => ['http://14.1.1.50:9200']
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => elastic
password => NfBxxxxxx
}
and restarted logstash but today's access_server index is not having above individual fields.
The difference between above two clusters is, first cluster which I created to test above scenario (v 7.13.2), is having only one index (it indexed old log files and there are no new log files getting created daily), and I have created Index pattern of it in kibana but in second cluster which was already running is having daily indices of access_server but I have not created index patterns of it so I looked into today's access_server index but it is not having desired individual fields but I think the main issue is because of configration in pipeline.conf file which I mentioned above.
Can you please let me know what is not correct in modified pipeline.conf file because of which I still can't see individual message fields in today's index.
Thanks,