Hello Experts,
Different type of logs going into single index, technically it should go to different new (defined) indexes.
I am differentiating 2 different log files with TAG and then applying filter on TAG in logstash.
my filebeat.yaml file config:
-
type: log
enabled: true
paths:- /xxx/system.log.2018-03-18_2148
tags: ["cassandra_test"]
exclude_lines: ['^DBG']
fields:
app_id: cassandralog
multiline.pattern: "^INFO|^WARN|^ERROR|^FATAL"
multiline.negate: true
multiline.match: after
add_locale: ~
- /xxx/system.log.2018-03-18_2148
-
type: log
paths:- /apth/all.log.2
tags: ["swift_proxy_test"]
fields:
app_id: swiftproxylog
multiline.pattern: "^Jan|^Feb|^Mar|^Apr|^May|^Jun|^Jul|^Aug|^Sep|^Oct|^Nov|^Dec"
multiline.negate: true
multiline.match: after
- /apth/all.log.2
ouput to logstash
In my Logstash I have 2 config file one for cassandralog and another one for swiftproxylog
Cassandra.conf
input {
beats {
port => 5044
}
}
filter {
if "cassandra_test" in [tags]{
grok {
patterns_dir => ["path"]
break_on_match => true
match => {
"message" => [
#"%{CASS_COMPACTION_LARGE_KEY}",
"%{CASS_SLAB_POOL_CLEANER_1}",
"%{CASS_SLAB_POOL_CLEANER_2}",
# "%{CASS_MEMTABLE_FLUSH_START}",
# "%{CASS_MEMTABLE_FLUSH_COMPLETE}",
"%{CASS_BATCH_STATEMENT}",
"%{CASS_SIMS_TOMBSTONE}",
"%{CASS_COMPACTION_COMPLETE}",
"%{CASS_GC_GRACE}",
"%{CASS_SERVICE_THREAD_PENDING}"
]
}
add_tag => [ "cass_parsed" ]
}
output {
elasticsearch {
hosts => "host_ip:9200"
index => "prd-log-%{+YYYY.MM.dd.HH}-000001"
template => "cass_log_sizing_2.json"
template_name => "cassandra_log"
template_overwrite => true
}
}
Swiftproxylog.conf
input {
beats {
port => 5044
}
}
filter {
if "swift_proxy_test" in [tags]{
grok {
patterns_dir => ["path"]
break_on_match => true
match => { "message" => [ "%{SWIFT_P_ALL}" ] }
add_tag => [ "swift_all_parsed" ]
}
if "swift_all_parsed" not in [tags] {
grok {
patterns_dir => ["path"]
match => { "message" => ["%{SWIFT_P_204_499}"] }
add_tag => [ "swift_rest" ]
}
}
output {
elasticsearch {
hosts => "host_ip:9200"
index => "swift-proxy-log-%{+YYYY.MM.dd.HH}"
manage_template => "false"
}
}
when ever I ran logstash on Cassandra.conf all 2 files data going into prd-log-* index and if I ran
Swiftproxylog.conf all the data going into swift-proxy-log-* index it supposed to go it's own defined index as per filter TAG and elasticsearch output config.
But I do notice one thing, eventhough it's going to same index but I see tags are correct the only problem I see is data mix into one index.
below is the mixing data image:
This index shouldn't have this data
This index should have only this data not above one
Please advice/correct me if I am doing anything wrong here.
Thanks
Chandra
