Data going into wrong index

Elastic Stack Beats
1

Hello Experts,

Different type of logs going into single index, technically it should go to different new (defined) indexes.

I am differentiating 2 different log files with TAG and then applying filter on TAG in logstash.

my filebeat.yaml file config:

  • type: log
    enabled: true
    paths:

    • /xxx/system.log.2018-03-18_2148
      tags: ["cassandra_test"]
      exclude_lines: ['^DBG']
      fields:
      app_id: cassandralog
      multiline.pattern: "^INFO|^WARN|^ERROR|^FATAL"
      multiline.negate: true
      multiline.match: after
      add_locale: ~

  • type: log
    paths:

    • /apth/all.log.2
      tags: ["swift_proxy_test"]
      fields:
      app_id: swiftproxylog
      multiline.pattern: "^Jan|^Feb|^Mar|^Apr|^May|^Jun|^Jul|^Aug|^Sep|^Oct|^Nov|^Dec"
      multiline.negate: true
      multiline.match: after

ouput to logstash

In my Logstash I have 2 config file one for cassandralog and another one for swiftproxylog

Cassandra.conf
input {
beats {
port => 5044
}
}

filter {
if "cassandra_test" in [tags]{
grok {
patterns_dir => ["path"]
break_on_match => true
match => {
"message" => [
#"%{CASS_COMPACTION_LARGE_KEY}",
"%{CASS_SLAB_POOL_CLEANER_1}",
"%{CASS_SLAB_POOL_CLEANER_2}",
# "%{CASS_MEMTABLE_FLUSH_START}",
# "%{CASS_MEMTABLE_FLUSH_COMPLETE}",
"%{CASS_BATCH_STATEMENT}",
"%{CASS_SIMS_TOMBSTONE}",
"%{CASS_COMPACTION_COMPLETE}",
"%{CASS_GC_GRACE}",
"%{CASS_SERVICE_THREAD_PENDING}"
]
}
add_tag => [ "cass_parsed" ]
}

output {
elasticsearch {
hosts => "host_ip:9200"
index => "prd-log-%{+YYYY.MM.dd.HH}-000001"
template => "cass_log_sizing_2.json"
template_name => "cassandra_log"
template_overwrite => true
}
}

Swiftproxylog.conf

input {
beats {
port => 5044
}
}

filter {
if "swift_proxy_test" in [tags]{
grok {
patterns_dir => ["path"]
break_on_match => true
match => { "message" => [ "%{SWIFT_P_ALL}" ] }
add_tag => [ "swift_all_parsed" ]
}

if "swift_all_parsed" not in [tags] {
grok {
patterns_dir => ["path"]
match => { "message" => ["%{SWIFT_P_204_499}"] }
add_tag => [ "swift_rest" ]
}
}

output {
elasticsearch {
hosts => "host_ip:9200"
index => "swift-proxy-log-%{+YYYY.MM.dd.HH}"
manage_template => "false"
}
}

when ever I ran logstash on Cassandra.conf all 2 files data going into prd-log-* index and if I ran
Swiftproxylog.conf all the data going into swift-proxy-log-* index it supposed to go it's own defined index as per filter TAG and elasticsearch output config.

But I do notice one thing, eventhough it's going to same index but I see tags are correct the only problem I see is data mix into one index.

below is the mixing data image:

This index shouldn't have this data

image
image.png1779×840 61.6 KB

This index should have only this data not above one

image
image.png1693×979 69.1 KB

Please advice/correct me if I am doing anything wrong here.
Thanks
Chandra

2

@ .. Can anyone help me on this?

3

I answered the version of this you put in the logstash thread.

4

Please keep your questions to one thread, having multiple makes it harder for us to assist you :slight_smile: Data going into wrong index-why