How to create multiple indexes based on conduction in logstash

Elastic Stack Logstash
1

Trying to create multiple indexes for elasticsearch in logstash. But my "if conduction" is not creating any single index, without if conduction it is working fine.

But if I'm using input as file and in logstash without using filebeat then it is working as per my expectation. Can anyone help me for resolution.

###filebeat.yml###
=============
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/user/vinit/pache/*.log
  fields:
    log_type: apache-log

- type: log
  enabled: true
  paths:
    - /home/user/vinit/boss/*.log
  fields:
    log_type: jboss-log
  fields_under_root: true

###pipeline-conf.conf###
==================

input {
  beats {
    port => 5044
  }
}

filter {
    grok {
           match => { "message" => "^%{IP:CLIENT_IP} (?:-|%{USER:IDEN}) (?:-|%{USER:AUTH}) \[%{HTTPDATE:CREATED_ON}\] \"(?:%{WORD:REQUEST_METHOD} (?:/|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})?|-)\" %{NUMBER:RESPONSE_CODE} (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
    add_field => {
                "LOG_TYPE" => "api-log"
        }
    overwrite => [ "message" ]
    }
    grok {
           match => { "message" => "%{HTTPDATE:CREATED_ON}%{NOTSPACE}%{SPACE} (?:-|%{IP:CLIENT_IP})%{SPACE} %{NOTSPACE}(?:-|%{WORD:REQUEST_METHOD}%{SPACE}) (?:-|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})%{NOTSPACE}(?:-|%{GREEDYDATA:OTHER_INFO}) (?:-|%{NUMBER:RESPONSE_CODE}) (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
    add_field => {
                "LOG_TYPE" => "web-log"
        }
    overwrite => [ "message" ]
    }
    grok {
           match => { "message" => "%{TIME:CREATED_ON}%{SPACE}\[(?<THREAD>[^\]]+)?\] %{WORD:METHOD}%{SPACE}%{JAVACLASS:CLASS} - (?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<extra>(.|\r?\n)+))?"}
    add_field => {
                "LOG_TYPE" => "jboss-log"
        }
    overwrite => [ "message" ]
    }
}
output {
   if [fields][log_type] == "apache-log"{
     elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "server-logs-apache"
     }
  }
   if [fields][log_type] == "jboss-log" {
     elasticsearch {
     hosts => ["localhost:9200"]
     manage_template => false
     index => "server-logs-jboss"
     }
  }
   stdout { codec => rubydebug }    
}

##Also Tried##
==============
output {
       if "apache-log" in [fields][log_type] {
         elasticsearch {
         hosts => ["localhost:9200"]
         manage_template => false
         index => "server-logs-apache"
         }
      }
       if "jboss-log" in [fields][log_type] {
         elasticsearch {
         hosts => ["localhost:9200"]
         manage_template => false
         index => "server-logs-jboss"
         }
      }
       stdout { codec => rubydebug }    
    }

I'm expecting result as indexes : server-logs-apache, server-logs-jboss but actual output is empty.

2

You have fields_under_root set to true, so the events should have [log_type] set to jboss-log, not [fields][log_type].

Note that field names are case sensitive, so log_type and LOG_TYPE are different fields.

Are you looking at stdout and checking what the actual events look like? If not, I suggest you do so.

3

Hi @Badger

Thanks for the reply.

stdout is showing _type is doc and log_type is still not working and the weird thing is I'm not getting any error.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/user/vinit/pache/*.log
  fields:
    log_type: apache-log

- type: log
  enabled: true
  paths:
    - /home/user/vinit/boss/*.log
  fields:
    log_type: jboss-log
  fields_under_root: true
4

In stdout, do you have [log_type] or do you have [fields][log_type]?

5

I ran this script to check data
curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty' and I didn't get neither [log_type] nor [fields][log_type].

I am getting old data which I was created first time this is another problem I'm facing.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.