Logstash 6.2 multiple input confused

Elastic Stack Logstash
1

env：logstash 6.2

I touch two config files in /etc/logstash/conf.d , eg：a.conf b.conf

a.conf input from kafka, setting type is 'aa'

b.conf input from file, setting type is 'bb'

Both output are same ES

The question is the data from aa's input into bb's output

eg： index_day write into ES index bb

My confuse that why aa.conf and bb.conf are not standalone like nginx!!!

aa.conf

input {
  kafka {
        bootstrap_servers => "kafka0:19092,kafka1:19093,kafka2:19094"
        topics => [ "aa" ]
        codec => "json"
        type => "aa"
        group_id => "aa"
        consumer_threads => 2

  }
}

filter {

  grok {
        match => [ "message" , "(20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME:time})\s+%{LOGLEVEL:level}" ]
  }

  mutate {
       add_field => [ "log_time","20%{year}-%{month}-%{day} %{time}" ]
  }

  date {
        match => [ "log_time","yyyy-MM-dd HH:mm:ss.SSS" ]
        target => "@timestamp"
  }

  ruby {
        code => [ "event.set('index_day', event.get('@timestamp').time.localtime.strftime('%Y.%m.%d'))" ]
  }

  mutate {
       remove_field => ["[beat][name]","[beat][version]","@version","offset","tmptime","log_time","year","month","day","time"]
  }

}

output {
  if [type] == "aa" {
      elasticsearch {
        codec => plain{ charset => "UTF-8" }
        hosts => "http://es1:9200"
        index => "%{[fields][log_topic]}-%{index_day}"
      }
  }
}

bb.conf

input {
  file {
      path => [ "/data/bba/mobile/*" ]
      start_position => "beginning"
      type => "bb"
      codec=> multiline {
         pattern => "^\[([0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})\]([0-9]{4}-[0-9]{2}-[0-9]{2})"
         negate => true
         what => "previous"
      }
  }
}



filter {

     grok {
           match => [ "message" , "\[%{TIME:time}\](20%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL}" ]
     }

     mutate {
          add_field => [ "log_time","20%{year}-%{month}-%{day} %{time}" ]
     }

     date {
           match => [ "log_time","yyyy-MM-dd HH:mm:ss.SSS" ]
           target => "@timestamp"
     }

}

output {

  if [type] == "bb" {
    elasticsearch {
      codec => plain{ charset => "UTF-8" }
      hosts => "http://es1:9200"
      index => "bb"
    }
  }

}
2

It looks like you want the Multiple Pipelines feature, which is used a bit differently.

When multiple files are placed in the conf.d directory, they are concatenated together and run as a single pipeline.

3

Thank you for your reply!!!

I settting it following in /etc/logstash/logstash.yml ,but it is not working neither cant writing data to ES:

pipeline.id: aa
path.config: "/etc/logstash/conf.d/aa.conf"

pipeline.id: bb
path.config: "/etc/logstash/conf.d/bb.conf"
4

Have a look at the pipeline-to-pipeline communication feature.
You can send aa's input to bb's pipeline and send it to ES in bb's output

5

Thank you for your reply!!!

I don't want to send aa's input into bb's output! I need them to be standalone!

The question is the data from aa's input into bb's output result in confused!

6

The docs that I linked call out that pipelines.yml is a separate config file.

You will also want to be careful of syntax (note the indentation and the yaml list items):

- pipeline.id: aa
  path.config: "/etc/logstash/conf.d/aa.conf"
- pipeline.id: bb
  path.config: "/etc/logstash/conf.d/bb.conf"
7

Thank you for your reply again!

but it is still not working

8

My best guess is that your pipelines.yml either isn't in the config directory, or isn't readable by the user under which Logstash is running.

What do your logs say is happening? You may need to enable debug-level logging (by adding --log.level debug command-line flag or by setting log.level: debug in your logstash.yml).

9

Thank you very much for your reply again!

I am sorry! I make a mistake! :slightly_frowning_face: I modify a wrong config file!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.