Problems in Elasticsearch combining two types of logs using two pipelines (Logstash)

Elastic Stack Logstash
,
1

Hi everyone, I need some help.
I have configured my ES stack inside a docker. I need to show two types of logs (syslog, log), in an Elastic index. I have created two configuration files

  • logstash.cong
  • logstash-syslog.conf

I also created two pipelines in logstash to manage the two files. Despite this Elasticsearch always shows only the logs of one of the two and never both together.

Configuration files below

logstash.cong

input {
  beats {
    port => 5044
  }
  tcp {
    port => 5000
  }

}

filter {
    mutate { replace => { "[host]" => "%{[host][name]}" } }
}

output {
  elasticsearch {

    hosts => "https://elasticsearch:9200"
    user => "user"
    password => "password"

    ecs_compatibility => disabled
    cacert => "config/elasticsearch-ca.pem"

    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "my-index"

  }
}

logstash-syslog.conf

input {
  tcp {
    port => 5044
    type => syslog
  }
  udp {
    port => 5044
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch {

    hosts => "https://elasticsearch:9200"
    user => "user"
    password => "password"

    ecs_compatibility => disabled
    cacert => "config/elasticsearch-ca.pem"

    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "my-index"

  }
}

pipelines.yml

- pipeline.id: pipeline_1
  path.config: "/usr/share/logstash/pipeline/logstash.conf"
  pipeline.workers: 3
- pipeline.id: pipeline_2
  path.config: "/usr/share/logstash/pipeline/logstash-syslog.conf"
  queue.type: persisted

How can i report the two types of logs in the elastic index?

Thanks in advance

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.