Logstash Configuration when removing specified type mapping

Elastic Stack Logstash
1

Hi,

There is probably a simple fix to this. We've recently upgrade our elastic stack from 5.3 to 6.4, and we're having the mapping issues that many other people are having but we seem to be using it in a slightly different way and therefore, we're a little lost.

We use filebeat to set the document_type based on the log file prospector - for example:

prospectors:
    -
      paths:
        - /var/log/messages
        - /var/log/syslog
      document_type: syslog

This allows us in Logstash to apply the following output:

else {
elasticsearch {
   hosts => [ "**" ]
   index => "%{type}-%{+YYYY.MM.dd}"
   template_overwrite => true
}

}

For other document_types we create other indexes, with templates etc.

Given that document_type is now not working - what would be the best solution for creating an index based on the document_type?

2

@Josh_cullum

Probably you can achieve it by adding type => syslog in the logstash input section ??

3

Hi @Makra, how do you mean?

Our input configuration looks like this:

input {
  beats {
    port => 6782
  }
  lumberjack {
    port => 5782
    ssl_certificate => "/etc/logstash/ssl/logstash-forwarder.crt"
    ssl_key => "/etc/logstash/ssl/logstash-forwarder.key"
  }
  tcp {
    codec => "json"
    type => "curator"
    port => "28778"
  }
  file {
    path => [ "/var/log/logstash/logstash-plain.log"]
    type => "logstash"

The logic of separating out to various doc_types is done on the filebeat configuration:

-
      paths:
        - /var/log/exim/main.log
      document_type: exim

    -
      paths:
        - "/var/log/httpd/*-access_log"
      document_type: httpd

and then the logstash.elasticsearch output plugin has this:

else if [type] == "delivery_report" {
    elasticsearch {
      hosts => [ "**" ]
      index => "%{type}-%{_year}.%{_month}"
      action => "update"
      document_id => "%{delivery_report_id}"
      doc_as_upsert => true
    }
  }
}
  else if [type] == "pmta_acct" {
    elasticsearch {
       hosts => [ "**" ]
       index => "pmta_acct-%{+YYYY.MM.dd}"
       manage_template => true
       template => "/etc/logstash/templates/pmta_acct.json"
       template_name => "pmta_acct"
       template_overwrite => true
    }
  }

etc etc. So the need to be able to keep the a 1:1 relationship between a document_type and an Index is essential without causing massive security holes in the firewalls etc etc.

4

@Makra I'm assuming that something like this: https://stackoverflow.com/questions/45974963/elasticsearch-filebeat-document-type-deprecated-issue using custom fields would allow us to specify the same thing but using a custom field instead of document_type?

5

Your answer is here

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.