Logstash configuration for new Version 6.5.4

Elastic Stack Logstash
1

Hello Experts,

I know this may look a basic question but its just brain draining for me to understand new logstash config as i'm getting below error:

[2019-01-29T21:27:30,230][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"syslog-2019.01.29", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x7e88287a>], :response=>{"index"=>{"_index"=>"syslog-2019.01.29", "_type"=>"doc", "_id"=>"zsY5nWgB6AmJPdJO_omb", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [syslog-2019.01.29] as the final mapping would have more than 1 type: [messages, doc]"}}}}

My logstash conf files with Old logstash and new logstash version:

Older Configuration with logstash 5.x:

[root@myelk04 ~]# cat /etc/logstash/conf.d/syslog.conf
input {
  file {
    path => [ "/data/SYSTEMS/*/messages.log" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "message", "_type", "_index", "_score", "path"]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
}
}
output {
        if [type] == "syslog" {
        elasticsearch {
                hosts => "myelk01:9200"
                manage_template => false
                index => "syslog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

Current configuration with 6.5.x:

I just removed the document_type => "messages" , Since then it's popping up the mention error.

[root@myelk04 ~]# cat /etc/logstash/conf.d/syslog.conf
input {
  file {
    path => [ "/data/SYSTEMS/*/messages.log" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => ["@version", "host", "message", "_type", "_index", "_score", "path"]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
}
}
output {
        if [type] == "syslog" {
        elasticsearch {
                hosts => "myelk01:9200"
                manage_template => false
                index => "syslog-%{+YYYY.MM.dd}"
  }
 }
}

How Could i change my new logstash file with 6.5 version.

OS VERSION: CentOS 7 and RHEL 7

2

Your 5.x configuration set the type of document to "messages"

"reason"=>"Rejecting mapping update to [syslog-2019.01.29] as the final mapping would have more than 1 type: [messages, doc]"

For an up to date elasticsearch output writing to elasticsearch clusters 6.x and above: the value of "doc" will be used.

You cannot have both in the same index. Basically you need to make the config change at the same time as the index rolls over.

3

Many thanks for your great Inputs, would you be able to provide the few example changes for my logstash.conf that will be grateful. However, if the still choose document_type it says its deprecated .

4

Yes, it is deprecated, but it still has a use, which is to force the document type to something other than doc. If you want to continue writing document to an index that already contains documents of type messages then you have to use it. As I said, you really need to remove document_type at the same time that you roll over to a new daily index.

5

Thanks @Badger , Is it Okay to use same version of Elastic , Kibana and Logstash or i should consider lower version of Logstash because as of now all are at 6.5.4.

However, I'm not sure if its supported in logstash 6.x if [type] == "syslog" { or this needs to replaced with some other new method.

6

I would keep the versions consistent if you can.

It's perfectly OK to have a field that tells you the document type. However, because elastic used the field "type" for that in the past and want to stop doing so, they warn you if you use the field type for that. But you can use another field, say "doctype" to tell you what your idea of the document type is. You just have to be consistent throughout to use the same field name.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.