Logstash default document type from logs to doc

Elastic Stack Logstash
1

Hi forum,
I would like to ask an advice for a smooth transition from Logstash version 5.6.4 to 6.8.6, given that I have recently updated succesfully both Elasticsearch and Kibana.

My main doubts are about the breaking change of the document type field, that was by default logs in the former and then doc in the latter version mentioned above, for consistency with Beats.

I am aware, reading other posts on the same topic, that I can use the mutate statement to make the change within the filter section as follows:

filter{
        ...
        mutate {
          replace => { "type" => "doc" }
        }
}

But I would like to check if there is a strategy to perform this operation is a smooth way, avoiding data loss, since I am creating daily indexes and ideally I would like to start the new day with the correct type across all indexes.

Therefore I cannot apply the mutate statement without embedding it in a statement that for example compares the timestamp and applies the modification only after midnight, to avoid indexing issues due to the multiple conflicting type [doc, logs] within the same index.

Does anybody have any idea how this problem could be tackled?

Thanks in advance.

1 Like
2

The Elasticsearch output plugin will detect the version of Elasticsearch it is talking to, and attempt to do the right thing:

  • If it is talking to Elasticsearch 8+ where document types are removed, it will not send a document type at all (even if it is explicitly asked for with document_type); else
  • if document_type is specified, it is used; else
  • if it is talking to Elasticsearch 7.x, it will send _doc as the document type; else
  • if it is talking to Elasticsearch 6.x, it will send doc as the document type; else
  • it the event has a type fields, it will be used as the document type; else
  • it will fall back to doc if the type is not specified.

Sources:

3

Thanks for the reply Ry,
my question was more related to how to guarantee a smooth transition if I make the upgrade at some point in time say x.

I would like to avoid the condition where messages indexed before x have logs as type and messages indexed after x have type doc.

This is how I was thinking to modify the logstash configuration file, given that x is some point in time on 6th February and new indexes are created at night when @timestamp hour is 01:00

filter{
...

  if [@metadata][type] == "system_logs"
  {
    mutate{
      # enable timestamp comparison adding a new field
      add_field => { "str_dt_new" => "2020-02-07T01:00:00.000Z"}
    }
    date {
      match => ["str_dt_new", "YYYY-MM-dd'T'HH:mm:ss.SSSZ"]
      target => "constant_date"
    }
  }
  if [@timestamp] <= [constant_date] {
    mutate {
      # for consistency with messages sent before changing config file, in the 
      # same day (i.e. same ES index)
      replace => { "type" => "logs" }
    }
  }
  else {
    mutate {
      # required from LS v6.x
      replace => { "type" => "doc" }
    }
  }

}

Does it make sense to you?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.