Logstash6 recommended way of filtering for the future?

Elastic Stack Logstash
1

Hi all,
GZ for Elastic Stack 6.
I am still a noobie in ELK and after reading most of the changelog I have a question:
In https://www.elastic.co/blog/elasticsearch-6-0-0-released it is said that elastic is already going to remove the mapping types in the future. I used to set a field called "document_type" in my filebeat prospectors to have a entry point for my logstash filter (if [type] == "xyz" { grok { ..... }} and so on).
I understand that there are multiple pipelines in logstash6, what is great. I also understand, that I can define these pipelines based on the inputs, like it is shown in https://www.elastic.co/blog/logstash-multiple-pipelines. But I ask myself, what would be the best way to separate and grok logs, which have the same input (e.g. beats on port 5044) but are different kinds (e.g. access logs, syslogs, error logs etc) when the types are removed in the future? Till now it is based on the document_type field in my environment.

If I think sth. wrong, please excuse me and correct me. I am still hard working in learning ELK as fast as I can :slight_smile:
Thanks and regards,
Marcus

2

You have a couple of options:

  • Use a different field to indicate the kind of log and let all events share the same document type.
  • Continue to use the type field and store each document type in its own index series.
  • Continue to use the type field but hardcode the document type in the elasticsearch output to a fixed value.
3

Hi Magnus,
thanks for response.
I understand point 1. But point 2 and 3, should I really continue using it, when it will be removed?

Elasticsearch 7.x
The type parameter in URLs are optional. For instance, indexing a document no longer requires a document type.
The GET|PUT _mapping APIs support a query string parameter (include_type_name) which indicates whether the body should include a layer for the type name. It defaults to true. 7.x indices which don’t have an explicit type will use the dummy type name _doc.
The default mapping type is removed.
Elasticsearch 8.x
The type parameter is no longer supported in URLs.
The include_type_name parameter defaults to false.
Elasticsearch 9.x
The include_in_type parameter is removed.

4

The type field in a Logstash event has a loose coupling to the type of the document. It just so happens that the elasticsearch output by default uses the type value as the document type, but you can certainly set the output's document_type option to something else and continue to use type as a classifier of your events.

5

Ok thanks a lot!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.