[6.0.0alpha] Packetbeat - multiple types

Elastic Stack Beats
1

While running fresh Elastic stack version 6.0.0 alpha, most Packetbeat events cannot be indexed.

I have loaded Packetbeat template into Elasticsearch before running Logstash and Packetbeat. Logstash only indexes events with one type and reports error when trying to index different type.

By default Elasticsearch 6.0.0 forces single type indices, while Packetbeat is using multiple types (http,icmp,flow).

Adding index.mapping.single_type : false to the index template fixes the problem.

2

Hi, just for clarity, you tested Packetbeat 6.0.0-alpha1 with ES 6.0.0-alpha1? I'd expect this to work, but Packetbeat 5.x with ES 6.0.0-alpha1 indeed doesn't work without that change. We are planning to have Packetbeat 5.5 work with ES 6.0 without any workarounds.

3

Hi, Tudor,
I have tested versions 6.0.0-alpha1 of all components, including Packetbeat.

To clarify it more: Packetbeat -> ES works. The "_type" is always "doc", which means it's single type.

Packetbeat -> Logstash -> ES does not work. Packetbeat also sends field named "type" that is automatically copied into field "_type" by Logstash. This is probably the cause.

4

Ah, gotcha. Which Logstash config did you use? We've recently updated our docs to not set a type explicitly.

5

The logstash's pipeline config was:

input {
  beats {
    port => "5044"
  }
}
filter {
}
output {
  elasticsearch {
    index => "packetbeat-6.0.0-alpha1-%{+YYYY.MM.dd}"
  }
}
6

Interesting, I don't seem to be able to reproduce that when using the master version. The type differs per document, but _type is always set to doc and Logstash doesn't seem to change it.

7

May be changed between versions we are using. I am using version 6.0.0-alpha1 available for download on elastic.co (release date May 09). Just tried it again, deleting ES directory and extracting new one. I can now link the warning from Logstash. If it's working on master version I believe it was fixed and we can close this.

[2017-05-29T16:18:14,245][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400,
:action=>["index", {:_id=>nil, :_index=>"packetbeat-6.0.0-alpha1-2017.05.29", :_type=>"http", :_routing=>nil}, 2017-05-2
9T14:18:12.702Z MyPC %{message}], :response=>{"index"=>{"_index"=>"packetbeat-6.0.0-alpha1-2017.05.29", "_type"=>
"http", "_id"=>"AVxUklE_UDJvpVdzoTzC", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecti
ng mapping update to [packetbeat-6.0.0-alpha1-2017.05.29] as the final mapping would have more than 1 type: [doc, http]"
}}}}
8

Yeah, i also suspect it was fixed in the meantime. We're going to release alpha2 relatively soon. Thanks for researching this!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.