Packetbeat compatibable with Elastic 2.4

Elastic Stack Beats
1

Is any of the versions of packetbeat compatible with Elasticsearch 2.4

I am getting a lot of "can't contain '." in the field name" I am running the de_dot filter but was hoping I am just running the wrong version of packetbeat (1.3)

2

Both Packetbeat 1.x and Packetbeat 5.x should be compatible with ES 2.4. What protocol and what field contains a dot in the field name?

3

can't be, in ELK 2.X they disallowed fields with dot's in the field name. (1.X and 5.X it was re-enabled)

I know when I upgraded from 1.3 last year i had to disable packetbeat cause of this. I am just now getting back to it to trouble shoot a production issue

I am working on getting the exact field. but I am logging alot so I will have to write another config to get the raw data

4

Ok pulled this from the logs, sorry had to blank alot of the data but here are the important parts of the capture.

as you can see the cookie being parsed is creating fields with "." in them.

 "request_headers"=>{"accept"=>"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
 "accept-encoding"=>"gzip", 
"accept-language"=>"ko,zh;q=0.8,en;q=0.6", 
"akamai-origin-hop"=>"",
 "cache-control"=>"no-cache, max-age=0",
 "connection"=>"Keep-Alive", 
"cookie"=>{"__cmbdomtm"=>"0", 
"__cmbtpvtm"=>"2326",
 "_sp_id.2b02"=>"XXXXXXX", 
"cart"=>"XXXXX", 
"cm.byogokaotfnub9vbngntuq4o.aotfnuuzhwwomen"=>"XXXXXX", "cm.byokzxaotfnub9vyupntuq4o.aotfnujuhfurla"=>"XXXXX",
 "cm.byoyduaotfnub9vyrrntvk3r.aotfnuu$h"=>"XXXXX", 

"@version"=>"1", 
"host"=>"hd1pxx24lx", 
"dst_index"=>"infra_packetbeat", 
"tags"=>["beats_input_raw_event", 
"_grokparsefailure"],
 "filename"=>"redirect.jsp", 
"kafka"=>{"msg_size"=>12708, 
"topic"=>"logstash", 
"consumer_group"=>"logstash", 
"partition"=>4, 
"offset"=>594050844, 
"key"=>nil}, 
"indextime"=>"2016-12-09T19:47:41Z"}, 
"type"]}>>], 
:response=>{"create"=>{"_index"=>"infra_packetbeat-2016.12.09", 
"_type"=>"http", "_id"=>"AVjlIJGVcnKzVuhX5iTY", 
"status"=>400, 
"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field name [_sp_id.2b02] cannot contain '.'"}}}, :level=>:warn}
5

The worst part is that that DE_DOT is not catching those fields even with nested turned on and it is fulling up my system
:frowning:

Probably try to turn off cookie parsing but that is the feature I really want .:frowning:

6

What is the configuration you use for the de_dot filter? By default it only works on the top level fields. You have to set the fields to get it to work on sub-fields.

7

I have nested set which seems the only option to deal with that

filter{
  de_dot{
	nested => true
  }
}
8

You could configure Elasticsearch 2.4 to allow dots in field names. See https://www.elastic.co/guide/en/elasticsearch/reference/2.4/dots-in-names.html#_enabling_support_for_dots_in_field_names

9

well that works as a solution but can't be implemented quickly as I have a 30TB in the cluster. :frowning:

but will add that option so I have it

THANKS! That solves a lot of issues. funny I never came across it.

10

This topic was automatically closed after 21 days. New replies are no longer allowed.