How to add other _types to the /etc/fields.yml in beats

Elastic Stack Beats
1

I have a beat that will have MANY different _types.

beats/libbeat/scripts/generate_template.py doesn't seem to support that?

Any help greatly appreciated.

It looks like you can add more _types with filebeat modles ect, that looks like what I want to do.

Thanks!

2

Packetbeat uses a different type for each protocol. Have a look at its fields.yml.

3

So I tried using multiple keys, but after running make update I believe that beats/libbeat/scripts/generate_template.py did not parse it correctly. I will try again.

4

I tried again and again it didn't seem to parse out he addtional _types with the key field

https://github.com/blacktop/brobeat/blob/master/etc/fields.yml

@andrewkroh is it because I am using the /etc/fields.yml and not the _meta folder?

I did a cookiecutter to init the beat, but maybe that was an older version?

Thanks

5

I blew my beat away and started from scratch and make update didn't pick up the extra -key _types :cry:

6

The vendored version of the script you have appears to be looking at _meta/fields.yml.

7

So I was able to figure out how to add more sub-fields except that the way that it works is it creates nested fields under the _default mapping.

I usually like to use document _types to pull apart.. well document types. Is there a reason you have designed it this way? I can see that you are going to start using the concept of modules. For example an nginx filebeat module that I assume would create fields like this:

_index: filebeat-*
_type: filebeat
nginx.access.request.method: GET

To me it makes sense to have it be like:

_index: filebeat-*
_type: nginx-access
request.method: GET

I just think more deeply nested field names are harder to query?

Thoughts?

8

This blog post should share some insights on types vs index: https://www.elastic.co/blog/index-vs-type In our case it didn't bring an advantage.

9

thank you @ruflin!

10

This topic was automatically closed after 21 days. New replies are no longer allowed.