How Do I Change The Format Of A Metric/File Beat Log Field?

[kss (Kurt Shull)]

I am using ElasticSearch FileBeat and MetricBeat to provide logging for my apps. I am using Grafana to visualize this log data.

I do have a question though regarding the host.hostname field....

This field is storing the hostname in the following format:

ip-10-109-4-123.us-west-2.compute.internal

I am wondering if if it possible to change this format so that it stores the hostname as follows:

ip-10-109-4-123

Or, can I add a new field to the this log record that is based on the hostname field but with the ".us-west-2.compute.internal" stripped off?

Thanks

[kavierkoo (Kavier)]

Hi Kss,

I know of few ways to do it:

1. you can use processors like Dissect for filebeat or metricbeat. This will add/modify the logs being stored into elasticsearch.
2. you can use Scripted field, this is on the fly hence no altering of logs but this is resource intensive on Kibana and might affect performance of Kibana.
3. If you have a logstash, it will be easier to filter with GROK, Dissect and etc. This will add/modify the logs being stored into elasticsearch.

Hope this helps you!

[legoguy1000 (Alex)]

hoat.hostname is populated by whatever is returned by the hostname command which is what's located in /etc/hostname. Is host.name populated? I would go with the dissect processor to generate the field u want from the fqdn as mentioned above.

[kss (Kurt Shull)]

Thanks Kavier, I'll give Dissect a try. I tried scripted fields, but it doesn't look like I can access them with Grafana.

[kss (Kurt Shull)]

I'm trying to get my brain around the syntax of dissect.

But first a question, does dissect create a new field, or does it operate on the source fields and modify it.

Second, what would the format be for the dissection. Based on the example in the doc this is what I'm coming up with.

Source string: ip-10-109-4-123.us-west-2.compute.internal
(I want only ip-10-109-4-123)

Would the format be: "%{ipaddress}.%{region}.%{level2domain}.%{topleveldomain}"

Would this ADD the three fields to the document: ipaddress, region, level2domain and topleveldomain?

[kss (Kurt Shull)]

This came in handy!
https://dissect-tester.jorgelbg.me/

[kss (Kurt Shull)]

Dissect works great. Thanks guys!