Filebeat for both system|service logs and application logs

[coudenysj (Jachim Coudenys)]

Hi,

I'm having trouble to pick a solution to use filebeat for both the default system|service log files and application log files.

I'm logging my application logs in json, so not much processing must be done, but I would like to store the application logs in a different index (instead of filebeat-*) because a lot of application log fields are not available in the filebeat ECS list.

Should I just add those fields to the filebeat template? Or should I start a different filebeat process to ship those logs to the different index (because filebeat cannot output to different indexes)?

[steffens (Steffen Siering)]

Filebeat can output to different indices. But you'd need to setup the template for other indices yourself (plus you'd might have to disable ILM).

e.g.

filebeat.inputs:
- input: log
  ...
  fields.index: applog  // output to 

- input: log
  ...

output.elasticsearch:
  index: '%{[fields.index}:filebeat}-%{[agent.version}}-%{+yyyy.MM.dd}'

[coudenysj (Jachim Coudenys)]

Thanks for the feedback!

It looks like my setup ignores the output.elasticsearch.index setting.

I had to enable specify the setup.template.name and setup.template.pattern (which I gave the default value), but filebeat is not creating the different indices.

I used this value (just to be sure it was changing):

index: "%{[fields.index]:filebeat}-%{[agent.version]}---%{+yyyy.MM.dd}"

I restarted filebeat multiple times, but nothing is changing (it keeps writing to the filebeat-7.2.0-2019.07.11-000001 index).

[coudenysj (Jachim Coudenys)]

I've got it working like this:

  indices:
    - index: "%{[fields.index]:filebeat}-%{[agent.version]}-%{+yyyy.MM.dd}"

But the normal index approach wasn't doing much, probably a bug?

[coudenysj (Jachim Coudenys)]

The problem now, is that the ILM is acting crazy. My filebeat logs aren't written into the -00001 index. Is this something I can fix?

[steffens (Steffen Siering)]

When using the index setting you have to disable ILM. Did you disable ILM? When using ILM one has to use a write alias. By default the write alias is filebeat-7.2.0. This alias creates indices named filebeat-7.2.0-<date>-0000001. You have one index with and one index without ILM configured.

If you want to use custom indices with ILM, then you will need to setup ILM yourself and configure index such the Beats will write to the write alias you've setup beforehand.

[coudenysj (Jachim Coudenys)]

Thanks for the reply @steffens!

So I'm probably better of just indexing my custom data (which also follows the ECS as much as possible) in the default filebeat indices?

That way the ILM is working as expected and the overhead of the filebeat mapping will be minimal, correct?

[steffens (Steffen Siering)]

So I'm probably better of just indexing my custom data (which also follows the ECS as much as possible) in the default filebeat indices?

That way the ILM is working as expected and the overhead of the filebeat mapping will be minimal, correct?

Well, as always it depends

But I think I would agree here. If you are unsure, add some meta-data to your events. This would allow you to filter and reindex said events in the future, in case you want to change something.