How to customize output format?

Elastic Stack Beats
1

Dear guys,

I wanted to customize output format for filebeat, like logback or log4j does. Does filebeat support it?

For example:

"%m" means output origin message

"%ip %m %t" means local ip, message and timestamp in a line

Thanks!

2

This is not supported by filebeat. Filebeat takes every single log line and forwards it without processing the content. For processing the content of log files, Logstash should be sued.

3

@jiming On a second thought: Were you above referring the the filebeat logging format?

4

@fuflin thanks for the reply.

I agree with you that filebeat should only do it's own job and let logstash do the rest.

What I expected is NOT filebeat logging format, I expect the output format.

The reason I want customize the output format is that I don't need the meta data added by filebeat. I just need my pure log message from log files.

If you do not think output format is necessory, at least please add a config, like "noBeatMeta" or "messageOnly", so the output is clean one.

Thanks!

Jiming

5

I think generic filtering support introduced in 5.x will help. See 5.0 alpha 3 docs.

One can use generic filtering to remove unwanted fields from events + drop events.

1 Like
6

@steffens Thanks. I read the docs. Yes, I can use this feature to filter out unnecessary fileds. But it still wrap in an json object. I expected a clean log entry.

For example, I have an log message:

"{ip:xxx.xxx.xxx, time:yyyy-mm-dd}"

I don't want it be changed to

{
message: "{ip:xxx.xxx.xxx, time:yyyy-mm-dd}"
}

I just want to get my origin message. Using logstash do convert is an option. But to a system with huge amount of logs, this wrap and unwrap make no sense but waste resource.

Thanks!

Jiming

7

Having a full object is on purpose, to make logs more searchable from within kibana. Main purpose of filebeat is to ship logs, not to process them. Elasticsearch 5.0 will get a so called ingest node, supporting even more filtering/processing.

8

@steffens

I can understand you define it on purpose. But my log item already contains everything I need to process next. The data which filebeat added is kind of like noisy. I think it is good idea to let users make the choice by themselves.

And BTW the timestamp that filebeat generated is not accurate if I read the document correct.

9

As I said, filebeat is not processing content. More elaborate processing needs to be done either via logstash or elasticsearch.

The timestamp generated by filebeat is the time filebeat did read the line from your log-file. In case filebeat has to deal with back-pressure from logstash/elasticsearch/... , it will be slowed down. If log-files contain timestamps themselves, one can use grok to parse the timestamp and compare log file timestamp with timestamp generated by filebeat to tell the difference.

10

This topic was automatically closed after 21 days. New replies are no longer allowed.