Add current timestamp to a logstash_processed_at field

Elastic Stack Logstash
1

Currently there are a lot of FIlebeat instances in our infrastructure.
All of them are sending tons of logs to a single Logstash endpoint.
Some logs are not in json and come as a text. For example, Postfix logs.

Each log line has a syslog timestamp, which is parsed by grok pattern
and gets converted to a timestamp field on Logstash side.
That timestamp field gets converted to @timestamp by Logstash.

Quite often when there is a heavy load, Logstash queue becomes huge.
A lot of events have syslog timestamp which happened few minutes ago.

We would like to have DateTime.now() in @timestamp field.
Something like - @timestamp is the date when document appeared in ElasticSearch.

First idea was to change global grok pattern

SYSLOGBASE2 (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:

and match syslog timestamp not into timestamp field, but to syslog_timestamp.
But it's to risky, because it can affect other projects.

Second idea was to add a date filter like

date {
    add_field => { "logstash_processed_at" => "%{DATESTAMP}" }
}

But DATESTAMP is just a pattern and won't have current date.

2

Just copy the initial @timestamp value to another field (before your date filter that overwrites @timestamp).

3

I thought that logs like

Nov  1 04:09:02 my-hostname postfix/cleanup[13509]: 3yRj7y3g8wz36b0: info: header X-MyHeader: 1585

are processed by Logstash automatically without extra configuration.

I think that Logstash uses some default grok patterns like SYSLOGBASE2 to extract timestamp and replace @timestamp with extracted value. So, I don't have any custom date filters.

Idea number 3:

ruby { 
    code => "event.set('logstash_processed_at', Time.now());"
}
4

So what does your current configuration look like?

A ruby filter is of course also an option.

5

Something like this:

6

Yes, but what inputs do you use? I'm asking because the syslog input has grok and date filters built into it so if that's what you use you'll never get a chance to save the original @timestamp.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.