Syslog output date format

Elastic Stack Logstash
1

Hi,

I am running Windows logs successfully to our SIEM using a kafka input and a logstash syslog output. The logs are parsing perfectly in the SIEM.

When running a virtually identical pipeline to poll a different topic from kafka (AWS Cloudtrail logs) and send to the SIEM using an identical output the logs won't parse due to an additional "." after the month in the syslog timestamp, for example:

<13>Feb. 15 22:35:33 abd1234.blah.com.au LOGSTASH-AWS[-]

Is there a known method to format the timestamp produced by the logstash-syslog-output plugin, or a field I can target with the date filter.

This seems to be non-configurable, but I'm hoping someone can tell me otherwise.

Thanks.

2

Are you saying that the . after Feb is in the message received from kafka and it causes a problem downstream?

3

Hi,

No, the problem appears to be in the conversion and formatting within the logstash-syslog-output. The kafka object is as follows, and contains only timestamps. I just need the date in the syslog output to be of the form <13>Feb 15 22:35:33

{
  "@timestamp": "2021-02-11T01:07:51.125Z",
  "@version": "1",
  "cloudfront_version": "1.0",
  "cloudfront_fields": "XXXX",
  "message": "2021-02-11\t01:02:22\tPER50-C1\t1500\XXX",
  "fields": {
    "type": "cloudfront"
  },
  "s3": {
    "last_modified": "2021-02-11T01:07:23.000Z",
    "file": "2021-02-11-01.95fbae1e.gz"
  }
}

Any ideas would be great! Thanks.

4

and yes, the "." causes a parsing error in the SIEM.

5

The syslog output uses a sprint pattern of "%{+MMM dd HH:mm:ss}". The sprintf code insert the date and time using a Joda DateTimeFormat, which in turn, I believe, uses this function. The text it inserts is locale specific. I am not aware of any locales where the short month name includes the period, but that is the only thing I can think of that cause two syslog outputs to produce different output.

6

Many thanks for your time and help. This seems to be related to a known JDK bug for Australian locales:

https://bugs.openjdk.java.net/browse/JDK-8208487

Frustrating!

7

OK, so according to the bug you link to you might have a workaround by editing jvm.options. You may be able to upgrade out of it, or you may be able to change JVM providers. Also, you could consider running your JVM with another locale setting instead of en_AU, maybe en_GB or en_US. Or let us never forget ... en_CA. Lastly, you could route the syslog output to another pipeline that I think could use a tcp input,

 filter { mutate { gsub => [ "message", "^(<\d+>\w{3})\.", "\1" ] } }

(or something like that), and then another tcp output. You might have issue with line endings that would require you to mess with message options or codec format options on the inputs/outputs.

8

Thanks.

Trying to upgrade out of it, but have also considered pipeline-to-pipeline to run a gsub ... might be expensine though :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.