I have an Elastic cluster setup with a server hosting Zeek and Filebeat. We are manually running PCAP files through Zeek, Filebeat is picking up the logs, and data is being parsed/indexed by Elastic and we can see all the Zeek log data. I would like to see if anyone has been able to get Filebeat to pass along the full timestamp field from the Zeek log to Elastic. Currently, it cuts off part of the timestamp and only goes to the millisecond, but I would like it to keep all the way to the nano second. I've tried changing the fields.yml file to say the date field is type date_nanos, but that did not work. I've also tried to create a new field and pass it the zeek.conn.ts field, which is the "ts" field from the Zeek connection log and it creates the new field in Elastic, but it is still the same truncated timestamp with no nano seconds. It's almost as though Filebeat cannot handle date/time that goes to the nano second.
We have a log ongoing work around nanosecond.
For beats and logstash
One of the PRs adds a
fraction of second formatting possibility for time formatting:
// S fraction of second nanoseconds yes 978000 // f fraction of seconds nanoseconds yes 123456789
Should be merged in 7.7.0.
Would you mind to share, how exactly does your field from the log file look like?
Here is a snipit of a line from our Zeek Connection log. It shows the full timestamp. We are using the default config and Elastic pipelines that come with Filebeat.