How to send all the content of the files as well as the creation time?

[anthony-o]

I would like to configure Filebeat so it would send the whole content of all files of a specific folder (the log files appear in this folder, one file per new event).

I have some problems configuring this use case because the last line of every file is not sent if there is no carriage return at the very last end of the file. As I can't add one, this is a blocker issue for me and it seems that it is simply not possible to configure Filebeat to send all the content anyway by looking at the FAQ...

I would like to know also if it is possible to add the creation time (and/or the modification time) of each file as a metadata or a field to the Logstash server?

Here is my current configuration of prospectors.d/http_queries.yml:

- type: log
  paths:
    - /var/log/http_queries/*
  multiline:
    pattern: ^POST|^GET
    negate: true
    match: after
  close_eof: true

[ruflin]

There is no standard way at the moment to send the complete file as one event if there is no newline as then filebeat assumes it's not a log event. You could "hack" around it potentially with multiline and a timeout.

For the meta data see https://github.com/elastic/beats/issues/1775 Currently not possible. For LS to enrich your event it would need also access to the same file system I think.

[anthony-o]

I tried to use the timeout and close_timeout parameters as well as define pattern: .* without any success (this last parameter is worse as it doesn't even send the content of the files previously sent).