We need to collect MT4 logs (MetaTrader 4 trading platform).
Log-saving "Feature" in MT4 is implemented according to the following algorithm:
When creating a log file (or adding new logs to the current one), the platform "reserves" a place for writing a log in the form of appending several megabytes of null-characters (Null character - Wikipedia - I will refer to them as [NULL] below) to the end of log file, which, as necessary, are replaced by an informative text of the log.
Thus, in the directory that filebeat looks at when a new day arrives, a file appears consisting of only a few megabytes of [NULL] characters.
- Filebeat tries to read them, sees that all lines (or almost all but the first few lines) consist of [NULL] and in log.json the corresponding offset corresponding to the end of the file at its current size.
Example of filebeat registry log:
MT4 starts writing an informative log to the file, replacing [NULL] characters with text. At the same time, the file size remains the same (only its contents change) and filebeat does not reread it.
When the text of the log occupies almost the entire log file, MT4 supplements the log file with a new portion of [NULL] characters. Filebeat tries to read new records starting from the last offset, but see again there are only [NULL] characters. Filebeat again reaches the end of the file without sending anything. And then again from point 2
As result - filebeat always seen only [NULL] characters and collect nothing.
Example of end of informative log (there is a about 3m [NULL] characters at the end of log file)
Cutted file example on gist: gist:0366bf186b89e9042459c3230c824f1b · GitHub (All [NULL] characters are invisible on line 20)
How can we harvest informative logs in this case?