We need to collect MT4 logs (MetaTrader 4 trading platform).
Log-saving "Feature" in MT4 is implemented according to the following algorithm:
When creating a log file (or adding new logs to the current one), the platform "reserves" a place for writing a log in the form of appending several megabytes of null-characters (Null character - Wikipedia - I will refer to them as [NULL] below) to the end of log file, which, as necessary, are replaced by an informative text of the log.
Thus, in the directory that filebeat looks at when a new day arrives, a file appears consisting of only a few megabytes of [NULL] characters.
- Filebeat tries to read them, sees that all lines (or almost all but the first few lines) consist of [NULL] and in log.json the corresponding offset corresponding to the end of the file at its current size.
Example of filebeat registry log:
{"k":"filestream::mt4-pro-dev::native::5767168-842797-2932532775","v":{"updated":[516029731620,1675202469],"cursor":{"offset":1545639},"meta":{"identifier_name":"native","source":"C:\\MetaTrader4Server\\logs\\20230131.log"},"ttl":1800000000000}}
{"op":"set","id":475283}
-
MT4 starts writing an informative log to the file, replacing [NULL] characters with text. At the same time, the file size remains the same (only its contents change) and filebeat does not reread it.
-
When the text of the log occupies almost the entire log file, MT4 supplements the log file with a new portion of [NULL] characters. Filebeat tries to read new records starting from the last offset, but see again there are only [NULL] characters. Filebeat again reaches the end of the file without sending anything. And then again from point 2
As result - filebeat always seen only [NULL] characters and collect nothing.
Example of end of informative log (there is a about 3m [NULL] characters at the end of log file)
Cutted file example on gist: gist:0366bf186b89e9042459c3230c824f1b · GitHub (All [NULL] characters are invisible on line 20)
How can we harvest informative logs in this case?