I have log file with this structure:
the file daily have about 40mil rows
I gather the files from production by filebeat
I need to merge all lines with same ID into 1 record,
all IDs are stored always together in multiple lines
the question is
?How to create MULTILINE configuration PATTERN to match lines with same ID?
would it be better solution to use agregate filter in logstash?
with aggreagate filter I concern of assumption to set workers to 1 .