First of all I'd like to do two things:
Apologize to all about my bad English.
Congratulate you about this amazing project (ELK).
My case is....
I have millions of text files with the following structure.
One "header" line, one "column header" line, multiple "data" lines and one "trailer" line.
The information i need to send to ES is the "data" lines, but the identification of the file is on the first line.
So, for each request (data line) I need to append the identification of the file before upload it to ES.
What is the proper way to do that?
I'm able to parse the text file using the 'csv' filter. But I don't know how to use data from the first line on other lines.
Maybe I can declare a variable in ruby when reading the first line and use it on the rest of the file?
How is that approach affected by multiple workers?
Some more info:
Example of a file:
v1234 10373 17/03/17 23:35:43 M DATE DATA1 DATA2 DATA3 26/10/16 06:40 1 1 (ok) 05 15/03/17 21:07 1 1 (ok) 00 2 89067EC2711277ABF279DC5B743BD9D250F78495 23296 1100
The above example is a file with one header, one column header, two data lines and one trailer.
I need to send both data lines to ES with the file identifier (10373).
Can anyone help me?
Thanks in advance.