Use information of first line in the rest of the file

[thix (Thiago Santos)]

Hi, everyone.

First of all I'd like to do two things:

1. Apologize to all about my bad English.

2. Congratulate you about this amazing project (ELK).

My case is....

I have millions of text files with the following structure.

One "header" line, one "column header" line, multiple "data" lines and one "trailer" line.

The information i need to send to ES is the "data" lines, but the identification of the file is on the first line.
So, for each request (data line) I need to append the identification of the file before upload it to ES.

What is the proper way to do that?

I'm able to parse the text file using the 'csv' filter. But I don't know how to use data from the first line on other lines.
Maybe I can declare a variable in ruby when reading the first line and use it on the rest of the file?
How is that approach affected by multiple workers?

Some more info:

Example of a file:

v1234	10373	17/03/17 23:35:43	M
DATE	DATA1	DATA2	DATA3
26/10/16 06:40	1	1 (ok)	05	
15/03/17 21:07	1	1 (ok)	00	
2	89067EC2711277ABF279DC5B743BD9D250F78495	23296	1100

The above example is a file with one header, one column header, two data lines and one trailer.

I need to send both data lines to ES with the file identifier (10373).

Can anyone help me?

Thanks in advance.

[thix (Thiago Santos)]

This is what I`'m trying:

filter {
        csv {
               columns => ["data1", "data2", "data3", "data4"]
               separator => "	"
        }
	if ([data1] == "v1234") {
		ruby{
			init => "@@userId = ''"
			code => "@@userId = event.get('[data2]')"
		}
		drop { }
	}
	ruby{
		code => "event.set('usuario', @@userId)"
	}
}

But the value of "usuario" is always nil

[thix (Thiago Santos)]

Solved my own problem.

I was removing "data2" in the CSV filter.

I'll keep it here for someone with the same problem.