Include One Identifier for each log entry

[mrashid (MBR)]

My aim is to include an identifier for logs of a few Jenkins Jobs that run in sequence.

Kindly give your opinion on the following --

1. Is there any way to append an ID such as '1234' to each document entry i.e each line of log by using logstash ? I am not planning to append any hash value , rather a readable ID .I can find ways to include an increment value for each log entry. However, I want to include only one ID such that a particular sequence of jobs can be identified.

2. Is it possible to pass a variable from a file to the filebeat index name ? The variable will hold an identifier such as a number.

Kindly help.

[mrashid (MBR)]

can someone kindly give his/her opinion on this ? Any idea if this is possible ?

[Christian_Dahlqvist]

Where do you get the value from? Is it available in all the documents, e.g. through a file path or something similar? How do you logically determine that two documents belong to the same job?

[mrashid (MBR)]

@Christian_Dahlqvist thanks for responding back. I get the value i.e the unique number in only one document entry of the first job of Jenkins. I don't get the number in every document entry. Had I got the number in every document entry I could have easily used Grok to filter and create a new Field. So comes the problem.

What I am planning to do -

1. Use grok to filter the number into a new field.
2. Use Ruby to write that field , which holds my unique number into a ruby file.
3. Use Ruby plugin to read from a path (the path will hold the ruby file with my unique number) and append a new field with my unique number to every document entry.

But I will have to create a new job at the beginning of the sequence of Jenkins jobs just to create the unique number.

Is my approach correct ? Any input that you would like to give?

[Christian_Dahlqvist]

Sounds tricky. Am not sure I have any good solution to recommend. I am not sure whether your solution would or wouldn't work either.

[mrashid (MBR)]

@Christian_Dahlqvist Any idea if I can write a field value into a .rb file ??

[mrashid (MBR)]

@Christian_Dahlqvist

But its not working

This is what the config file looks like in logstash-

input {
beats {
port => 5044
ssl => true
ssl_certificate => "/security/logstash.crt"
ssl_key => "/security/logstash.key"
}
}
filter {
grok {
match => { "message" => "^The build ID of the run is : %{NUMBER:filteredValues}" }
}
}
filter {
ruby {
code =>'File.open(/etc/logstash/output.rb, 'w') { |file| file.write("%(filteredValues)") }'
path => "/etc/logstash/test.rb"
script_params => { "percentage" => 100 }
}
}

It is the below grok filter which holds my unique number in the filed "filteredValues"

match => { "message" => "^The build ID of the run is : %{NUMBER:filteredValues}" }