Connect the agent and agent running task in Logstash (help for parsing)

Elastic Stack Logstash
1

Hi all,
i am new to logstash and ask for the help in the following condition.

i will use a engine to run different java programs. each engine will have a session id and trigger the java program sequentially with job . But the java program don't have any knowledge about which engine is trigger the java program. So in the engine log , it will look like:

[INFO] Engine section:123456 job:123 start
[INFO] Java program A: start
[INFO] java program A: running
[INFO] java program A: end
[INFO] Engine section:123456 job:123 end

[INFO] Engine section:123456 job:124 start
[INFO] Java program B: start
[INFO] java program B: error
[INFO] Engine section:123456 job:124 end

i am wondering how we can use logstash plugins (if there are any), which may group the Engine job and engine run java program together (e.g use the plugins and add the section and job info to the java program log). So in kibana i may use the query and easily search all the log related to "section 123456 , job 123" instead of use timestamp for help

thx

capho

2

Use the multiline codec, that'll do it :slight_smile:

3

Thx .

just have one more question. Since the multiline codec will merge the several event to one based on the pattern. I would like to know if there are some other plugins which i may put the section and job id in the java program. Since i would like to keep the log as a individual . But also add the section and job id inside the java program log so that i may use it as a key for filter in Kibana

capho

4

to be more specified , what i want is just add the session id and job id into the json when logstash parse the log (i cannot do it in source code level since the java program don't have any information about the session and job)
i want to like to make it like from:

[INFO] Engine section:123456 job:123 start
[INFO] Java program A: start
[INFO] java program A: running
[INFO] java program A: end
[INFO] Engine section:123456 job:123 end
[INFO] Engine section:123456 job:124 start
[INFO] Java program B: start
[INFO] java program B: error
[INFO] Engine section:123456 job:124 end

to

[INFO] Engine section:123456 job:123 start
[INFO] Engine section:123456 job:123 Java program A: start
[INFO] Engine section:123456 job:123 java program A: running
[INFO] Engine section:123456 job:123 java program A: end
[INFO] Engine section:123456 job:123 end
[INFO] Engine section:123456 job:124 start
[INFO] Engine section:123456 job:124 Java program B: start
[INFO] Engine section:123456 job:124 java program B: error
[INFO] Engine section:123456 job:124 end

by doing this , i can have a field call session and job in the event so that kibana can use to display it with checking the time range.

Can some of you have any suggestion?

5

Not sure if that is possible sorry.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.