I have a log file in which i will search with a specific word which will give me that line but i need to get whole message. (started from "waiting for message" to another "waiting for message")

18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessageXLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230] 18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015
4:[0704] 18/07/04 00:00:00 | main Waiting For A Message...

hi,
i think that you should parse you log file and then insert it to elasticsearch,
you can use filebeat or logstasg, below is an example conf of filebeat:

Multiline options

multiline.pattern: .*Waiting For A Message.*
multiline.negate: true
multiline.match: before

In first picture I have edit the multiline option in filebeat config file

In second picture when I search from "osama" it will just get me that particular line not the entire message

its not working i want the whole message
Thanks in advance

you should delete one of the parameters: multiline.match and set after or before
and try to change pattern with starting ^

filebeat.yml:

filebeat.prospectors:

  • type: log
    enabled: true
    paths:

    • /var/log/nginx/*

    multiline.pattern: "^.*This is a new row.*"
    multiline.negate: true
    multiline.match: before

and used next test data:

18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessageXLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230] 18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015
4:[0704] 18/07/04 00:00:00 | main This is a new row....

1 Like

after editing in filebeat file kibana is not loading the latest the file which i have edited in my folder and filebeat status failed to start. please help out. Thanks

can you send your filebeat.yml?
to kanagat.nugusbayev@gmail.com

its resolve thanks a lot its very helpful.
Now i need to know can i break the line when the next line starts while seeing the messages on kibana
Thanks Man in Advance

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.