I have a log file in which i will search with a specific word which will give me that line but i need to get whole message. (started from "waiting for message" to another "waiting for message")


(osama ikhlas) #1

18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessageXLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230] 18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015
4:[0704] 18/07/04 00:00:00 | main Waiting For A Message...


(Kanagat) #2

hi,
i think that you should parse you log file and then insert it to elasticsearch,
you can use filebeat or logstasg, below is an example conf of filebeat:

Multiline options

multiline.pattern: .*Waiting For A Message.*
multiline.negate: true
multiline.match: before


(osama ikhlas) #3

In first picture I have edit the multiline option in filebeat config file

In second picture when I search from "osama" it will just get me that particular line not the entire message

its not working i want the whole message
Thanks in advance


(Kanagat) #4

you should delete one of the parameters: multiline.match and set after or before
and try to change pattern with starting ^

filebeat.yml:

filebeat.prospectors:

  • type: log
    enabled: true
    paths:

    • /var/log/nginx/*

    multiline.pattern: "^.*This is a new row.*"
    multiline.negate: true
    multiline.match: before

and used next test data:

18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessageXLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230] 18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015
4:[0704] 18/07/04 00:00:00 | main This is a new row....


(osama ikhlas) #5

after editing in filebeat file kibana is not loading the latest the file which i have edited in my folder and filebeat status failed to start. please help out. Thanks


(Kanagat) #6

can you send your filebeat.yml?
to kanagat.nugusbayev@gmail.com


(osama ikhlas) #7

its resolve thanks a lot its very helpful.
Now i need to know can i break the line when the next line starts while seeing the messages on kibana
Thanks Man in Advance


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.