I have a log file in which i will search with a specific word which will give me that line but i need to get whole message. (started from "waiting for message" to another "waiting for message")

Elastic Stack Elasticsearch
1

18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessageXLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230] 18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main Waiting For A Message...
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015
4:[0704] 18/07/04 00:00:00 | main Waiting For A Message...

2

hi,
i think that you should parse you log file and then insert it to elasticsearch,
you can use filebeat or logstasg, below is an example conf of filebeat:

Multiline options

multiline.pattern: .*Waiting For A Message.*
multiline.negate: true
multiline.match: before

3

In first picture I have edit the multiline option in filebeat config file

Screenshot%20from%202018-10-22%2017-03-11
Screenshot%20from%202018-10-22%2017-03-11758×459 58.5 KB

In second picture when I search from "osama" it will just get me that particular line not the entire message

Screenshot%20from%202018-10-22%2017-03-27
Screenshot%20from%202018-10-22%2017-03-271366×671 146 KB

its not working i want the whole message
Thanks in advance

4

you should delete one of the parameters: multiline.match and set after or before
and try to change pattern with starting ^

image
image1812×801 127 KB

filebeat.yml:

filebeat.prospectors:

  • type: log
    enabled: true
    paths:

    • /var/log/nginx/*

    multiline.pattern: "^.*This is a new row.*"
    multiline.negate: true
    multiline.match: before

and used next test data:

18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessageXLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230] 18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015 4:[0704]
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main This is a new row....
18/07/04 00:00:00 | main G_GetMsg XLI-38159714: Message Length Received [230]
18/07/04 00:00:00 | main XLI-38159714: MessageSource [0028]
18/07/04 00:00:00 | main company
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-011 6:[234548]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-012 6:[234748]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-013 4:[0703]
18/07/04 00:00:00 | ProcessMessage XLI-38159714: DE-015
4:[0704] 18/07/04 00:00:00 | main This is a new row....

5

after editing in filebeat file kibana is not loading the latest the file which i have edited in my folder and filebeat status failed to start. please help out. Thanks

Screenshot%20from%202018-10-23%2012-22-34
Screenshot from 2018-10-23 12-22-34.png737×486 99.8 KB

6

can you send your filebeat.yml?
to kanagat.nugusbayev@gmail.com

7

its resolve thanks a lot its very helpful.
Now i need to know can i break the line when the next line starts while seeing the messages on kibana
Thanks Man in Advance