White space breaks multi line nagate behavior

[madurad (Madura)]

Hi,

I'm trying to send Error stack to ELK server using filebeat installed in one of my server. Following is the sample log entry.

And following is the multiline.patter I have configured,

multiline.pattern: '^\d{4}-(0?[1-9]|1[0-2])-(0?[1-9]|[12][0-9]|3[01])'
multiline.negate: true
multiline.match: after

But when I do restart filebeat and see the result in Kibana, I can see the Error stack is separated for each line one-by-one as given below.

Question : How can I get all those Error stack in to a one single entry in Kibana ?

Any help would be greatly appreciated.

Thanks.

[kvch (Noémi Ványi)]

Could you please share an example log as text, so it can be copy pasted?

[madurad (Madura)]

test file are not allowed to upload here, anyway I have copied and pasted log entry here and , you can see image has been attached in initial question.

2018-05-02 06:45:10 | [http-nio-9092-exec-2] | 514 | INCENTIVO_CONPOS_SILVERWARE | 8f771490c9a8597e | 8f771490c9a8597e | ERROR | c.i.c.s.a.service.UserService | Error occurred while updating user in silverware: 500 Internal Server Error | {"First-Name":"Kalhandk","Last-Name":"Dino","External-User-id":"58919","Client-id":"f33e5562-b971-4895-b7d1-0161724608a0"} |

    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:597)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:475)
    at com.incentivio.connector.silverware.application.service.UserService.updateUserInSilverware(UserService.java:264)
    at com.incentivio.connector.silverware.api.UserController.updateUser(UserController.java:44)
    at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:832)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerA`

[kvch (Noémi Ványi)]

I tested your multiline config and it seems to work. Could you enable debug logging (./filebeat -e -d "*") to see what messages are published?

[madurad (Madura)]

Here is the Debugging information :

"@timestamp": "2018-05-02T09:20:04.073Z", "beat": { "hostname": "nodeD", "name": "nodeD", "version": "5.5.0" }, "input_type": "log", "message": "2018-05-02 08:48:10 | [http-nio-9092-exec-2] | 514 | INCENTIVO_CONPOS_SILVERWARE | 8f771490c9a8597e | 8f771490c9a8597e | ERROR | c.i.c.s.a.service.UserService | Error occurred while updating user in silverware: 500 Internal Server Error | {\"First-Name\":\"Kalhandk\",\"Last-Name\":\"Dino\",\"External-User-id\":\"58919\",\"Client-id\":\"f33e5562-b971-4895-b7d1-0161724608a0\"} |", "offset": 14587, "source": "/opt/ALLMODULESLOG/test/test.log", "type": "mixlog" }

2018-05-02T09:20:04Z DBG Publish: { "@timestamp": "2018-05-02T09:20:04.073Z", "beat": { "hostname": "nodeD", "name": "nodeD", "version": "5.5.0" }, "input_type": "log", "message": "org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error", "offset": 14670, "source": "/opt/ALLMODULESLOG/test/test.log", "type": "mixlog" }

2018-05-02T09:20:04Z DBG Publish: { "@timestamp": "2018-05-02T09:20:04.073Z", "beat": { "hostname": "nodeD", "name": "nodeD", "version": "5.5.0" }, "input_type": "log", "message": " at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94)", "offset": 14789, "source": "/opt/ALLMODULESLOG/test/test.log", "type": "mixlog" }

2018-05-02T09:20:04Z DBG Publish: { "@timestamp": "2018-05-02T09:20:04.073Z", "beat": { "hostname": "nodeD", "name": "nodeD", "version": "5.5.0" }, "input_type": "log", "message": " at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:641)", "offset": 14882, "source": "/opt/ALLMODULESLOG/test/test.log", "type": "mixlog" }

For more information, I've added sample screenshot here.

[kvch (Noémi Ványi)]

Is the multiline pattern inside '?

Your config should look like the following, without ':

multiline.pattern: ^\d{4}-(0?[1-9]|1[0-2])-(0?[1-9]|[12][0-9]|3[01])
multiline.negate: true
multiline.match: after

[madurad (Madura)]

Still getting the same output, no luck

[kvch (Noémi Ványi)]

Well, for me it works.
Could you please provide the full config formatted using </>? You might have an indentation problem somewhere.

[madurad (Madura)]

filebeat.prospectors:

- input_type: log

paths:
- /opt/ALLMODULESLOG/test/test.log

document_type: mixlog

multiline.pattern: ^\d{4}-(0?[1-9]|1[0-2])-(0?[1-9]|[12][0-9]|3[01])

# Defines if the pattern set under pattern should be negated or not. Default is false. multiline.negate: true

# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash multiline.match: after

output.logstash:
# The Logstash hosts
hosts: ["xx.xx.xx.xx:5044"]

logging.level: debug

`

[kvch (Noémi Ványi)]

Could you please format it so everything is in the same code block. In this format I cannot see if the indentation is correct.

[madurad (Madura)]

filebeat.prospectors:

- input_type: log

paths:
- /opt/ALLMODULESLOG/test/test.log

document_type: mixlog

multiline.pattern: ^\d{4}-(0?[1-9]|1[0-2])-(0?[1-9]|[12][0-9]|3[01])

# Defines if the pattern set under pattern should be negated or not. Default is false. multiline.negate: true

# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash multiline.match: after

output.logstash:
# The Logstash hosts
hosts: ["xx.xx.xx.xx:5044"]

logging.level: debug

[kvch (Noémi Ványi)]

I corrected the whitespace error I have seen. Could you try this config?

filebeat.prospectors:

- input_type: log
  paths:
    - /opt/ALLMODULESLOG/test/test.log

  document_type: mixlog

  multiline.pattern: ^\d{4}-(0?[1-9]|1[0-2])-(0?[1-9]|[12][0-9]|3[01])

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after

output.logstash:
  # The Logstash hosts
  hosts: ["xx.xx.xx.xx:5044"]

logging.level: debug

[madurad (Madura)]

That's great, it's working now. Thanks a lot for your support.

But still I couldn't realize what was the change I had ? As per your last comment, it was already there, and I added single space in at the beginning of the line. (Ex : multiline.pattern: xxxxxx).

And now I can see two entries for each log line

Anyway thanks for your great help...!!!