Multiline pattern with backtick in content

(Olivier Boudet) #1

Hello,

I am trying to use kubernetes autodiscover with multiline pattern matching. It seems working for some java stacktrace but not for all. I found some logs having issue with multiline matching have backtick in the content.

Do I have a way to debug multiline matching in filebeat 6.6.0 ?

For example theses log lines are not grouped together :

2019-02-13 10:55:40.441 432f2c313174a7e737a975bfbeab2b21 ERROR 1 --- [nio-8080-exec-5] c.c.c.ws.dao.DatawarehouseDao  : Les données du rapport tauxPerteStructure n'ont pas pu être récupérés.

org.springframework.web.client.RestClientException: Error while extracting response for type [class com.mycompany.myproject.ws.domain.dw.DWResponse] and content type [application/json;charset=UTF-8]; nested exception is org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Cannot construct instance of `com.mycompany.myproject.ws.domain.dw.DWResponse` (although at least one Creator exists): no int/Int-argument constructor/factory method to deserialize from Number value (406); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `com.mycompany.myproject.ws.domain.dw.DWResponse` (although at least one Creator exists): no int/Int-argument constructor/factory method to deserialize from Number value (406)
 at [Source: (PushbackInputStream); line: 1, column: 1]
	at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:115) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
	at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:959) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
	at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:942) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:689) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:644) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

This is the configuration I use with filbeat 6.6.0 :

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat:
      config:
        modules:
          path: ${path.config}/modules.d/*.yml
      autodiscover:
        providers:
          - type: kubernetes
            templates:
              - condition:
                  equals:
                    kubernetes.labels.type: java
                config:
                  - type: docker
                    containers.ids:
                      - "${data.kubernetes.container.id}"
                    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} ' 
                    multiline.negate: true 
                    multiline.match: after
                    processors: 
                      - add_kubernetes_metadata: 
                          in_cluster: true 
              - condition:
                  contains:
                    kubernetes.labels.app: nginx
                config:
                  - module: nginx
                    access:
                      input:
                        type: docker
                        containers.stream: stdout
                        containers.ids:
                          - "${data.kubernetes.container.id}"
                        processors: 
                          - add_kubernetes_metadata: 
                              in_cluster: true 
                    error:
                      input:
                        type: docker
                        containers.stream: stderr
                        containers.ids:
                          - "${data.kubernetes.container.id}"
                        processors: 
                          - add_kubernetes_metadata: 
                              in_cluster: true 
              - config:
                  - type: docker
                    containers.ids:
                      - "${data.kubernetes.container.id}"
                    processors: 
                      - add_kubernetes_metadata: 
                          in_cluster: true 
    setup:
      template:
        name: "filebeat"
        pattern: "filebeat-*"
    
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']  
      index: "%{[kubernetes.namespace]:nonamespace}-filebeat-%{+yyyy.MM.dd}"
      pipelines:
        - pipeline: java-logs-pipeline
          when.equals:
            kubernetes.labels.type: java
        - pipeline: mongodb-logs-pipeline
          when.equals:
            kubernetes.labels.app: mongo-pod
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  labels:
    app: filebeat
data:
  kubernetes.yml: |-
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-modules
  labels:
    app: filebeat
data:
  nginx.yml: |-
    - module: nginx
      access:
        enabled: true
      error:
        enabled: true
(system) closed #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.