Multiline error

egonzalez · August 16, 2017, 1:36pm

I have problem with filebeat multiline options.

/usr/bin/filebeat -version
filebeat version 5.5.1 (amd64), libbeat 5.5.1

Fragment of my server.log:

10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3)        servletPath=
10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3)           isSecure=true
10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3) ---------------------------------------------------------------
10:25:14,704 INFO  [stdout] (ajp-/172.16.15.61:8509-3) Inicialice el provider de MedioPagoConfigService EJB
10:25:15,249 INFO  [org.hibernate.engine.internal.StatisticalLoggingSessionEventListener] (ajp-/172.16.15.61:8509-3) Session Metrics {
    19772077 nanoseconds spent acquiring 8 JDBC connections;
    378558 nanoseconds spent releasing 8 JDBC connections;
    23685207 nanoseconds spent preparing 10 JDBC statements;
    59644248 nanoseconds spent executing 10 JDBC statements;
    0 nanoseconds spent executing 0 JDBC batches;
    0 nanoseconds spent performing 0 L2C puts;
    0 nanoseconds spent performing 0 L2C hits;
    0 nanoseconds spent performing 0 L2C misses;
    23947144 nanoseconds spent executing 3 flushes (flushing a total of 589 entities and 18 collections);
    8098 nanoseconds spent executing 2 partial-flushes (flushing a total of 0 entities and 0 collections)
}
10:25:15,291 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3) ---------------------------------------------------------------
12:51:10,842 INFO  [service.MotorReglas.ejecutarRegla] (ajp-/172.16.15.61:8109-7) Service  : MotorReglas, Operation: ejecutarRegla
Ext. ref: 1502812269900194
[[
Creamos un FlujoResponse con un Servicio de Usuario.
]]
12:51:10,843 INFO  [service.MotorReglas.ejecutarRegla] (ajp-/172.16.15.61:8109-7) Service  : MotorReglas, Operation: ejecutarRegla
Ext. ref: 1502812269900194
[[
Creamos un FlujoResponse con una Regla Formulario.
]]

My filebeat config:

multiline:
      match: after
      negate: true
      pattern: ^[[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}

This config don´t work.

I tested my config and log with this site https://play.golang.org/ and it works well.

package main

import (
	"fmt"
	"regexp"
	"strings"
)

var pattern = `^[[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}`
var negate = true

var content = `10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3)        servletPath=
10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3)           isSecure=true
10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3) ---------------------------------------------------------------
10:25:14,704 INFO  [stdout] (ajp-/172.16.15.61:8509-3) Inicialice el provider de MedioPagoConfigService EJB
10:25:15,249 INFO  [org.hibernate.engine.internal.StatisticalLoggingSessionEventListener] (ajp-/172.16.15.61:8509-3) Session Metrics {
    19772077 nanoseconds spent acquiring 8 JDBC connections;
    378558 nanoseconds spent releasing 8 JDBC connections;
    23685207 nanoseconds spent preparing 10 JDBC statements;
    59644248 nanoseconds spent executing 10 JDBC statements;
    0 nanoseconds spent executing 0 JDBC batches;
    0 nanoseconds spent performing 0 L2C puts;
    0 nanoseconds spent performing 0 L2C hits;
    0 nanoseconds spent performing 0 L2C misses;
    23947144 nanoseconds spent executing 3 flushes (flushing a total of 589 entities and 18 collections);
    8098 nanoseconds spent executing 2 partial-flushes (flushing a total of 0 entities and 0 collections)
}
10:25:15,291 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3) ---------------------------------------------------------------
12:51:10,842 INFO  [service.MotorReglas.ejecutarRegla] (ajp-/172.16.15.61:8109-7) Service  : MotorReglas, Operation: ejecutarRegla
Ext. ref: 1502812269900194
[[
Creamos un FlujoResponse con un Servicio de Usuario.
]]
12:51:10,843 INFO  [service.MotorReglas.ejecutarRegla] (ajp-/172.16.15.61:8109-7) Service  : MotorReglas, Operation: ejecutarRegla
Ext. ref: 1502812269900194
[[
Creamos un FlujoResponse con una Regla Formulario.
]]
`

func main() {
	regex, err := regexp.Compile(pattern)
	if err != nil {
		fmt.Println("Failed to compile pattern: ", err)
		return
	}

	lines := strings.Split(content, "\n")
	fmt.Printf("matches\tline\n")
	for _, line := range lines {
		matches := regex.MatchString(line)
		if negate {
			matches = !matches
		}
		fmt.Printf("%v\t%v\n", matches, line)
	}
}

Result:

matches	line
false	10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3)        servletPath=
false	10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3)           isSecure=true
false	10:25:14,694 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3) ---------------------------------------------------------------
false	10:25:14,704 INFO  [stdout] (ajp-/172.16.15.61:8509-3) Inicialice el provider de MedioPagoConfigService EJB
false	10:25:15,249 INFO  [org.hibernate.engine.internal.StatisticalLoggingSessionEventListener] (ajp-/172.16.15.61:8509-3) Session Metrics {
true	    19772077 nanoseconds spent acquiring 8 JDBC connections;
true	    378558 nanoseconds spent releasing 8 JDBC connections;
true	    23685207 nanoseconds spent preparing 10 JDBC statements;
true	    59644248 nanoseconds spent executing 10 JDBC statements;
true	    0 nanoseconds spent executing 0 JDBC batches;
true	    0 nanoseconds spent performing 0 L2C puts;
true	    0 nanoseconds spent performing 0 L2C hits;
true	    0 nanoseconds spent performing 0 L2C misses;
true	    23947144 nanoseconds spent executing 3 flushes (flushing a total of 589 entities and 18 collections);
true	    8098 nanoseconds spent executing 2 partial-flushes (flushing a total of 0 entities and 0 collections)
true	}
false	10:25:15,291 INFO  [org.apache.catalina.core.ContainerBase.[jboss.web]] (ajp-/172.16.15.61:8509-3) ---------------------------------------------------------------
false	12:51:10,842 INFO  [service.MotorReglas.ejecutarRegla] (ajp-/172.16.15.61:8109-7) Service  : MotorReglas, Operation: ejecutarRegla
true	Ext. ref: 1502812269900194
true	[[
true	Creamos un FlujoResponse con un Servicio de Usuario.
true	]]
false	12:51:10,843 INFO  [service.MotorReglas.ejecutarRegla] (ajp-/172.16.15.61:8109-7) Service  : MotorReglas, Operation: ejecutarRegla
true	Ext. ref: 1502812269900194
true	[[
true	Creamos un FlujoResponse con una Regla Formulario.
true	]]

steffens · August 17, 2017, 9:11pm

multiline settings are per prospector and not global level. From snippet you posted it seems like you didn't add the multiline settings into the correct section.
You are recommend to quote regular expressions in yaml configurations. See: https://www.elastic.co/guide/en/beats/filebeat/current/regexp-support.html#regexp-support

egonzalez · August 17, 2017, 9:24pm

Hi Steffens, today fix my error. I had to generate a grok pattern multiline.
MULTILINE (?s:.*).

Thanks!

system · September 14, 2017, 9:25pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.