Regexp pattern not working

Elastic Stack Beats
1

Hi,

I'm using this configuration to multiline all logs till a timestamp or an ip is found.
I started with timestamp, but it ignores the pattern.

filebeat:
 # List of prospectors to fetch data.
 prospectors:
 -
  paths:
   - /opt/tomcat/logs/tomcat.*.log
  input_type: log
  document_type: filebeat
  scan_frequency: 60s
  fields:
   host: xxxx
  fields_under_root: true
  multiline:
    pattern: ^([0-9]{4})(-?)(1[0-2]|0[1-9])(-?)(3[01]|0[1-9]|[12][0-9])[[:space:]](2[0-3]|[01][0-9]):?([0-5][0-9]):?([0-5][0-9])$
    negate: true
    match: after
    max_lines: 200
    timeout: 2s
  tail_files: true
  registry_file: /var/lib/filebeat/registry

output:
  logstash:
   hosts: ["x.x.x.x:5044"]
shipper:
  logging:
   files:
    rotateeverybytes: 10485760 # = 10MB

logstash receives the log in multiline format, but ignores if it receives a timestamp and marge all content.

What am i doing wrong?

Thanks

2

And a couple of example lines of input, as requested earlier?

3

Hi Magnus,

2016-02-15 16:06:00 ERROR BingoChampions [pool-54-thread-1] [StandardIntegrationImpl.java:650] Finalizado proceso de historificacion de transacciones
        at sun.reflect.GeneratedConstructorAccessor594.newInstance(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.6.0_34]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:534) ~[na:1.6.0_34]

If the log looks like that, never sends to logstash, because when i disable multiline and restart filebeat, it sends all messages remaining.

Thanks

4

Your multiline pattern doesn't match the input line. There might be several reasons for that, but the most obvious one is that you end the pattern with time and a dollar sign,

[[:space:]](2[0-3]|[01][0-9]):?([0-5][0-9]):?([0-5][0-9])$

but the actual line from the log contains extra text after the time.

5

Hi Magnus, thanks for early response.
As in logstash multiline filter, i though that if it finds the pattern (just a piece of it) it processes. I did like this and worked in logstash.
I mean, it finds the pattern at the beggining of the line, nevermind what comes after that because it finds the pattern at the beggingin fo the line.
Am i wrong? if so, how to proceed?

Thanks again

6

The dollar sign means "end of string". Your string doesn't end after the time marker. Remove the dollar sign.

7

Magnus, as always ou solved, that was the pb...
Starting this topic, i write that i wanted to check if theres a timestamp or an ip. I have ip regexp, but i tried to put it with and or between both regexp (timestampt and ip) and not working.

((?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)