Filebeat regexp

[O.Shulha (Oleksandr)]

I have some log files in folder. I tried to send files to the logstash.

filebeat.yml

filebeat.inputs:
- input_type: log
  enabled: true
  paths:
    - /opt/nso/ncs-run/logs/rollback*
  multiline.pattern: 'ncs:*'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '^/s}'

output:
  logstash:
    hosts: [ "host:5002"]

Log file example:

# Created by: admin
# Date: 2019-07-18 17:10:35
# Via: rest
# Type: delta
# Label: 
# Comment: 
# No: 10087

ncs:services {
    ap:attachment {
        ap:pe asr9006a {
            ap:vrf {
                ap:vrf 9af34f86ec7448f6b223ab0ee77b8413 {
                    delete:
                    ap:bgp-peer 172.101.0.54;
                }
             }
         }
     }
 }

I want to send part of this log. Start from ncs: and end of the file "}"

logstash.conf

input {
  beats {
    port => 5002
  }
}

filter {}

output {
  elasticsearch {
    hosts => [ "elasticsearch:9200" ]
    index => "logstash-beats-%{+YYYY.MM.dd}"
  }
}

Errors:

[2019-07-23T09:20:55,280][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-syslog-2019.07.23", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x34fc23b>], :response=>{"index"=>{"_index"=>"logstash-syslog-2019.07.23", "_type"=>"_doc", "_id"=>"9MciHmwBv7eJVn6lOJd9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id '9MciHmwBv7eJVn6lOJd9'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:75"}}}}}

[2019-07-23T09:20:55,281][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-syslog-2019.07.23", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x6cde57c2>], :response=>{"index"=>{"_index"=>"logstash-syslog-2019.07.23", "_type"=>"_doc", "_id"=>"-sciHmwBv7eJVn6lOJd9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id '-sciHmwBv7eJVn6lOJd9'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:75"}}}}}

All versions are = 7.2
And filebeat don`t sends the whole message. Just some parts. How to solve it??

Thank you a lot

[pierhugues (Pier-Hugues Pellerin)]

Hello @O.Shulha,

Looking the regular expression you are using I think it's almost correct.
The pattern should be:

  multiline.pattern: '^ncs:.*'

looking a the other errors which come from the elasticsearch output, I think its a mapping issues, and there is some work around and solution in Logstash errors after upgrading to filebeat-6.3.0

[2019-07-23T09:20:55,281][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-syslog-2019.07.23", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x6cde57c2>], :response=>{"index"=>{"_index"=>"logstash-syslog-2019.07.23", "_type"=>"_doc", "_id"=>"-sciHmwBv7eJVn6lOJd9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id '-sciHmwBv7eJVn6lOJd9'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:75"}}}}}