Logstash file input plugin no output response

Elastic Stack Logstash
1

I am new to ELK so any comment is welcome.

I am now using logstash to parse logs and I want to combine different lines of into one line by using codec => multiline plugin. Here is my config.

input {
  #beats {
  #  port => 5044
  #}
  file {
    path => "C:\ELK\log_190712\mq.log"
    start_position => "beginning"
    codec => multiline {
      # grok pattern are valid! :)
      pattern => "^(?<logTimestamp>\d{2} \w{3} \d{4} \d{2}:\d{2}:\d{2},\d{3})"
      negate => true
      what => "previous"
    }
  }
}
filter {
  grok {		
     match => [ "message", "(?<logTimestamp>\d{2} \w{3} \d{4} \d{2}:\d{2}:\d{2},\d{3}) %{DATA:logType} %{DATA:ip} \[%{NUMBER:random}\] \[%{DATA:username}\] \[\[%{DATA:threadstate}\] %{DATA:thread}\] %{DATA:logger} %{GREEDYDATA:detail}" ]
  }
  mutate {
    add_tag => ["logstash"]
  }
  date {
    match => [ "logTimestamp", "dd MMM yyyy HH:mm:ss,SSS" ]
    target => "logTimestamp"
  }  
}
output {
  
  # file {
  #   path => "C:\ELK\mq_change.log"
	#   codec => line { format => "custom format: %{message}"}
  #   #message_format => "%{message}"
  # }
  stdout {
      codec => rubydebug
  }
}

I have tested the codec => multiline plugin using stdin and it works fine, but when I switch to file plugin, then no responsive even using debug mode. I have tried to delete .sincedb and i also don't work.

The debug message looks like this,

image
image.png962×684 16.8 KB

Please take a look at my case and help me with this problem. Thx.

2

Firstly, use forward slash instead of backslash in a file input.

Secondly, when you use 'negate => true' an event is not pushed until there is a second line that does match the pattern. So you may need to use a timeout, such as 'auto_flush_interval => 5'.

3

Thanks for your comment :).

I have changed that but it didn't work. Finally, by fixing the path and adding timeout mechanism plus adding 'mode = "read"', it finally run as I hope. But I don't know why it works and how it works. Can you explain about it. Thanks.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.