File input yields no stdout output

Elastic Stack Logstash
1

Continuing the discussion from New to logstash: file input and stdout output not working:

I have the same problem except that deleting the sincedb file, and setting the sincedb_path to /dev/null and rerunning still doesn't output anything. I'm even testing on a very small file (100 lines).

2

What's your config look like?

3

Well I progressed past that problem (I had at one point added an extra match section (that just matched /.*/ ) and then forgot to remove it, and threw out every document without a timestamp, so of course I wasn't getting anything.

But now I'm running into an out of memory error.

config is:

input {
  file {
    path => [
         "/tmp/logstash/scannet70/ag_log",
         "/tmp/logstash/scannet78/ag_log",
         "/tmp/logstash/scannet86/ag_log",
         "/tmp/logstash/scannet89/ag_log",
         "/tmp/logstash/scannet93/ag_log",
         "/tmp/logstash/scannet94/ag_log"
            ]
    stat_interval => 60
    codec => line
    start_position => "beginning"
  }
}

filter {
  multiline {
    pattern => "^(\[|started|Resetting)"
    negate => true
    what => "previous"
  }
  grok {
    patterns_dir => "/etc/logstash/conf.d/patterns"
    match => { "message" => [
                             "started at %{NUMBER:timestamp}, local %{TIMESTAMP_ISO8601:localtime}",
                             "Resetting the coordinator, waiting 30 seconds\.\.\.%{RESET_MULTILINE:line}",
                             "\[%{TIME_FORMAT:timestamp}: %{NONNEGINT:device}\] %{DEVICE_MESSAGE:line}",
                             "\[%{TIME_FORMAT:timestamp}: %{MOTE:mote}\] %{MOTE_MESSAGE:line}",
                             "\[%{TIME_FORMAT:timestamp}: %{MOTE:mote}\] %{CHILD_MESSAGE:line}",
                             "\[%{TIME_FORMAT:timestamp}: (?<command>mote command)\] %{MOTE_COMMAND:line}",
                             "\[%{TIME_FORMAT:timestamp}: (?<command>aggregation)\] entered isInNwkSession %{NONNEGINT:session}",
                             "\[%{TIME_FORMAT:timestamp}: (?<command>network)\] %{NETWORK_MESSAGE:line}%{NETWORK_MULTILINE:multiline}",
                             "\[%{TIME_FORMAT:timestamp}: (?<command>ms cmd)\] %{MS_CMD:line}",
                             "\[%{TIME_FORMAT:timestamp}: (?<command>aggregator)\]%{AGGREGATOR_MESSAGE:line}",
                             "\[%{TIME_FORMAT:timestamp}: (?<command>aggregator)\] %{AGGREGATOR_MESSAGE_TYPE_2:line}"
                            ] }
  }
  if ([timestamp] !~ /.+/) {
      drop {}
    }
    mutate {
      convert => { "acc_rate" => "float" }
      convert => { "address" => "integer" }
      convert => { "attempt" => "integer" }
      # many more converts removed in order to fit here
      add_field => { "gateway" => "%{path}" }
      gsub =>  [ "gateway", ".*scannet([0-9]+).*", ""]
      gsub =>  [ "timestamp", "(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)", "2015 \1"]
    }
    translate {
      field => "gateway"
      destination => "project"
      dictionary => {
        "70" => "P247"
        "78" => "P237"
        "86" => "P269"
        "89" => "P263"
        "93" => "P272"
        "94" => "P273"
      }   
    }
    translate {
      field => "mote"
      destination => "location_id"
      dictionary => {
        "0000000000000000" => "80B95DA0-899B-4948-8E3A-54518034041F"
        "0001000013B6A834" => "66CF9E84-7FAB-4998-B890-F4C2F9E6AFFF"
        #many more lines like above here
      }
    }
    date {
      match => [ "timestamp", "yyyy MMM dd, HH:mm:ss", "UNIX"]
    }
  }

  output {
    amazon_es{secrets&stuff}
  }
4

Are you sure the multiline filter is correct? If it's not, Logstash will just store everything in memory in anticipation of something that satisfies the condition so it can flush the built-up events.

5

It turned out to be an issue with using file input with the line codec. As soon as I removed that it started working.