No log data arriving after update of filebeats (1.0.0-beta4 (386) -> 1.0.1 (386))

Moxl · 2016-02-02 17:20:01 UTC

Hi everyone,

i was having trouble with file locks with filebeat 1.0.0-beta4. Thus, i updated to 1.0.1 but now my Logstash server does not receive any logs anymore...

My setting:
I am running ELK on a debian server and filebeats on a bunch of windows boxes (windows server 2012 R2)

Before the update my filebeats config looked like this:

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    -      
      paths:
        - d:\ivu\fleet.data.adc\log\console.log
      fields:
        log_type: ADC console
      ignore_older: 1h
      force_close_files: true

  spool_size: 1024

  idle_timeout: 5s

  registry_file: "C:/ProgramData/filebeat/registry"

output:

  logstash:
    enabled: true

    hosts: ["IVU-MONITOR:5000"]

    tls:

      disabled: true

shipper:
  name: "IVU-FLEET"

After the update to 1.0.1 i realized that the config file syntax slightly changed - thus, i changed my config to:


############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    -      
      paths:
        - d:\ivu\fleet.data.adc\log\console.log
      fields:
        log_type: ADC console
      ignore_older: 1h
      force_close_files: true

  spool_size: 1024

  idle_timeout: 5s

  registry_file: "C:/ProgramData/filebeat/registry"


output:

  logstash:

    hosts: ["IVU-MONITOR:5000"]

shipper:

  name: "IVU-FLEET"

(sorry for those long snippets, i just wanna make sure my problems are not caused by any typos in my config)

If i start filebeats in command line (filebeat.exe -c filebeat.yml -e -v) the output looks fine to me:

2016/02/02 17:12:40.655667 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/02/02 17:12:40.664696 outputs.go:111: INFO Activated logstash as output plugin.
2016/02/02 17:12:40.665697 publish.go:249: INFO Publisher name: IVU-FLEET
2016/02/02 17:12:40.670678 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1
2016/02/02 17:12:40.674681 beat.go:133: INFO filebeat sucessfully setup. Start running.
2016/02/02 17:12:40.674681 registrar.go:66: INFO Registry file set to: C:\ProgramData\filebeat\registry
2016/02/02 17:12:40.676682 registrar.go:76: INFO Loading registrar data from C:\ProgramData\filebeat\registry
2016/02/02 17:12:40.678685 spooler.go:77: INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016/02/02 17:12:40.681686 log.go:62: INFO Harvester started for file: d:\ivu\fleet\server\bon5\jboss-4.2.3.GA\server\bon5-server\log\import_version_20160129_135758_2.log
2016/02/02 17:12:40.682687 log.go:62: INFO Harvester started for file: d:\ivu\fleet\server\bon5\jboss-4.2.3.GA\server\bon5-server\log\import_version_20160201_153252_2.log
2016/02/02 17:12:40.682687 log.go:62: INFO Harvester started for file: d:\ivu\fleet\server\bon5\jboss-4.2.3.GA\server\bon5-server\log\import_version_20160129_140258_3.log
2016/02/02 17:12:40.682687 log.go:62: INFO Harvester started for file: d:\ivu\fleet\server\bon5\jboss-4.2.3.GA\server\bon5-server\log\import_version_20160201_153752_3.log
2016/02/02 17:12:40.687690 log.go:62: INFO Harvester started for file: d:\ivu\fleet\server\bon2-STO\wrapperbon.log
2016/02/02 17:12:40.688691 log.go:62: INFO Harvester started for file: d:\ivu\fleet\server\bon2-IVU\wrapperbon.log
2016/02/02 17:12:40.691693 crawler.go:78: INFO All prospectors initialised with 7 states to persist
2016/02/02 17:12:40.692694 registrar.go:83: INFO Starting Registrar
2016/02/02 17:12:40.692694 filebeat.go:122: INFO Start sending events to output
2016/02/02 17:12:40.691693 log.go:62: INFO Harvester started for file: d:\ivu\fleet\server\bon5\jboss-4.2.3.GA\server\bon5-server\log\server.log
2016/02/02 17:12:42.382666 filebeat.go:134: INFO Events sent: 1024
2016/02/02 17:12:42.393641 registrar.go:157: INFO Registry file updated. 7 states written.
2016/02/02 17:12:48.316098 filebeat.go:134: INFO Events sent: 85
2016/02/02 17:12:48.326080 registrar.go:157: INFO Registry file updated. 7 states written.
2016/02/02 17:12:53.919972 filebeat.go:134: INFO Events sent: 203
2016/02/02 17:12:53.929934 registrar.go:157: INFO Registry file updated. 7 states written.
2016/02/02 17:12:58.244103 filebeat.go:134: INFO Events sent: 14
2016/02/02 17:12:58.252138 registrar.go:157: INFO Registry file updated. 7 states written.

However, no data arrives at my logstash server

Any ideas?

Thanks a lot in advance!

Cheers,

Max

tudor · 2016-02-02 21:15:07 UTC

You might need to also upgrade the logstash beats-input plugin. You can check the current version with:

./bin/plugin list --verbose beats

Please also let us know which Logstash version you have.

Moxl · 2016-02-03 09:03:11 UTC

Thanks for the quick reply. I feel kinda stupid right now - could have thought about updating the beats plugin myself... Will definitely try that.

Right now all my ELKs are running on logstash_1.5.4-1

The plugin was 0.9.6

Moxl · 2016-02-03 11:24:14 UTC

Hm, looks like my logstash version will not accept any newer beats plugin than the one i am using

at least updating it via plugin binary does not work

steffens · 2016-02-03 15:49:40 UTC

how can you tell logstash not receiving any events? Not sure, but I think(hope) the plugin (besides a little dated) should work. The filebeat logs look just fine.

Can you share your logstash config. Have you tried to use the stdout output only in logstash?

Moxl · 2016-02-03 16:00:06 UTC

Here's what i did today:

I tried to update the beats plugin to something newer than 0.9.6 -> didn't work. I just didnt manage to update it via the plugin binary. To me it looks like my logstash version was too old for a 2.* beats plugin

I rechecked the release notes of filebeat releases and actually checked every release that is newer than mine (1.0.0.beta4).
According to the release notes the file locking problem was solved in the release after the one i am using (1.0.0 release candidate one). Unfortunately logstash does seem to receive any data from any filebeat that is newer than beta4.

What i was doing: i checked kibana and set it to a high refresh rate. Onm the windows box i switched through all filebeat versions by replacing the filebeat.exe files (and slightly modified the filebeat.yml when needed).

With beta4 my logstash receives log data and i can see it in kibana right after i start filebeat on the command line. With each newer filebeat version i see jack.

Moxl · 2016-02-03 16:02:45 UTC

my logstash config consists of one file for the input definition:

input {
	beats {
		port => 5000
		type => "logs"
	}
}

several files for product specific filtering - one example:

filter {
  if [fields][log_type] == "IVU ESH Wildfly Logs" {
    
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:time} %{WORD:loglevel}.*\[%{DATA:java_class}\] %{GREEDYDATA:stripped_message}" }
    }
    
    date {
    	match => [ "time" , "YYYY-MM-dd HH:mm:ss,SSS" ]
    }
    
    if [message] =~ ".*java.lang.OutOfMemoryError.*" {
      mutate {
        add_tag => [ "nagios_check" ]
        add_field => {
          "nagios_host" => "%{shipper}"
          "nagios_service" => "logstash"
        }
      }
    }
  }
}

and one file for the output:

output {
  elasticsearch {
    host => localhost
    cluster => "elasticsearch-whatsoever"
  }

  if "nagios_check" in [tags] {
    if "nagios_warning_only" in [tags] {
      nagios {
        commandfile => "/var/run/icinga2/cmd/icinga2.cmd"
        nagios_level => "1"
      }
    }
    else {
      nagios {
        commandfile => "/var/run/icinga2/cmd/icinga2.cmd"
      }
    }
  }
}

thanks a lot for your effort!

tudor · 2016-02-03 19:26:40 UTC

That's right, plugin version newer than 2.0.0 need Logstash 2.X, so 0.9.6 is the latest that you can use. I thought you might have 0.9.2, that's why I asked.

I just tried Logstash 1.5.4 with Filebeat 1.1.0 (we just released 1.1.0, you might just want to upgrade to that one) and seems to work fine for me. I used the following minimal LS config:

input {
    beats {
        port => 5044
    }
}


output {
    stdout {}
}

Can you perhaps do a similar test with a minimal LS config?

Moxl · 2016-02-04 08:24:38 UTC

Of course I can

thanks a lot for your help.

Will give a shout as soon as i tested it.

Moxl · 2016-02-04 10:11:16 UTC

Weird: minimal setup works: i am sending logs from my windows machine via filebeat1.1.0 and i can see it on std on my logstash debian machine.

When i changed back to my original setup i can only see logs from windows machines that are running on beta4 version in kibana, again.

Then i added stdout as an additional output -> i can see logs from my filebeat1.1.0 machine

Now I'm confused: how can the filebeat version on the windows machines influence whether logs are sent to elasticsearch?

any ideas?

Thanks a lot in advance!

Moxl · 2016-02-04 11:01:34 UTC

do not really understand it by now, but deleting the kibana index via curator (and my logstash indices) kinda solved the problem -> i see logs arriving in kibana

steffens · 2016-02-04 16:47:20 UTC

Maybe it was due to some mapping conflict.

Moxl · 2016-02-05 09:04:29 UTC

Maybe. Glad it's working again now
thanks a lot!

system · 2017-07-05 21:56:01 UTC