You are mixing a prospector configuration with autodiscover (which is a top level thing). If you plan to use autodiscover only you can drop prospectors.yml and put autodiscover settings in the main filebeat.yml. I created a sample gist doing so (disclaimer: I didn't test it):
cool beans. I'll just keep putting updates and stuff here then. I have a question below:
Our goal is to capture logs from our microservices and not all the crap that comes from K8s, so I've limited filebeats to the namespaces we care about and so far so good with that.
We also send everything to Logstash instead. Here's what I got so far for the filebeat-config ConfigMap:
I pass the actual LOGSTASH_HOST and _PORT later on in the DaemonSet configuration.
I commented out the rest of the prospector references in the original YAML as well since we aren't using any of that.
Question: Is there a need for me to specify filebeat.registry_file ? and have that be a physical location on the host (via hostPath)?
If something happens to the filebeat pod and it gets restarted, it's going to resend all the same logs from the namespaces I've identified. To avoid that, I need that registry file to persist, correct?
You don't need to update registry_file setting, as it defaults to the data volume. It's recommended to mount the data volume as a hostPath, so we persist the resgistry file across pod changes. That will avoid us to send logs again on restart.
Gotcha. FYI for anyone googlin, to persist the data volume, change the data volume definition from emptyDir to this:
- name: data
hostPath:
path: /var/lib/filebeat
of course you need the directory /var/lib/filebeat to exist on every node in your cluster and then the registry file should persist no matter what happens to the pods.
It appears only the last condition statement works. Meaning in the above configuration, only the "stage" namespace is being looked at.
Our logs are sent by the apps are multi-line but are not being sent by Filebeat via multi-line no matter what I try to make the multiline.pattern. Is that not observed with autodiscover?
I'm still having issues with getting multiline.pattern to work though...will keep at it until I pass out here. Any advice you can give me on that (when I awake from my coma) would be appreciated!
Here's a sample of the logs that our application is sending:
{"log":"\u001b[40m\u001b[32minfo\u001b[39m\u001b[22m\u001b[49m: Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor[1]\n","stream":"stdout","time":"2018-03-15T17:58:43.269647888Z"}
{"log":" Executing ObjectResult, writing value Microsoft.AspNetCore.Mvc.ControllerContext.\n","stream":"stdout","time":"2018-03-15T17:58:43.269673063Z"}
I have this under providers:
multiline.pattern: '^\\u001b'
multiline.negate: true
multiline.match: after
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.