How can we verify that filebeat parsed all the data? is there any mecanism that can permit this.
How can we do the intergration and unit test that concerns the integration of logs from filebeat to kibana. so that we can be sure that we have all our logs in elastic indices and kibana dashboards
What are your requirements for such a test application? Do your log messages have an ID so you can find the corresponding messages in ElasticSearch?
Basically, you could either write your own application reading the messages from the logs and reading the data from ElasticSearch. Then, you could compare the data.
You could also use LogStash for this: Create a LogStash pipeline which gets the data from the LogFiles. Add an elasticsearch filter and read the corresponding message from ElasticSearch. For comparison you could either compare the data using a ruby filter or add a fingerprint filter to create the fingerprint from all relevant fields from both documents separately. If the fingerprint is the same you pipeline was correct.
The solution you are proposing is interesting but I may be interested by other solutions. something more clean and easy. because developping a logstash pipeline for all the specific logs won't be much of fun.
when you said "write your own application" would you please give some more details please. "I'm not very familiar with that".
could you also please tell me if there's any others solutions or an elastic plugin that can help with testing.
If you do not have an ID to find corresponding entries: How do you know which documents to compare?
With your own application I meant that you could take any programming language of your choice and get the data using the ELasticSearch Search API and do the comparison yourself.
Maybe another user of this forum has an idea how to solve that in a simpler way.
well maybe by counting the number of lines in the source (filebeat) and compare that with the count of the documents in elasticseach, make sure that the dates match, type of the logs,... I don't know, some stuff like that.
Have you ever done any application to test the elastic data? is it complex to do that ?
I have done somthing similar but we had IDs in them which helped us to compare the correct entries:
our logs from a database have an ID field so they can be uniquely identified
our application logs from Java do not have a unique ID but we use the Elastic APM with trace correlation enabled so we have the Trace ID from APM in our Logs. This trace ID in combination with the date was sufficiently unique for us to do a comparision.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
