Is there a mecanism to check that the data coming from the source (filebeat) is the same that we have in elastic

Elastic Stack Elasticsearch
1

Hello everyone,

We are using logstash to parse our application log and index to elastic search. Following is the high level architecture

file beat -----> Logstash ----> Elastic search ------> kibana.

How can we verify that filebeat parsed all the data? is there any mecanism that can permit this.
How can we do the intergration and unit test that concerns the integration of logs from filebeat to kibana. so that we can be sure that we have all our logs in elastic indices and kibana dashboards

let me know if you need any other information.

Thanks in advance.

2

Hi,

What are your requirements for such a test application? Do your log messages have an ID so you can find the corresponding messages in ElasticSearch?

Basically, you could either write your own application reading the messages from the logs and reading the data from ElasticSearch. Then, you could compare the data.

You could also use LogStash for this: Create a LogStash pipeline which gets the data from the LogFiles. Add an elasticsearch filter and read the corresponding message from ElasticSearch. For comparison you could either compare the data using a ruby filter or add a fingerprint filter to create the fingerprint from all relevant fields from both documents separately. If the fingerprint is the same you pipeline was correct.

Best regards
Wolfram

3

Hello,

thanks for the reply.

No we don't have any ID for the logs.

The solution you are proposing is interesting but I may be interested by other solutions. something more clean and easy. because developping a logstash pipeline for all the specific logs won't be much of fun.
when you said "write your own application" would you please give some more details please. "I'm not very familiar with that".

could you also please tell me if there's any others solutions or an elastic plugin that can help with testing.

Thank you.

4

HI,

If you do not have an ID to find corresponding entries: How do you know which documents to compare?

With your own application I meant that you could take any programming language of your choice and get the data using the ELasticSearch Search API and do the comparison yourself.

Maybe another user of this forum has an idea how to solve that in a simpler way.

Best regards
Wolfram

5

Hi,

thanks for the reply.

well maybe by counting the number of lines in the source (filebeat) and compare that with the count of the documents in elasticseach, make sure that the dates match, type of the logs,... I don't know, some stuff like that.

Have you ever done any application to test the elastic data? is it complex to do that ?

thank you.

regards
duk

6

Hi,

I have done somthing similar but we had IDs in them which helped us to compare the correct entries:

  • our logs from a database have an ID field so they can be uniquely identified
  • our application logs from Java do not have a unique ID but we use the Elastic APM with trace correlation enabled so we have the Trace ID from APM in our Logs. This trace ID in combination with the date was sufficiently unique for us to do a comparision.

Best regards
Wolfram

7

Hi,

Thank you for you help :slight_smile:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.