Save the last fetch/execution time in a persistent file

Elastic Stack Beats
1

I’m developing a metricbeat module which quries data from a table.

As I would like to fetch the data only once, I want store the last fetch time in a persistent store so that I can pull the required data after a planned or unplanned outage.

Is there such a functionality in libbeat/metricbeat 6.1 which I can use for that (like registry file in filebeat)?

2

There is no out of the box solution for this at the moment as we have different implementations for different use cases at the moment. We plan to unify this as part of https://github.com/elastic/beats/issues/5755 This could make it also possible to be used from modules which I think is a pretty interesting idea. As far as I understand you are looking for a disk cache instead of just a memory cache.

Can you share a bit more about the system you are integrating with?

3

I want develop an oraclebeat on basis on metricbeat. In some metricsets I get records from a table which I don't want receive twice if the beat is planned or unplanned restartet.

Yes, you are right I need something like a disk cache. At the moment I use the simple library diskv for my use case. My first draft is already provided on github.com.

@ruflin Thanks for providing the link.

4

Very interesting to see that there is golang lib to connect to oracle: https://github.com/go-goracle/goracle/tree/v2.1.12

For now you are probably best of to use your own implementation until we provide an interface / method you can use.

Let me know if you hit other issues or if I can help. Really looking forward to see this out in the wild.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.