Best way to Send data from influxdb to elastic

Elastic Observability
1

Hi
I have influxdb in production that store metric from different source now question is:

1-How can i send these metric to elastic for realtime analysis?

2-which is better? read data with elastic from influx, or configure influxdb send metric to elastic?

3-is it logical to send these metric from influxdb to elastic? Or i should use elastic agent, or opentelemetry?

Any idea?
Thanks

2

Hi @Indeed2000

In short, I think you'd be better off sending your metrics directly to elasticsearch with the elastic agent or open telemetry rather than sending it to influxdb first and then trying to pull it into elastic, especially if you're interested in near real time.

3

@stephenb here is my stack:

Jboss,mq,db2,my java app.

all metrics send directly from my app and other services that i mention to influxdb, these service locate in different host.

Now this is current issue, if i want to send metrics directly need to reconfigure all of them, but if i use influxdb data, benfit is all metrics located same place.

And there is no universal agent for all of them.

4

Ok so now I am confused... So if #3 is not really an option... If you already have all your metrics in one place, and you like / want it that way...why do you want to send them to elasticsearch.. what are you trying to accomplish?

Yes there are typically not a single agent for everything... But most agents can be pointed at elasticsearch as a metrics store.

Curious about what agents are you using today?

The telegraf agent can point to more than one output.

Elasticsearch does not read data from other systems so you will need to pull data from influxdb and push it to elasticsearch... You will probably need to use an ETL like tool or write some custom code.

Then of course you will be trying to keep 2 data stores in sync.

Looks like influxdb has a REST API you might be able to use logstash with the HTTP input.

So again what are you actually trying to accomplish? Perhaps that will help us guide you better

5

using elastic machine learning to detect realtime anomaly.

6

send directly metrics from myapp to influx and use telegraf as agent to send system metric.

7

Hi @Indeed2000

Ok for anomaly detection / alerting makes sense.

Telegraf can send to more than one output.

Otherwise your choice will be to use some tool to pull the data from influxdb (unless it has a push feature) and push telemetry data into elasticsearch. That will typically involve some polling interval.

Or You could re-architect and put something in the middle like Kafka and then publish the data to both influxdb and Elasticsearch

I think those are your 3 high level choices unless you want to switch over to elasticsearch all together.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.