Pulling data from a statsd server

I need to pull data from a statsd server,whats the best approach to do this? Im trying to import the statsd metrics from a spark jobs using metricbeats but metricbeat is getting the following error

river_LiveListenerBus_queue_shared_numDroppedEvents":{"count":0},"spark-5a9e0fdbde2e44b38a075a6bdbb67ecb_driver_LiveListenerBus_queue_shared_size":{"value":0.000000}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(ni
l)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Limit of total fields [10000] has been exceeded while adding new fields [3621]"}}

Is there a workarround to this? Is there a better way to ingests these metrics?

Thanks in advance

TLDR this is managed by Mapping limit settings | Elasticsearch Guide [7.14] | Elastic, and it looks like you have already increased it from the default of 1000. Increasing it more isn't a great idea as it causes mapping explosions and inefficiencies.

What other data are you pulling in from Metricbeat?
Can you put data from different sources into different indices?

Im trying to pull data from some sparks jobs via statsd. I think that the whole issue is about metrics mappings, without it the ELK stack is trying to index the metrics with the sparkIds as a field thats why the number of records exceds the limits.

Regarding to mappings how do i translate a graphite map like this

- match: '*.*.LiveListenerBus.listenerProcessingTime.org.apache.spark.sql.execution.ui.*.*'
  name: Live_Listener_Bus_ExecutionUI
    application: $1
    executor_id: $2
    type: $3
    qty: $4

Currently Im starting to use a graphite exporter to get de metrics from spark, and translate them to prometheus metrics labels. But I would really like to try Elastic as a metrics collector


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.