Send metricbeat via logstash as datastream

Elastic Observability Metrics
1

Hi
I'm trying to send metricbeat data to logstash then store it as datastream into elasticsearch.
I already have an datastream in elasticsearch "metricbeat-8.6.1", in my dashboards I'm using "metricbeat*" index pattern
My problem is that logstash is trying to send to a datastream "metrics" because datastream type only accepts "logs", "metrics", "traces", and "synthetics".
How can I force logstash to send the metrics into metricbeat-8.6.1 datastream?
Thanks!
A

2

I beleive, This page shows the correct logstash configuration

3

I already tried this and it fails because my "metricbeat-8.6.1" is a data stream, not a classic index, what I want is to store it as datastream, but I can't set data_stream_type into "metricbeat"

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"metricbeat-8.6.1", :routing=>nil}, 
{"ecs"=>{"version"=>"8.0.0"}, "tags"=>["beats_input_raw_event"], "@version"=>"1", "@timestamp"=>2023-02-13T16:01:02.611Z, 
"system"=>{"core"=>{"system"=>{"pct"=>0.0594}, "user"=>{"pct"=>0.011}, "idle"=>{"pct"=>0.9297}, "id"=>5, "total"=>{"pct"=>0.0703}}}, 
"metricset"=>{"name"=>"core", "period"=>20000}, "service"=>{"type"=>"system"},
"event"=>{"module"=>"system", "dataset"=>"system.core"}, "type"=>"metricbeat", "version"=>"8.6.1", 
"id"=>"5cb113ea-54aa-4291-9779-65e611401fbe"}}], :response=>{"index"=>{"_index"=>"metricbeat-8.6.1", 
"_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}
4

Hi @A_Mightiev I don't think you looked close...

I have personally tested Metricbeat -> Logstash -> Elasticsearch Cloud 8.x (although I have not tried 8.6.1) today I can do that later...

action => "create" <!----- Not Setting this is cause of your issue

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" <!----- Not Setting this is cause of your issue
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
      action => "create" <!----- Not Setting this is cause of your issue
      user => "elastic"
      password => "secret"
    }
  }
1 Like
5

Indeed! thanks! It is working!

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.