Outputing Logstash logs to Elastic Index fails

Elastic Stack Elasticsearch
1

I am using Logstash version 7.11 and and trying to output logs from logstash pipeline to Elastic version 8.*.
I am using hosts, index, api_key, ssl and action params with ssl => true and action => "create" .

output {
    elasticsearch {
       hosts => ["***"]
       index => "***"
       api_key => "***"
       ssl => true
       action => "create"
    }
}

I don't see any logs in Elastic. and see this error : "only write ops with an op_type of create are allowed in data streams"
Can someone help with any pointers what could be wrong/missing.

2

Hi @aashini!
You are working in with elastic in the cloud or installed locally?
Can you give all the log you get when you start the pipeline?

3

I work with elastic in the cloud. And I found the fix by using cloud id param instead of hosts param. Thanks for response.

4

Hi @aashini
this is kinda of down deep in the weeds ... complicated because you are using such an old version of Logstash.

Where is the data originating from? a Beat? something else?

Are you purposely trying to write to a data stream?

If you are The problem is that the incoming data / documents have a field [@metadata][op_type] that is probably set to index and that is why it is failing...

You can try removing that field or set it to create

Or you can manually create and index in 8.x elasticsearch and the output to that.

You could also try updated to an 8.x logstash ....

5

@stephenb yes, looks like old logstash version doesn't support writing to data stream (which was preferable). So I am writing logs to an Index now and planning to upgrade to 8.x logstash soon.
Thanks for your reply.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.