Can't create indices

Elastic Stack Logstash
1

setup elk version 7.9.3 cannot create index logstash show me warning

[logstash.outputs.elasticsearch][main][27fb9d77028509eff9b3e4ec584ce0c8528c70f76c962e4e94e144a725a5b964] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logs-packetbeat-flow-2020.11.04", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x35a2514c>], :response=>{"index"=>{"_index"=>"logs-packetbeat-flow-2020.11.04", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}

my pipleline file is

output {
 if[agent][type]=="packetbeat"
 {
if[type]=="flow"
   {
    elasticsearch {
    hosts => ["coordinate2:9200"]
    manage_template => false
    index => "logs-packetbeat-flow-%{+YYYY.MM.dd}"
  }
  }
2

Hi,

It seems you are using data streams instead of indexes in your installation:

Although I have not used data streams myself I know that datastreams cannot be updated - they only allow inserts of new documents. Therefore, I think setting the action setting to create (defaults to index) on your Elasticsearch output should help you.

Best regards
Wolfram

1 Like
3

i actually don't know what is the data stream in elasticsearch.In elk 7.8 version i didn't see this error and when i upgraded to 7.9 version it can't show me indices although it is showing but it is hidden.
now it is working
i changed my pipeline file

output {
 if[agent][type]=="packetbeat"
 {
if[type]=="flow"
   {
    elasticsearch {
    hosts => ["coordinate2:9200"]
    manage_template => false
    index => "logs-packetbeat-flow-%{+YYYY.MM.dd}"
    action => "create"
  }
  }
}
}
5 Likes
4

Great to hear that is working!

For more information about datastreams have a look here: https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams.html

5

The data streams seems to be confusing and i dont want to use it is there any want to not using of data stream.

6

I guess this is caused by the new index templates: The Elastic Stack creates a new index template with pattern logs-- which defines a data stream. See here for the relevant breaking change in 7.9.

To solve that, you would have to create your own template with a higher priority to store the data in indizes instead of datastreams. Unfortunately, I don't know if it is possible to convert the datastream back to "raw" indizes...

7

yes you are saying right and my logstash index is also start with logs- prefix thats why it automatically uses the data streams

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.