Only write ops with an op_type of create are allowed in data streams"}}}}

Not sure what to do here, have tried a bunch of suggestions from Github for this issue..

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    data_stream => "true"
    data_stream_timestamp => "@timestamp"
    data_stream_type => "logs"
    hosts => ["http://10.0.0.48:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

Sometimes Ill have a nice and formatted data stream. A few moments later its just a big mess and starts spitting this error.

"host"=>{"name"=>"ELK", "hostname"=>"ELK", "architecture"=>"x86_64", "os"=>{"name"=>"Debian GNU/Linux", "platform"=>"debian", "type"=>"linux", "version"=>"11 (bullseye)", "codename"=>"bullseye", "kernel"=>"5.10.0-15-amd64", "family"=>"debian"}, "containerized"=>false, "ip"=>["10.0.0.48", "2601:541:300:5d40::3440", "fe80::a00:27ff:fef7:ff01"], "id"=>"294c9f13bce94d27b6168ab14ffd752e", "mac"=>["08:00:27:f7:ff:01"]}}], :response=>{"index"=>{"_index"=>"filebeat-8.2.3", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}

I have changed my config file to look like this; I am no longer getting errors, but also am not getting any logs

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    user => "elastic"
    password => "mypass"
    "action" => "create"
    data_stream => "true"
    data_stream_timestamp => "@timestamp"
    data_stream_type => "logs"
    hosts => ["http://10.0.0.48:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}
1 Like

Try this there is an error in the docs...
Clean up the data stream and try again...

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      pipeline => "%{[@metadata][pipeline]}"
      user => "elastic"
      password => "password"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      user => "elastic"
      password => "password"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
    }
  }
} 

I provided a bit of an explanation here...
Could not index event to Elasticsearch DataStream - #4 by stephenb and here

2 Likes

For a moment the logs looked good, then it went back to these errors.

Jun 25 20:59:25 ELK logstash[475]: [2022-06-25T20:59:14,055][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-8.2.3", :routing=>nil}, {"tags"=>["beats_input_codec_plain_applied"], "event"=>{"module"=>"logstash", "dataset"=>"logstash.log", "timezone"=>"-04:00", "original"=>"[2022-06-25T20:57:51,342][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\"index\", {:_id=>nil, :_index=>\"filebeat-8.2.3\", :routing=>nil}, {\"tags\"=>[\"beats_input_codec_plain_applied\"], \"event\"=>{\"module\"=>\"logstash\", \"dataset\"=>\"logstash.log\", \"timezone\"=>\"-04:00\", \"original\"=>\"[2022-06-25T20:56:50,854][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\\\"index\\\", {:_id=>nil, :_index=>\\\"filebeat-8.2.3\\\", :routing=>nil}, {\\\"input\\\"=>{\\\"type\\\"=>\\\"filestream\\\"}, \\\"tags\\\"=>[\\\"beats_input_codec_plain_applied\\\"], \\\"ecs\\\"=>{\\\"version\\\"=>\\\"8.0.0\\\"}, \\\"agent\\\"=>{\\\"ephemeral_id\\\"=>\\\"7c802340-a895-4511-972c-1bd8402da5f5\\\", \\\"type\\\"=>\\\"filebeat\\\", \\\"version\\\"=>\\\"8.2.3\\\", \\\"id\\\"=>\\\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\\\", \\\"name\\\"=>\\\"ELK\\\"}, \\\"event\\\"=>{\\\"original\\\"=>\\\"Jun 25 20:54:22 ELK logstash[475]: [2022-06-25T20:54:22,350][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\\\\\\\"index\\\\\\\", {:_id=>nil, :_index=>\\\\\\\"filebeat-8.2.3\\\\\\\", :routing=>nil}, {\\\\\\\"input\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"filestream\\\\\\\"}, \\\\\\\"tags\\\\\\\"=>[\\\\\\\"beats_input_codec_plain_applied\\\\\\\"], \\\\\\\"ecs\\\\\\\"=>{\\\\\\\"version\\\\\\\"=>\\\\\\\"8.0.0\\\\\\\"}, \\\\\\\"agent\\\\\\\"=>{\\\\\\\"ephemeral_id\\\\\\\"=>\\\\\\\"7c802340-a895-4511-972c-1bd8402da5f5\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"filebeat\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"8.2.3\\\\\\\", \\\\\\\"id\\\\\\\"=>\\\\\\\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}, \\\\\\\"event\\\\\\\"=>{\\\\\\\"original\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\"}, \\\\\\\"@version\\\\\\\"=>\\\\\\\"1\\\\\\\", \\\\\\\"@timestamp\\\\\\\"=>2022-06-26T00:54:21.276Z, \\\\\\\"log\\\\\\\"=>{\\\\\\\"file\\\\\\\"=>{\\\\\\\"path\\\\\\\"=>\\\\\\\"/var/log/dpkg.log\\\\\\\"}, \\\\\\\"offset\\\\\\\"=>252218}, \\\\\\\"message\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\", \\\\\\\"host\\\\\\\"=>{\\\\\\\"mac\\\\\\\"=>[\\\\\\\"08:00:27:f7:ff:01\\\\\\\"], \\\\\\\"hostname\\\\\\\"=>\\\\\\\"ELK\\\\\\\", \\\\\\\"architecture\\\\\\\"=>\\\\\\\"x86_64\\\\\\\", \\\\\\\"os\\\\\\\"=>{\\\\\\\"family\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"platform\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"linux\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"11 (bullseye)\\\\\\\", \\\\\\\"codename\\\\\\\"=>\\\\\\\"bullseye\\\\\\\", \\\\\\\"kernel\\\\\\\"=>\\\\\\\"5.10.0-15-amd64\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"Debian GNU/Linux\\\\\\\"}, \\\\\\\"containerized\\\\\\\"=>false, \\\\\\\"ip\\\\\\\"=>[\\\\\\\"10.0.0.48\\\\\\\", \\\\\\\"2601:541:300:5d40::3440\\\\\\\", \\\\\\\"fe80::a00:27ff:fef7:ff01\\\\\\\"], \\\\\\\"id\\\\\\\"=>\\\\\\\"294c9f13bce94d27b6168ab14ffd752e\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}}], :response=>{\\\\\\\"index\\\\\\\"=>{\\\\\\\"_index\\\\\\\"=>\\\\\\\"filebeat-8.2.3\\\\\\\", \\\\\\\"_id\\\\\\\"=>nil, \\\\\\\"status\\\\\\\"=>400, \\\\\\\"error\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"illegal_argument_exception\\\\\\\", \\\\\\\"reason\\\\\\\"=>\\\\\\\"only write ops with an op_type of create are allowed in data streams\\\\\\\"}}}}\\\"}, \\\"@version\\\"=>\\\"1\\\", \\\"@timestamp\\\"=>2022-06-26T00:56:49.590Z, \\\"log\\\"=>{\\\"file\\\"=>{\\\"path\\\"=>\\\"/var/log/daemon.log\\\"}, \\\"offset\\\"=>31054620}, \\\"message\\\"=>\\\"Jun 25 20:54:22 ELK logstash[475]: [2022-06-25T20:54:22,350][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\\\\\\\"index\\\\\\\", {:_id=>nil, :_index=>\\\\\\\"filebeat-8.2.3\\\\\\\", :routing=>nil}, {\\\\\\\"input\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"filestream\\\\\\\"}, \\\\\\\"tags\\\\\\\"=>[\\\\\\\"beats_input_codec_plain_applied\\\\\\\"], \\\\\\\"ecs\\\\\\\"=>{\\\\\\\"version\\\\\\\"=>\\\\\\\"8.0.0\\\\\\\"}, \\\\\\\"agent\\\\\\\"=>{\\\\\\\"ephemeral_id\\\\\\\"=>\\\\\\\"7c802340-a895-4511-972c-1bd8402da5f5\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"filebeat\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"8.2.3\\\\\\\", \\\\\\\"id\\\\\\\"=>\\\\\\\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}, \\\\\\\"event\\\\\\\"=>{\\\\\\\"original\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\"}, \\\\\\\"@version\\\\\\\"=>\\\\\\\"1\\\\\\\", \\\\\\\"@timestamp\\\\\\\"=>2022-06-26T00:54:21.276Z, \\\\\\\"log\\\\\\\"=>{\\\\\\\"file\\\\\\\"=>{\\\\\\\"path\\\\\\\"=>\\\\\\\"/var/log/dpkg.log\\\\\\\"}, \\\\\\\"offset\\\\\\\"=>252218}, \\\\\\\"message\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\", \\\\\\\"host\\\\\\\"=>{\\\\\\\"mac\\\\\\\"=>[\\\\\\\"08:00:27:f7:ff:01\\\\\\\"], \\\\\\\"hostname\\\\\\\"=>\\\\\\\"ELK\\\\\\\", \\\\\\\"architecture\\\\\\\"=>\\\\\\\"x86_64\\\\\\\", \\\\\\\"os\\\\\\\"=>{\\\\\\\"family\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"platform\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"linux\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"11 (bullseye)\\\\\\\", \\\\\\\"codename\\\\\\\"=>\\\\\\\"bullseye\\\\\\\", \\\\\\\"kernel\\\\\\\"=>\\\\\\\"5.10.0-15-amd64\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"Debian GNU/Linux\\\\\\\"}, \\\\\\\"containerized\\\\\\\"=>false, \\\\\\\"ip\\\\\\\"=>[\\\\\\\"10.0.0.48\\\\\\\", \\\\\\\"2601:541:300:5d40::3440\\\\\\\", \\\\\\\"fe80::a00:27ff:fef7:ff01\\\\\\\"], \\\\\\\"id\\\\\\\"=>\\\\\\\"294c9f13bce94d27b6168ab14ffd752e\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}}], :response=>{\\\\\\\"index\\\\\\\"=>{\\\\\\\"_index\\\\\\\"=>\\\\\\\"filebeat-8.2.3\\\\\\\", \\\\\\\"_id\\\\\\\"=>nil, \\\\\\\"status\\\\\\\"=>400, \\\\\\\"error\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"illegal_argument_exception\\\\\\\", \\\\\\\"reason\\\\\\\"=>\\\\\\\"only write ops with an op_type of create are allowed in data streams\\\\\\\"}}}}\\\", \\\"host\\\"=>{\\\"architecture\\\"=>\\\"x86_64\\\", \\\"mac\\\"=>[\\\"08:00:27:f7:ff:01\\\"], \\\"name\\\"=>\\\"ELK\\\", \\\"os\\\"=>{\\\"name\\\"=>\\\"Debian GNU/Linux\\\", \\\"platform\\\"=>\\\"debian\\\", \\\"type\\\"=>\\\"linux\\\", \\\"version\\\"=>\\\"11 (bullseye)\\\", \\\"codename\\\"=>\\\"bullseye\\\", \\\"kernel\\\"=>\\\"5.10.0-15-amd64\\\", \\\"family\\\"=>\\\"debian\\\"}, \\\"containerized\\\"=>false, \\\"ip\\\"=>[\\\"10.0.0.48\\\", \\\"2601:541:300:5d40::3440\\\", \\\"fe80::a00:27ff:fef7:ff01\\\"], \\\"hostname\\\"=>\\\"ELK\\\", \\\"id\\\"=>\\\"294c9f13bce94d27b6168ab14ffd752e\\\"}}], :response=>{\\\"index\\\"=>{\\\"_index\\\"=>\\\"filebeat-8.2.3\\\", \\\"_id\\\"=>nil, \\\"status\\\"=>400, \\\"error\\\"=>{\\\"type\\\"=>\\\"illegal_argument_exception\\\", \\\"reason\\\"=>\\\"only write ops with an op_type of create are allowed in data streams\\\"}}}}\"}, \"@timestamp\"=>2022-06-26T00:57:28.121Z, \"log\"=>{\"file\"=>{\"path\"=>\"/var/log/logstash/logstash-plain.log\"}, \"offset\"=>29916530}, \"message\"=>\"[2022-06-25T20:56:50,854][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to
Jun 25 20:59:44 ELK logstash[475]: [2022-06-25T20:59:43,352][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-8.2.3", :routing=>nil}, {"input"=>{"type"=>"filestream"}, "tags"=>["beats_input_codec_plain_applied"], "ecs"=>{"version"=>"8.0.0"}, "agent"=>{"ephemeral_id"=>"7c802340-a895-4511-972c-1bd8402da5f5", "type"=>"filebeat", "version"=>"8.2.3", "id"=>"fa2cfb33-e7d8-4005-baff-bfa622d0182c", "name"=>"ELK"}, "event"=>{"original"=>"Jun 25 20:54:27 ELK logstash[475]: [2022-06-25T20:54:27,201][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\"index\", {:_id=>nil, :_index=>\"filebeat-8.2.3\", :routing=>nil}, {\"input\"=>{\"type\"=>\"filestream\"}, \"tags\"=>[\"beats_input_codec_plain_applied\"], \"ecs\"=>{\"version\"=>\"8.0.0\"}, \"agent\"=>{\"ephemeral_id\"=>\"7c802340-a895-4511-972c-1bd8402da5f5\", \"type\"=>\"filebeat\", \"version\"=>\"8.2.3\", \"id\"=>\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\", \"name\"=>\"ELK\"}, \"event\"=>{\"original\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\"}, \"@version\"=>\"1\", \"@timestamp\"=>2022-06-26T00:54:26.821Z, \"log\"=>{\"file\"=>{\"path\"=>\"/var/log/dpkg.log\"}, \"offset\"=>666691}, \"message\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\", \"host\"=>{\"mac\"=>[\"08:00:27:f7:ff:01\"], \"architecture\"=>\"x86_64\", \"id\"=>\"294c9f13bce94d27b6168ab14ffd752e\", \"os\"=>{\"name\"=>\"Debian GNU/Linux\", \"platform\"=>\"debian\", \"type\"=>\"linux\", \"version\"=>\"11 (bullseye)\", \"codename\"=>\"bullseye\", \"kernel\"=>\"5.10.0-15-amd64\", \"family\"=>\"debian\"}, \"containerized\"=>false, \"ip\"=>[\"10.0.0.48\", \"2601:541:300:5d40::3440\", \"fe80::a00:27ff:fef7:ff01\"], \"hostname\"=>\"ELK\", \"name\"=>\"ELK\"}}], :response=>{\"index\"=>{\"_index\"=>\"filebeat-8.2.3\", \"_id\"=>nil, \"status\"=>400, \"error\"=>{\"type\"=>\"illegal_argument_exception\", \"reason\"=>\"only write ops with an op_type of create are allowed in data streams\"}}}}"}, "@version"=>"1", "@timestamp"=>2022-06-26T00:59:19.543Z, "log"=>{"file"=>{"path"=>"/var/log/daemon.log"}, "offset"=>56480448}, "message"=>"Jun 25 20:54:27 ELK logstash[475]: [2022-06-25T20:54:27,201][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\"index\", {:_id=>nil, :_index=>\"filebeat-8.2.3\", :routing=>nil}, {\"input\"=>{\"type\"=>\"filestream\"}, \"tags\"=>[\"beats_input_codec_plain_applied\"], \"ecs\"=>{\"version\"=>\"8.0.0\"}, \"agent\"=>{\"ephemeral_id\"=>\"7c802340-a895-4511-972c-1bd8402da5f5\", \"type\"=>\"filebeat\", \"version\"=>\"8.2.3\", \"id\"=>\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\", \"name\"=>\"ELK\"}, \"event\"=>{\"original\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\"}, \"@version\"=>\"1\", \"@timestamp\"=>2022-06-26T00:54:26.821Z, \"log\"=>{\"file\"=>{\"path\"=>\"/var/log/dpkg.log\"}, \"offset\"=>666691}, \"message\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\", \"host\"=>{\"mac\"=>[\"08:00:27:f7:ff:01\"], \"architecture\"=>\"x86_64\", \"id\"=>\"294c9f13bce94d27b6168ab14ffd752e\", \"os\"=>{\"name\"=>\"Debian GNU/Linux\", \"platform\"=>\"debian\", \"type\"=>\"linux\", \"version\"=>\"11 (bullseye)\", \"codename\"=>\"bullseye\", \"kernel\"=>\"5.10.0-15-amd64\", \"family\"=>\"debian\"}, \"containerized\"=>false, \"ip\"=>[\"10.0.0.48\", \"2601:541:300:5d40::3440\", \"fe80::a00:27ff:fef7:ff01\"], \"hostname\"=>\"ELK\", \"name\"=>\"ELK\"}}], :response=>{\"index\"=>{\"_index\"=>\"filebeat-8.2.3\", \"_id\"=>nil, \"status\"=>400, \"error\"=>{\"type\"=>\"illegal_argument_exception\", \"reason\"=>\"only write ops with an op_type of create are allowed in data streams\"}}}}", "host"=>{"architecture"=>"x86_64", "id"=>"294c9f13bce94d27b6168ab14ffd752e", "mac"=>["08:00:27:f7:ff:01"], "os"=>{"name"=>"Debian GNU/Linux", "platform"=>"debian", "type"=>"linux", "version"=>"11 (bullseye)", "codename"=>"bullseye", "kernel"=>"5.10.0-15-amd64", "family"=>"debian"}, "containerized"=>false, "ip"=>["10.0.0.48", "2601:541:300:5d40::3440", "fe80::a00:27ff:fef7:ff01"], "hostname"=>"ELK", "name"=>"ELK"}}], :response=>{"index"=>{"_index"=>"filebeat-8.2.3", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}

now its good again. What could be causing intermittent issues?

Thank you so much for this however.. This is significant progress!

This is working! Thank you!!

Now its time to figure out how to filter out the alerts I dont really care for, and I guess I need to give the server a bit more resources... Keep getting a message saying my system is too slow xD

However, it is logging as intended at this point. Lots of alerts I want to get rid of (Beginning next scan being one of the main ones!)

I really appreciate this, you are a life saver :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.