Only write ops with an op_type of create are allowed in data streams"}}}}

Elastic Stack Beats
1

Not sure what to do here, have tried a bunch of suggestions from Github for this issue..

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    data_stream => "true"
    data_stream_timestamp => "@timestamp"
    data_stream_type => "logs"
    hosts => ["http://10.0.0.48:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}

Sometimes Ill have a nice and formatted data stream. A few moments later its just a big mess and starts spitting this error.

"host"=>{"name"=>"ELK", "hostname"=>"ELK", "architecture"=>"x86_64", "os"=>{"name"=>"Debian GNU/Linux", "platform"=>"debian", "type"=>"linux", "version"=>"11 (bullseye)", "codename"=>"bullseye", "kernel"=>"5.10.0-15-amd64", "family"=>"debian"}, "containerized"=>false, "ip"=>["10.0.0.48", "2601:541:300:5d40::3440", "fe80::a00:27ff:fef7:ff01"], "id"=>"294c9f13bce94d27b6168ab14ffd752e", "mac"=>["08:00:27:f7:ff:01"]}}], :response=>{"index"=>{"_index"=>"filebeat-8.2.3", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}

I have changed my config file to look like this; I am no longer getting errors, but also am not getting any logs

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    user => "elastic"
    password => "mypass"
    "action" => "create"
    data_stream => "true"
    data_stream_timestamp => "@timestamp"
    data_stream_type => "logs"
    hosts => ["http://10.0.0.48:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}
1 Like
2

Try this there is an error in the docs...
Clean up the data stream and try again...

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      pipeline => "%{[@metadata][pipeline]}"
      user => "elastic"
      password => "password"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      user => "elastic"
      password => "password"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
    }
  }
}

I provided a bit of an explanation here...
Could not index event to Elasticsearch DataStream - #4 by stephenb and here

2 Likes
3

For a moment the logs looked good, then it went back to these errors.

Jun 25 20:59:25 ELK logstash[475]: [2022-06-25T20:59:14,055][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-8.2.3", :routing=>nil}, {"tags"=>["beats_input_codec_plain_applied"], "event"=>{"module"=>"logstash", "dataset"=>"logstash.log", "timezone"=>"-04:00", "original"=>"[2022-06-25T20:57:51,342][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\"index\", {:_id=>nil, :_index=>\"filebeat-8.2.3\", :routing=>nil}, {\"tags\"=>[\"beats_input_codec_plain_applied\"], \"event\"=>{\"module\"=>\"logstash\", \"dataset\"=>\"logstash.log\", \"timezone\"=>\"-04:00\", \"original\"=>\"[2022-06-25T20:56:50,854][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\\\"index\\\", {:_id=>nil, :_index=>\\\"filebeat-8.2.3\\\", :routing=>nil}, {\\\"input\\\"=>{\\\"type\\\"=>\\\"filestream\\\"}, \\\"tags\\\"=>[\\\"beats_input_codec_plain_applied\\\"], \\\"ecs\\\"=>{\\\"version\\\"=>\\\"8.0.0\\\"}, \\\"agent\\\"=>{\\\"ephemeral_id\\\"=>\\\"7c802340-a895-4511-972c-1bd8402da5f5\\\", \\\"type\\\"=>\\\"filebeat\\\", \\\"version\\\"=>\\\"8.2.3\\\", \\\"id\\\"=>\\\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\\\", \\\"name\\\"=>\\\"ELK\\\"}, \\\"event\\\"=>{\\\"original\\\"=>\\\"Jun 25 20:54:22 ELK logstash[475]: [2022-06-25T20:54:22,350][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\\\\\\\"index\\\\\\\", {:_id=>nil, :_index=>\\\\\\\"filebeat-8.2.3\\\\\\\", :routing=>nil}, {\\\\\\\"input\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"filestream\\\\\\\"}, \\\\\\\"tags\\\\\\\"=>[\\\\\\\"beats_input_codec_plain_applied\\\\\\\"], \\\\\\\"ecs\\\\\\\"=>{\\\\\\\"version\\\\\\\"=>\\\\\\\"8.0.0\\\\\\\"}, \\\\\\\"agent\\\\\\\"=>{\\\\\\\"ephemeral_id\\\\\\\"=>\\\\\\\"7c802340-a895-4511-972c-1bd8402da5f5\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"filebeat\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"8.2.3\\\\\\\", \\\\\\\"id\\\\\\\"=>\\\\\\\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}, \\\\\\\"event\\\\\\\"=>{\\\\\\\"original\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\"}, \\\\\\\"@version\\\\\\\"=>\\\\\\\"1\\\\\\\", \\\\\\\"@timestamp\\\\\\\"=>2022-06-26T00:54:21.276Z, \\\\\\\"log\\\\\\\"=>{\\\\\\\"file\\\\\\\"=>{\\\\\\\"path\\\\\\\"=>\\\\\\\"/var/log/dpkg.log\\\\\\\"}, \\\\\\\"offset\\\\\\\"=>252218}, \\\\\\\"message\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\", \\\\\\\"host\\\\\\\"=>{\\\\\\\"mac\\\\\\\"=>[\\\\\\\"08:00:27:f7:ff:01\\\\\\\"], \\\\\\\"hostname\\\\\\\"=>\\\\\\\"ELK\\\\\\\", \\\\\\\"architecture\\\\\\\"=>\\\\\\\"x86_64\\\\\\\", \\\\\\\"os\\\\\\\"=>{\\\\\\\"family\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"platform\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"linux\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"11 (bullseye)\\\\\\\", \\\\\\\"codename\\\\\\\"=>\\\\\\\"bullseye\\\\\\\", \\\\\\\"kernel\\\\\\\"=>\\\\\\\"5.10.0-15-amd64\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"Debian GNU/Linux\\\\\\\"}, \\\\\\\"containerized\\\\\\\"=>false, \\\\\\\"ip\\\\\\\"=>[\\\\\\\"10.0.0.48\\\\\\\", \\\\\\\"2601:541:300:5d40::3440\\\\\\\", \\\\\\\"fe80::a00:27ff:fef7:ff01\\\\\\\"], \\\\\\\"id\\\\\\\"=>\\\\\\\"294c9f13bce94d27b6168ab14ffd752e\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}}], :response=>{\\\\\\\"index\\\\\\\"=>{\\\\\\\"_index\\\\\\\"=>\\\\\\\"filebeat-8.2.3\\\\\\\", \\\\\\\"_id\\\\\\\"=>nil, \\\\\\\"status\\\\\\\"=>400, \\\\\\\"error\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"illegal_argument_exception\\\\\\\", \\\\\\\"reason\\\\\\\"=>\\\\\\\"only write ops with an op_type of create are allowed in data streams\\\\\\\"}}}}\\\"}, \\\"@version\\\"=>\\\"1\\\", \\\"@timestamp\\\"=>2022-06-26T00:56:49.590Z, \\\"log\\\"=>{\\\"file\\\"=>{\\\"path\\\"=>\\\"/var/log/daemon.log\\\"}, \\\"offset\\\"=>31054620}, \\\"message\\\"=>\\\"Jun 25 20:54:22 ELK logstash[475]: [2022-06-25T20:54:22,350][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\\\\\\\"index\\\\\\\", {:_id=>nil, :_index=>\\\\\\\"filebeat-8.2.3\\\\\\\", :routing=>nil}, {\\\\\\\"input\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"filestream\\\\\\\"}, \\\\\\\"tags\\\\\\\"=>[\\\\\\\"beats_input_codec_plain_applied\\\\\\\"], \\\\\\\"ecs\\\\\\\"=>{\\\\\\\"version\\\\\\\"=>\\\\\\\"8.0.0\\\\\\\"}, \\\\\\\"agent\\\\\\\"=>{\\\\\\\"ephemeral_id\\\\\\\"=>\\\\\\\"7c802340-a895-4511-972c-1bd8402da5f5\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"filebeat\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"8.2.3\\\\\\\", \\\\\\\"id\\\\\\\"=>\\\\\\\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}, \\\\\\\"event\\\\\\\"=>{\\\\\\\"original\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\"}, \\\\\\\"@version\\\\\\\"=>\\\\\\\"1\\\\\\\", \\\\\\\"@timestamp\\\\\\\"=>2022-06-26T00:54:21.276Z, \\\\\\\"log\\\\\\\"=>{\\\\\\\"file\\\\\\\"=>{\\\\\\\"path\\\\\\\"=>\\\\\\\"/var/log/dpkg.log\\\\\\\"}, \\\\\\\"offset\\\\\\\"=>252218}, \\\\\\\"message\\\\\\\"=>\\\\\\\"2022-06-25 16:55:26 status half-installed libtss2-sys1:amd64 3.0.3-2\\\\\\\", \\\\\\\"host\\\\\\\"=>{\\\\\\\"mac\\\\\\\"=>[\\\\\\\"08:00:27:f7:ff:01\\\\\\\"], \\\\\\\"hostname\\\\\\\"=>\\\\\\\"ELK\\\\\\\", \\\\\\\"architecture\\\\\\\"=>\\\\\\\"x86_64\\\\\\\", \\\\\\\"os\\\\\\\"=>{\\\\\\\"family\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"platform\\\\\\\"=>\\\\\\\"debian\\\\\\\", \\\\\\\"type\\\\\\\"=>\\\\\\\"linux\\\\\\\", \\\\\\\"version\\\\\\\"=>\\\\\\\"11 (bullseye)\\\\\\\", \\\\\\\"codename\\\\\\\"=>\\\\\\\"bullseye\\\\\\\", \\\\\\\"kernel\\\\\\\"=>\\\\\\\"5.10.0-15-amd64\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"Debian GNU/Linux\\\\\\\"}, \\\\\\\"containerized\\\\\\\"=>false, \\\\\\\"ip\\\\\\\"=>[\\\\\\\"10.0.0.48\\\\\\\", \\\\\\\"2601:541:300:5d40::3440\\\\\\\", \\\\\\\"fe80::a00:27ff:fef7:ff01\\\\\\\"], \\\\\\\"id\\\\\\\"=>\\\\\\\"294c9f13bce94d27b6168ab14ffd752e\\\\\\\", \\\\\\\"name\\\\\\\"=>\\\\\\\"ELK\\\\\\\"}}], :response=>{\\\\\\\"index\\\\\\\"=>{\\\\\\\"_index\\\\\\\"=>\\\\\\\"filebeat-8.2.3\\\\\\\", \\\\\\\"_id\\\\\\\"=>nil, \\\\\\\"status\\\\\\\"=>400, \\\\\\\"error\\\\\\\"=>{\\\\\\\"type\\\\\\\"=>\\\\\\\"illegal_argument_exception\\\\\\\", \\\\\\\"reason\\\\\\\"=>\\\\\\\"only write ops with an op_type of create are allowed in data streams\\\\\\\"}}}}\\\", \\\"host\\\"=>{\\\"architecture\\\"=>\\\"x86_64\\\", \\\"mac\\\"=>[\\\"08:00:27:f7:ff:01\\\"], \\\"name\\\"=>\\\"ELK\\\", \\\"os\\\"=>{\\\"name\\\"=>\\\"Debian GNU/Linux\\\", \\\"platform\\\"=>\\\"debian\\\", \\\"type\\\"=>\\\"linux\\\", \\\"version\\\"=>\\\"11 (bullseye)\\\", \\\"codename\\\"=>\\\"bullseye\\\", \\\"kernel\\\"=>\\\"5.10.0-15-amd64\\\", \\\"family\\\"=>\\\"debian\\\"}, \\\"containerized\\\"=>false, \\\"ip\\\"=>[\\\"10.0.0.48\\\", \\\"2601:541:300:5d40::3440\\\", \\\"fe80::a00:27ff:fef7:ff01\\\"], \\\"hostname\\\"=>\\\"ELK\\\", \\\"id\\\"=>\\\"294c9f13bce94d27b6168ab14ffd752e\\\"}}], :response=>{\\\"index\\\"=>{\\\"_index\\\"=>\\\"filebeat-8.2.3\\\", \\\"_id\\\"=>nil, \\\"status\\\"=>400, \\\"error\\\"=>{\\\"type\\\"=>\\\"illegal_argument_exception\\\", \\\"reason\\\"=>\\\"only write ops with an op_type of create are allowed in data streams\\\"}}}}\"}, \"@timestamp\"=>2022-06-26T00:57:28.121Z, \"log\"=>{\"file\"=>{\"path\"=>\"/var/log/logstash/logstash-plain.log\"}, \"offset\"=>29916530}, \"message\"=>\"[2022-06-25T20:56:50,854][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to

Jun 25 20:59:44 ELK logstash[475]: [2022-06-25T20:59:43,352][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-8.2.3", :routing=>nil}, {"input"=>{"type"=>"filestream"}, "tags"=>["beats_input_codec_plain_applied"], "ecs"=>{"version"=>"8.0.0"}, "agent"=>{"ephemeral_id"=>"7c802340-a895-4511-972c-1bd8402da5f5", "type"=>"filebeat", "version"=>"8.2.3", "id"=>"fa2cfb33-e7d8-4005-baff-bfa622d0182c", "name"=>"ELK"}, "event"=>{"original"=>"Jun 25 20:54:27 ELK logstash[475]: [2022-06-25T20:54:27,201][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\"index\", {:_id=>nil, :_index=>\"filebeat-8.2.3\", :routing=>nil}, {\"input\"=>{\"type\"=>\"filestream\"}, \"tags\"=>[\"beats_input_codec_plain_applied\"], \"ecs\"=>{\"version\"=>\"8.0.0\"}, \"agent\"=>{\"ephemeral_id\"=>\"7c802340-a895-4511-972c-1bd8402da5f5\", \"type\"=>\"filebeat\", \"version\"=>\"8.2.3\", \"id\"=>\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\", \"name\"=>\"ELK\"}, \"event\"=>{\"original\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\"}, \"@version\"=>\"1\", \"@timestamp\"=>2022-06-26T00:54:26.821Z, \"log\"=>{\"file\"=>{\"path\"=>\"/var/log/dpkg.log\"}, \"offset\"=>666691}, \"message\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\", \"host\"=>{\"mac\"=>[\"08:00:27:f7:ff:01\"], \"architecture\"=>\"x86_64\", \"id\"=>\"294c9f13bce94d27b6168ab14ffd752e\", \"os\"=>{\"name\"=>\"Debian GNU/Linux\", \"platform\"=>\"debian\", \"type\"=>\"linux\", \"version\"=>\"11 (bullseye)\", \"codename\"=>\"bullseye\", \"kernel\"=>\"5.10.0-15-amd64\", \"family\"=>\"debian\"}, \"containerized\"=>false, \"ip\"=>[\"10.0.0.48\", \"2601:541:300:5d40::3440\", \"fe80::a00:27ff:fef7:ff01\"], \"hostname\"=>\"ELK\", \"name\"=>\"ELK\"}}], :response=>{\"index\"=>{\"_index\"=>\"filebeat-8.2.3\", \"_id\"=>nil, \"status\"=>400, \"error\"=>{\"type\"=>\"illegal_argument_exception\", \"reason\"=>\"only write ops with an op_type of create are allowed in data streams\"}}}}"}, "@version"=>"1", "@timestamp"=>2022-06-26T00:59:19.543Z, "log"=>{"file"=>{"path"=>"/var/log/daemon.log"}, "offset"=>56480448}, "message"=>"Jun 25 20:54:27 ELK logstash[475]: [2022-06-25T20:54:27,201][WARN ][logstash.outputs.elasticsearch][main][ddb8c1e5e4dccf837f9fe5bbbd0dd7b5f3d3c01cafb695c48aec992f36c73114] Could not index event to Elasticsearch. {:status=>400, :action=>[\"index\", {:_id=>nil, :_index=>\"filebeat-8.2.3\", :routing=>nil}, {\"input\"=>{\"type\"=>\"filestream\"}, \"tags\"=>[\"beats_input_codec_plain_applied\"], \"ecs\"=>{\"version\"=>\"8.0.0\"}, \"agent\"=>{\"ephemeral_id\"=>\"7c802340-a895-4511-972c-1bd8402da5f5\", \"type\"=>\"filebeat\", \"version\"=>\"8.2.3\", \"id\"=>\"fa2cfb33-e7d8-4005-baff-bfa622d0182c\", \"name\"=>\"ELK\"}, \"event\"=>{\"original\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\"}, \"@version\"=>\"1\", \"@timestamp\"=>2022-06-26T00:54:26.821Z, \"log\"=>{\"file\"=>{\"path\"=>\"/var/log/dpkg.log\"}, \"offset\"=>666691}, \"message\"=>\"2022-06-25 16:58:03 configure pulseaudio:amd64 14.2-2 <none>\", \"host\"=>{\"mac\"=>[\"08:00:27:f7:ff:01\"], \"architecture\"=>\"x86_64\", \"id\"=>\"294c9f13bce94d27b6168ab14ffd752e\", \"os\"=>{\"name\"=>\"Debian GNU/Linux\", \"platform\"=>\"debian\", \"type\"=>\"linux\", \"version\"=>\"11 (bullseye)\", \"codename\"=>\"bullseye\", \"kernel\"=>\"5.10.0-15-amd64\", \"family\"=>\"debian\"}, \"containerized\"=>false, \"ip\"=>[\"10.0.0.48\", \"2601:541:300:5d40::3440\", \"fe80::a00:27ff:fef7:ff01\"], \"hostname\"=>\"ELK\", \"name\"=>\"ELK\"}}], :response=>{\"index\"=>{\"_index\"=>\"filebeat-8.2.3\", \"_id\"=>nil, \"status\"=>400, \"error\"=>{\"type\"=>\"illegal_argument_exception\", \"reason\"=>\"only write ops with an op_type of create are allowed in data streams\"}}}}", "host"=>{"architecture"=>"x86_64", "id"=>"294c9f13bce94d27b6168ab14ffd752e", "mac"=>["08:00:27:f7:ff:01"], "os"=>{"name"=>"Debian GNU/Linux", "platform"=>"debian", "type"=>"linux", "version"=>"11 (bullseye)", "codename"=>"bullseye", "kernel"=>"5.10.0-15-amd64", "family"=>"debian"}, "containerized"=>false, "ip"=>["10.0.0.48", "2601:541:300:5d40::3440", "fe80::a00:27ff:fef7:ff01"], "hostname"=>"ELK", "name"=>"ELK"}}], :response=>{"index"=>{"_index"=>"filebeat-8.2.3", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}

now its good again. What could be causing intermittent issues?

Thank you so much for this however.. This is significant progress!

4

This is working! Thank you!!

Now its time to figure out how to filter out the alerts I dont really care for, and I guess I need to give the server a bit more resources... Keep getting a message saying my system is too slow xD

However, it is logging as intended at this point. Lots of alerts I want to get rid of (Beginning next scan being one of the main ones!)

I really appreciate this, you are a life saver :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.