Versioning with version_type external

Elastic Stack Logstash
1

Hello,

I want to use an external versioning system with logstash so a document only gets updated when the version is GTE
The config looks like this

output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => [ "hcd1515g:9200" ]
action => "update"
doc_as_upsert => "true"
document_id => "%{name}"
document_type => "mks-files"
index => "logstash-mks-file"
version => "%{Rev}"
version_type => "external_gte"
}
}

But i get this error as soon as version_type is set

[2017-08-26T10:33:59,958][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::OutputDelega
tor:0x7dc93e01 @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x730edfc8 @metric=#<LogStash::Instrument::Me
tric:0x77315571 @collector=#<LogStash::Instrument::Collector:0x3233d5f3 @agent=nil, @metric_store=#<LogStash::Instrument
::MetricStore:0x6bedc82 @store=#<Concurrent::map:0x000000000627a8 entries=2 default_proc=nil>, @structured_lookup_mutex=
#Mutex:0x34ba302, @fast_lookup=#<Concurrent::map:0x000000000627ac entries=57 default_proc=nil>>>>, @namespace_name=[:s
tats, :pipelines, :main, :plugins, :outputs, :"73aba349a5abf8e2465258fc520b2bb30369f619-10"]>, @metric=#<LogStash::Ins
trument::NamespacedMetric:0x521c2abc @metric=#<LogStash::Instrument::Metric:0x77315571 @collector=#<LogStash::Instrument
::Collector:0x3233d5f3 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x6bedc82 @store=#<Concurrent::map:
0x000000000627a8 entries=2 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x34ba302, @fast_lookup=#<Concurrent::Ma
p:0x000000000627ac entries=57 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @l
ogger=#<LogStash::Logging::Logger:0x7cbb5246 @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x68e14285>, @strategy=#
<LogStash::OutputDelegatorStrategies::Shared:0x56529409 @output=<LogStash::Outputs::Elasticsearch hosts=>[//hcd1515g:920
0], action=>"update", doc_as_upsert=>true, document_id=>"%{name}", document_type=>"mks-files", index=>"logstash-m
ks-file", version=>"%{Rev}", version_type=>"external", id=>"73aba349a5abf8e2465258fc520b2bb30369f619-10", enable_
metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_497afcb1-b4ab-4206-8f2c-12366e3b3e66", enable_metric=>true, c
harset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, idle_flus
h_time=>1, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry
_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ssl_certificate_verification=>true, sniffing=>false,
sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=

10000, http_compression=>false>>, @id="73aba349a5abf8e2465258fc520b2bb30369f619-10", @metric_events=#<LogStash::Instr
ument::NamespacedMetric:0x4146b35c @metric=#<LogStash::Instrument::Metric:0x77315571 @collector=#<LogStash::Instrument::
Collector:0x3233d5f3 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x6bedc82 @store=#<Concurrent::map:0x
000000000627a8 entries=2 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x34ba302, @fast_lookup=#<Concurrent::map:
0x000000000627ac entries=57 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"73a
ba349a5abf8e2465258fc520b2bb30369f619-10", :events]>, @output_class=LogStash::Outputs::Elasticsearch>", :error=>"Extern
al versioning is not supported by the update action. See Elasticsearch Guide [8.11] | Elastic
/docs-update.html."}
[2017-08-26T10:33:59,973][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<LogStash::Confi
gurationError: External versioning is not supported by the update action. See https://www.elastic.co/guide/en/elasticsea
rch/reference/current/docs-update.html.>, :backtrace=>["C:/contiio_stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-
output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:82:in build'", "C:/contiio_st ack/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-java/lib/logstash/outputs/elasticsearch.rb :230:in build_client'", "C:/contiio_stack/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.2-jav
a/lib/logstash/outputs/elasticsearch/common.rb:28:in register'", "C:/contiio_stack/logstash/logstash-core/lib/logstash/ output_delegator_strategies/shared.rb:9:in register'", "C:/contiio_stack/logstash/logstash-core/lib/logstash/output_del
egator.rb:41:in register'", "C:/contiio_stack/logstash/logstash-core/lib/logstash/pipeline.rb:268:in register_plugin'"
, "C:/contiio_stack/logstash/logstash-core/lib/logstash/pipeline.rb:279:in register_plugins'", "org/jruby/RubyArray.jav a:1613:in each'", "C:/contiio_stack/logstash/logstash-core/lib/logstash/pipeline.rb:279:in register_plugins'", "C:/con tiio_stack/logstash/logstash-core/lib/logstash/pipeline.rb:288:in start_workers'", "C:/contiio_stack/logstash/logstash-
core/lib/logstash/pipeline.rb:214:in run'", "C:/contiio_stack/logstash/logstash-core/lib/logstash/agent.rb:398:in star
t_pipeline'"]}
[2017-08-26T10:34:00,036][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-08-26T10:34:03,000][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}

2

This is your problem right here

Pipeline aborted due to error {:exception=>#<LogStash::Confi
gurationError: External versioning is not supported by the update action.

Try changing your output plugin action to index, it should start up normally afterwards.

Edit: Changed suggested action to proper one, was a typo

3

Thanks for yout time, with insert i got this error

Action 'insert' is invalid! Pick one of ["index", "delete", "create", "update"]`

it seems to be working with "index"
it indexes the documents, but the @version is not set with
version => "%{Rev}"

i just want a conditional update, that the document only gets indext, if its Version or Rev is GTE to the document in elasticsearch.

Edit:
It works now thanks.

@version isn't the right field it's _version it works now as intended

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.