Output to elasticsearch hitting error 409

[tbrettinger (Thomas Brettinger)]

Hi,

i'm rather new to elasticsearch, so I guess I am missing something obvious.

I am trying to migrate data from opensearch to elasticsearch, following allong this article:

Migrating 1 billion log lines from OpenSearch to Elasticsearch — Elastic...

After some fiddling, I got logstash to read the opensearch data, and also connect to my local elasticsearch. However, it gets 409 errors and exits:

[WARN ] 2024-11-13 14:26:53.709 [main] runner - 'pipeline.buffer.type' setting is not explicitly defined.Before moving to 9.x set it to 'heap' and tune heap size upward, or set it to 'direct' to maintain existing behavior.
[INFO ] 2024-11-13 14:26:53.718 [main] runner - Starting Logstash {"logstash.version"=>"8.16.0", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +jit [x86_64-linux]"}
[INFO ] 2024-11-13 14:26:53.723 [main] runner - JVM bootstrap flags: [-Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[INFO ] 2024-11-13 14:26:53.737 [main] settings - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2024-11-13 14:26:53.740 [main] settings - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[INFO ] 2024-11-13 14:26:54.181 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"c4b8452e-701d-432f-ad32-bb4b82924d93", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2024-11-13 14:26:55.480 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2024-11-13 14:26:56.203 [Converge PipelineAction::Create<reindex-os-es>] Reflections - Reflections took 238 ms to scan 1 urls, producing 149 keys and 523 values
[WARN ] 2024-11-13 14:26:56.849 [Converge PipelineAction::Create<reindex-os-es>] elasticsearch - You are using a deprecated config setting "ssl" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_enabled' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"ssl", :plugin=><LogStash::Outputs::ElasticSearch password=><password>, id=>"6cca87c341c79d421766aeae3e7c36603130c475a86ddba2f61b806427a96319", document_id=>"%{[@metadata][doc][_id]}", ssl=>false, user=>"elastic", hosts=>[http://elasticsearch.dmarc.svc.cluster.local:9200], enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_9a002eb6-d92c-46eb-bbdd-4e7295571c57", enable_metric=>true, charset=>"UTF-8">, workers=>1, ssl_certificate_verification=>true, ssl_verification_mode=>"full", sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>true, compression_level=>1, retry_initial_interval=>2, retry_max_interval=>64, dlq_on_failed_indexname_interpolation=>true, data_stream_type=>"logs", data_stream_dataset=>"generic", data_stream_namespace=>"default", data_stream_sync_fields=>true, data_stream_auto_routing=>true, manage_template=>true, template_overwrite=>false, template_api=>"auto", doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy">}
[INFO ] 2024-11-13 14:26:56.885 [Converge PipelineAction::Create<reindex-os-es>] javapipeline - Pipeline `reindex-os-es` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO ] 2024-11-13 14:26:56.942 [[reindex-os-es]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch.dmarc.svc.cluster.local:9200"]}
[INFO ] 2024-11-13 14:26:57.197 [[reindex-os-es]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch.dmarc.svc.cluster.local:9200/]}}
[WARN ] 2024-11-13 14:26:57.680 [[reindex-os-es]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch.dmarc.svc.cluster.local:9200/"}
[INFO ] 2024-11-13 14:26:57.682 [[reindex-os-es]-pipeline-manager] elasticsearch - Elasticsearch version determined (8.16.0) {:es_version=>8}
[WARN ] 2024-11-13 14:26:57.683 [[reindex-os-es]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[INFO ] 2024-11-13 14:26:57.711 [[reindex-os-es]-pipeline-manager] elasticsearch - Data streams auto configuration (`data_stream => auto` or unset) resolved to `true`
[INFO ] 2024-11-13 14:26:57.794 [[reindex-os-es]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"reindex-os-es", "pipeline.workers"=>4, "pipeline.batch.size"=>1000, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>4000, "pipeline.sources"=>["/etc/logstash/pipelines/pipeline.conf"], :thread=>"#<Thread:0x72dce014 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:139 run>"}
[INFO ] 2024-11-13 14:26:59.168 [[reindex-os-es]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.37}
[INFO ] 2024-11-13 14:27:00.098 [[reindex-os-es]-pipeline-manager] opensearch - ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[INFO ] 2024-11-13 14:27:00.100 [[reindex-os-es]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"reindex-os-es"}
[INFO ] 2024-11-13 14:27:00.121 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:"reindex-os-es"], :non_running_pipelines=>[]}
[WARN ] 2024-11-13 14:27:00.388 [[reindex-os-es]>worker0] elasticsearch - Failed action {:status=>409, :action=>["create", {:_id=>"ukTbIJMBdQq6jxjn33PZ", :_index=>"logs-generic-default", :routing=>nil}, {"date_end"=>"2024-11-11T23:59:59+00:00", "report_id"=>"17870870857350716043", "dkim_aligned"=>true, "spf_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "scope"=>"mfrom"}], "@timestamp"=>2024-11-13T14:27:00.182996121Z, "envelope_from"=>"nms.cool.de", "org_extra_contact_info"=>"https://support.google.com/a/answer/2466580", "date_begin"=>"2024-11-11T00:00:00+00:00", "message_count"=>9, "xml_schema"=>"draft", "source_base_domain"=>"cool.de", "source_country"=>"DE", "header_from"=>"nms.cool.de", "source_reverse_dns"=>"mail.cool.de", "spf_aligned"=>true, "disposition"=>"none", "published_policy"=>{"fo"=>"0", "sp"=>"quarantine", "p"=>"quarantine", "adkim"=>"r", "domain"=>"cool.de", "pct"=>100, "aspf"=>"r"}, "date_range"=>["2024-11-11T00:00:00+00:00", "2024-11-11T23:59:59+00:00"], "source_ip_address"=>"2a01:4f8:c2c:2621::1", "source_name"=>"cool.de", "org_email"=>"noreply-dmarc-support@google.com", "dkim_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "selector"=>"dkim"}], "@version"=>"1", "passed_dmarc"=>true, "org_name"=>"google.com", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[ukTbIJMBdQq6jxjn33PZ]: version conflict, document already exists (current version [1])", "index_uuid"=>"68qqpnUuTU-cs6uQYVpgfw", "shard"=>"0", "index"=>".ds-logs-generic-default-2024.11.13-000001"}}}}
[WARN ] 2024-11-13 14:27:00.388 [[reindex-os-es]>worker2] elasticsearch - Failed action {:status=>409, :action=>["create", {:_id=>"tkTbIJMBdQq6jxjn3nNP", :_index=>"logs-generic-default", :routing=>nil}, {"date_end"=>"2024-11-11T23:59:59+00:00", "report_id"=>"17870870857350716043", "dkim_aligned"=>true, "spf_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "scope"=>"mfrom"}], "@timestamp"=>2024-11-13T14:27:00.182158683Z, "envelope_from"=>"nms.cool.de", "org_extra_contact_info"=>"https://support.google.com/a/answer/2466580", "date_begin"=>"2024-11-11T00:00:00+00:00", "message_count"=>14, "xml_schema"=>"draft", "source_base_domain"=>"cool.de", "source_country"=>"DE", "header_from"=>"nms.cool.de", "source_reverse_dns"=>"mail.cool.de", "spf_aligned"=>true, "disposition"=>"none", "published_policy"=>{"fo"=>"0", "sp"=>"quarantine", "p"=>"quarantine", "adkim"=>"r", "domain"=>"cool.de", "pct"=>100, "aspf"=>"r"}, "date_range"=>["2024-11-11T00:00:00+00:00", "2024-11-11T23:59:59+00:00"], "source_ip_address"=>"157.90.115.1", "source_name"=>"cool.de", "org_email"=>"noreply-dmarc-support@google.com", "dkim_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "selector"=>"dkim"}], "@version"=>"1", "passed_dmarc"=>true, "org_name"=>"google.com", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[tkTbIJMBdQq6jxjn3nNP]: version conflict, document already exists (current version [1])", "index_uuid"=>"68qqpnUuTU-cs6uQYVpgfw", "shard"=>"0", "index"=>".ds-logs-generic-default-2024.11.13-000001"}}}}
[WARN ] 2024-11-13 14:27:00.390 [[reindex-os-es]>worker2] elasticsearch - Failed action {:status=>409, :action=>["create", {:_id=>"uETbIJMBdQq6jxjn33MI", :_index=>"logs-generic-default", :routing=>nil}, {"date_end"=>"2024-11-11T23:59:59+00:00", "report_id"=>"17870870857350716043", "dkim_aligned"=>false, "spf_results"=>[{"domain"=>"vpn-fra.cool.de", "result"=>"none", "scope"=>"mfrom"}], "@timestamp"=>2024-11-13T14:27:00.182632395Z, "envelope_from"=>"vpn-fra.cool.de", "org_extra_contact_info"=>"https://support.google.com/a/answer/2466580", "date_begin"=>"2024-11-11T00:00:00+00:00", "message_count"=>1, "xml_schema"=>"draft", "source_base_domain"=>"cool.de", "source_country"=>"GB", "header_from"=>"vpn-fra.cool.de", "source_reverse_dns"=>"vpn-fra.cool.de", "spf_aligned"=>false, "disposition"=>"quarantine", "published_policy"=>{"fo"=>"0", "sp"=>"quarantine", "p"=>"quarantine", "adkim"=>"r", "domain"=>"cool.de", "pct"=>100, "aspf"=>"r"}, "date_range"=>["2024-11-11T00:00:00+00:00", "2024-11-11T23:59:59+00:00"], "source_ip_address"=>"213.200.90.103", "source_name"=>"cool.de", "org_email"=>"noreply-dmarc-support@google.com", "@version"=>"1", "passed_dmarc"=>false, "org_name"=>"google.com", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[uETbIJMBdQq6jxjn33MI]: version conflict, document already exists (current version [1])", "index_uuid"=>"68qqpnUuTU-cs6uQYVpgfw", "shard"=>"0", "index"=>".ds-logs-generic-default-2024.11.13-000001"}}}}
[WARN ] 2024-11-13 14:27:00.388 [[reindex-os-es]>worker1] elasticsearch - Failed action {:status=>409, :action=>["create", {:_id=>"ekRXJJMBdQq6jxjn5XTR", :_index=>"logs-generic-default", :routing=>nil}, {"date_end"=>"2024-11-12T00:00:00+00:00", "report_id"=>"58fe38f9e0e64d60a3042ddb28b7f93d", "dkim_aligned"=>true, "spf_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "scope"=>"mfrom"}], "@timestamp"=>2024-11-13T14:27:00.184274623Z, "envelope_from"=>"nms.cool.de", "envelope_to"=>"outlook.de", "date_begin"=>"2024-11-11T00:00:00+00:00", "message_count"=>24, "xml_schema"=>"1.0", "source_base_domain"=>"cool.de", "source_country"=>"DE", "header_from"=>"nms.cool.de", "source_reverse_dns"=>"mail.cool.de", "spf_aligned"=>true, "disposition"=>"none", "published_policy"=>{"fo"=>"0", "sp"=>"quarantine", "p"=>"quarantine", "adkim"=>"r", "domain"=>"cool.de", "pct"=>100, "aspf"=>"r"}, "date_range"=>["2024-11-11T00:00:00+00:00", "2024-11-12T00:00:00+00:00"], "source_ip_address"=>"157.90.115.1", "source_name"=>"cool.de", "org_email"=>"dmarcreport@microsoft.com", "dkim_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "selector"=>"dkim"}], "@version"=>"1", "passed_dmarc"=>true, "org_name"=>"outlook.com", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[ekRXJJMBdQq6jxjn5XTR]: version conflict, document already exists (current version [1])", "index_uuid"=>"68qqpnUuTU-cs6uQYVpgfw", "shard"=>"0", "index"=>".ds-logs-generic-default-2024.11.13-000001"}}}}
[WARN ] 2024-11-13 14:27:00.387 [[reindex-os-es]>worker3] elasticsearch - Failed action {:status=>409, :action=>["create", {:_id=>"rkTbIJMBdQq6jxjnynMa", :_index=>"logs-generic-default", :routing=>nil}, {"date_end"=>"2024-11-11T23:59:59+00:00", "report_id"=>"289ade2d68ed4818a6ef89f2b225a3cc", "dkim_aligned"=>true, "spf_results"=>[{"domain"=>"cool.de", "result"=>"pass", "scope"=>"mfrom"}], "@timestamp"=>2024-11-13T14:27:00.174197568Z, "envelope_from"=>"cool.de", "org_extra_contact_info"=>"https://postmaster.web.de/en/case?c=r2002", "date_begin"=>"2024-11-11T00:00:00+00:00", "message_count"=>2, "xml_schema"=>"1.0", "source_base_domain"=>"cool.de", "source_country"=>"DE", "header_from"=>"cool.de", "source_reverse_dns"=>"mail.cool.de", "spf_aligned"=>true, "disposition"=>"none", "published_policy"=>{"fo"=>"0", "sp"=>"quarantine", "p"=>"quarantine", "adkim"=>"r", "domain"=>"cool.de", "pct"=>100, "aspf"=>"r"}, "date_range"=>["2024-11-11T00:00:00+00:00", "2024-11-11T23:59:59+00:00"], "source_ip_address"=>"157.90.115.1", "source_name"=>"cool.de", "org_email"=>"noreply-dmarc@sicher.web.de", "dkim_results"=>[{"domain"=>"cool.de", "result"=>"pass", "selector"=>"dkim"}], "@version"=>"1", "passed_dmarc"=>true, "org_name"=>"web.de", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[rkTbIJMBdQq6jxjnynMa]: version conflict, document already exists (current version [1])", "index_uuid"=>"68qqpnUuTU-cs6uQYVpgfw", "shard"=>"0", "index"=>".ds-logs-generic-default-2024.11.13-000001"}}}}
[WARN ] 2024-11-13 14:27:00.394 [[reindex-os-es]>worker3] elasticsearch - Failed action {:status=>409, :action=>["create", {:_id=>"dURXJJMBdQq6jxjn3nTW", :_index=>"logs-generic-default", :routing=>nil}, {"date_end"=>"2024-11-12T00:00:00+00:00", "report_id"=>"58fe38f9e0e64d60a3042ddb28b7f93d", "dkim_aligned"=>true, "spf_results"=>[{"domain"=>"nms.cool.de", "result"=>"fail", "scope"=>"mfrom"}], "@timestamp"=>2024-11-13T14:27:00.183448317Z, "envelope_from"=>"nms.cool.de", "envelope_to"=>"outlook.de", "date_begin"=>"2024-11-11T00:00:00+00:00", "message_count"=>1, "xml_schema"=>"1.0", "source_base_domain"=>"cool.de", "source_country"=>"DE", "header_from"=>"nms.cool.de", "source_reverse_dns"=>"mail.cool.de", "spf_aligned"=>false, "disposition"=>"none", "published_policy"=>{"fo"=>"0", "sp"=>"quarantine", "p"=>"quarantine", "adkim"=>"r", "domain"=>"cool.de", "pct"=>100, "aspf"=>"r"}, "date_range"=>["2024-11-11T00:00:00+00:00", "2024-11-12T00:00:00+00:00"], "source_ip_address"=>"157.90.115.1", "source_name"=>"cool.de", "org_email"=>"dmarcreport@microsoft.com", "dkim_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "selector"=>"dkim"}], "@version"=>"1", "passed_dmarc"=>true, "org_name"=>"outlook.com", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[dURXJJMBdQq6jxjn3nTW]: version conflict, document already exists (current version [1])", "index_uuid"=>"68qqpnUuTU-cs6uQYVpgfw", "shard"=>"0", "index"=>".ds-logs-generic-default-2024.11.13-000001"}}}}
[WARN ] 2024-11-13 14:27:00.395 [[reindex-os-es]>worker2] elasticsearch - Failed action {:status=>409, :action=>["create", {:_id=>"eERXJJMBdQq6jxjn5HRo", :_index=>"logs-generic-default", :routing=>nil}, {"date_end"=>"2024-11-12T00:00:00+00:00", "report_id"=>"58fe38f9e0e64d60a3042ddb28b7f93d", "dkim_aligned"=>false, "spf_results"=>[{"domain"=>"nms.cool.de", "result"=>"pass", "scope"=>"mfrom"}], "@timestamp"=>2024-11-13T14:27:00.183886265Z, "envelope_from"=>"nms.cool.de", "envelope_to"=>"outlook.de", "date_begin"=>"2024-11-11T00:00:00+00:00", "message_count"=>1, "xml_schema"=>"1.0", "source_base_domain"=>"cool.de", "source_country"=>"DE", "header_from"=>"nms.cool.de", "source_reverse_dns"=>"mail.cool.de", "spf_aligned"=>true, "disposition"=>"none", "published_policy"=>{"fo"=>"0", "sp"=>"quarantine", "p"=>"quarantine", "adkim"=>"r", "domain"=>"cool.de", "pct"=>100, "aspf"=>"r"}, "date_range"=>["2024-11-11T00:00:00+00:00", "2024-11-12T00:00:00+00:00"], "source_ip_address"=>"157.90.115.1", "source_name"=>"cool.de", "org_email"=>"dmarcreport@microsoft.com", "dkim_results"=>[{"domain"=>"nms.cool.de", "result"=>"temperror", "selector"=>"dkim"}], "@version"=>"1", "passed_dmarc"=>true, "org_name"=>"outlook.com", "data_stream"=>{"type"=>"logs", "dataset"=>"generic", "namespace"=>"default"}}], :response=>{"create"=>{"status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[eERXJJMBdQq6jxjn5HRo]: version conflict, document already exists (current version [1])", "index_uuid"=>"68qqpnUuTU-cs6uQYVpgfw", "shard"=>"0", "index"=>".ds-logs-generic-default-2024.11.13-000001"}}}}
[INFO ] 2024-11-13 14:27:00.713 [[reindex-os-es]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"reindex-os-es"}
[INFO ] 2024-11-13 14:27:01.132 [Converge PipelineAction::Delete<reindex-os-es>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:"reindex-os-es"}
[INFO ] 2024-11-13 14:27:01.140 [LogStash::Runner] runner - Logstash shut down.

During search, I found the hint to include 'retry_on_conflict' in my output config:

	        elasticsearch {
            hosts => "http://elasticsearch.dmarc.svc.cluster.local:9200"
            ssl => false
            retry_on_conflict => 5

            user => "${ELASTICSEARCH_USERNAME}"
            password => "${ELASTICSEARCH_PASSWORD}"

            document_id => "%{[@metadata][doc][_id]}"

            data_stream => "true"
            data_stream_type => "logs"
            data_stream_dataset => "dmarc_aggregate"
            data_stream_namespace => "reindex"

this however is of no help:

[WARN ] 2024-11-13 14:37:08.436 [main] runner - 'pipeline.buffer.type' setting is not explicitly defined.Before moving to 9.x set it to 'heap' and tune heap size upward, or set it to 'direct' to maintain existing behavior.
[INFO ] 2024-11-13 14:37:08.446 [main] runner - Starting Logstash {"logstash.version"=>"8.16.0", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +jit [x86_64-linux]"}
[INFO ] 2024-11-13 14:37:08.452 [main] runner - JVM bootstrap flags: [-Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[INFO ] 2024-11-13 14:37:08.468 [main] settings - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2024-11-13 14:37:08.470 [main] settings - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[INFO ] 2024-11-13 14:37:09.198 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"4718324d-43ea-490e-9dab-e48568561828", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2024-11-13 14:37:10.537 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2024-11-13 14:37:11.309 [Converge PipelineAction::Create<reindex-os-es>] Reflections - Reflections took 287 ms to scan 1 urls, producing 149 keys and 523 values
[WARN ] 2024-11-13 14:37:11.942 [Converge PipelineAction::Create<reindex-os-es>] elasticsearch - You are using a deprecated config setting "ssl" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Set 'ssl_enabled' instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"ssl", :plugin=><LogStash::Outputs::ElasticSearch password=><password>, hosts=>[http://elasticsearch.dmarc.svc.cluster.local:9200], id=>"b894bd2e548a06ca16a8a6de4f9b8d97fa3fbeaff8d26ef06ba9dd84376c52f7", document_id=>"%{[@metadata][doc][_id]}", ssl=>false, retry_on_conflict=>5, user=>"elastic", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_fbd5700a-473b-44b7-b8d6-f7e8ebdfa0d1", enable_metric=>true, charset=>"UTF-8">, workers=>1, ssl_certificate_verification=>true, ssl_verification_mode=>"full", sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>true, compression_level=>1, retry_initial_interval=>2, retry_max_interval=>64, dlq_on_failed_indexname_interpolation=>true, data_stream_type=>"logs", data_stream_dataset=>"generic", data_stream_namespace=>"default", data_stream_sync_fields=>true, data_stream_auto_routing=>true, manage_template=>true, template_overwrite=>false, template_api=>"auto", doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, ilm_enabled=>"auto", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy">}
[INFO ] 2024-11-13 14:37:12.029 [Converge PipelineAction::Create<reindex-os-es>] javapipeline - Pipeline `reindex-os-es` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO ] 2024-11-13 14:37:12.064 [[reindex-os-es]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch.dmarc.svc.cluster.local:9200"]}
[INFO ] 2024-11-13 14:37:12.301 [[reindex-os-es]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch.dmarc.svc.cluster.local:9200/]}}
[WARN ] 2024-11-13 14:37:12.718 [[reindex-os-es]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch.dmarc.svc.cluster.local:9200/"}
[INFO ] 2024-11-13 14:37:12.719 [[reindex-os-es]-pipeline-manager] elasticsearch - Elasticsearch version determined (8.16.0) {:es_version=>8}
[WARN ] 2024-11-13 14:37:12.720 [[reindex-os-es]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[INFO ] 2024-11-13 14:37:12.737 [[reindex-os-es]-pipeline-manager] elasticsearch - Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"retry_on_conflict"=>5}
[INFO ] 2024-11-13 14:37:12.737 [[reindex-os-es]-pipeline-manager] elasticsearch - Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[INFO ] 2024-11-13 14:37:12.785 [[reindex-os-es]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"reindex-os-es", "pipeline.workers"=>4, "pipeline.batch.size"=>1000, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>4000, "pipeline.sources"=>["/etc/logstash/pipelines/pipeline.conf"], :thread=>"#<Thread:0x2e05cbb2 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:139 run>"}
[INFO ] 2024-11-13 14:37:12.788 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.9-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:164] elasticsearch - Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[INFO ] 2024-11-13 14:37:14.029 [[reindex-os-es]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.24}
[INFO ] 2024-11-13 14:37:15.180 [[reindex-os-es]-pipeline-manager] opensearch - ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[INFO ] 2024-11-13 14:37:15.181 [[reindex-os-es]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"reindex-os-es"}
[INFO ] 2024-11-13 14:37:15.201 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:"reindex-os-es"], :non_running_pipelines=>[]}
[INFO ] 2024-11-13 14:37:17.750 [[reindex-os-es]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"reindex-os-es"}
[INFO ] 2024-11-13 14:37:18.219 [Converge PipelineAction::Delete<reindex-os-es>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:"reindex-os-es"}
[INFO ] 2024-11-13 14:37:18.229 [LogStash::Runner] runner - Logstash shut down.

I am running into walls currently, whatever I try. I would be grateful if someone could give me a hint what I could do to get that data across

logstash config:

input {
    opensearch {
      hosts => ["opensearch.dmarc.svc.cluster.local:9200"]
      ssl => true
      user => "${OPENSEARCH_USERNAME}"
      password => "${OPENSEARCH_PASSWORD}"
      index => "${SOURCE_INDEX_NAME}"
      slices => "${SOURCE_SLICES}"
      size => "${SOURCE_PAGE_SIZE}"
      scroll => "5m"
      docinfo => true
      docinfo_target => "[@metadata][doc]"
    }
}

filter {
    mutate {
    }
}

output {
    elasticsearch {
        hosts => "http://elasticsearch.dmarc.svc.cluster.local:9200"
        ssl => false
        retry_on_conflict => 5
        user => "${ELASTICSEARCH_USERNAME}"
        password => "${ELASTICSEARCH_PASSWORD}"
        document_id  => "%{[@metadata][doc][_id]}"
     }
}

Thanks in advance for any hint!

Regards
Thomas

[leandrojmp (Leandro Pereira)]

Hello and welcome,

Can you share what is the elasticsearch output you are using? You shard two different elasticsearch outputs, one with the data_stream settings and another without it.

The error you shared is because Logstash is writing into data streams and data streams are append only, it is for some reason trying to index a document with a _id that was already indexed.

Did you run this pipeline multiple times? Your opensearch _id seems to be auto-generated, so for logstash to try to index it multiple times the only way I can think is if this pipeline was already executed and the same document was already indexed.

[tbrettinger (Thomas Brettinger)]

multiple times probably is an understatment, I guess while testing, I ran that a million times

the second output (without datastream) matches the config I posted. For some reason, it does not recogonize the data_stream in that case.

If I can figure out how (I haven't yet), I guess I could delete the indices logstash was creating during the trial runs and give it another shot?

Regards,
Thomas

[tbrettinger (Thomas Brettinger)]

add-on: I am not even sure, if I should use datastream here. The data in opensearch is split among several indices, one per 24h.

It's a total of 46 indices, each between 40kb and 200kb (so, it's not a lot of data, sharing this in case if it matters for other ideas how to proceed)

Regards
Thomas

[tbrettinger (Thomas Brettinger)]

Update: my problem is solved. Thanks leandro very much for giving advice, the fact that data streams can only be added to made me look in the right places.

I did not use data streams for the migration now (the source did not have them either).
I deleted the indices/data streams that were created by my migration attempts and started all over with a refined config and it finally worked.

Kind regards,
Thomas