Logstash does not update document

Elastic Stack Logstash
1

Hi,

I have a problem with data update in logstash.
I need to update specific field in some document according to sql data.
My data is looking like this:

testid-field1-field2-field3-testtype-time
1-1-1-1-1-2023/05
1-null-null-2-2-2023/06
1-null-null-3-2-2023/07
Time field is an example (It is normal in my sql data)
I need to have it as
1-1-1-3-2
in the end.

I have a logstash configuration like this:

input {
    jdbc {
        jdbc_driver_library => "${LOGSTASH_JDBC_DRIVER_JAR_LOCATION}"
        jdbc_driver_class => "${LOGSTASH_JDBC_DRIVER}"
        jdbc_connection_string => "${LOGSTASH_JDBC_URL}"
        jdbc_user => "${LOGSTASH_JDBC_USERNAME}"
        jdbc_password => "${LOGSTASH_JDBC_PASSWORD}"
        schedule => "/10 * * * * *"
        statement => "select * from testdata where time
        > :sql_last_value ORDER BY time NULLS LAST"
        use_column_value => true
        tracking_column => "time"
        tracking_column_type => "timestamp"
        jdbc_paging_enabled => true
        jdbc_paging_mode => "explicit"
        jdbc_page_size => 100000
        clean_run => true
    }
}

filter {
    if [testtype] == 2 {mutate {remove_field => ["field1", "field2"]}}
}

output {
    elasticsearch {
        hosts => ["${LOGSTASH_ELASTICSEARCH_HOST}"]
        user => "${ELASTIC_USER}"
        password => "${ELASTIC_PASSWORD}"
        index => "testdata"
        document_id => "%{testid}"
        doc_as_upsert => true
        action => "update"
        retry_initial_interval => 10
        retry_max_interval => 300
        retry_on_conflict => 25
    }
    stdout { codec => json_lines }
}

But with this configuration elasticsearch store it as first row of data which is
1-1-1-1-1

What do I do wrong?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.