Logstash JDBC Input infinite loop

[mc1392]

I'm using Logstash 5.6.3 and I think I am stuck in an infinite loop.

I'm using the upsert functionality, with my Id field as the document_id.

Seems the same record is getting updated multiple times.
The process never stops.

Here us my config:

input {
  jdbc {
    jdbc_driver_library => "/usr/share/java/mysql-connector-java-5.1.38.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://database_url"
    jdbc_user => "user"
    jdbc_password => "password"
    use_column_value=> true
    tracking_column=>"Id"

    statement => "select Id,.....;"
  }
}


output {

stdout {codec=>json}

        elasticsearch{
                hosts=>["https://elastic_cloud_instance:9243"]
                index=>"my-data"
                user=>"user"
                password=>"password"

                doc_as_upsert=>true
                document_id=>"%{id}"
                action=>"update"


        }
}

[Christian_Dahlqvist]

Have a look at this thread.

[mc1392]

Are you saying that I have to use a timestamp instead?

[Christian_Dahlqvist]

No, but make sure you sort it correctly and lowercase the tracking_column name.

[guyboertje]

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-lowercase_column_names

Column names are lower cased by default. This is because all logstash examples and docs are always shown with lower_case_and_underscored_field_names. Its a convention that is useful in large collaborations when different people over time add or change the logstash config expecting lower case.

[mc1392]

A permissions issue caused the process to restart.
Couldn't write to a directory.

Once I ran with sudo, no errors.

In any case, here is the config file:

input {
  jdbc {
    jdbc_driver_library => "/usr/share/java/mysql-connector-java-5.1.38.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://database_url"
    jdbc_user => "user"
    jdbc_password => "password"
    tracking_column=>"Id"
    statement => "select Id,....from.... order by Id;"
  }
}


output {

stdout {}

        elasticsearch{
                hosts=>["https://elastic_cloud_instance:9243"]
                index=>"my-data"
                user=>"user"
                password=>"password"
                doc_as_upsert=>true
                document_id=>"%{id}"
                action=>"update"
        }
}