Logstash jdbc input plugin not indexing all data to elasticsearch

Hi to everyone!

I'm new to elasticsearch and my english is very bad so I will try to explain my issue as well as possible! :grinning:

I got a mysql table with more than 300.000 records, but the logstash is indexing only 3.000 records. I have modified the configuration of the jdbc input several times after searching in google (clear_run, jdbc_page_size, jdbc_fetch_size) but I have not obtained the expected results.

My *.conf file:


input {
    jdbc {
        jdbc_driver_library => '/path/to/my/driver/mysql-connector-java-5.1.42-bin.jar'
        jdbc_driver_class => 'com.mysql.jdbc.Driver'
        jdbc_connection_string => 'jdbc:mysql://127.0.0.1:3306/ln_crm'
        jdbc_user => 'ln_user'
        jdbc_password => '123456'
        statement => 'SELECT * FROM ebooks'
        jdbc_paging_enabled => 'true'
        jdbc_page_size => 300000
    }
}

output {
    elasticsearch {
        hosts => ['127.0.0.1:9200']
        index => 'ln_items'
        document_type => 'items'
        document_id => '%{id_item}'
    }
    stdout { codec => rubydebug }
}

The column "id_item" is a uuid string. Next i show the stdout with one record for check the data model:


{
			       "id_item" => "11f7ea75-06fd-0035-ae80-55dff7c81a4e",
             "tipo_articulo" => "eBook",
        "id_sugar_editorial" => "e2699fb9-ffef-d718-579f-55dea61faa11",
                   "prologo" => nil,
                  "@version" => "1",
            "id_sugar_autor" => "126aefcf-bed7-5c85-4ea8-55dff73b3e35",
     "autor_fecha_defuncion" => nil,
       "imagen_banner_movil" => nil,
           "autor_ceudonimo" => nil,
                "@timestamp" => 2018-01-06T14:48:54.543Z,
           "autor_biografia" => "",
            "no_capitulos_c" => nil,
           "nombre_articulo" => "lorem ipsum dolor",
                     "valor" => 65000.0,
                   "edicion" => "Primera",
              "mas_imagenes" => nil,
                    "indice" => nil,
                   "paginas" => nil,
          "nombre_editorial" => "lorem ipsum dolor",
         "etiqueta_articulo" => "eBook",
          "editorial_imagen" => nil,
                       "plu" => "123456789",               
              "nombre_autor" => "lorem ipsum dolor",
          "recurso_externo1" => nil,
        "proteccion_digital" => "lorem ipsum dolor",
                  "preventa" => false,
            "acepta_ofertas" => false,
              "calificacion" => nil,
                "autor_foto" => nil,
                  "id_autor" => 12345,
                      "isbn" => "123456789",
            "autor_sitioweb" => nil,
         "fecha_publicacion" => 2013-11-25T05:00:00.000Z,
        "autor_nacionalidad" => nil,
             "traducido_por" => nil,
             "imagen_banner" => nil,
                    "idioma" => "Aleman",
              "id_editorial" => 54321,
      "descripcion_articulo" => "lorem ipsum dolor	",
               "pais_origen" => "Germany",
                    "tipo_c" => "EPUB - Publicación Electrónica",
        "editorial_historia" => nil,
    "autor_fecha_nacimiento" => nil,
          "previsualizacion" => true,
            "imagen_portada" => "http://example.com/an-example.jpg",
                      "alto" => nil,
                     "ancho" => nil,
      "imagen_contraportada" => nil,
          "recurso_externo2" => nil
}

My logstash.yml is with the default configuration (logstash 6.1.1). I appreciate your help in advance!

Blessings and a happy new year!

Try the solution here and turn paging off.

Otherwise read the other comments in the issue.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.