Logstash jdbc input plugin not indexing all data to elasticsearch

Elastic Stack Logstash
1

Hi to everyone!

I'm new to elasticsearch and my english is very bad so I will try to explain my issue as well as possible! :grinning:

I got a mysql table with more than 300.000 records, but the logstash is indexing only 3.000 records. I have modified the configuration of the jdbc input several times after searching in google (clear_run, jdbc_page_size, jdbc_fetch_size) but I have not obtained the expected results.

My *.conf file:

input {
    jdbc {
        jdbc_driver_library => '/path/to/my/driver/mysql-connector-java-5.1.42-bin.jar'
        jdbc_driver_class => 'com.mysql.jdbc.Driver'
        jdbc_connection_string => 'jdbc:mysql://127.0.0.1:3306/ln_crm'
        jdbc_user => 'ln_user'
        jdbc_password => '123456'
        statement => 'SELECT * FROM ebooks'
        jdbc_paging_enabled => 'true'
        jdbc_page_size => 300000
    }
}

output {
    elasticsearch {
        hosts => ['127.0.0.1:9200']
        index => 'ln_items'
        document_type => 'items'
        document_id => '%{id_item}'
    }
    stdout { codec => rubydebug }
}

The column "id_item" is a uuid string. Next i show the stdout with one record for check the data model:

{
			       "id_item" => "11f7ea75-06fd-0035-ae80-55dff7c81a4e",
             "tipo_articulo" => "eBook",
        "id_sugar_editorial" => "e2699fb9-ffef-d718-579f-55dea61faa11",
                   "prologo" => nil,
                  "@version" => "1",
            "id_sugar_autor" => "126aefcf-bed7-5c85-4ea8-55dff73b3e35",
     "autor_fecha_defuncion" => nil,
       "imagen_banner_movil" => nil,
           "autor_ceudonimo" => nil,
                "@timestamp" => 2018-01-06T14:48:54.543Z,
           "autor_biografia" => "",
            "no_capitulos_c" => nil,
           "nombre_articulo" => "lorem ipsum dolor",
                     "valor" => 65000.0,
                   "edicion" => "Primera",
              "mas_imagenes" => nil,
                    "indice" => nil,
                   "paginas" => nil,
          "nombre_editorial" => "lorem ipsum dolor",
         "etiqueta_articulo" => "eBook",
          "editorial_imagen" => nil,
                       "plu" => "123456789",               
              "nombre_autor" => "lorem ipsum dolor",
          "recurso_externo1" => nil,
        "proteccion_digital" => "lorem ipsum dolor",
                  "preventa" => false,
            "acepta_ofertas" => false,
              "calificacion" => nil,
                "autor_foto" => nil,
                  "id_autor" => 12345,
                      "isbn" => "123456789",
            "autor_sitioweb" => nil,
         "fecha_publicacion" => 2013-11-25T05:00:00.000Z,
        "autor_nacionalidad" => nil,
             "traducido_por" => nil,
             "imagen_banner" => nil,
                    "idioma" => "Aleman",
              "id_editorial" => 54321,
      "descripcion_articulo" => "lorem ipsum dolor	",
               "pais_origen" => "Germany",
                    "tipo_c" => "EPUB - Publicación Electrónica",
        "editorial_historia" => nil,
    "autor_fecha_nacimiento" => nil,
          "previsualizacion" => true,
            "imagen_portada" => "http://example.com/an-example.jpg",
                      "alto" => nil,
                     "ancho" => nil,
      "imagen_contraportada" => nil,
          "recurso_externo2" => nil
}

My logstash.yml is with the default configuration (logstash 6.1.1). I appreciate your help in advance!

Blessings and a happy new year!

2

Try the solution here and turn paging off.

Otherwise read the other comments in the issue.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.