The records in the elastic index are missing frequenlty

Elastic Stack Logstash
1

I had a record in the elastic index which was indexed by logstash from a sql data source. This record which is already came to elastic index is not available after sometime. After updating the record in the database does the logstash removes the existed record in the elastic index and it will re-index the record with the new updated record ?

2

What does your Logstash config look like?

3

it is a normal config file that indexes the sql table data. it contains input jdbc plugin,filter json, output es

4

Without seeing the file, at least elasticsearch output and sections setting fields used in this, I can not do much to help.

How are you establishing that documents are missing?

5

Here already the document is indexed after some time updating the document, it is not available in elastic index. how once indexed document suddenly disappers.

6

How are you querying for the document?

Are there any errors in the Logstash or Elasticsearch logs?

What does the relevant parts of your Logstash config look like?

7 
input { 

 jdbc { 
    jdbc_connection_string =>""
    jdbc_user =>""
    jdbc_password =>""
    jdbc_driver_class =>"com.mysql.jdbc.Driver"
    schedule =>"* * * * *"
    statement =>"SELECT * FROM blog_pro WHERE last_modified_time  >:sql_last_value"
    use_column_value =>true
    tracking_column =>last_modified_time
    tracking_column_type => "timestamp"
    } 
 }
filter{
json {
    source => "post"
    target => "Post"    
    remove_field => ["post"]
 }
output 
{ 
    #output to elasticsearch    
    elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "idx_index"
        document_id => "%{id}"
    }

}

it looks like this

8

What about the other questions I had?

9
  1. I am querying the document with the document id in the elastic head
  2. There are no errors in elastic & logstash logs
10

Logstash does not delete anything. If a new document with the same ID as an existing document comes in it will simply overwrite that document. I can therefore not see any reason why a document would vanish. If something failed in the update I would expect you to continue seeing the old version.

Which version of Logstash and Elasticsearch are you using? How many documents does your index contain? How many nodes do you have in the cluster?

11

logstash version 6.4.2 and elastic version 6.4.2

I'm indexing 8000 documents and i had only one node in cluster

12

What does the output of the cat indices API look like? What does the query you are running to find the document look like?

13

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

green open idx_index nYFWZEO7TUiOjLQXBaYJpA 6474 0 0 0 260mb 260mb

query: {

                    bool: {

                        must: [

                            {

                                term: {

                                    "Id.keyword": "1453"

                                }

                            },

                            {

                                term: {

                                    "IsActive": true,

                               

                                }

                            }

                        ]

                    }

                }
14

What do you get if you just run GET idx_index/doc/1453 (Might be _doc instead of doc depending on version used..)?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.