New documents are not getting added to an already existing index

Elastic Stack Elasticsearch
1

Hi,
I am new to the ELK stack workflow, currently I am doing a project that pulls live feed from twitter every time a user requests and I want to display the results using Kibana. In my project instead of using Logstash to collect data I have used tweepy library in python, and then connected it to Elastic Search where I created an index to store my incoming data.

The problem:
The code is working but my problem is that the incoming doccuments are getting stored only if I give a new index name every other time my new set of live feeds come in. If I wanted to use the same index name to add all incoming documents to the existing ones the new documents are not updating.

I have attached my code snipped below:

def filterKeys(document):
            return {key: document[key] for key in use_these_keys }
 from elasticsearch import Elasticsearch
 from elasticsearch import helpers
 es_client = Elasticsearch(http_compress=True)
 def doc_generator(df):
            df_iter = df.iterrows()
            #index, document = next(df_iter)
            for index, document in df_iter:
                yield {
                        "_index": 'twitter_req',
                        "_type": "twitter_twp",
                        "_id" : f"{document['ID']}",
                        "_source": filterKeys(document),
                    }
            #raise StopIteration
 helpers.bulk(es_client, doc_generator(twitter_feeds))

Can any of you kindly tell me where I am going wrong and how to make it so that incoming documents get automatically updated every time the code runs.

2

Given that you have said you are not using logstash I do not understand why you would post this in the logstash forum. Perhaps you should edit it to move it to the elasticsearch forum.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.