Hello everyone, once again I ran into a wall with ELK stack that I'm unable to overcome by myself so I came here to ask for your help. In my project , I'm creating new index , putting data into it via logstash and then evaluating the data in angular app. From my understanding elasticsearch index takes a few secs to be created and more time to index data. Since I can't really set a fix timeout for my evaluating function because I don't know if I will get a log file of 20 lines or log file o 200k lines I would need a way to check if elasticsearch index is searchable as the title says. I tried this method .
Made a request
index/_field_caps?fields=*
fields here contain searchable parameter but from testing even on search : true, index is not searchable. I looked into _source field aswell but this one has searchable : false even if I get valid output from my function that fetches data from index. Is my only option to periodically fetch data from index until I don't get error output ? Thank you.
I am not really following why you need to do this, can you elaborate what the actual problem you are trying to solve is?
Hi Mark , what I'm trying to achieve is have a clean index with no duplicate values and with input from only one user at the time as I am evaluating data from client that uploads file to my client app. I can't have my data mixed up with other user inputs etc. Thats why I opted for option to create new index every time user uploads a file. My whole code for this is basically done except the part of checking for index searchability without setting fixed timeout. Im going assign unique ID to the index every time its created and delete it every time user closes his web app.
That seems kind of wasteful for Elasticsearch. Is there a reason you don't keep the index/data after the session?
As I said it is really easy way to prevent duplicates on my indices as I can't really rely on the user to not upload the same file multiple times. I'm not sure if this will even go to production as this is only part of my school project where Im using ELK stack. But just for my own interest what would be the 100% correct way of handling this ? I think I saw that you can set expiry date on index so maybe assign unique ID to the index for user and delete the index after set amount of time ? This would leave 2 problems which I can think of right away.
- Problem , you see, this software is like an online chart building tool, you dont need to register or anything so Im not sure if I have a way of indentifying new user and tying this user to some id (maybe IP?). If I had a way of tying a user to index ID I would be able to create index on new user arrival and then do my stuff with Elastic.
- Problem , duplicate values , I was searching through elastic forums quite a bit and I found proposed solution of digital fingerprint on each value, however to be honest I didnt really get it right away so I didnt loook very deep into it , its just something I remember from top of my head.
This is why I opted for the slower but far easier solution of creating new index -> posting data to logstash -> logstash to elastic -> fetching data from elastic -> evaluating data -> delete index. This means no identification of user , no duplicate management etc. , to be honest its far easier solution for my project if only I could figure out how to check if index is serchable via some query other than cycling through trying until I get valid result.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.