Continuing the discussion from App Search Limitations:
This is my first post so I apologize if this question was already answered. I did some searching on this topic and the only article I could find was the one linked above which was never really answered...
I am self-hosting an elasticsearch, kibana, and appsearch instance. I am trying to index a large number of logs so I can search through them for certain keywords. I have written scripts that convert the logs to the proper json format for appsearch and runs the curl command to post them. That all works great but after about an hour or two app search stops acknowledging the total documents number going up. Most of the time it stops around 1,000,000 docs but I once got it to 3,000,000 docs before it stopped. Once it stops, no matter how many docs I post the number won't go up. Looking at the api logs, it still recognizes the api requests to post every few seconds and they still say status 200 that they are going through correctly, but no more total documents. I'm not positive if it's actually taking the documents in and just not changing the number on the dashboard or just not taking in any more documents. Is there some limit I'm running into I don't know about?
I have looked through the config yml but couldn't find any limit correlating to number of documents. Even so, I have changed all the limits in the config to about 4x original - just in case. I have tried deleting the engine and starting over probably at least 10 times with no real change to the number of docs it stops at - it's just random somewhere between 1-3 million.
Any advice would be greatly appreciated! Thanks!