App Search Document Limit?

[andrewlavigne]

Continuing the discussion from App Search Limitations:

Hello all!

This is my first post so I apologize if this question was already answered. I did some searching on this topic and the only article I could find was the one linked above which was never really answered...

I am self-hosting an elasticsearch, kibana, and appsearch instance. I am trying to index a large number of logs so I can search through them for certain keywords. I have written scripts that convert the logs to the proper json format for appsearch and runs the curl command to post them. That all works great but after about an hour or two app search stops acknowledging the total documents number going up. Most of the time it stops around 1,000,000 docs but I once got it to 3,000,000 docs before it stopped. Once it stops, no matter how many docs I post the number won't go up. Looking at the api logs, it still recognizes the api requests to post every few seconds and they still say status 200 that they are going through correctly, but no more total documents. I'm not positive if it's actually taking the documents in and just not changing the number on the dashboard or just not taking in any more documents. Is there some limit I'm running into I don't know about?

I have looked through the config yml but couldn't find any limit correlating to number of documents. Even so, I have changed all the limits in the config to about 4x original - just in case. I have tried deleting the engine and starting over probably at least 10 times with no real change to the number of docs it stops at - it's just random somewhere between 1-3 million.

Any advice would be greatly appreciated! Thanks!

[JasonStoltz (Jason Stoltzfus)]

Hey @andrewlavigne,

I'm not positive if it's actually taking the documents in and just not changing the number on the dashboard or just not taking in any more documents. Is there some limit I'm running into I don't know about?

Thank you for posting. I suspect we have an issue with document counts on our dashboard. The number shown there may sometimes lag behind the actual number of documents indexed, and that issue may be exacerbated by the large volume of documents being indexed.

However, your documents should have been indexed properly, as we do not have a limit on documents.

[andrewlavigne]

Thanks for the reply! I think you're right, in the meantime since posting this, I decided to keep indexing documents anyway and see what happens. The number did eventually go up to 12 million on the dashboard - although this was nowhere near proportional to the actual number of documents I indexed. Then I discovered kibana's elasticstack monitoring and setting that up instantly said I had 45 million docs which I believe is right. So yeah from what I can tell, elasticsearch is storing the documents, the app search dashboard number just doesn't recognize that high volume.

Thanks for the input!