Filebeat is not sending complete logs


(Sinwar Prem) #1

I am using docker elk for logs and forwarding logs using filebeat but some logs were sent or some not?

my filebeat.yml -> https://ghostbin.com/paste/rrjeh

filebeat logs in debug mode -> https://ghostbin.com/paste/fm2om


(ruflin) #2

Filebeat persists the state. So if you started filebeat in the past it only sends the new events. Looking at the log file during startup it finds a lot of files it has a state for it before.

Can you share some more details on which events / logs are not send or does the above already answer your question?


(Sinwar Prem) #3

Thanks for quick response

So can you direct me how can I state the files again like. I want all the files to be stated again?


(Sinwar Prem) #4

and in my elasticsearch log -

Failed to execute phase [query], all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:855)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:833)
at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:387)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


(ruflin) #5

If you want to reindex all files, you need to stop filebeat and then remove the registry_file here. In your case it is found here: https://ghostbin.com/paste/rrjeh#L164

Not sure how the elasticsearch query is related to this?


(system) #6

This topic was automatically closed after 21 days. New replies are no longer allowed.