Logstash crashed 2 days ago, now not parsing files

[jamesp220291 (james)]

Hi

My Elastic search crashed 2 days ago, I have since restarted it and all the services are running.

However logstash does not seem to be parsing the build up of files, or any new files.

New files/logs are fetched from a remote server and put in a directory every 10 minutes, obviously there is quite alot built up.

I've checked the logstash logs and nothing is being logged from the pipeline start -

[2019-03-05T13:08:28,135][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-03-05T13:08:28,139][INFO ][filewatch.observingread ] START, creating Discoverer, Watch with file and sincedb collections
[2019-03-05T13:08:28,338][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Does anyone know why its doing this and why there are no errors anywhere, i could work with errors but it just seems to be doing nothing.

[A_B]

Just curious (as the startup messages look healthy)... You say Logstash does not seem to be parsing. What do you base this on? A few times when I have has log buildup or lag that needs to be processed I have thought there are problems but in fact Logstash has been doing exactly what it was supposed to, it just took a while to process the backlog. The new logs were shipped to Elasticsearch with their original timestamp so they show up in "past" indices at first. Just a possibility.

How busy is the Logstash process? CPU wise...

[jamesp220291 (james)]

Hi

Thanks for your reply

The process seems busy, but nothing has been added to Kibana since 2nd of March at 11pm.

I'd expect it to start adding them in bits as they come through (as it does if i was make it re-index)

I wouldnt expect it process it all in 1 batch.

It's the fact nothing is being logged, that is making me thing it is doing nothing.

[A_B]

What does the monitoring in Kibana show? If you have x-pack monitoring that is. Sometimes I use that to figure things out... You can sort indices based on Index Rate ...

If that doesn't help shed some light on your problem, it would probably be good to list some things about your setup, like

• Logstash version
• OS you run Logstash on
• Elasticsearch version
• etc...

Posting your Logstash config would not hurt either.

Do Elasticsearch logs show anything that might be related?

[jamesp220291 (james)]

Hi

Kibana is now showing logs from the 3rd of march, nothing for 4/5/6th though.

It could just be like you said it seems busy, but I wouldn't expect it to be so slow, its running on quite a beefy dedicated box.

I enabled the monitoring and ES is 6.5.1

I noticed this -
JVM Heap

99.71%

987.1 MB / 989.9 MB

The server has 32GB of RAM, but doesnt seem to be allocating it all the JVM heap?

Should I increase the heap size?

so - Increased heap size, now getting -
{"message":"all shards failed: [search_phase_execution_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}

On Kibana, curl -v http://localhost:9200 shows elastic search is responding

Elastic search logs show -
2019-03-06T08:53:53,435][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52603] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:53:55,456][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52604] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:53:57,519][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52605] overhead, spent [2s] collecting in the last [2s]
[2019-03-06T08:53:59,569][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52606] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:54:01,603][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52607] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:54:03,628][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52608] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:54:05,690][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52609] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:54:07,707][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52610] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:54:09,757][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52611] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:54:11,783][WARN ][o.e.m.j.JvmGcMonitorService] [nre3ddu] [gc][52612] overhead, spent [1.9s] collecting in the last [2s]
[2019-03-06T08:53:19,578][DEBUG][o.e.a.b.TransportShardBulkAction] [nre3ddu] [stackpathcdn-2019.03.03][0] failed to execute bulk item (index) index

[A_B]

@jamesp220291

987.1 MB / 989.9 MB

The above suggest that it is configured to be only 989.9 MB. Might be that Logstash defaults to 1GB. I would increase it.

With ps or systemctl you should be able to see how much heap Logstash is configured to use. I have -Xms8g -Xmx8g on one of my machines as an example. That seems to be set in /etc/logstash/jvm.options

[jamesp220291 (james)]

Hi

I have done this and it seems to be working now.

Only issue is if i restart elastic search, Kibana seems to die for about 45 minutes, which is strange with the below -

{"message":"all shards failed: [search_phase_execution_exception] all shards failed","statusCode":503,"error":"Service Unavailable"}