Big logs files with TFS elasticsearch

Elastic Stack Elasticsearch
1

Hi,

We have a TFS 2018 and since a few days elasticsearch create 5Go logs file every days.
I don't know what is the matter with it.
The log contain 2 type of WARN "cluster.action.shard" and "indices.cluster"
Can someone help me?

[codesearchshared_1_0][[codesearchshared_1_0][9]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException;
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
	at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
	at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: [codesearchshared_1_0][[codesearchshared_1_0][9]] EngineCreationFailureException[failed to create engine]; nested: EOFException;
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
	... 5 more
Caused by: java.io.EOFException
	at org.apache.lucene.store.InputStreamDataInput.readByte(InputStreamDataInput.java:37)
	at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
	at org.apache.lucene.store.DataInput.readLong(DataInput.java:157)
	at org.elasticsearch.index.translog.Checkpoint.<init>(Checkpoint.java:54)
	at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:83)
	at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:337)
	at org.elasticsearch.index.translog.Translog.<init>(Translog.java:179)
	at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
	... 11 more
[2020-09-07 09:23:49,864][WARN ][cluster.action.shard     ] [SCHRODINGER] [codesearchshared_1_0][9] received shard failed for target shard [[codesearchshared_1_0][9], node[BOxuVCxDRkK8Yq_sDeLLzg], [P], v[215], s[INITIALIZING], a[id=q_QzCNrTSBm3GET9AlvyrA], unassigned_info[[reason=ALLOCATION_FAILED], at[2020-09-07T07:23:49.802Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]]], indexUUID [6o5xtwntTGG74aLcO99cSw], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]
[codesearchshared_1_0][[codesearchshared_1_0][9]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException;
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
	at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
	at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: [codesearchshared_1_0][[codesearchshared_1_0][9]] EngineCreationFailureException[failed to create engine]; nested: EOFException;
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
	at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
	at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1509)
	at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1493)
	at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:966)
	at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:938)
	at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
	... 5 more
Caused by: java.io.EOFException
	at org.apache.lucene.store.InputStreamDataInput.readByte(InputStreamDataInput.java:37)
	at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
	at org.apache.lucene.store.DataInput.readLong(DataInput.java:157)
	at org.elasticsearch.index.translog.Checkpoint.<init>(Checkpoint.java:54)
	at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:83)
	at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:337)
	at org.elasticsearch.index.translog.Translog.<init>(Translog.java:179)
	at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:205)
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:148)
	... 11 more
2

What's TFS? What does it have to do with Elasticsearch in this context?

3

TFS is Team Foundation Server
This is a Microsoft software that use elasticsearch for some of the search that you can do.

I was talking about TFS, because we didn't install elasticsearch, we install TFS that install and use elasticsearch

4

This looks to be a positively ancient version of Elasticsearch, somewhere in the 2.3 or 2.4 range, so literally years past the end of its supported life. It also looks like the data on disk is fundamentally broken. I'm not sure what to suggest apart from a reinstall since we don't know how TFS is using it and whether it can repair the broken data. Microsoft support are the right people to ask about that. You should also upgrade this to a more recent version of Elasticsearch as a matter of some urgency.

1 Like
5

They said that

Hi, we don’t give support to these kind of issues because it is related to a third-party product. Please contact them directly for assistance. Best regards!

6

Sorry, I haven't got any better advice than to upgrade to a non-ancient version of Elasticsearch and reinstall whatever needs reinstalling to rebuild the lost data.

7

Tx, since the search in TFS seems to work fine, we have setup the log on error for now.
We will try to add a new ElasticSearch server on day, but not now.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.