I have set up elk version 6.3.2, all the services can work normally, but the index of es is automatically deleted. In each index, I can only view the last 5 records, and more than 5 will be deleted.
This can not happen.
You have something or someone who is calling the DELETE API.
Can you share your full elasticsearch logs please?
Sorry, what I described earlier is not very accurate. There should be only one record in each index, and I did not perform any delete operations on es.
All services are running normally and the logs are at the info level
There should be only one record in each index
Why? What are you exactly doing?
All services are running normally and the logs are at the info level
Yes but please share them
This is a deprecation. Log for a portion of the primary node
++++++++++++++++++++++++++++++++++++++++++++++++++++++
[2018-08-10T14:47:05,461][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T14:47:14,875][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T14:47:21,854][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T14:47:27,849][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:01:58,776][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:02:05,209][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:02:12,496][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:02:17,898][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:02:23,713][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:02:29,763][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:02:29,765][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:03:07,397][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:03:08,434][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:03:09,699][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:03:37,509][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:03:37,510][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:04:04,077][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:04:04,869][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:04:06,113][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:33:10,193][WARN ][o.e.d.c.ParseField ] Deprecated field [all_fields] used, replaced by [Set [fields] to * instead]
[2018-08-10T15:33:10,197][WARN ][o.e.d.c.ParseField ] Deprecated field [inline] used, expected [source] instead
[2018-08-10T15:33:10,335][WARN ][o.e.d.c.ParseField ] Deprecated field [all_fields] used, replaced by [Set [fields] to * instead]
[2018-08-10T15:33:10,369][WARN ][o.e.d.c.ParseField ] Deprecated field [inline] used, expected [source] instead
[2018-08-10T15:33:50,410][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:33:51,805][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:44:24,395][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:44:24,404][WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2018-08-10T15:44:24,411][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:44:24,415][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:44:24,425][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T15:44:24,429][WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2018-08-10T15:44:24,437][WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2018-08-10T15:44:24,441][WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2018-08-10T16:27:29,785][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T16:27:29,788][WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2018-08-10T16:27:29,790][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T16:27:29,814][WARN ][o.e.d.i.m.MapperService ] [default] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
[2018-08-10T16:31:51,833][WARN ][o.e.d.c.ParseField ] Deprecated field [all_fields] used, replaced by [Set [fields] to * instead]
[2018-08-10T16:31:51,836][WARN ][o.e.d.c.ParseField ] Deprecated field [inline] used, expected [source] instead
[2018-08-10T16:31:51,869][WARN ][o.e.d.c.ParseField ] Deprecated field [all_fields] used, replaced by [Set [fields] to * instead]
[2018-08-10T16:31:51,877][WARN ][o.e.d.c.ParseField ] Deprecated field [inline] used, expected [source] instead
[2018-08-10T17:59:43,980][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
[2018-08-10T17:59:49,246][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
This is the es startup log
+++++++++++++++++++++++++++++++++++++++++
[2018-08-10T18:14:39,857][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/opt/elk/elasticsearch-6.3.2/config/roles.yml]
[2018-08-10T18:14:40,373][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/8025] [Main.cc@109] controller (64 bit): Version 6.3.2 (Build 903094f295d249) Copyright (c) 2018 Elasticsearch BV
[2018-08-10T18:14:40,701][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-08-10T18:14:41,129][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen]
[2018-08-10T18:14:41,881][INFO ][o.e.n.Node ] [node-1] initialized
[2018-08-10T18:14:41,882][INFO ][o.e.n.Node ] [node-1] starting ...
[2018-08-10T18:14:42,030][INFO ][o.e.t.TransportService ] [node-1] publish_address {10.2.29.149:9300}, bound_addresses {0.0.0.0:9300}
[2018-08-10T18:14:42,078][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-08-10T18:14:45,189][INFO ][o.e.c.s.MasterService ] [node-1] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {node-1}{j3U2uzfJTgqYmNKeBPmo5g}{eIcKFmVFQb2_Oy4wyLLiHg}{10.2.29.149}{10.2.29.149:9300}{ml.machine_memory=16658153472, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-08-10T18:14:45,193][INFO ][o.e.c.s.ClusterApplierService] [node-1] new_master {node-1}{j3U2uzfJTgqYmNKeBPmo5g}{eIcKFmVFQb2_Oy4wyLLiHg}{10.2.29.149}{10.2.29.149:9300}{ml.machine_memory=16658153472, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {node-1}{j3U2uzfJTgqYmNKeBPmo5g}{eIcKFmVFQb2_Oy4wyLLiHg}{10.2.29.149}{10.2.29.149:9300}{ml.machine_memory=16658153472, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2018-08-10T18:14:45,212][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] publish_address {10.2.29.149:9200}, bound_addresses {0.0.0.0:9200}
[2018-08-10T18:14:45,212][INFO ][o.e.n.Node ] [node-1] started
[2018-08-10T18:14:45,314][INFO ][o.w.a.d.Monitor ] try load config from /opt/elk/elasticsearch-6.3.2/config/analysis-ik/IKAnalyzer.cfg.xml
[2018-08-10T18:14:45,967][INFO ][o.e.c.s.ClusterSettings ] [node-1] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2018-08-10T18:14:46,429][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [node-1] Failed to clear cache for realms [[]]
[2018-08-10T18:14:46,451][INFO ][o.e.l.LicenseService ] [node-1] license [21858741-fe03-4b67-a1c1-bc348c2c8e42] mode [basic] - valid
[2018-08-10T18:14:46,462][INFO ][o.e.g.GatewayService ] [node-1] recovered [37] indices into cluster_state
[2018-08-10T18:14:49,326][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-6-2018.08.10][0]] ...]).
[2018-08-10T18:14:49,394][WARN ][o.e.c.r.a.AllocationService] [node-1] [.monitoring-kibana-6-2018.08.10][0] marking unavailable shards as stale: [cNP8Nsn0S3-SgY1VJ2-YZg]
[2018-08-10T18:14:50,595][INFO ][o.e.c.s.MasterService ] [node-1] zen-disco-node-join, reason: added {{node-2}{paXrLJM9QtC1zB88IyW8Hg}{9yXpRSlkQKCR4n0fREH7wA}{10.2.29.148}{10.2.29.148:9300}{ml.machine_memory=16658153472, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}
[2018-08-10T18:14:51,729][INFO ][o.e.c.s.ClusterApplierService] [node-1] added {{node-2}{paXrLJM9QtC1zB88IyW8Hg}{9yXpRSlkQKCR4n0fREH7wA}{10.2.29.148}{10.2.29.148:9300}{ml.machine_memory=16658153472, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {node-1}{j3U2uzfJTgqYmNKeBPmo5g}{eIcKFmVFQb2_Oy4wyLLiHg}{10.2.29.149}{10.2.29.149:9300}{ml.machine_memory=16658153472, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [57] source [zen-disco-node-join]])
[2018-08-10T18:14:51,744][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node-1] updating number_of_replicas to [1] for indices [.kibana6, .monitoring-es-6-2018.08.10, .monitoring-kibana-6-2018.08.10]
[2018-08-10T18:14:51,792][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node-1] [.kibana6/5lfqoBJCTk2W--rDxZCQ2g] auto expanded replicas to [1]
[2018-08-10T18:14:51,792][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node-1] [.monitoring-es-6-2018.08.10/OT95y3CTQlmwWBENma-0OQ] auto expanded replicas to [1]
[2018-08-10T18:14:51,792][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [node-1] [.monitoring-kibana-6-2018.08.10/abVTNt84Tyq-BuS4rhs7jQ] auto expanded replicas to [1]
[2018-08-10T18:14:57,196][WARN ][o.e.c.r.a.AllocationService] [node-1] [app-m-dal-2018.08.10][1] marking unavailable shards as stale: [F0L0R9jMQm2XRKp35FuOqA]
[2018-08-10T18:14:57,261][WARN ][o.e.c.r.a.AllocationService] [node-1] [app-show-error-2018.08.10][0] marking unavailable shards as stale: [AWKPequUQQ-QfIYlfatx0A]
[2018-08-10T18:14:57,397][WARN ][o.e.c.r.a.AllocationService] [node-1] [app-b-error-2018.08.10][3] marking unavailable shards as stale: [aGVLvKjsQ4y8LeOfAjzBxQ]
[2018-08-10T18:15:00,908][WARN ][o.e.c.r.a.AllocationService] [node-1] [app-task-error-2018.08.10][2] marking unavailable shards as stale: [FMLRbLeOTWm3xWscJStolw]
[2018-08-10T18:16:31,039][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2018.08.10][0]] ...]).
Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
What do you mean by
In each index, I can only view the last 5 records, and more than 5 will be deleted.
Can you explain that with a script?
Sorry, I am sorry because my reasons have delayed your time. I am a rookie, thank you very much for taking the time to answer my questions. I am going to practice it myself and see where the problem is. Thank you again for taking the time to answer my question, thank you. Forgive me for my bad English. Thank you.If I find a problem, I will reply the reason to the bottom.
Thank you very much for your patient answers and instructions on the format of my question. I have found the cause of the problem, because I used an error operation to specify document_id in the logstash configuration, so I only have one id all the time. When I delete the id, it's automatically generated. Thank you very much. Thank you.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.