Getting metricbeat error on 7.14 v

cool999 · August 23, 2021, 9:11pm

Hi,

I have ansible config files which was installing all elasticsearch components of 7.4 version. I have changed it to download and use 7.14 version.

I am getting below error, elasticsearch service is not getting started.

Unable to start service elasticsearch: Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

elasticsearch service output,

Aug 23 20:28:21  systemd-entrypoint[22215]: Error: A fatal exception has occurred. Program will exit.
Aug 23 20:28:21  systemd-entrypoint[22215]: at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOpt...:119)
Aug 23 20:28:21  systemd-entrypoint[22215]: at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(...a:81)
Aug 23 20:28:21  systemd-entrypoint[22215]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErg...a:38)
Aug 23 20:28:21  systemd-entrypoint[22215]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions...:135)
Aug 23 20:28:21  systemd-entrypoint[22215]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOp...a:86)
Aug 23 20:28:21  systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Aug 23 20:28:21  systemd[1]: Failed to start Elasticsearch.
Aug 23 20:28:21  systemd[1]: Unit elasticsearch.service entered failed state.
Aug 23 20:28:21  systemd[1]: elasticsearch.service failed.

elasticsearch logs -

[2021-08-23T12:21:02,356][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [ping_server-2021.08.23/adVQSjVASH6NqwfTsaKMfA] create_mapping [_doc]
[2021-08-23T12:21:09,275][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [access_ping_server-2021.08.23/_rspEZtyRPqGCX4rZ4d40A] update_mapping [_doc]
[2021-08-23T12:21:09,471][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [filebeat-7.4.0-2021.08.23/I0L3_I7ZRLuEHCSrgT84ew] update_mapping [_doc]
[2021-08-23T12:21:20,043][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [ping_server-2021.08.23/adVQSjVASH6NqwfTsaKMfA] update_mapping [_doc]
[2021-08-23T12:21:20,354][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [access_ping_server-2021.08.23/_rspEZtyRPqGCX4rZ4d40A] update_mapping [_doc]
[2021-08-23T12:21:20,557][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [filebeat-7.4.0-2021.08.23/I0L3_I7ZRLuEHCSrgT84ew] update_mapping [_doc]
[2021-08-23T12:21:40,373][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [nextgenpsd2bank-api_app_server-2021.08.23/yEa9I-8zTHesLcp6b_fCVw] update_mapping [_doc]
[2021-08-23T12:21:40,718][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [filebeat-7.4.0-2021.08.23/I0L3_I7ZRLuEHCSrgT84ew] update_mapping [_doc]
[2021-08-23T12:21:54,479][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elasticsearch_1] adding template [apm-7.4.0] for index patterns [apm-7.4.0*]
[2021-08-23T12:21:54,551][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elasticsearch_1] adding index lifecycle policy [apm-7.4.0-span]
[2021-08-23T12:21:54,670][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elasticsearch_1] adding template [apm-7.4.0-span] for index patterns [apm-7.4.0-span*]
[2021-08-23T12:21:54,752][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticsearch_1] [apm-7.4.0-span-000001] creating index, cause [api], templates [apm-7.4.0-span, apm-7.4.0], shards [1]/[1], mappings [_doc]
[2021-08-23T12:21:54,951][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elasticsearch_1] adding index lifecycle policy [apm-7.4.0-transaction]
[2021-08-23T12:21:55,015][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elasticsearch_1] adding template [apm-7.4.0-transaction] for index patterns [apm-7.4.0-transaction*]
[2021-08-23T12:21:55,093][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticsearch_1] [apm-7.4.0-transaction-000001] creating index, cause [api], templates [apm-7.4.0-transaction, apm-7.4.0], shards [1]/[1], mappings [_doc]
[2021-08-23T12:21:55,266][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elasticsearch_1] adding index lifecycle policy [apm-7.4.0-error]
[2021-08-23T12:21:55,347][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elasticsearch_1] adding template [apm-7.4.0-error] for index patterns [apm-7.4.0-error*]
[2021-08-23T12:21:55,387][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticsearch_1] [apm-7.4.0-error-000001] creating index, cause [api], templates [apm-7.4.0-error, apm-7.4.0], shards [1]/[1], mappings [_doc]
[2021-08-23T12:21:55,697][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elasticsearch_1] adding index lifecycle policy [apm-7.4.0-metric]
[2021-08-23T12:21:55,785][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elasticsearch_1] adding template [apm-7.4.0-metric] for index patterns [apm-7.4.0-metric*]
[2021-08-23T12:21:55,900][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticsearch_1] [apm-7.4.0-metric-000001] creating index, cause [api], templates [apm-7.4.0-metric, apm-7.4.0], shards [1]/[1], mappings [_doc]
[2021-08-23T12:21:56,485][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticsearch_1] [apm-7.4.0-onboarding-2021.08.23] creating index, cause [auto(bulk api)], templates [apm-7.4.0], shards [1]/[1], mappings [_doc]
[2021-08-23T12:22:06,833][INFO ][o.e.m.j.JvmGcMonitorService] [elasticsearch_1] [gc][458] overhead, spent [412ms] collecting in the last [1s]
[2021-08-23T12:22:33,683][INFO ][o.e.c.m.MetaDataCreateIndexService] [elasticsearch_1] [metricbeat-7.4.0-2021.08.23] creating index, cause [auto(bulk api)], templates [metricbeat-7.4.0], shards [1]/[1], mappings [_doc]
[2021-08-23T12:22:34,006][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [metricbeat-7.4.0-2021.08.23/hSXx1WScRJCFnn9gpQQ5Dw] update_mapping [_doc]
[2021-08-23T12:23:16,685][INFO ][o.e.c.m.MetaDataMappingService] [elasticsearch_1] [access_ping_server-2021.08.23/_rspEZtyRPqGCX4rZ4d40A] update_mapping [_doc]
[2021-08-23T12:23:43,682][INFO ][o.e.x.s.a.f.FileUserPasswdStore] [elasticsearch_1] users file [/etc/elasticsearch/users] changed. updating users... )
[2021-08-23T12:23:43,714][INFO ][o.e.x.s.a.f.FileUserRolesStore] [elasticsearch_1] users roles file [/etc/elasticsearch/users_roles] changed. updating users roles...
[2021-08-23T12:23:43,715][INFO ][o.e.x.s.a.s.FileRolesStore] [elasticsearch_1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-08-23T12:23:43,716][INFO ][o.e.x.s.a.s.FileRolesStore] [elasticsearch_1] updated roles (roles file [/etc/elasticsearch/roles.yml] changed)
[2021-08-23T12:34:28,534][ERROR][o.e.x.i.IndexLifecycleRunner] [elasticsearch_1] policy [heartbeat-7.4.0] for index [heartbeat-7.4.0-2021.08.23] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [heartbeat-7.4.0] does not point to index [heartbeat-7.4.0-2021.08.23]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92) [x-pack-core-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133) [x-pack-ilm-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:274) [x-pack-ilm-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:213) [x-pack-ilm-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:175) [x-pack-core-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:203) [x-pack-core-7.4.0.jar:7.4.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-08-23T12:34:28,652][ERROR][o.e.x.i.IndexLifecycleRunner] [elasticsearch_1] policy [metricbeat-7.4.0] for index [metricbeat-7.4.0-2021.08.23] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [metricbeat-7.4.0] does not point to index [metricbeat-7.4.0-2021.08.23]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92) [x-pack-core-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133) [x-pack-ilm-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:274) [x-pack-ilm-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:213) [x-pack-ilm-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:175) [x-pack-core-7.4.0.jar:7.4.0]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:203) [x-pack-core-7.4.0.jar:7.4.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:830) [?:?]
[2021-08-23T12:51:41,712][INFO ][o.e.n.Node               ] [elasticsearch_1] stopping ...
[2021-08-23T12:51:41,740][INFO ][o.e.x.w.WatcherService   ] [elasticsearch_1] stopping watch service, reason [shutdown initiated]
[2021-08-23T12:51:41,741][INFO ][o.e.x.w.WatcherLifeCycleService] [elasticsearch_1] watcher has stopped and shutdown
[2021-08-23T12:51:41,864][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elasticsearch_1] [controller/7651] [Main.cc@150] Ml controller exiting
[2021-08-23T12:51:41,868][INFO ][o.e.x.m.p.NativeController] [elasticsearch_1] Native controller process has stopped - no new native processes can be started
[2021-08-23T12:51:44,621][INFO ][o.e.n.Node               ] [elasticsearch_1] stopped
[2021-08-23T12:51:44,621][INFO ][o.e.n.Node               ] [elasticsearch_1] closing ...
[2021-08-23T12:51:44,652][INFO ][o.e.n.Node               ] [elasticsearch_1] closed

As can see above custom index are getting created along with filebeat, metricbeat etc.. index but they are getting created with version 7.4.0 and not 7.14.0 version.

Not sure from where it is picking up version as 7.4.0 and not 7.14.0

Below is the logstash file,

input {
  beats {
    port => 5044
  }
}

if [log_type] == "admin_app_server" and [app_id] == "node"
  {
    grok { match => { "message" => "%{SYSLOGBASE} %{GREEDYDATA:json_message}"  } } json { source =>  "json_message" }
    mutate {
             replace => {
               "[type]" => "admin_app_server"
             }
           }
  }

output {
 if [log_type] == "admin_app_server" {
  elasticsearch {
    hosts => ['http://10.10.10.242:9200']
    index => "%{type}-%{+YYYY.MM.dd}"
        user => elastic
    password => XXX
      }
 }

  elasticsearch {
    hosts => ['http://10.10.10.242:9200']
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => elastic
    password => XXX
  }
}

I think its because of the above line in elasticsearch output i.e index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" which is causing issue.

Can someone please tell from where version is picking value as 7.4.0 and not 7.14.0

Thanks,

warkolm · August 23, 2021, 10:23pm

That won't stop Elasticsearch from starting/running. As your log shows, Elasticsearch is definitely starting up fine.

Correct.
7.14 defaults to using ILM, which means you would need to write to the ILM policy alias.

TLDR you may want to disable ILM in Metricbeat, get things working and then look at enabling ILM.

cool999 · August 24, 2021, 11:23am

Hi @Warkolm,

elasticsearch was in failed state as you can see in first output above i.e in elasticsearch service output above.

I had to cleanup everything as there were some indices got created with v 7.4.0 for metricbet, heartbeat etc.. as I was not able to remove those indices as elasticsearch service was not started properly.

I ran the installation again by disabling ilm for heartbeat and metricbeat.

setup.ilm.enabled: false

The installation is completed. I can login to kibana and see our application logs using filebeat.

I still see these beats indices are created and have version of 7.4.0 and not 7.14.0.

yellow open   apm-7.4.0-metric-000001                    kXiCgqnrS4WC2LMCIBRjKQ   1   1          0            0       283b           283b
yellow open   metricbeat-7.4.0-2021.08.23-000001         fdCu3hGSQnqnvqgFEdjSrg   1   1          0            0       283b           283b
yellow open   apm-7.4.0-transaction-000001               qI6xo3OGQWK9JGOQadHS_w   1   1          0            0       283b           283b
yellow open   metricbeat-7.4.0-2021.08.23                hSXx1WScRJCFnn9gpQQ5Dw   1   1       4829            0      3.6mb          3.6mb
yellow open   apm-7.4.0-onboarding-2021.08.23            6aqB4RRbT2aMEk1UhHp1SA   1   1          1            0      6.4kb          6.4kb
yellow open   apm-7.4.0-span-000001                      dtGpgmUOQqeaduxvrENj-Q   1   1          0            0       283b           283b
yellow open   heartbeat-7.4.0-2021.08.24                 Fie6wGTyTrCyDs2i51QHiA   1   1        320            0    395.2kb        395.2kb
yellow open   heartbeat-7.4.0-2021.08.23                 y1v0zYXtRbeu3QCLM8NGVg   1   1       6006            0        2mb            2mb
yellow open   filebeat-7.4.0-2021.08.23                  I0L3_I7ZRLuEHCSrgT84ew   1   1    2947735            0      1.4gb          1.4gb
yellow open   apm-7.4.0-error-000001                     1BhQhc4pSk2a5sR6pnQqGQ   1   1          0            0       283b           283b

Can you let me know from where in below line it's taking value as 7.4.0 and not 7.14.0?

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

Also how can fix the above said error by keeping ilm enabled (which is default) and pointing it correctly.

[2021-08-23T12:34:28,652][ERROR][o.e.x.i.IndexLifecycleRunner] [elasticsearch_1] policy [metricbeat-7.4.0] for index [metricbeat-7.4.0-2021.08.23] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [metricbeat-7.4.0] does not point to index [metricbeat-7.4.0-2021.08.23]

Thanks,

cool999 · August 24, 2021, 12:12pm

Hi @stephenb,

Sorry to include you directly.

Below is the open case having similar error.

only difference is that was completely on v7.4.0 but here I am installing all components with v7.14.0 and changing any ref of 7.4 to 7.14 in our automation tool.

Output of GET metricbeat-7.4.0/_alias as requested there.

#! this request accesses aliases with names reserved for system indices: [.kibana_task_manager, .security, .kibana_task_manager_7.14.0, .kibana, .kibana_7.14.0], but in a future major version, direct access to system indices and their aliases will not be allowed
{
  "metricbeat-7.4.0-2021.08.23-000001" : {
    "aliases" : {
      "metricbeat-7.4.0" : {
        "is_write_index" : true
      }
    }
  }
}

Its showing index with 7.4 even if 7.14 v is installed.

[root@ip-10-10-10-242 ~]# rpm -qa |grep metricbeat
metricbeat-7.14.0-1.x86_64

[root@ip-10-0-1-142 ~]# rpm -qa |grep heartbeat
heartbeat-elastic-7.14.0-1.x86_64
[root@ip-10-10-10-242 ~]#

Thanks,

stephenb · August 24, 2021, 4:50pm

What is the output of

metricbeat version

If you are still getting 7.4 is there another host with metricbeat running?

If you are using 7.14 and want ILM etc etc. you should run

metricbeat setup -e

first while metricbeat is pointing to elasticsearch this will set up all the assets ILM etc...

Then change metricbeat.yml to point to logstash and use the pipeline below.

(do the same for filebeat)

You can look at this thread where I explain this

Your logstash should be of the form (you can add your extra filtering in)

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

system · September 21, 2021, 6:51pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed to start Elasticsearch 7.11.2 Elasticsearch	36	1288	July 6, 2023
Unable to start FileBeat "sends log files to Logstash or directly to Elasticsearch" Beats	1	2723	March 20, 2020
Failed to start Elasticsearch.7.15.2 Elasticsearch	7	2548	December 15, 2021
Systemctl failing to start elasticsearch Elasticsearch	5	5224	August 2, 2021
Failed to start Elasticsearch Elasticsearch	2	283	February 12, 2023

Getting metricbeat error on 7.14 v

Related topics