Elasticsearch with x-pack installation

Elastic Stack Elasticsearch
1

[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [aggs-matrix-stats]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [ingest-common]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [lang-expression]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [lang-groovy]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [lang-mustache]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [lang-painless]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [parent-join]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [percolator]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [reindex]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [transport-netty3]
[2018-04-06T06:15:12,473][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded module [transport-netty4]
[2018-04-06T06:15:12,474][INFO ][o.e.p.PluginsService ] [1e85IX5] loaded plugin [x-pack]
[2018-04-06T06:15:46,295][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/59] [Main.cc@128] controller (64 bit): Version 5.6.8 (Build f703b1d52613ea) Copyright (c) 2018 Elasticsearch BV
[2018-04-06T06:15:47,189][INFO ][o.e.d.DiscoveryModule ] [1e85IX5] using discovery type [zen]
[2018-04-06T06:16:00,250][INFO ][o.e.n.Node ] initialized
[2018-04-06T06:16:00,281][INFO ][o.e.n.Node ] [1e85IX5] starting ...
[2018-04-06T06:16:02,939][INFO ][o.e.t.TransportService ] [1e85IX5] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-04-06T06:16:03,100][WARN ][o.e.b.BootstrapChecks ] [1e85IX5] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-04-06T06:16:04,850][INFO ][o.e.m.j.JvmGcMonitorService] [1e85IX5] [gc][3] overhead, spent [620ms] collecting in the last [1.6s]
[2018-04-06T06:16:06,779][INFO ][o.e.c.s.ClusterService ] [1e85IX5] new_master {1e85IX5}{1e85IX5KTQOYaUUZ228FPg}{ahfpP3NJSGOIqCDBkLrE6A}{127.0.0.1}{127.0.0.1:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-04-06T06:16:07,126][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [1e85IX5] publish_address {172.17.0.3:9200}, bound_addresses {[::]:9200}
[2018-04-06T06:16:07,127][INFO ][o.e.n.Node ] [1e85IX5] started
[2018-04-06T06:16:10,721][ERROR][o.e.x.m.c.i.IndexStatsCollector] [1e85IX5] collector [index-stats] failed to collect data
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];

[2018-04-06T06:16:14,575][INFO ][o.e.g.GatewayService ] [1e85IX5] recovered [3] indices into cluster_state
[2018-04-06T06:16:14,870][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [1e85IX5] successfully created .ml-state index template
[2018-04-06T06:16:15,179][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [1e85IX5] successfully created .ml-meta index template
[2018-04-06T06:16:15,529][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [1e85IX5] successfully created .ml-notifications index template
[2018-04-06T06:16:16,875][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [1e85IX5] successfully created .ml-anomalies- index template
[2018-04-06T06:16:25,649][INFO ][o.e.c.r.a.AllocationService] [1e85IX5] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0], [apt-2018.04.06][3]] ...]).
[2018-04-06T06:16:26,026][INFO ][o.e.l.LicenseService ] [1e85IX5] license [2daa5bb9-fd0d-49b0-871a-b3838a458443] mode [trial] - valid
[2018-04-06T06:16:26,657][INFO ][o.e.m.j.JvmGcMonitorService] [1e85IX5] [gc][24] overhead, spent [353ms] collecting in the last [1.3s]
[2018-04-06T06:16:27,249][INFO ][o.e.c.m.MetaDataMappingService] [1e85IX5] [.kibana/elWkJqqaSBO4eUYf5xnvGg] create_mapping [graph-workspace]
[2018-04-06T06:16:32,923][INFO ][o.e.c.m.MetaDataCreateIndexService] [1e85IX5] [.monitoring-es-6-2018.04.06] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
[2018-04-06T06:16:33,871][INFO ][o.e.c.m.MetaDataCreateIndexService] [1e85IX5] [.watches] creating index, cause [auto(bulk api)], templates [watches], shards [1]/[1], mappings [watch]
[2018-04-06T06:16:35,554][INFO ][o.e.c.m.MetaDataMappingService] [1e85IX5] [.watches/V8v3AnsHTlewKN1M1OnKTw] update_mapping [watch]
[2018-04-06T06:16:51,199][INFO ][o.e.m.j.JvmGcMonitorService] [1e85IX5] [gc][48] overhead, spent [637ms] collecting in the last [1.4s]
[2018-04-06T06:16:53,801][INFO ][o.e.c.m.MetaDataCreateIndexService] [1e85IX5] [.monitoring-kibana-6-2018.04.06] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[1], mappings [doc]
[2018-04-06T06:17:39,991][INFO ][o.e.c.m.MetaDataCreateIndexService] [1e85IX5] [.monitoring-alerts-6] creating index, cause [auto(bulk api)], templates [.monitoring-alerts], shards [1]/[1], mappings [doc]
[2018-04-06T06:17:40,138][INFO ][o.e.c.m.MetaDataCreateIndexService] [1e85IX5] [.watcher-history-6-2018.04.06] creating index, cause [auto(bulk api)], templates [.watch-history-6], shards [1]/[1], mappings [doc]
[2018-04-06T06:17:40,291][INFO ][o.e.c.m.MetaDataMappingService] [1e85IX5] [.watcher-history-6-2018.04.06/rH-pC1axSjOk5KMvAf7xAg] update_mapping [doc]
[2018-04-06T06:17:40,410][INFO ][o.e.c.m.MetaDataMappingService] [1e85IX5] [.watcher-history-6-2018.04.06/rH-pC1axSjOk5KMvAf7xAg] update_mapping [doc]
[2018-04-06T06:19:44,063][WARN ][o.e.d.r.RestController ] Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.
[2018-04-06T06:19:45,384][WARN ][o.e.d.r.RestController ] Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.
[2018-04-06T06:19:50,517][WARN ][o.e.d.r.RestController ] Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.
[2018-04-06T06:19:51,744][WARN ][o.e.d.r.RestController ] Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header.
[2018-04-06T06:22:43,241][INFO ][o.e.m.j.JvmGcMonitorService] [1e85IX5] [gc][399] overhead, spent [365ms] collecting in the last [1.3s]

When I try to install x-pack on elasticserach it is throws this error I am not able to figure out what is wrong the main problem is my config file is not able to pull the logs because it says some connection refuse problem and status shown is 401. Please do let me know where M I going wrong

2

You have to set vm.max_map_count to 262144. What OS are you running this on ?

3

The ELK setup is hosted on docker

4

apart from the memory thing are there any changes required in the elasticsearch.yml file if we use x-pack feature

5

I don't think you will need any change in config to install x-pack.
How much memory is your docker using right now ?

6

around 120 gb
My problem is that logstash is not able to pull logs after a certain point when I install x-pack on the ELK stack can u help me out as I am not able to figure out where I am going wrong

7

Looking at your logs, the issue that I noticed is with max_map_count.
Why don't you try fixing that and see what logs are you saying.
Try sharing the complete log file.

10

how do I change the virtual memory

11

If you are using mac. Try this https://stackoverflow.com/questions/41192680/update-max-map-count-for-elasticsearch-docker-container-mac-host

12

not using MAC

13

I suspect logstash fails because it couldn't connect to elastic search.

14

That is happening only when i install x-pack on the elk stack

15

What does the content type refer to

16

If you are making any GET requests, then it is expecting header like 'Content-Type: application/json'

did your x-pack installation succeed ?

17

yes it did

18

Capture12

It did all this has come on the kibana window

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.