Ingest-user-agent


(David Pilato) #4

Can you share elasticsearch logs?


(naphtali) #5

The log is quite big. Any thing in specific?


(David Pilato) #6

It shouldn't be that big. Share everything that happened at the restart time please.
If too big, share on gist.github.com.


(naphtali) #7

[2018-07-05T13:39:04,906][INFO ][o.e.n.Node ] [MyNode] initializing ...
[2018-07-05T13:39:04,953][INFO ][o.e.e.NodeEnvironment ] [MyNode] using [1] data paths, mounts [[Data (D:)]], net usable_space [35.2gb], net total_space [64.8gb], types [NTFS]
[2018-07-05T13:39:04,953][INFO ][o.e.e.NodeEnvironment ] [MyNode] heap size [2.7gb], compressed ordinary object pointers [true]
[2018-07-05T13:39:09,156][INFO ][o.e.n.Node ] [MyNode] node name [MyNode], node ID [gHrJD8P0QUuUaV7NyuxyBw]
[2018-07-05T13:39:09,156][INFO ][o.e.n.Node ] [MyNode] version[6.3.0], pid[6688], build[default/zip/424e937/2018-06-11T23:38:03.357887Z], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-07-05T13:39:09,156][INFO ][o.e.n.Node ] [MyNode] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=C:\Users\046111~1\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -Xmx2816m, -Xms2816m, -Delasticsearch, -Des.path.home=D:\kits\ELK\elasticsearch-6.3.0, -Des.path.conf=D:\ELK\config, -Des.distribution.flavor=default, -Des.distribution.type=zip, exit, -Xms2816m, -Xmx2816m, -Xss1024k]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [aggs-matrix-stats]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [analysis-common]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [ingest-common]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [lang-expression]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [lang-mustache]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [lang-painless]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [mapper-extras]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [parent-join]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [percolator]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [rank-eval]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [reindex]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [repository-url]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [transport-netty4]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [tribe]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-core]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-deprecation]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-graph]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-logstash]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-ml]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-monitoring]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-rollup]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-security]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-sql]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-upgrade]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] loaded module [x-pack-watcher]
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] no plugins loaded
[2018-07-05T13:39:14,969][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/2012] [Main.cc@109] controller (64 bit): Version 6.3.0 (Build 0f0a34c67965d7) Copyright (c) 2018 Elasticsearch BV
[2018-07-05T13:39:15,188][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-07-05T13:39:16,297][INFO ][o.e.d.DiscoveryModule ] [MyNode] using discovery type [zen]
[2018-07-05T13:39:17,000][INFO ][o.e.n.Node ] [MyNode] initialized
[2018-07-05T13:39:17,000][INFO ][o.e.n.Node ] [MyNode] starting ...
[2018-07-05T13:39:17,172][INFO ][o.e.t.TransportService ] [MyNode] publish_address {1.1.1.1:9300}, bound_addresses {[fe80::a174:26c2:27db:59d1]:9300}, {1.1.1.1:9300}
[2018-07-05T13:39:17,469][INFO ][o.e.b.BootstrapChecks ] [MyNode] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-07-05T13:39:20,516][INFO ][o.e.c.s.MasterService ] [MyNode] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {MyNode}{gHrJD8P0QUuUaV7NyuxyBw}{UFtTGwU_TPG439vIB68U8A}{MyNode}{1.1.1.1:9300}{ml.machine_memory=17179262976, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-07-05T13:39:20,516][INFO ][o.e.c.s.ClusterApplierService] [MyNode] new_master {MyNode}{gHrJD8P0QUuUaV7NyuxyBw}{UFtTGwU_TPG439vIB68U8A}{MyNode}{1.1.1.1:9300}{ml.machine_memory=17179262976, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {MyNode}{gHrJD8P0QUuUaV7NyuxyBw}{UFtTGwU_TPG439vIB68U8A}{MyNode}{1.1.1.1:9300}{ml.machine_memory=17179262976, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2018-07-05T13:39:20,547][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [MyNode] publish_address {1.1.1.1:9200}, bound_addresses {[fe80::a174:26c2:27db:59d1]:9200}, {1.1.1.1:9200}
[2018-07-05T13:39:20,547][INFO ][o.e.n.Node ] [MyNode] started


(naphtali) #8

2018-07-05T13:39:25,079][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [MyNode] Failed to clear cache for realms [[]]
[2018-07-05T13:39:25,126][INFO ][o.e.l.LicenseService ] [MyNode] license [88457102-585d-468a-b7f4-aae0ee33d124] mode [basic] - valid
[2018-07-05T13:39:25,141][INFO ][o.e.g.GatewayService ] [MyNode] recovered [185] indices into cluster_state
[2018-07-05T13:39:21,969][INFO ][o.e.c.s.ClusterSettings ] [MyNode] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2018-07-05T13:39:35,907][DEBUG][o.e.a.s.TransportSearchAction] [MyNode] All shards failed for phase: [query]
[2018-07-05T13:39:35,907][WARN ][r.suppressed ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:210) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:189) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_151]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_151]


(David Pilato) #9

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.

Anyway, we can see that plugins are not loaded here. Which means not installed or not installed in the expected dir.

Could you check what you have in plugins dir?


(naphtali) #10

This is what I have in the plugin dir
image

And as I mentioned earlier, running, elasticsearch-plugin list, returns the plugins.
image


(Christian Dahlqvist) #11

Have you restarted the node following the installation?


(naphtali) #12

Yes, I restarted the elasticsearch server after installing the plugins


(David Pilato) #13
[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] no plugins loaded

None of the plugins is loaded. Wondering if you a right access issue on this dir.
Or if the plugins dir is expected to be actually somewhere else...


(naphtali) #14

Hi, Thanks for the reply.
I am an admin on the server, so I have right access to the dir, and the plugins are installed to the es-home dir.
In addition, running elasticsearch-plugin list does return them, so elasticsearch does recognize them.
What could prevent the plugins from loading?


(David Pilato) #15

I'm sure elasticsearch is looking inside another dir. I remember some past issues where people were setting path.plugins but this property was not read by the plugin manager.

So plugin manager thought it installed the plugin but elasticsearch did not see them.

I'm almost sure that's the case here.

May be change the plugin manager class to debug or check elasticsearch settings.

ÉDIT: just realized you are using 6.3.0. So it should work. Debug mode would help then.


(naphtali) #16

How do I start debug mode?


(David Pilato) #17

https://www.elastic.co/guide/en/elasticsearch/plugins/current/_other_command_line_parameters.html


(naphtali) #18

It looks like it ran without problems


(David Pilato) #19

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

Please share all the output


(naphtali) #20
-> Downloading file:////D:/kits/ELK/ingest-geoip-6.3.0.zip
Retrieving zip from file:////D:/kits/ELK/ingest-geoip-6.3.0.zip
[=================================================] 100%  
- Plugin information:
Name: ingest-geoip
Description: Ingest processor that uses looksup geo data based on ip adresses using the Maxmind geo database
Version: 6.3.0
Elasticsearch Version: 6.3.0
Java Version: 1.8
Native Controller: false
Extended Plugins: []
 * Classname: org.elasticsearch.ingest.geoip.IngestGeoIpPlugin
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip

(David Pilato) #21

Can you share your elasticsearch.yml file please?


(naphtali) #22
bootstrap.memory_lock: false
cluster.name: dev-log01
http.port: 9200
network.host: devlog01
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
node.name: devlog01
path.data: D:\ELK\data
path.logs: D:\ELK\logs
path.repo: D:\ELK\backup
transport.tcp.port: 9300

(David Pilato) #23

@Martijn_Laarman have you heard about such an issue on Windows?