Ingest-user-agent

naphtali · July 5, 2018, 8:13am

Hi,
I have the elk pack version 6.3.0. One node, on windows.
I'm trying to use file beat with the iis module. For that I needed to install the ingest-user-agent and the ingest-geoip plugins.
I run the install of the plugins, and when I run "elasticsearch-plugin list", they are both listed.
But when I run the elasticsearch api "_nodes/plugins" i get "plugins": []
Also, when I run "GET _nodes/ingest", the user agent and geoip are not listed.
I should mention that my server is dissconected from the internet. I Installed after downloading the plugin.
Do I need to install anything else?
Naphtali

dadoonet · July 5, 2018, 8:40am

May be you did not restart your nodes?

naphtali · July 5, 2018, 8:43am

I restarted the service after installing and it didn't help

dadoonet · July 5, 2018, 8:56am

Can you share elasticsearch logs?

naphtali · July 5, 2018, 9:32am

The log is quite big. Any thing in specific?

dadoonet · July 5, 2018, 10:09am

It shouldn't be that big. Share everything that happened at the restart time please.
If too big, share on gist.github.com.

naphtali · July 5, 2018, 10:40am

[2018-07-05T13:39:04,906][INFO ][o.e.n.Node [2018-07-05T13:39:04,953][INFO ][o.e.e.NodeEnvironment [2018-07-05T13:39:04,953][INFO ][o.e.e.NodeEnvironment [2018-07-05T13:39:09,156][INFO ][o.e.n.Node [2018-07-05T13:39:09,156][INFO ][o.e.n.Node [2018-07-05T13:39:09,156][INFO ][o.e.n.Node [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService [2018-07-05T13:39:14,969][INFO ][o.e.x.m.j.p.l.CppLog [2018-07-05T13:39:15,188][DEBUG][o.e.a.ActionModule [2018-07-05T13:39:16,297][INFO ][o.e.d.DiscoveryModule [2018-07-05T13:39:17,000][INFO ][o.e.n.Node [2018-07-05T13:39:17,000][INFO ][o.e.n.Node [2018-07-05T13:39:17,172][INFO ][o.e.t.TransportService [2018-07-05T13:39:17,469][INFO ][o.e.b.BootstrapChecks [2018-07-05T13:39:20,516][INFO ][o.e.c.s.MasterService [2018-07-05T13:39:20,516][INFO ][o.e.c.s.ClusterApplierService] [2018-07-05T13:39:20,547][INFO ][o.e.x.s.t.n.Security [2018-07-05T13:39:20,547][INFO ][o.e.n.Node ] [MyNode] initializing ...
] [MyNode] using [1] data paths, mounts [[Data (D:)]], net usable_space [35.2gb], net total_space [64.8gb], types [NTFS]
] [MyNode] heap size [2.7gb], compressed ordinary object pointers [true]
] [MyNode] node name [MyNode], node ID [gHrJD8P0QUuUaV7NyuxyBw]
] [MyNode] version[6.3.0], pid[6688], build[default/zip/424e937/2018-06-11T23:38:03.357887Z], OS[Windows Server 2016/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
] [MyNode] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=C:\Users\046111~1\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -Xmx2816m, -Xms2816m, -Delasticsearch, -Des.path.home=D:\kits\ELK\elasticsearch-6.3.0, -Des.path.conf=D:\ELK\config, -Des.distribution.flavor=default, -Des.distribution.type=zip, exit, -Xms2816m, -Xmx2816m, -Xss1024k]
] [MyNode] loaded module [aggs-matrix-stats]
] [MyNode] loaded module [analysis-common]
] [MyNode] loaded module [ingest-common]
] [MyNode] loaded module [lang-expression]
] [MyNode] loaded module [lang-mustache]
] [MyNode] loaded module [lang-painless]
] [MyNode] loaded module [mapper-extras]
] [MyNode] loaded module [parent-join]
] [MyNode] loaded module [percolator]
] [MyNode] loaded module [rank-eval]
] [MyNode] loaded module [reindex]
] [MyNode] loaded module [repository-url]
] [MyNode] loaded module [transport-netty4]
] [MyNode] loaded module [tribe]
] [MyNode] loaded module [x-pack-core]
] [MyNode] loaded module [x-pack-deprecation]
] [MyNode] loaded module [x-pack-graph]
] [MyNode] loaded module [x-pack-logstash]
] [MyNode] loaded module [x-pack-ml]
] [MyNode] loaded module [x-pack-monitoring]
] [MyNode] loaded module [x-pack-rollup]
] [MyNode] loaded module [x-pack-security]
] [MyNode] loaded module [x-pack-sql]
] [MyNode] loaded module [x-pack-upgrade]
] [MyNode] loaded module [x-pack-watcher]
] [MyNode] no plugins loaded
MessageHandler] [controller/2012] [Main.cc@109] controller (64 bit): Version 6.3.0 (Build 0f0a34c67965d7) Copyright (c) 2018 Elasticsearch BV
] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
] [MyNode] using discovery type [zen]
] [MyNode] initialized
] [MyNode] starting ...
] [MyNode] publish_address {1.1.1.1:9300}, bound_addresses {[fe80::a174:26c2:27db:59d1]:9300}, {1.1.1.1:9300}
] [MyNode] bound or publishing to a non-loopback address, enforcing bootstrap checks
] [MyNode] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {MyNode}{gHrJD8P0QUuUaV7NyuxyBw}{UFtTGwU_TPG439vIB68U8A}{MyNode}{1.1.1.1:9300}{ml.machine_memory=17179262976, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[MyNode] new_master {MyNode}{gHrJD8P0QUuUaV7NyuxyBw}{UFtTGwU_TPG439vIB68U8A}{MyNode}{1.1.1.1:9300}{ml.machine_memory=17179262976, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {MyNode}{gHrJD8P0QUuUaV7NyuxyBw}{UFtTGwU_TPG439vIB68U8A}{MyNode}{1.1.1.1:9300}{ml.machine_memory=17179262976, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
Netty4HttpServerTransport] [MyNode] publish_address {1.1.1.1:9200}, bound_addresses {[fe80::a174:26c2:27db:59d1]:9200}, {1.1.1.1:9200}
] [MyNode] started

naphtali · July 5, 2018, 10:43am

2018-07-05T13:39:25,079][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [MyNode] Failed to clear cache for realms [[]]
[2018-07-05T13:39:25,126][INFO ][o.e.l.LicenseService ] [MyNode] license [88457102-585d-468a-b7f4-aae0ee33d124] mode [basic] - valid
[2018-07-05T13:39:25,141][INFO ][o.e.g.GatewayService ] [MyNode] recovered [185] indices into cluster_state
[2018-07-05T13:39:21,969][INFO ][o.e.c.s.ClusterSettings ] [MyNode] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2018-07-05T13:39:35,907][DEBUG][o.e.a.s.TransportSearchAction] [MyNode] All shards failed for phase: [query]
[2018-07-05T13:39:35,907][WARN ][r.suppressed ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:210) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:189) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_151]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_151]

dadoonet · July 5, 2018, 12:29pm

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.

Anyway, we can see that plugins are not loaded here. Which means not installed or not installed in the expected dir.

Could you check what you have in plugins dir?

naphtali · July 5, 2018, 12:41pm

This is what I have in the plugin dir

And as I mentioned earlier, running, elasticsearch-plugin list, returns the plugins.

Christian_Dahlqvist · July 5, 2018, 12:43pm

Have you restarted the node following the installation?

naphtali · July 5, 2018, 12:58pm

Yes, I restarted the elasticsearch server after installing the plugins

dadoonet · July 5, 2018, 7:38pm

[2018-07-05T13:39:11,219][INFO ][o.e.p.PluginsService ] [MyNode] no plugins loaded

None of the plugins is loaded. Wondering if you a right access issue on this dir.
Or if the plugins dir is expected to be actually somewhere else...

naphtali · July 8, 2018, 8:39am

Hi, Thanks for the reply.
I am an admin on the server, so I have right access to the dir, and the plugins are installed to the es-home dir.
In addition, running elasticsearch-plugin list does return them, so elasticsearch does recognize them.
What could prevent the plugins from loading?

dadoonet · July 8, 2018, 9:51am

I'm sure elasticsearch is looking inside another dir. I remember some past issues where people were setting path.plugins but this property was not read by the plugin manager.

So plugin manager thought it installed the plugin but elasticsearch did not see them.

I'm almost sure that's the case here.

May be change the plugin manager class to debug or check elasticsearch settings.

ÉDIT: just realized you are using 6.3.0. So it should work. Debug mode would help then.

naphtali · July 8, 2018, 10:17am

How do I start debug mode?

dadoonet · July 8, 2018, 11:01am

https://www.elastic.co/guide/en/elasticsearch/plugins/current/_other_command_line_parameters.html

naphtali · July 8, 2018, 11:21am

It looks like it ran without problems

dadoonet · July 8, 2018, 11:34am

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

Please share all the output

naphtali · July 8, 2018, 11:35am

-> Downloading file:////D:/kits/ELK/ingest-geoip-6.3.0.zip
Retrieving zip from file:////D:/kits/ELK/ingest-geoip-6.3.0.zip
[=================================================] 100%  
- Plugin information:
Name: ingest-geoip
Description: Ingest processor that uses looksup geo data based on ip adresses using the Maxmind geo database
Version: 6.3.0
Elasticsearch Version: 6.3.0
Java Version: 1.8
Native Controller: false
Extended Plugins: []
 * Classname: org.elasticsearch.ingest.geoip.IngestGeoIpPlugin
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip

Topic		Replies	Views
How to install/enable ingest-geoip and ingest-user-agent in ELK STACK 7.1 Elasticsearch	4	5094	July 16, 2019
Elasticsearch 6.3 ingest-user-agent Elasticsearch	3	339	August 7, 2018
Filebeat (v6.2.3) complaining about ES plugins (v6.2.3) missing Beats filebeat	6	2727	April 20, 2018
Install ingest-user-agent on ELK Logstash	3	548	November 26, 2018
Are the following plugins (attachment, geoip, user-agent) for ingest node already been shipped with ES 5.4.0? Elasticsearch	6	549	June 29, 2017

Ingest-user-agent

Related topics