Failed to create elasticsearch client


(David) #1

Hi, I am trying to learn about Elastic and fscrawler so I can catalogue and search my pdf’s on a win7 machine. I am going up a very steep learning curve haha

I have installed elastic version 6.4.2 using the suggested installer and its working “as a service” and accessing via http://127.0.0.1:9202, I get for name:”MESH-PC” the following

|cluster_name|"elasticsearch"|
|---|---|
|cluster_uuid|"GhGmJvzKTwK_uktjKa3vjQ"|
|version||
|number|"6.4.2"|
|build_flavor|"unknown"|
|build_type|"unknown"|
|build_hash|"04711c2"|
|build_date|"2018-09-26T13:34:09.098244Z"|
|build_snapshot|false|
|lucene_version|"7.4.0"|
|minimum_wire_compatibility_version|"5.6.0"|
|minimum_index_compatibility_version|"5.0.0"|
|tagline|"You Know, for Search"|

I am using port 9202 as 9200 is being used (information from ipconfig)

I have had numerous problems:

  1. I set-up c:\tmp\es manually but changing .json gives an invalid code message when I replace “/tmp/es” with “c:\tmp\es”

  2. The fscrawler message said that I would find the .json file in c:/users/david/.fscrawler/{job_name}/_settings.json but it is the a folder called “c:/vtroot”

I am also unable to create elasticsearch client, see trace below

C:\Program Files\fscrawler-2.5\bin>fscrawler --trace mesh-pc
21:06:23,662 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Mapping [2/_settings.json] already exists
21:06:23,667 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Mapping [2/_settings_folder.json] already exists
21:06:23,668 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Mapping [5/_settings.json] already exists
21:06:23,669 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Mapping [5/_settings_folder.json] already exists
21:06:23,671 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Mapping [6/_settings.json] already exists
21:06:23,672 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Mapping [6/_settings_folder.json] already exists
21:06:23,674 DEBUG [f.p.e.c.f.c.FsCrawler] Starting job [mesh-pc]...
21:06:24,171 TRACE [f.p.e.c.f.c.FsCrawler] settings used for this crawler: [{
  "name" : "MESH-PC",
  "fs" : {
    "url" : "/tmp/es",
    "update_rate" : "15m",
    "excludes" : [ "*/~*" ],
    "json_support" : false,
    "filename_as_id" : false,
    "add_filesize" : true,
    "remove_deleted" : true,
    "add_as_inner_object" : false,
    "store_source" : false,
    "index_content" : true,
    "attributes_support" : false,
    "raw_metadata" : true,
    "xml_support" : false,
    "index_folders" : true,
    "lang_detect" : false,
    "continue_on_error" : false,
    "pdf_ocr" : true,
    "ocr" : {
      "language" : "eng"
    }
  },
  "elasticsearch" : {
    "nodes" : [ {
      "host" : "127.0.0.1",
      "port" : 9202,
      "scheme" : "HTTP"
    } ],
    "bulk_size" : 100,
    "flush_interval" : "5s",
    "byte_size" : "10mb"
  },
  "rest" : {
    "scheme" : "HTTP",
    "host" : "127.0.0.1",
    "port" : 8080,
    "endpoint" : "fscrawler"
  }
}]
21:06:25,593 WARN  [f.p.e.c.f.c.ElasticsearchClientManager] failed to create elasticsearch client, disabling crawler...
21:06:25,593 FATAL [f.p.e.c.f.c.FsCrawler] Fatal error received while running the crawler: [Permission denied: no further information]
21:06:25,594 DEBUG [f.p.e.c.f.c.FsCrawler] error caught
java.io.IOException: Permission denied: no further information
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:728) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:283) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.setElasticsearchBehavior(ElasticsearchClient.java:291) ~[fscrawler-elasticsearch-client-2.5.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClientManager.start(ElasticsearchClientManager.java:90) ~[fscrawler-elasticsearch-client-2.5.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawler.main(FsCrawler.java:260) [fscrawler-cli-2.5.jar:?]
Caused by: java.net.SocketException: Permission denied: no further information
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_191]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_191]
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171) ~[httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145) ~[httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192) ~[httpasyncclient-4.1.2.jar:4.1.2]
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.2.jar:4.1.2]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]
21:06:25,621 DEBUG [f.p.e.c.f.FsCrawlerImpl] Closing FS crawler [MESH-PC]
21:06:25,622 DEBUG [f.p.e.c.f.c.ElasticsearchClientManager] Closing Elasticsearch client manager
21:06:25,674 DEBUG [f.p.e.c.f.c.ElasticsearchClient] Closing REST client
21:06:25,710 DEBUG [f.p.e.c.f.FsCrawlerImpl] ES Client Manager stopped
21:06:25,712 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [MESH-PC] stopped
21:06:25,718 DEBUG [f.p.e.c.f.FsCrawlerImpl] Closing FS crawler [MESH-PC]
21:06:25,720 DEBUG [f.p.e.c.f.c.ElasticsearchClientManager] Closing Elasticsearch client manager
21:06:25,720 DEBUG [f.p.e.c.f.c.ElasticsearchClient] Closing REST client
21:06:25,721 DEBUG [f.p.e.c.f.FsCrawlerImpl] ES Client Manager stopped

C:\Program Files\fscrawler-2.5\bin>

Sorry for the naive question, but I could do with some help


Failed to create elasticsearch
(David) #2

Perhaps I am not being patient enough or doesn't anyone help newbies :slight_smile:
Either I have not configured fscrawler/java correctly or there is a problem with the
ElasticsearchClientManager
Its probably not the best vehicle but I am on a Win7 machine
Any help would be welcome
Thanks


(Byron Voorbach) #3

I don't think it's that nobody wants to help out newbies, but I'm not sure if there are a lot of people running ES on Windows7 in combination with fscrawler.

Your first issue I think is caused because of invalid json syntax. You'll have to escape the input. (c:\\tmp\\es)

I don't have any personal experience with fsscrawler so can't help you out there :frowning:
Maybe @dadoonet can help out.


(David Pilato) #4

For whatever reason no client can be created... Never seen this message before.

What gives http://127.0.0.1:9202 on your machine?


(David) #5

Thanks [byronvoorbach] you could be right about the usage. I have changed the syntax but still get the error message


(David) #6

Thanks David, Another application is using 9200, hence using that alternative port instead. Could that be the reason? As you can see I have electricsearch on that port

ElasticsearchClientManager is one program, does that make any system assumptions that I have not set in windows7 perhaps?


(David) #7

that output for http://127.0.0.1:9202 is above


(David Pilato) #8

Could you try this version? https://oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-es6/2.6-SNAPSHOT/fscrawler-es6-2.6-20181114.143519-6.zip

I don't know if something has been fixed in the meantime... Might worth a try.


(David) #9

I tried the new version as suggested but got
> C:\fscrawler-es6-2.6-SNAPSHOT\bin>fscrawler.bat --debug MESH-PC1

 no main manifest attribute, in C:\fscrawler-es6-2.6-SNAPSHOT/lib/fscrawler-cli-2.6-SNAPSHOT.jar

(David Pilato) #10

OMG! I did something bad apparently...


(David) #11

Haha if you have a problem then someone with my skill set is doomed haha


(David Pilato) #12

Fixed:

To workaround it, could you edit bin/fscrawler.bat and replace:

%JAVA% %JAVA_OPTS% -cp "%FS_CLASSPATH%" -jar "%FS_HOME%/lib/fscrawler-cli-2.6-SNAPSHOT.jar" !newparams!

by

%JAVA% %JAVA_OPTS% -cp "%FS_CLASSPATH%" -jar "%FS_HOME%/lib/fscrawler-es6-2.6-SNAPSHOT.jar" !newparams!

(David) #13

No luck sorry

C:\fscrawler-es6-2.6-SNAPSHOT\bin>fscrawler.bat
16:07:46,974 INFO  [f.p.e.c.f.c.FsCrawlerCli] No job specified. Here is the list of existing jobs:
16:07:47,030 INFO  [f.p.e.c.f.c.FsCrawlerCli] [1] - data
16:07:47,030 INFO  [f.p.e.c.f.c.FsCrawlerCli] [2] - david
16:07:47,031 INFO  [f.p.e.c.f.c.FsCrawlerCli] [3] - edata
16:07:47,031 INFO  [f.p.e.c.f.c.FsCrawlerCli] [4] - e_drive
16:07:47,031 INFO  [f.p.e.c.f.c.FsCrawlerCli] [5] - fedata
16:07:47,031 INFO  [f.p.e.c.f.c.FsCrawlerCli] [6] - MESH-PC
16:07:47,031 INFO  [f.p.e.c.f.c.FsCrawlerCli] [7] - MESH-PC1
16:07:47,031 INFO  [f.p.e.c.f.c.FsCrawlerCli] [8] - test
16:07:47,031 INFO  [f.p.e.c.f.c.FsCrawlerCli] [9] - test1
16:07:47,032 INFO  [f.p.e.c.f.c.FsCrawlerCli] [10] - test12
16:07:47,032 INFO  [f.p.e.c.f.c.FsCrawlerCli] [11] - TEST123
16:07:47,032 INFO  [f.p.e.c.f.c.FsCrawlerCli] [12] - test3
16:07:47,032 INFO  [f.p.e.c.f.c.FsCrawlerCli] Choose your job [1-12]...
7
16:07:59,293 WARN  [f.p.e.c.f.c.v.ElasticsearchClientV6] failed to create elasticsearch client, disabling crawler...
16:07:59,293 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
java.io.IOException: Permission denied: no further information
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:948) ~[elasticsearch-rest-client-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227) ~[elasticsearch-rest-client-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1256) ~[elasticsearch-rest-high-level-client-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1231) ~[elasticsearch-rest-high-level-client-6.4.2.jar:6.4.2]
        at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:393) ~[elasticsearch-rest-high-level-client-6.4.2.jar:6.4.2]
        at fr.pilato.elasticsearch.crawler.fs.client.v6.ElasticsearchClientV6.getVersion(ElasticsearchClientV6.java:171) ~[fscrawler-elasticsearch-client-v6-2.6-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.checkVersion(ElasticsearchClient.java:202) ~[fscrawler-elasticsearch-client-base-2.6-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.v6.ElasticsearchClientV6.start(ElasticsearchClientV6.java:144) ~[fscrawler-elasticsearch-client-v6-2.6-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:259) [fscrawler-cli-2.6-SNAPSHOT.jar:?]
Caused by: java.net.SocketException: Permission denied: no further information
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_191]
        at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) ~[?:1.8.0_191]
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:171) ~[httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145) ~[httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[httpcore-nio-4.4.5.jar:4.4.5]
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192) ~[httpasyncclient-4.1.2.jar:4.1.2]
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.2.jar:4.1.2]
        at java.lang.Thread.run(Unknown Source) ~[?:1.8.0_191]
16:07:59,304 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [MESH-PC1] stopped
16:07:59,305 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [MESH-PC1] stopped

(David Pilato) #14

Is there any firewall or anything like that which prevents connecting the Java application (FSCrawler) to Elasticsearch? If you have curl can you run curl 127.0.0.1:9202 from the command line?


(David) #15

The firewall is asleep.

Here is the curl output

{
  "name" : "MESH-PC1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "GhGmJvzKTwK_uktjKa3vjQ",
  "version" : {
    "number" : "6.4.3",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "fe40335",
    "build_date" : "2018-10-30T23:17:19.084789Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

(David) #16

Can I invoke a more detailed trace to highlight any issues,so that I can resolve the problem?


(David) #17

I have been doing some Google searches to do with the error messages ```
java.io.IOException: Permission denied: and java.net.SocketException: Permission denied: and they seem to be associated with

  1. The incompatibility, or lack of correct computer settings, when new sockets are created
  2. IPv4 issues
  3. Mis-specified heaps. I notice in the ES files there is a JVM file, but I don't see one in the FS suite

I did have a problem when installing ES from the MSI that I downloaded in that the JVM.options file could not be evaluated and adding ES_JVM_OPTIONS seemed to solve it . Adding that has not had a knock on effect, or do I need something similar for FS?

Please excuse my persistence.


(David Pilato) #18

I have really no idea.

Are you 100% sure that you don't block anything on a network level? ie Microsoft firewall on Java process?