# FSCrawler not able to connect to elastic search

**URL:** <https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530>\
**Category:** Elasticsearch\
**Created:** [September 7, 2021, 12:29pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530 "2021-09-07T12:29:59Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Reghu\_Shanker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reghu_shanker/32/77664_2.png) [@Reghu\_Shanker](https://discuss.elastic.co/u/Reghu_Shanker)\
**Post date:** [September 7, 2021, 12:29pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/1 "2021-09-07T12:29:59Z")

</div>

Hi,

I have an AWS Elastic Search Instance. I am trying to us FSCrawler along with it. My elastic search version is 7.10 and FSCrawler is [fscrawler-es7-2.7.zip](https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler-es7/2.7/fscrawler-es7-2.7.zip)

I am getting the following error:

```auto
17:51:21,439 e[33mWARN e[m [f.p.e.c.f.c.v.ElasticsearchClientV7] failed to create elasticsearch client on Elasticsearch{nodes=[https://search-reghu-personal-3-dlvlsq4wm7qa7fmgrckjtep66y.us-east-2.es.amazonaws.com], index='elasticSearchPoc', indexFolder='elasticSearchPoc_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, username='null', pipeline='null', pathPrefix='null', sslVerification='false'}, disabling crawler...

17:51:21,440 e[1;31mFATALe[m [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.

org.elasticsearch.ElasticsearchException: Invalid or missing build flavor [oss]

at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2084) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]

at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1732) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]

at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1717) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]

at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1684) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]

at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:825) ~[elasticsearch-rest-high-level-client-7.14.0.jar:7.14.0]

at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.getVersion(ElasticsearchClientV7.java:180) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]

at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.checkVersion(ElasticsearchClient.java:193) ~[fscrawler-elasticsearch-client-base-2.7.jar:?]

at fr.pilato.elasticsearch.crawler.fs.client.v7.ElasticsearchClientV7.start(ElasticsearchClientV7.java:153) ~[fscrawler-elasticsearch-client-v7-2.7.jar:?]

at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerManagementServiceElasticsearchImpl.start(FsCrawlerManagementServiceElasticsearchImpl.java:63) ~[fscrawler-core-2.7.jar:?]

at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:116) ~[fscrawler-core-2.7.jar:?]

at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:322) [fscrawler-cli-2.7.jar:?]

at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:298) [fscrawler-cli-2.7.jar:?]

```

My AWS ES instance is ip restricted. I get the following response from the end point when I try to hit it from browser:

```auto
{
  "name" : "30cf6100daa7d88d855a9bd1b6cf0cba",
  "cluster_name" : "656830870718:reghu-personal-3",
  "cluster_uuid" : "bAzRn39rTlSWW6yNV2qzIQ",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2021-07-08T21:51:39.338286Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

```

What could be the issue here?

Thanks,  
Regs

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 7, 2021, 12:40pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/2 "2021-09-07T12:40:26Z")

</div>

You're using the AWS fork of Elasticsearch (a.k.a. [OpenSearch](https://aws.amazon.com/blogs/opensource/introducing-opensearch/)) which isn't 100% compatible with Elasticsearch and won't work with the latest FSCrawler. Not sure if there's an AWS-compatible fork of FSCrawler, you'll have to speak to the OpenSearch folks about that. If not, you can try using an older version of FSCrawler or else move to Elasticsearch proper (e.g. using [Elastic Cloud: Hosted Elasticsearch, Hosted Search | Elastic](https://www.elastic.co/cloud/) if you want to stick with a managed service)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 7, 2021, 12:57pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/3 "2021-09-07T12:57:20Z")

</div>

I can confirm that the current version of FSCrawler needs to run with Elasticsearch 7.14. AFAIK this version is only available on [cloud.elastic.co](http://cloud.elastic.co).

---

<div class="post-metadata">

**Author:** ![Reghu\_Shanker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reghu_shanker/32/77664_2.png) [@Reghu\_Shanker](https://discuss.elastic.co/u/Reghu_Shanker)\
**Post date:** [September 7, 2021, 1:01pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/4 "2021-09-07T13:01:10Z")

</div>

Just a little confused. My AWS console says it has Elastic Search Engine

 ![Screenshot 2021-09-07 at 6.30.17 PM](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2bf03f4c72afda60495aca0ad9ffaa9949441904.png)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 7, 2021, 1:03pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/5 "2021-09-07T13:03:05Z")

</div>

Yes, it's unfortunate and very confusing that AWS Elasticsearch uses the same name for a different thing. That's one of the reasons why [they're renaming it](https://aws.amazon.com/blogs/opensource/introducing-opensearch/).

---

<div class="post-metadata">

**Author:** ![Reghu\_Shanker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reghu_shanker/32/77664_2.png) [@Reghu\_Shanker](https://discuss.elastic.co/u/Reghu_Shanker)\
**Post date:** [September 7, 2021, 1:24pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/6 "2021-09-07T13:24:30Z")

</div>

I was going through open search documentation, AWS says Opensearch is still not available... !  
[Screenshot 2021-09-07 at 6.53.00 PM|690x70](https://discuss.elastic.co/uploads/short-url/a5NbtplqWv9jpeDyTgZishdElw9.png)

> **[What is OpenSearch - Amazon Web Services](https://aws.amazon.com/elasticsearch-service/the-elk-stack/what-is-opensearch/#:~:text=Q:%20Is%20OpenSearch%20available%20in%20Amazon%20Elasticsearch%20Service?)**
>
> OpenSearch is a distributed, open source search and analytics suite used for a broad set of use cases like log analytics.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 7, 2021, 1:38pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/7 "2021-09-07T13:38:05Z")

</div>

Short term response. If you want to run FSCrawler with Elasticsearch, look at [Cloud by Elastic](https://www.elastic.co/cloud), also available if needed from [AWS Marketplace](https://aws.amazon.com/marketplace/pp/Elasticsearch-Inc-Elasticsearch-Service-on-Elastic/B01N6YCISK), [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.ec-azure?tab=Overview) and [Google Cloud Marketplace](https://console.cloud.google.com/marketplace/details/endpoints/elasticsearch-service.gcpmarketplace.elastic.co)?

Cloud by elastic is one way to have access to **all features** , all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named [Observability](https://www.elastic.co/observability), [Security](https://www.elastic.co/security), [Enterprise Search](https://www.elastic.co/enterprise-search) and what is coming next 🙂 ...

You can also run FSCrawler with Workplace Search if you want to have a UI and search not only your local files but also from Google Drive, Dropbox...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 5, 2021, 1:38pm UTC](https://discuss.elastic.co/t/fscrawler-not-able-to-connect-to-elastic-search/283530/8 "2021-10-05T13:38:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.