FS River not working in elasticsearch cluster

imdhmd · May 23, 2013, 8:58am

I am using:

FS River version: 0.0.3
Elasticsearch: 0.20.6

Setup:

Two nodes of elasticsearch in a cluster, es-one & es-two
A local file system on es-one at /tmp/data
A remotely mounted (via fuse-sshfs) file system on es-two at /tmp/data

SCENARIO 1:
Issued the following rest call to create fsriver:
curl -XPUT es-one:9200/_river/data-river/_meta -d '{
"type": "fs",
"fs": {
"url": "/tmp/data",
"update_rate": 120000
},
"index": {
"index": "data",
"type": "doc",
"bulk_size": 50
}
}'

Expectation:

River with name [data-river] should be created
All files under /tmp/data should be indexed

Actual:

[data-river] wasn't created
Expected files didn't get indexed

SCENARIO 2:
After SCENARIO-1, I brought down elasticsearch server at es-two.
Then,

[data-river] got created
all expected files got indexed

Question:
For FS river to get created and start indexing, do we require to have only
one node running in the cluster?

Thanks,
Imdad/Vidhi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · May 23, 2013, 11:08am

Did you install the FSRiver plugin on both nodes and restart them both?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 23 mai 2013 à 10:58, Imdad Ahmed imdhmd@gmail.com a écrit :

I am using:

FS River version: 0.0.3

Elasticsearch: 0.20.6

Setup:

Two nodes of elasticsearch in a cluster, es-one & es-two

A local file system on es-one at /tmp/data

A remotely mounted (via fuse-sshfs) file system on es-two at /tmp/data

SCENARIO 1:
Issued the following rest call to create fsriver:
curl -XPUT es-one:9200/_river/data-river/_meta -d '{
"type": "fs",
"fs": {
"url": "/tmp/data",
"update_rate": 120000
},
"index": {
"index": "data",
"type": "doc",
"bulk_size": 50
}
}'

Expectation:

River with name [data-river] should be created

All files under /tmp/data should be indexed

Actual:

[data-river] wasn't created

Expected files didn't get indexed

SCENARIO 2:
After SCENARIO-1, I brought down elasticsearch server at es-two.
Then,

[data-river] got created

all expected files got indexed

Question:
For FS river to get created and start indexing, do we require to have only one node running in the cluster?

Thanks,
Imdad/Vidhi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

imdhmd · August 23, 2013, 8:35am

This is a late reply to this thread, to bring it to closure.

Yes, Pilato, we did follow the correct steps. But the problem was something
else.
Elasticsearch user on es-two didnt have proper access to /tmp/data.
Once we corrected accessibility issue, this problem no longer occured.

On Thursday, May 23, 2013 4:38:32 PM UTC+5:30, David Pilato wrote:

Did you install the FSRiver plugin on both nodes and restart them both?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 23 mai 2013 à 10:58, Imdad Ahmed <imd...@gmail.com <javascript:>> a
écrit :

I am using:

FS River version: 0.0.3

Elasticsearch: 0.20.6

Setup:

Two nodes of elasticsearch in a cluster, es-one & es-two

A local file system on es-one at /tmp/data

A remotely mounted (via fuse-sshfs) file system on es-two at /tmp/data

SCENARIO 1:
Issued the following rest call to create fsriver:
curl -XPUT es-one:9200/_river/data-river/_meta -d '{
"type": "fs",
"fs": {
"url": "/tmp/data",
"update_rate": 120000
},
"index": {
"index": "data",
"type": "doc",
"bulk_size": 50
}
}'

Expectation:

River with name [data-river] should be created

All files under /tmp/data should be indexed

Actual:

[data-river] wasn't created

Expected files didn't get indexed

SCENARIO 2:
After SCENARIO-1, I brought down elasticsearch server at es-two.
Then,

[data-river] got created

all expected files got indexed

Question:
For FS river to get created and start indexing, do we require to have only
one node running in the cluster?

Thanks,
Imdad/Vidhi

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · August 23, 2013, 8:36am

Thanks for the feedback.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 23 août 2013 à 10:35, Imdad Ahmed imdhmd@gmail.com a écrit :

This is a late reply to this thread, to bring it to closure.

Yes, Pilato, we did follow the correct steps. But the problem was something else.
Elasticsearch user on es-two didnt have proper access to /tmp/data.
Once we corrected accessibility issue, this problem no longer occured.

On Thursday, May 23, 2013 4:38:32 PM UTC+5:30, David Pilato wrote:
Did you install the FSRiver plugin on both nodes and restart them both?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 23 mai 2013 à 10:58, Imdad Ahmed imd...@gmail.com a écrit :

I am using:

FS River version: 0.0.3

Elasticsearch: 0.20.6

Setup:

Two nodes of elasticsearch in a cluster, es-one & es-two

A local file system on es-one at /tmp/data

A remotely mounted (via fuse-sshfs) file system on es-two at /tmp/data

SCENARIO 1:
Issued the following rest call to create fsriver:
curl -XPUT es-one:9200/_river/data-river/_meta -d '{
"type": "fs",
"fs": {
"url": "/tmp/data",
"update_rate": 120000
},
"index": {
"index": "data",
"type": "doc",
"bulk_size": 50
}
}'

Expectation:

River with name [data-river] should be created

All files under /tmp/data should be indexed

Actual:

[data-river] wasn't created

Expected files didn't get indexed

SCENARIO 2:
After SCENARIO-1, I brought down elasticsearch server at es-two.
Then,

[data-river] got created

all expected files got indexed

Question:
For FS river to get created and start indexing, do we require to have only one node running in the cluster?

Thanks,
Imdad/Vidhi

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jamai_Ali · April 16, 2016, 12:20pm

Stp j utilise fs river pour indexer des journaux sur lesquels j px mener une recherche sur le contenu ultérieurement . Apres une annee je me suis trouver avec un problem que elastic n indec plus leq nouveaux doc ou accuse un grand retard pour le faire je compte sur votre aide

jamai_Ali · April 16, 2016, 12:21pm

Juste une precision un noeud un cluster

dadoonet · April 16, 2016, 12:35pm

Please use english here. There is a dedicated French area.
Also don't hijack old threads and create your own.

You can also open an issue in the project.