How do i connect to s3 and index files on s3

Siddartha_Reddy_2 · February 20, 2015, 6:12pm

Hi,

I installed elastic search today, i am basically testing the es with one
node. I am trying to configure the yml file, since the s3 gateway is
deprecated where do i specify the bucket name? currently my config file
looks like this :

cloud:
aws:
access_key: xxxxxxxxxxxxxxxx
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
region: us-west-2

gateway:
recover_after_nodes: 1
recover_after_time: 5m

And once i start the service will it start indexing all the files in my
bucket? is there an option to specify certain filenames to index in my
bucket?

Thanks,
Sid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6173bd78-6673-4b51-a1d3-0d79e66767fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 20, 2015, 6:23pm

You should not do it.

If you really want to do it, I think the option could be to mount a S3 volume on your EC2 instance. I guess it’s doable.
Digital (dis)content: HOWTO AWS: mount S3 buckets from a Linux EC2 instance with s3fs http://juliensimon.blogspot.fr/2013/08/howto-aws-mount-s3-buckets-from-linux.html

But really, prefer local disks.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 20 févr. 2015 à 19:12, Siddartha Reddy siddartha.nagireddy@gmail.com a écrit :

Hi,

I installed Elasticsearch today, i am basically testing the es with one node. I am trying to configure the yml file, since the s3 gateway is deprecated where do i specify the bucket name? currently my config file looks like this :

cloud:
aws:
access_key: xxxxxxxxxxxxxxxx
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
region: us-west-2

gateway:
recover_after_nodes: 1
recover_after_time: 5m

And once i start the service will it start indexing all the files in my bucket? is there an option to specify certain filenames to index in my bucket?

Thanks,
Sid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6173bd78-6673-4b51-a1d3-0d79e66767fa%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/6173bd78-6673-4b51-a1d3-0d79e66767fa%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/579789DA-8F9C-48A9-86D8-79FB4FC15E79%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Siddartha_Reddy_2 · February 20, 2015, 6:29pm

Thanks for reply david,

It's not one file, i have thousands of files on s3. Is there a way to
automate this? can i write some code in python to index documents?

On Friday, February 20, 2015 at 10:23:23 AM UTC-8, David Pilato wrote:

You should not do it.

If you really want to do it, I think the option could be to mount a S3
volume on your EC2 instance. I guess it’s doable.

Digital (dis)content: HOWTO AWS: mount S3 buckets from a Linux EC2 instance with s3fs

But really, prefer local disks.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 20 févr. 2015 à 19:12, Siddartha Reddy <siddartha...@gmail.com
<javascript:>> a écrit :

Hi,

I installed Elasticsearch today, i am basically testing the es with one
node. I am trying to configure the yml file, since the s3 gateway is
deprecated where do i specify the bucket name? currently my config file
looks like this :

cloud:
aws:
access_key: xxxxxxxxxxxxxxxx
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
region: us-west-2

gateway:
recover_after_nodes: 1
recover_after_time: 5m

And once i start the service will it start indexing all the files in my
bucket? is there an option to specify certain filenames to index in my
bucket?

Thanks,
Sid

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6173bd78-6673-4b51-a1d3-0d79e66767fa%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6173bd78-6673-4b51-a1d3-0d79e66767fa%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e80c8be-a247-45ce-8968-da19c302d08d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 20, 2015, 6:37pm

I thought you were after storing indices on S3.

Sorry.

Look at logstash. There is a S3 input IIRC.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 20 févr. 2015 à 19:29, Siddartha Reddy siddartha.nagireddy@gmail.com a écrit :

Thanks for reply david,

It's not one file, i have thousands of files on s3. Is there a way to automate this? can i write some code in python to index documents?

On Friday, February 20, 2015 at 10:23:23 AM UTC-8, David Pilato wrote:
You should not do it.

If you really want to do it, I think the option could be to mount a S3 volume on your EC2 instance. I guess it’s doable.
Digital (dis)content: HOWTO AWS: mount S3 buckets from a Linux EC2 instance with s3fs

But really, prefer local disks.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 20 févr. 2015 à 19:12, Siddartha Reddy siddartha...@gmail.com a écrit :

Hi,

I installed Elasticsearch today, i am basically testing the es with one node. I am trying to configure the yml file, since the s3 gateway is deprecated where do i specify the bucket name? currently my config file looks like this :

cloud:
aws:
access_key: xxxxxxxxxxxxxxxx
secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
region: us-west-2

gateway:
recover_after_nodes: 1
recover_after_time: 5m

And once i start the service will it start indexing all the files in my bucket? is there an option to specify certain filenames to index in my bucket?

Thanks,
Sid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6173bd78-6673-4b51-a1d3-0d79e66767fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4e80c8be-a247-45ce-8968-da19c302d08d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/D2E60AC9-9C8F-4039-9257-53FB2C6BCE83%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Siddartha_Reddy_2 · February 20, 2015, 6:59pm

Thanks David, I worked on logstash and was able to index a file into es.
But since I am working on a single node, the status changes to yellow and I
am not able to index anymore files using logstash. Is there any workaround
to handle this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9fb001ef-1683-475e-a25b-1bbf8106fac6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 20, 2015, 7:48pm

Yellow is not something bad. It just means that you have replica shards not allocated. Obviously you have one single node.

That should not stop you from indexing. I guess something is wrong with your logstash config but this is something you should ask on logstash mailing list IMO.

David

Le 20 févr. 2015 à 19:59, Siddartha Reddy siddartha.nagireddy@gmail.com a écrit :

Thanks David, I worked on logstash and was able to index a file into es. But since I am working on a single node, the status changes to yellow and I am not able to index anymore files using logstash. Is there any workaround to handle this?

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9fb001ef-1683-475e-a25b-1bbf8106fac6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B1836136-6E2C-466B-82C1-07F7CC544CAE%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

warkolm · February 20, 2015, 8:55pm

Logstash defaults to 5 shards, 1 replica. The replica is the reason for the
yellow status.
Just remove it and you will be fine - curl -XPUT localhost:9200/*/_settings
-d '{ "index" : { "number_of_replicas" : 0 } }'

On 21 February 2015 at 06:48, David Pilato david@pilato.fr wrote:

Yellow is not something bad. It just means that you have replica shards
not allocated. Obviously you have one single node.

That should not stop you from indexing. I guess something is wrong with
your logstash config but this is something you should ask on logstash
mailing list IMO.

David

Le 20 févr. 2015 à 19:59, Siddartha Reddy siddartha.nagireddy@gmail.com
a écrit :

Thanks David, I worked on logstash and was able to index a file into es.
But since I am working on a single node, the status changes to yellow and I
am not able to index anymore files using logstash. Is there any workaround
to handle this?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9fb001ef-1683-475e-a25b-1bbf8106fac6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9fb001ef-1683-475e-a25b-1bbf8106fac6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/B1836136-6E2C-466B-82C1-07F7CC544CAE%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/B1836136-6E2C-466B-82C1-07F7CC544CAE%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8z4%2BB69io4wpzOD%3D89Amx2ujWo%2B3Mv3Apaa_A04wp0wg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
How do i connect to S3 and index files on s3 Elasticsearch	1	345	July 6, 2017
Running on EC2 S3 vs EBS Elasticsearch	3	797	July 6, 2017
How to add ec2 s3 or other gateway after index is created? Elasticsearch	8	390	July 6, 2017
Backups of Nodes? Elasticsearch	4	311	July 6, 2017
How to index s3 files and transfer that index to my ES server Elasticsearch	4	1196	April 7, 2020

How do i connect to s3 and index files on s3

Thanks David, I worked on logstash and was able to index a file into es. But since I am working on a single node, the status changes to yellow and I am not able to index anymore files using logstash. Is there any workaround to handle this?

Related topics