Elasticsearch + attachment plugin + Kibana + couchbase


(Nadav Hashimshony) #1

Hi,

I'm new to the group, hope ill find what i need and share my experience as
i go along..

im using ES with the attachment-plugin in order to store and search files.
when i set the mapping right and insert the file data in a Base64 manner
I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to Couchbase
and use XDRC to replicate it to ES, i can't query the Data with Kibana.
it looks like the mapping of the index created in ES doesn't index well the
data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Itamar Syn-Hershko) #2

The XDCR plugin indexes the data using an envelope document. Long story
short, make sure you use the latest XDCR plugin as older ones are missing
lots of important functions, and use templates and dynamic templates with
proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-day-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nadavh@gmail.com wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience as
i go along..

im using ES with the attachment-plugin in order to store and search files.
when i set the mapping right and insert the file data in a Base64 manner
I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to Couchbase
and use XDRC to replicate it to ES, i can't query the Data with Kibana.
it looks like the mapping of the index created in ES doesn't index well
the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zs5OwXJe9aT1pPNu9vuooXO10Z3Mx7xc8CJh77EN9s%3DCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Nadav Hashimshony) #3

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
"files": {
"properties": {
"file": {
"type": "attachment",
"path": "full",
"fields": {
"content_type": {
"type": "string",
"store": true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long story
short, make sure you use the latest XDCR plugin as older ones are missing
lots of important functions, and use templates and dynamic templates with
proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-day-managing-index-mappings-like-a-pro

http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony <nad...@gmail.com
<javascript:>> wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience
as i go along..

im using ES with the attachment-plugin in order to store and search files.
when i set the mapping right and insert the file data in a Base64 manner
I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to Couchbase
and use XDRC to replicate it to ES, i can't query the Data with Kibana.
it looks like the mapping of the index created in ES doesn't index well
the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Itamar Syn-Hershko) #4

Like I said, you need the mapping to "catch" before the XDCR plugin begins
the replication - so you need to put a template with this mapping that will
override XDCR's

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nadavh@gmail.com wrote:

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
"files": {
"properties": {
"file": {
"type": "attachment",
"path": "full",
"fields": {
"content_type": {
"type": "string",
"store": true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long story
short, make sure you use the latest XDCR plugin as older ones are missing
lots of important functions, and use templates and dynamic templates with
proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-
day-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-
day-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience
as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64 manner
I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to Couchbase
and use XDRC to replicate it to ES, i can't query the Data with Kibana.
it looks like the mapping of the index created in ES doesn't index well
the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zuh7HHK8XmdznuHnw7E01ffXV8BC-49D70ekMc1-YhQCA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Nadav Hashimshony) #5

ok, just to be clear.

the steps i did was as followed:

  1. create the index with the mapping.
  2. define the XDCR to replicate my bucket with the index in ES.
  3. insert data to couchbase.
  4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.

  1. crate the index in ES
    and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to "catch" before the XDCR plugin begins
the replication - so you need to put a template with this mapping that will
override XDCR's

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony <nad...@gmail.com
<javascript:>> wrote:

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
"files": {
"properties": {
"file": {
"type": "attachment",
"path": "full",
"fields": {
"content_type": {
"type": "string",
"store": true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long story
short, make sure you use the latest XDCR plugin as older ones are missing
lots of important functions, and use templates and dynamic templates with
proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-
day-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-
day-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my experience
as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data with
Kibana.
it looks like the mapping of the index created in ES doesn't index well
the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Itamar Syn-Hershko) #6

Yes. Just make sure the template reflects the actual document structure -
as I said XDCR wraps your document in an envelope document

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:12 PM, Nadav Hashimshony nadavh@gmail.com wrote:

ok, just to be clear.

the steps i did was as followed:

  1. create the index with the mapping.
  2. define the XDCR to replicate my bucket with the index in ES.
  3. insert data to couchbase.
  4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.

  1. crate the index in ES
    and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to "catch" before the XDCR plugin
begins the replication - so you need to put a template with this mapping
that will override XDCR's

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
"files": {
"properties": {
"file": {
"type": "attachment",
"path": "full",
"fields": {
"content_type": {
"type": "string",
"store": true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long story
short, make sure you use the latest XDCR plugin as older ones are missing
lots of important functions, and use templates and dynamic templates with
proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-d
ay-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-d
ay-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my
experience as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data with
Kibana.
it looks like the mapping of the index created in ES doesn't index
well the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zuoj0kZmmY%2BeS_mH54nBp39HUSGDETJemEU4kdFLcjdCg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Nadav Hashimshony) #7

ok, ill try.

this "envelope document", is it something i need to be concerned about when
I'm querying via Kibana?

On Thursday, February 12, 2015 at 5:14:39 PM UTC+2, Itamar Syn-Hershko
wrote:

Yes. Just make sure the template reflects the actual document structure -
as I said XDCR wraps your document in an envelope document

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:12 PM, Nadav Hashimshony <nad...@gmail.com
<javascript:>> wrote:

ok, just to be clear.

the steps i did was as followed:

  1. create the index with the mapping.
  2. define the XDCR to replicate my bucket with the index in ES.
  3. insert data to couchbase.
  4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.

  1. crate the index in ES
    and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to "catch" before the XDCR plugin
begins the replication - so you need to put a template with this mapping
that will override XDCR's

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
"files": {
"properties": {
"file": {
"type": "attachment",
"path": "full",
"fields": {
"content_type": {
"type": "string",
"store": true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long
story short, make sure you use the latest XDCR plugin as older ones are
missing lots of important functions, and use templates and dynamic
templates with proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-d
ay-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-d
ay-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my
experience as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data with
Kibana.
it looks like the mapping of the index created in ES doesn't index
well the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/96f30e99-30dc-4af2-a51b-b12b0dee9e53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Itamar Syn-Hershko) #8

Yes, that too :slight_smile:

Also if its a time based data, you will not be able to use kibana's date
filtering etc - because it lacks the @timestamp field. Basically, the XDCR
elasticsearch plugin was built around the XDCR / Couchbase realm and not
around Elasticsearch's. Unfortunately this means many ES features are
unavailable / hard to use, e.g.

I can help fixing this on the XDCR plugin if you'd like - ping me privately
and we can work something out (or I can convince you to avoid using the
XDCR replication)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:18 PM, Nadav Hashimshony nadavh@gmail.com wrote:

ok, ill try.

this "envelope document", is it something i need to be concerned about
when I'm querying via Kibana?

On Thursday, February 12, 2015 at 5:14:39 PM UTC+2, Itamar Syn-Hershko
wrote:

Yes. Just make sure the template reflects the actual document structure -
as I said XDCR wraps your document in an envelope document

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:12 PM, Nadav Hashimshony nad...@gmail.com
wrote:

ok, just to be clear.

the steps i did was as followed:

  1. create the index with the mapping.
  2. define the XDCR to replicate my bucket with the index in ES.
  3. insert data to couchbase.
  4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.

  1. crate the index in ES
    and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to "catch" before the XDCR plugin
begins the replication - so you need to put a template with this mapping
that will override XDCR's

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
"files": {
"properties": {
"file": {
"type": "attachment",
"path": "full",
"fields": {
"content_type": {
"type": "string",
"store": true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar Syn-Hershko
wrote:

The XDCR plugin indexes the data using an envelope document. Long
story short, make sure you use the latest XDCR plugin as older ones are
missing lots of important functions, and use templates and dynamic
templates with proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-d
ay-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-d
ay-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Hi,

I'm new to the group, hope ill find what i need and share my
experience as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data with
Kibana.
it looks like the mapping of the index created in ES doesn't index
well the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef
8-4249-8e5d-acff8281a81a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/96f30e99-30dc-4af2-a51b-b12b0dee9e53%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/96f30e99-30dc-4af2-a51b-b12b0dee9e53%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt%3Drs%2BnOhoHP2UPM5ZmV%2BWQet_8FrmiGHFpf_Ykokdxyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Nadav Hashimshony) #9

thanks, sent you a pm

On Thursday, February 12, 2015 at 5:24:33 PM UTC+2, Itamar Syn-Hershko
wrote:

Yes, that too :slight_smile:

Also if its a time based data, you will not be able to use kibana's date
filtering etc - because it lacks the @timestamp field. Basically, the XDCR
elasticsearch plugin was built around the XDCR / Couchbase realm and not
around Elasticsearch's. Unfortunately this means many ES features are
unavailable / hard to use, e.g.
https://github.com/couchbaselabs/elasticsearch-transport-couchbase/issues/63

https://github.com/couchbaselabs/elasticsearch-transport-couchbase/issues/64

I can help fixing this on the XDCR plugin if you'd like - ping me
privately and we can work something out (or I can convince you to avoid
using the XDCR replication)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:18 PM, Nadav Hashimshony <nad...@gmail.com
<javascript:>> wrote:

ok, ill try.

this "envelope document", is it something i need to be concerned about
when I'm querying via Kibana?

On Thursday, February 12, 2015 at 5:14:39 PM UTC+2, Itamar Syn-Hershko
wrote:

Yes. Just make sure the template reflects the actual document structure

  • as I said XDCR wraps your document in an envelope document

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 5:12 PM, Nadav Hashimshony nad...@gmail.com
wrote:

ok, just to be clear.

the steps i did was as followed:

  1. create the index with the mapping.
  2. define the XDCR to replicate my bucket with the index in ES.
  3. insert data to couchbase.
  4. try to query with kibana

What you suggest is to Add another BEFORE step 1:
0. create a template to include my mapping.

  1. crate the index in ES
    and so on...

did i get it right?

Thanks.
Nadav.

On Thursday, February 12, 2015 at 5:04:24 PM UTC+2, Itamar Syn-Hershko
wrote:

Like I said, you need the mapping to "catch" before the XDCR plugin
begins the replication - so you need to put a template with this mapping
that will override XDCR's

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:59 PM, Nadav Hashimshony nad...@gmail.com
wrote:

Thanks you for the response

i am using mapping, i created the following index
PUT /storage/files/_mapping
{
"files": {
"properties": {
"file": {
"type": "attachment",
"path": "full",
"fields": {
"content_type": {
"type": "string",
"store": true
}
}
}
}
}
}

when i insert data via ES and query it, all is fine.
the problem is when data is inserted to Couchbase..

Nadav

On Thursday, February 12, 2015 at 4:03:01 PM UTC+2, Itamar
Syn-Hershko wrote:

The XDCR plugin indexes the data using an envelope document. Long
story short, make sure you use the latest XDCR plugin as older ones are
missing lots of important functions, and use templates and dynamic
templates with proper field paths for this to work correctly

http://code972.com/blog/2015/02/80-elasticsearch-one-tip-a-d
ay-managing-index-mappings-like-a-pro
http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-d
ay-using-dynamic-templates-to-avoid-rigorous-mappings

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 3:59 PM, Nadav Hashimshony <nad...@gmail.com

wrote:

Hi,

I'm new to the group, hope ill find what i need and share my
experience as i go along..

im using ES with the attachment-plugin in order to store and search
files.
when i set the mapping right and insert the file data in a Base64
manner I'm able to query my data via Kibana.

my problem is this.

if i create the index + mapping in ES, then insert the data to
Couchbase and use XDRC to replicate it to ES, i can't query the Data with
Kibana.
it looks like the mapping of the index created in ES doesn't index
well the data it gets from Couchbase.

has anyone encounter such an issue?

Thanks You

Nadav.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef
8-4249-8e5d-acff8281a81a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8092eaf5-0ef8-4249-8e5d-acff8281a81a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d9c8ce5-116f-40cc-a5e3-6ebe47191850%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8628ef07-2e10-423a-9de0-13ebaa37a0e8%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/96f30e99-30dc-4af2-a51b-b12b0dee9e53%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/96f30e99-30dc-4af2-a51b-b12b0dee9e53%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d4a1ccd2-d389-4218-b54c-349976de2a46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Alper Kanat) #10

Hi @Itamar_Syn_Hershko,

I'm using 5.4.1 for ES & Couchbase and the XDCR transport plugin. I'm able to query my dataset but I cannot use Kibana's time filtering. During index pattern creation, I checked the checkbox for time series data and it was able to show me doc.createdAt which is the datetime field (iso8601) in my docs. However no timepicker or histogram in discover tab appears.

I couldn't really find a pointer about this issue. Should I have a field with the exact same name: @timestamp?