Can 2 ElasticSearch plugins interact?

Kiran_Koirala · March 30, 2015, 7:07pm

I have a situation where we need to index a contents of a PDF file in
Elastic Search. This can be easily achieved using a FS river plugin but the
pdf file is stored as a BLOB object in a database, this means a JDBC river
plugin is also in picture now.

Please let me know your thoughts on this. I will have a look on Attachments
Mapper plugin to see if it can replace a FS river plugin and read a BLOB
directly.

Thanks in Advance..!!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/906f07be-50d9-4307-abf7-2d46f096f7f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · March 30, 2015, 8:10pm

You can combine JDBC plugin and attachment mapper if your database can
convert the blob to base64 string.

Jörg

On Mon, Mar 30, 2015 at 9:07 PM, Kiran Koirala kkoirala@manageforce.com
wrote:

I have a situation where we need to index a contents of a PDF file in
Elastic Search. This can be easily achieved using a FS river plugin but the
pdf file is stored as a BLOB object in a database, this means a JDBC river
plugin is also in picture now.

Please let me know your thoughts on this. I will have a look on
Attachments Mapper plugin to see if it can replace a FS river plugin and
read a BLOB directly.

Thanks in Advance..!!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/906f07be-50d9-4307-abf7-2d46f096f7f3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/906f07be-50d9-4307-abf7-2d46f096f7f3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEttxJZmR-oegFFTtCrDQuB0NyWY9Y-0Azm7OX%3DgUi37w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

jprante · March 30, 2015, 9:25pm

With MySQL blob, you do not need base64 conversion by DB, because JDBC
driver works correct. The base64 conversion is performed by JDBC river when
indexing. Maybe other DB work like MySQL too.

Demo:

gist.github.com

https://gist.github.com/jprante/22c28851bed9aff8d9da

mysql-blob-river.sh

#!/bin/sh

# MySQL 5.1+ with http://dev.mysql.com/doc/refman/5.1/en/string-functions.html#function_load-file
# ES 1.4.4+
# plugins required: jdbc, mapper attachment
# test.pdf is a PDF that can be parsed by Tika

/usr/local/mysql/bin/mysql -u root test <<EOT
drop table test;
create table test (

This file has been truncated. show original

result.txt

$ bash mysql-blob-river.sh 
{"error":"TypeMissingException[[_all] type[[my_test_river]] missing: No index has the type.]","status":404}{"acknowledged":true}{"_index":"_river","_type":"my_test_river","_id":"_meta","_version":1,"created":true}sleeping while river should run...
{
  "mytest" : {
    "mappings" : {
      "mydocs" : {
        "properties" : {
          "content" : {
            "type" : "attachment",
            "path" : "full",

This file has been truncated. show original

Jörg

On Mon, Mar 30, 2015 at 10:10 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

You can combine JDBC plugin and attachment mapper if your database can
convert the blob to base64 string.

Jörg

On Mon, Mar 30, 2015 at 9:07 PM, Kiran Koirala kkoirala@manageforce.com
wrote:

I have a situation where we need to index a contents of a PDF file in
Elastic Search. This can be easily achieved using a FS river plugin but the
pdf file is stored as a BLOB object in a database, this means a JDBC river
plugin is also in picture now.

Please let me know your thoughts on this. I will have a look on
Attachments Mapper plugin to see if it can replace a FS river plugin and
read a BLOB directly.

Thanks in Advance..!!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/906f07be-50d9-4307-abf7-2d46f096f7f3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/906f07be-50d9-4307-abf7-2d46f096f7f3%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH3eBz%3Dp_T-3rjPV0q7qGYTNrhr4_Fv5yMxeVmAnpUf-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.