Elasticsearch mapper attachment issues

Kennedy_Kan1 · June 10, 2016, 5:08pm

I have created an index in Sense like this

PUT /test/person/_mapping
{
    "person" : {
        "properties" : {
            "file" : {
                "type" : "attachment",
                "fields" : {
                    "content" : {"index" : "no"},
                    "title" : {"store" : "yes"},
                    "date" : {"store" : "yes"},
                    "author" : {"store" : "yes"},
                    "keywords" : {"store" : "yes"},
                    "content_type" : {"store" : "yes"},
                    "content_length" : {"store" : "yes"},
                    "name" : {"store" : "yes"}
                }
            }
        }
    }
}

PUT /test/person/1
{
    "my_attachment" : {
        "_content_type" : "application/pdf",
        "_name" : "resource/name/of/my.pdf",
        "_language" : "en",
        "_content" : "44CK5pyA5L2O5bel6LOH5qKd76a144CL77yI56ysIDYwOCDnq6DvvInjgIzlt6XkvZzntpPpqZflrbjlk6HjgI3l"
    }
}

PUT /test/person/2
{
    "my_attachment" : {
        "_content_type" : "application/pdf",
        "_title" : "Letter_of_confirmation_work_experience_student",
        "_name" : "Confirmation of student status for “work experience student” ",
        "_author" : "Labour Department",
        "_date" : "11/06/2016",
        "_keywords" : "工作經驗學員",
        "_content" : "44CK5pyA5L2O5bel6LOH5qKd76a144CL77yI56ysIDYwOCDnq6DvvInjgIzlt6XkvZzntpPpqZflrbjlk6HjgI3lrbjnlJ/ouqvku73norroqo3mm7gNCkNvbmZpcm1hdGlvbiBvZiBzdHVkZW50IHN0YXR1cyBmb3Ig4oCcd29yayBleHBlcmllbmNlIHN0dWRlbnTigJ0NCnVuZGVyIHRoZSBNaW5pbXVtIFdhZ2UgT3JkaW5hbmNlIChDYXAuIDYwOCkNCumgiOefpeS6i+mghQ0KTm90ZQ0KMSDjgIzlt6XkvZzntpPpqZflrbj2IOatsuOAgg0KVGhlIENvbmZpcm1hdGlvbiBvZiBzdHVkZW50IHN0YXR1cyBmb3Ig4oCcd29yayBleHBlcmllbmNlIHN0dWRlbnTigJ0gKOKAnENvbmZpcm1hdGlvbuKAnSkgYXBwbGllcyB0byBhIHdvcmsNCmVpcyB1bmRlciB0aGUgYWdlIG9mIDI2IHllYXJzLg0KMiDms5XlrprmnIDkvY7lt6Xos4fvpafpgannlKjmlrzmraPomZXmlrznjbLosYHlhY3lrbjnlJ/lg7HnlKjmnJ/nmoTlt6XkvZzntpPpqZflrbjlk6HvvIzlg7HkuLvmoLnmk5rjgIrlg7Hlgq3mop3vprXjgIvvvIjnrKw1N+eroO+8iemgiA0K5YKZ5a2Y5pWZ6IKy5qmf5qeL55m85Ye655qE5paH5Lu25oiW5YW25Ymv5pys77yM5Lul6aGv56S66Kmy5bel5L2c57aT6am"
    }

I have got 2 questions for that
1. After I create the index in Sense, I try also to create index in kibana. But why couldn't I see any incoming hits in the "Discover" Page.
![Create test index in Kibana]]1

Q2. How can I read back the decode format of the file content?

dadoonet · June 10, 2016, 8:24pm

Q1. May be you define an index with a timestamp?

May be run a query like GET test/_search and see what is coming from this?

If it has not been indexed, you probably got an error.

Q2. You can't. You have to decode yourself the BASE64 content. Or you have to store the content field (means you need to change the mapping).

Another thing: this plugin is deprecated. Prepare yourself to move to ingest-attachment plugin.

Kennedy_Kan1 · June 11, 2016, 4:44am

How can I define the index with timestamp?
Also, when I follow the document of mapper attachment https://www.elastic.co/guide/en/elasticsearch/plugins/master/mapper-attachments-usage.html

    PUT /test/person/_mapping
    {
        "person" : {
            "properties" : {
                "file" : {
                    "type" : "attachment",
                    "fields" : {
                        "content" : {"index" : "no"},
                        "title" : {"store" : "yes"},
                        "date" : {"store" : "yes"},
                        "author" : {"analyzer" : "myAnalyzer"},
                        "keywords" : {"store" : "yes"},
                        "content_type" : {"store" : "yes"},
                        "content_length" : {"store" : "yes"},
                        "language" : {"store" : "yes"}
                    }
                }
            }
        }
    }

It gives me the result

{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "analyzer [MyAnalyzer] not found for field [author]"
}
],
"type": "mapper_parsing_exception",
"reason": "analyzer [MyAnalyzer] not found for field [author]"
},
"status": 400
}

How this happen? And why the content could not be indexed?

dadoonet · June 11, 2016, 5:38am

1/ I don't know what it means and how it's related to mapper attachment plugin.

2/ you did not create an analyzer called MyAnalyzer, right?

Kennedy_Kan1 · June 11, 2016, 5:54am

For q1, I want to find back my record so can I create dashboard in kibana for my records.
For q2, Yes I didnt create it. How should I create one? And from the mapping template, what does the word “store” mean and how can I create an timestamp because when I try to insert timestamp in mapping, it gives me error result

dadoonet · June 11, 2016, 7:26am

You don't need a timestamp to visualize data in Kibana.

If you don't need a specific analyzer, don't define it.

Store means that the content of this field will be stored in Lucene.

Kennedy_Kan1 · June 11, 2016, 7:41am

Then what is the analyzer used for? And why could not the content be set to “index”: “yes”?

dadoonet · June 11, 2016, 8:08am

You might want to read: https://www.elastic.co/guide/en/elasticsearch/guide/current/languages.html

About store, what is the error message you are getting?

Kennedy_Kan1 · June 11, 2016, 9:20am

Yea, I have solved the analyzer problem.

As for the store and index problem, it is as shown as the follow

As seen, I can't create index on content.

dadoonet · June 11, 2016, 9:37am

Please don't copy images.

Try this

    PUT /test/person/_mapping
    {
        "person" : {
            "properties" : {
                "file" : {
                    "type" : "attachment",
                    "fields" : {
                        "content" : { "store" : true }
                    }
                }
            }
        }
    }

Kennedy_Kan1 · June 11, 2016, 10:57am

Is there any ways I could get the data created in elasticsearch to Kibana for making dashboard. Although I have made data in Sense and input to elasticsearch now, I could not get back any data in Kibana

dadoonet · June 11, 2016, 11:11am

Probably depends on how you set your index in Kibana.

Better to ask IMO in #kibana room...

Kennedy_Kan1 · June 11, 2016, 3:39pm

PUT /test/person/2
{
    "my_attachment" : {
        "_content_type" : "application/pdf",
        "_title" : "Letter_of_confirmation_work_experience_student",
        "_name" : "Confirmation of student status for “work experience student” ",
        "_author" : "Labour Department",
        "_date" : "11/06/2016",
        "_keywords" : "工作經驗學員",
        "_content" : "44CK5pyA5L2O5bel6LOH5qKd76a144CL77yI56ysIDYwOCDnq6DvvInjgIzlt6XkvZzntpPpqZflrbjlk6HjgI3lrbjnlJ/ouqvku73norroqo3mm7gNCkNvbmZpcm1hdGlvbiBvZiBzdHVkZW50IHN0YXR1cyBmb3Ig4oCcd29yayBleHBlcmllbmNlIHN0dWRlbnTigJ0NCnVuZGVyIHRoZSBNaW5pbXVtIFdhZ2UgT3JkaW5hbmNlIChDYXAuIDYwOCkNCumgiOefpeS6i+mghQ0KTm90ZQ0KMSDjgIzlt6XkvZznt"
    }

I would wanna also know why it will give me error of "reason": "No content is provided." when I miss out the _ before the 'content'.

dadoonet · June 11, 2016, 4:59pm

It's the way it works and have been designed.

Kennedy_Kan1 · June 12, 2016, 9:18am

How can I set the analyzer to both chinese or english?

dadoonet · June 12, 2016, 1:13pm

IIRC in recent versions multifield is not supported anymore.
I'd either wait for 5.0.0 with the ingest-attachment plugin or do the text extraction in my application (wusing Apache Tika if you are using Java).

Or you need to create 2 indices or 2 fields, one with english analyzer the other one with chinese.
Then send the BASE64 content twice. But that's really not efficient...