Single field object to Elasticsearch using Logstash

[KrishK (K P)]

Hi Support,

I have group concat field values like

Document_Type
1,2,3,4,5,6

In Elasticsearch field mapping with object type

"Document_Type": {
        "properties": {
          "d": {
            "type": "short"
          }
        }
 }

When I run Logstash getting this "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [Document_Type] tried to parse field [Document_Type] as object, but found a concrete value"}}}}

Can you please help me?

Thanks

[Fabio-sama (Fabio)]

Hi there,

it means that in your mapping you defined that Document_Type as a object (in fact it looks like an object) but you're passing it as something different (in fact what you wrote there under that bold Document_Type is an array of numbers, not an object).

Can you please share a sample of input doc and the complete mapping for that index?

[KrishK (K P)]

Hi Fabio,

Thanks for the reply.

Yes, You understood well.

Below is the mapping and example.

DELETE testindex
PUT testindex
PUT testindex/_mappings
{
"properties": {    
      "Document_Type": {
        "type": "object",
        "properties": {
          "d": {
            "type": "short"
          }
        }
      },
	  "filename": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "fileid": {
        "type": "long"
      }
    }
}
POST testindex/_doc
{
  "Document_Type":[{"d":1},{"d":1}],
  "fileid" : 1,
  "filename" :"elasticsearch filename.pdf"
}
GET testindex/_search

Above example working fine, But when indexing using logstash.

input {
  jdbc { 
    jdbc_connection_string => "jdbc:mysql://localhost:3306/test?useCursorFetch=true"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "mysql-connector-java-8.0.18.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
       # our query
    statement => "SELECT a.fileid, a.filename, group_concat(b.document_type) as Document_Type FROM file_table AS a LEFT JOIN file_doctype AS b ON a.fileid = b.fileid group by a.fileid order by a.fileid"    
    }
  }
  
output {
stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["localhost:9200"]
    index => ["testindex"]
  }
}

I am getting this "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [Document_Type] tried to parse field [Document_Type] as object, but found a concrete value"}}}}

Now can you clear me on this?

Thanks

[KrishK (K P)]

Any Hope?

[Fabio-sama (Fabio)]

Hi there,

I'm sorry I've been kinda busy and didn't see the notification. Anyway, can you share the output of logstash? I mean, you're sending to both elasticsearch and standard output.

What is written in the stdout?

[KrishK (K P)]

Hi Fabio,

I have resolved it.

Thanks

[Fabio-sama (Fabio)]

Very good!

Would you mind posting here your solution and mark it as the solution? It might be useful for future users.

[KrishK (K P)]

Sure Fabio.

Below is the solution.

 ruby {
  code => "
    r = []
	data = event.get('Document_Type').split(',')
    	data.each { |values|        
        item = {
            'd' => values,            
        }
        r << item
    }
    event.set('Document_Type', r)
  "
	}