Put Jackson JsonNode object for indexing

Elastic Stack Elasticsearch
1

Hi All.

I'm trying to put my data to elastic search using java library.

I need to keep some json data in the field, so some field is set to object type like below.
"bid_request.request.body":{"type":"object"},
...
"bid_request.response.body":{"type":"object"}

I put my data using below code
IndexRequestBuilder indexBuilder = transportClient.prepareIndex("inventory","logs", "12345");
IndexResponse response = indexBuilder.setSource(aggregateData.toString()).get();

and I got MapperParsingException from some of fields that I set as object.

Exception in thread "main" org.elasticsearch.index.mapper.MapperParsingException: failed to parse [bids.bid_request.response.body.bid_response.no_bid]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)

Caused by: org.elasticsearch.common.jackson.core.JsonParseException: Current token (VALUE_TRUE) not numeric, can not use numeric value accessors
at [Source: UNKNOWN; line: 1, column: 26576]
at org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1487)
at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
at org.elasticsearch.common.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:800)
at org.elasticsearch.common.jackson.core.base.ParserBase.getLongValue(ParserBase.java:671)
at org.elasticsearch.common.xcontent.json.JsonXContentParser.doLongValue(JsonXContentParser.java:173)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:147)
at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)

In the message, error might be raised during json parsing using jackson and my data is JsonNode type in Jackson library.
I'm not sure, but error could be prevent if I put the jackson json object instead of put serialized string to elastic java library.
Is there way use Jackson JsonNode object for indexing data ?

Thanks
Ducheol

2

May be you can share your mapping and the doc you are sending or the code you are using to generate the doc?

3

Below is my mappings.

{
	"mappings" : {
		"logs" : {
			"properties":{
				"trace_id" : {"type": "long"},
				"request.timestamp" : {"type":"long"},
			    "response.headers": {"type":"object"},
			    "response.body": {"type":"string"},
				"bids":{
					"type": "nested",
					"properties":{
						"bid_request.host":{"type":"string"},
						"bid_request.request.uri":{"type":"string"},
						"bid_request.request.headers":{"type":"object"},
						"bid_request.request.body":{"type":"object"},
						"bid_request.response.headers":{"type":"object"},
						"bid_request.response.body":{"type":"object"}
					}
				}
			}
		}
	}
}

You can download doc data from below link
https://dl.dropboxusercontent.com/u/32218991/input.json
( I can't write here because of max character restriction - 5000 character )
( I updated some critical info - IP, url ) to dummy value, but other is exactly same )

And below is stack tract message.

Thanks
Ducheol

Exception in thread "main" org.elasticsearch.index.mapper.MapperParsingException: failed to parse [bids.bid_request.response.body.bid_response.no_bid]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:487)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:487)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeNonDynamicArray(ObjectMapper.java:685)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:604)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:489)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:493)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:192)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.jackson.core.JsonParseException: Current token (VALUE_TRUE) not numeric, can not use numeric value accessors
at [Source: UNKNOWN; line: 1, column: 15233]
at org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1487)
at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
at org.elasticsearch.common.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:800)
at org.elasticsearch.common.jackson.core.base.ParserBase.getLongValue(ParserBase.java:671)
at org.elasticsearch.common.xcontent.json.JsonXContentParser.doLongValue(JsonXContentParser.java:173)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:147)
at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
... 20 more

4

Some comments:

  • in elasticsearch 2.0, you can not use dots in field names anymore
  • some inner objects are conflicting. For example no_bid is first set to 0 then to true which causes conflicts.

Please try to adjust your mapping and provide a full script recreation such as:

PUT logs
PUT logs/logs/_mapping
{
  "logs": {
    "properties": {
      "trace_id": {
        "type": "long"
      },
      "timestamp": {
        "type": "long"
      },
      "headers": {
        "type": "object"
      },
      "body": {
        "type": "string"
      },
      "bids": {
        "type": "nested",
        "properties": {
          "host": {
            "type": "string"
          },
          "uri": {
            "type": "string"
          },
          "reqheaders": {
            "type": "object"
          },
          "reqbody": {
            "type": "object"
          },
          "respheaders": {
            "type": "object"
          },
          "respbody": {
            "type": "object"
          }
        }
      }
    }
  }
}
PUT /logs/logs/1
{
    // YOUR DOC HERE
}

It will be then easier to find the issue (and probably you'll be able to fix it by yourself).

Also, please upgrade to 2.0 if you are starting a project or at least read all breaking changes in 2.0 so you won't have to reindex when you will upgrade.

5

Thanks for answer.

I have one more question.

Has we had same json schema for inner object ?
This data is coming from different companies, so it can't has same schema.
Is there any alternative way to save whole json data in a field ?

Regards
Ducheol

6

Yes as long you don't need to index and search for its content.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html

7

Thanks for the answer, but we need to search for that data. :sweat:

8

This will be hard then.

If the same data can be either true or 45, it will be hard to search for it, right?
You really ned to control your mapping and make some decisions IMO.

9

Got it.

How about mapped as string instead of object and put encoded json string like below ?

            "bidder_response-body": "{\"bid_response\":{\"responses\":[{\"member_id\":104,\"exclusive\":false,\"no_bid\":true,\"no_price_reduction\":false,\"price\":0.0,\"bidder_revshare\":0.0,\"bidder_minimum_cpm\":0.0,\"creative_id\":0,\"custom_macros\":[],\"buyer_currency\":\"USD\",\"auction_id_64\":7475650378184842386,\"pixel_type\":\"null\"}]}}"

I put the data successfully, can I search it using value in the json data ? ( e.g : no_bid : true ) )

Thanks
Ducheol

10

May be if you find the right analyzer but I doubt it's an easy task.

Searching for what you proposed is super hard IMO if you don't index fields properly.

11

Got it.

Thanks for helping me.

Regards
Ducheol