Put Jackson JsonNode object for indexing

Ducheol_Kim · November 11, 2015, 10:56pm

Hi All.

I'm trying to put my data to elastic search using java library.

I need to keep some json data in the field, so some field is set to object type like below.
"bid_request.request.body":{"type":"object"},
...
"bid_request.response.body":{"type":"object"}

I put my data using below code
IndexRequestBuilder indexBuilder = transportClient.prepareIndex("inventory","logs", "12345");
IndexResponse response = indexBuilder.setSource(aggregateData.toString()).get();

and I got MapperParsingException from some of fields that I set as object.

Exception in thread "main" org.elasticsearch.index.mapper.MapperParsingException: failed to parse [bids.bid_request.response.body.bid_response.no_bid]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)

Caused by: org.elasticsearch.common.jackson.core.JsonParseException: Current token (VALUE_TRUE) not numeric, can not use numeric value accessors
at [Source: UNKNOWN; line: 1, column: 26576]
at org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1487)
at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
at org.elasticsearch.common.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:800)
at org.elasticsearch.common.jackson.core.base.ParserBase.getLongValue(ParserBase.java:671)
at org.elasticsearch.common.xcontent.json.JsonXContentParser.doLongValue(JsonXContentParser.java:173)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:147)
at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)

In the message, error might be raised during json parsing using jackson and my data is JsonNode type in Jackson library.
I'm not sure, but error could be prevent if I put the jackson json object instead of put serialized string to elastic java library.
Is there way use Jackson JsonNode object for indexing data ?

Thanks
Ducheol

dadoonet · November 12, 2015, 3:20am

May be you can share your mapping and the doc you are sending or the code you are using to generate the doc?

Ducheol_Kim · November 12, 2015, 3:25pm

Below is my mappings.

{
	"mappings" : {
		"logs" : {
			"properties":{
				"trace_id" : {"type": "long"},
				"request.timestamp" : {"type":"long"},
			    "response.headers": {"type":"object"},
			    "response.body": {"type":"string"},
				"bids":{
					"type": "nested",
					"properties":{
						"bid_request.host":{"type":"string"},
						"bid_request.request.uri":{"type":"string"},
						"bid_request.request.headers":{"type":"object"},
						"bid_request.request.body":{"type":"object"},
						"bid_request.response.headers":{"type":"object"},
						"bid_request.response.body":{"type":"object"}
					}
				}
			}
		}
	}
}

You can download doc data from below link
https://dl.dropboxusercontent.com/u/32218991/input.json
( I can't write here because of max character restriction - 5000 character )
( I updated some critical info - IP, url ) to dummy value, but other is exactly same )

And below is stack tract message.

Thanks
Ducheol

Exception in thread "main" org.elasticsearch.index.mapper.MapperParsingException: failed to parse [bids.bid_request.response.body.bid_response.no_bid]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:487)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:487)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeNonDynamicArray(ObjectMapper.java:685)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:604)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:489)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:493)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:192)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.jackson.core.JsonParseException: Current token (VALUE_TRUE) not numeric, can not use numeric value accessors
at [Source: UNKNOWN; line: 1, column: 15233]
at org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1487)
at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
at org.elasticsearch.common.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:800)
at org.elasticsearch.common.jackson.core.base.ParserBase.getLongValue(ParserBase.java:671)
at org.elasticsearch.common.xcontent.json.JsonXContentParser.doLongValue(JsonXContentParser.java:173)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:147)
at org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:288)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:239)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
... 20 more

dadoonet · November 12, 2015, 4:27pm

Some comments:

in elasticsearch 2.0, you can not use dots in field names anymore
some inner objects are conflicting. For example no_bid is first set to 0 then to true which causes conflicts.

Please try to adjust your mapping and provide a full script recreation such as:

PUT logs
PUT logs/logs/_mapping
{
  "logs": {
    "properties": {
      "trace_id": {
        "type": "long"
      },
      "timestamp": {
        "type": "long"
      },
      "headers": {
        "type": "object"
      },
      "body": {
        "type": "string"
      },
      "bids": {
        "type": "nested",
        "properties": {
          "host": {
            "type": "string"
          },
          "uri": {
            "type": "string"
          },
          "reqheaders": {
            "type": "object"
          },
          "reqbody": {
            "type": "object"
          },
          "respheaders": {
            "type": "object"
          },
          "respbody": {
            "type": "object"
          }
        }
      }
    }
  }
}
PUT /logs/logs/1
{
    // YOUR DOC HERE
}

It will be then easier to find the issue (and probably you'll be able to fix it by yourself).

Also, please upgrade to 2.0 if you are starting a project or at least read all breaking changes in 2.0 so you won't have to reindex when you will upgrade.

Ducheol_Kim · November 12, 2015, 4:39pm

Thanks for answer.

I have one more question.

Has we had same json schema for inner object ?
This data is coming from different companies, so it can't has same schema.
Is there any alternative way to save whole json data in a field ?

Regards
Ducheol

dadoonet · November 12, 2015, 5:00pm

Yes as long you don't need to index and search for its content.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html

Ducheol_Kim · November 12, 2015, 5:59pm

Thanks for the answer, but we need to search for that data.

dadoonet · November 12, 2015, 8:09pm

This will be hard then.

If the same data can be either true or 45, it will be hard to search for it, right?
You really ned to control your mapping and make some decisions IMO.

Ducheol_Kim · November 13, 2015, 7:01pm

Got it.

How about mapped as string instead of object and put encoded json string like below ?

            "bidder_response-body": "{\"bid_response\":{\"responses\":[{\"member_id\":104,\"exclusive\":false,\"no_bid\":true,\"no_price_reduction\":false,\"price\":0.0,\"bidder_revshare\":0.0,\"bidder_minimum_cpm\":0.0,\"creative_id\":0,\"custom_macros\":[],\"buyer_currency\":\"USD\",\"auction_id_64\":7475650378184842386,\"pixel_type\":\"null\"}]}}"

I put the data successfully, can I search it using value in the json data ? ( e.g : no_bid : true ) )

Thanks
Ducheol

dadoonet · November 13, 2015, 8:39pm

May be if you find the right analyzer but I doubt it's an easy task.

Searching for what you proposed is super hard IMO if you don't index fields properly.

Ducheol_Kim · November 13, 2015, 10:25pm

Got it.

Thanks for helping me.

Regards
Ducheol