Need advice on my Open Source project using elastic search and question on MapperParsingException

Elastic Stack Elasticsearch
1

Hi all,

New to ES here. I am currently working on an open source project called PoE Merchant.

It is a "forum shops" indexer for a game called Path of Exile.

I have an issue and a question below. But I would really appreciate any further advice on how should I proceed.

Project goals:

  1. Index all the "Items for sale" in Path of Exile forums shops (e.g. http://www.pathofexile.com/forum/view-forum/561).
    1. At least finish re/indexing every 15mins.
  2. Run application as a stand-alone Desktop (I'll use Java SE).
  3. Provide a Lightning fast search in a very convenient way.
  4. Provide some other Quality of Life features.

So far, my setup is:

  1. Windows 7
  2. Eclipse Luna 4.4.1
  3. Jsoup for parsing the HTML.
  4. ES 1.4
    1. start ES via elasticsearch.bat

More context:

  1. Path of Exile is a Free to Play Action RPG - it's like Diablo 3 but for hardcore gamers and it is not Pay to Win.
  2. Path of Exile only provides the forums for player to setup "Shops".
  3. A player makes a forum thread under on the shop subforums - 1 subforum will be equivalent to 1 Index.
  4. A thread contains 1 or more "Item for sale" - this will be my Document
  5. A player will update this shop thread and add new Items - this is why i need to re-index every x minutes.
  6. Here is a sample shop: http://www.pathofexile.com/forum/view-thread/1127160
    1. The bulk of the data I'll need to index is located in the last tag.

Problem currently:

The data I'm gonna be scraping from the forums is fortunately contained in a . Currently I am trying things out and I encountered this issue am I curious of:

The following is valid JSON data, but am I getting a org.elasticsearch.index.mapper.MapperParsingException: failed to parse [x.user]

{

"user": [

    0,

    {

        "x": 1

    }

]

}

Why can't elasticsearch index this? Is there a guide on what JSON
format/structure to use for ES?
Given that one of my goals is fast index and fast search., are there any
considerations regarding structuring my data?

Full stack trace:

org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[x.user]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:415)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:555)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeNonDynamicArray(ObjectMapper.java:686)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:623)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.putDynamicMapper(ObjectMapper.java:661)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:581)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:490)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:541)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:490)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:392)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:198)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:511)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: unknown
property [x]
at
org.elasticsearch.index.mapper.core.LongFieldMapper.innerParseCreateField(LongFieldMapper.java:290)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:235)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:405)
... 16 more

Thanks a lot!
Thirdy

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/229d5b30-4e8c-4f30-b0ae-2492524e9290%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

sorry about the formatting, I pasted the JSON first, and the color went
yellow. whoops.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/70a509b3-59c0-4e8c-b672-64ecfa0e71be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Code to reproduce the exception:

@Test
public void testESIndex() throws Exception {
String json = "{" +
""user":[0, {"x":1}]" +
"}";
Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress(
"localhost", 9300));
IndexResponse indexResponse = index(client, "a", "y", "{"x":" + json +
"}");
logger.info(indexResponseToString(indexResponse));
client.close();
}
public static IndexResponse index(Client client, String index, String
type, String json) {
IndexResponse actionGet = client.prepareIndex(index, type)
.setSource(json).execute().actionGet();
return actionGet;
}

public static String indexResponseToString(IndexResponse actionGet) {
return actionGet.getIndex() + "/" + actionGet.getType() + "/"

  • actionGet.getId() + " isCreated: " + actionGet.isCreated();
    }

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b558488-1064-4056-b666-25e6226fdd66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Unsure but your JSON looks incorrect to me. You are trying to send in an array a number 0 and an object.

David

Le 5 févr. 2015 à 22:53, Vicente de Rivera III thirdy.derivera@gmail.com a écrit :

Code to reproduce the exception:

@Test
public void testESIndex() throws Exception {
String json = "{" +
""user":[0, {"x":1}]" +
"}";
Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress(
"localhost", 9300));

  IndexResponse indexResponse = index(client, "a", "y", "{\"x\":" + json + "}");
  
  logger.info(indexResponseToString(indexResponse));
  
  client.close();

}

public static IndexResponse index(Client client, String index, String type, String json) {
IndexResponse actionGet = client.prepareIndex(index, type)
.setSource(json).execute().actionGet();
return actionGet;
}

public static String indexResponseToString(IndexResponse actionGet) {
return actionGet.getIndex() + "/" + actionGet.getType() + "/"
+ actionGet.getId() + " isCreated: " + actionGet.isCreated();
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b558488-1064-4056-b666-25e6226fdd66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/626A9565-D5E7-4488-AF36-11F6E15C0D9E%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

5

Thank you for the response.

Both jsonlint.com and JSON Validator - JSONLint tool to validate JSON data says this JSON
is valid:

{
"x": {
"user": [
0,
{
"x": 1
}
]
}
}

I guess I'll parse that raw js data, build POJOs out of it and then pass
these pojos into ES. (Was hoping I could dump this data into Elasticsearch
without any trouble).

Also, sorry to ask this, is Elastic search a good choice for my goals? I've
never used any search technology before.

On Friday, February 6, 2015 at 6:02:40 AM UTC+8, David Pilato wrote:

Unsure but your JSON looks incorrect to me. You are trying to send in an
array a number 0 and an object.

David

Le 5 févr. 2015 à 22:53, Vicente de Rivera III <thirdy....@gmail.com
<javascript:>> a écrit :

Code to reproduce the exception:

@Test
public void testESIndex() throws Exception {
String json = "{" +
""user":[0, {"x":1}]" +
"}";
Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress(
"localhost", 9300));
IndexResponse indexResponse = index(client, "a", "y", "{"x":" + json +
"}");
logger.info(indexResponseToString(indexResponse));
client.close();
}
public static IndexResponse index(Client client, String index, String
type, String json) {
IndexResponse actionGet = client.prepareIndex(index, type)
.setSource(json).execute().actionGet();
return actionGet;
}

public static String indexResponseToString(IndexResponse actionGet) {
return actionGet.getIndex() + "/" + actionGet.getType() + "/"

  • actionGet.getId() + " isCreated: " + actionGet.isCreated();
    }

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1b558488-1064-4056-b666-25e6226fdd66%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1b558488-1064-4056-b666-25e6226fdd66%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9358cf2e-c72e-4455-9b45-e7b443a124db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

It's valid in term of JS not for Elasticsearch.
I mean that Elasticsearch needs to create a mapping for each field.

Your field user can not be at the same time a number and an object.
You can write this (pseudo JSON):

User : {
age : 10,
Name: Joe
}

Or this:

User : [{
age : 10,
Name: Joe
},{
age : 12,
Name: John
}]

HTH

David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 6 févr. 2015 à 01:26, Vicente de Rivera III thirdy.derivera@gmail.com a écrit :

Thank you for the response.

Both jsonlint.com and JSON Validator - JSONLint tool to validate JSON data says this JSON is valid:

{
"x": {
"user": [
0,
{
"x": 1
}
]
}
}

I guess I'll parse that raw js data, build POJOs out of it and then pass these pojos into ES. (Was hoping I could dump this data into Elasticsearch without any trouble).

Also, sorry to ask this, is Elastic search a good choice for my goals? I've never used any search technology before.

On Friday, February 6, 2015 at 6:02:40 AM UTC+8, David Pilato wrote:
Unsure but your JSON looks incorrect to me. You are trying to send in an array a number 0 and an object.

David

Le 5 févr. 2015 à 22:53, Vicente de Rivera III thirdy....@gmail.com a écrit :

Code to reproduce the exception:

@Test
public void testESIndex() throws Exception {
String json = "{" +
""user":[0, {"x":1}]" +
"}";
Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress(
"localhost", 9300));

  IndexResponse indexResponse = index(client, "a", "y", "{\"x\":" + json + "}");
  
  logger.info(indexResponseToString(indexResponse));
  
  client.close();

}

public static IndexResponse index(Client client, String index, String type, String json) {
IndexResponse actionGet = client.prepareIndex(index, type)
.setSource(json).execute().actionGet();
return actionGet;
}

public static String indexResponseToString(IndexResponse actionGet) {
return actionGet.getIndex() + "/" + actionGet.getType() + "/"
+ actionGet.getId() + " isCreated: " + actionGet.isCreated();
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1b558488-1064-4056-b666-25e6226fdd66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9358cf2e-c72e-4455-9b45-e7b443a124db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0AA64161-1385-4DDE-957B-8D7B63495A8D%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.