Mapper_parsing_exception with JSON event in custom Beat

Elastic Stack Beats
1

I am writing a Beat that generates the following event and sends to ES. I have included a sample event below. The data does not make it into ES - there is an error occurring:

WARN Can not index event (status=400): {"type":"mapper_parsing_exception",
"reason":"failed to parse [weather]",
"caused_by":{"type":"illegal_argument_exception",
"reason":"unknown property [cod]"}}

I didn't define a mapping because I assumed it would dynamically add any unknown properties. Any help would be much appreciated!

{
  "@timestamp": "2016-09-14T05:17:29.406Z",
  "beat": {
    "hostname": "9a0500dc0bb5",
    "name": "9a0500dc0bb5"
  },
  "type": "weatherbeat",
  "weather": {
    "cod": "200",
    "count": 1,
    "list": [
      {
        "clouds": {
          "all": 0
        },
        "coord": {
          "lat": 40.71455,
          "lon": -74.00714
        },
        "dt": 1.473830208e+09,
        "id": 0,
        "main": {
          "humidity": 62,
          "pressure": 1019.6,
          "temp": 69.51,
          "temp_max": 73,
          "temp_min": 66
        },
        "name": "New York",
        "sys": {
          "country": "United States"
        },
        "weather": [
          {
            "description": "Sky is Clear",
            "icon": "01n",
            "id": 800,
            "main": "Clear"
          }
        ],
        "wind": {
          "deg": 213.006,
          "speed": 9.63
        }
      }
    ],
    "message": "accurate"
  }
}
2

Happy to hear you are building the Weatherbeat. Looking forward to have another community Beat!

You need to tell Elasticsearch in advance what type each field is, especially for special types. If you don't define the mappings, then Elasticsearch will try to guess the types based on the first data pushed. For example, if you have a float field, but the first value that it's pushed to Elasticsearch is integer, then Elasticsearch will consider it as integer. When you push a float value, Elasticsearch will throw an exception as it expects to be integer, and not float. To avoid these problems, it's better to define the type of each field beforehand. Starting with 5.0, you just need to define the type of each field under etc/fields.yml, and then run make update in your Beat. This will generate the index template for you, that it's loaded automatically when the Beat starts.

NOTE: Please make sure to delete the current index from Elasticsearch with the old values before generating the new index template, and starting the Beat.

Did you use the Beat generator to generate your Beat?

3

@monica Thanks for the reply! I did use Beat generator, but underestimated the importance of fields.yml. I was trying to define all the fields but could not find any docs on how to properly use that file. Instead, I added a placeholder in fields.yml and removed any actual fields:

- key: weatherbeat
  title: weatherbeat
  description:
  fields:
   - name: test
     type: text

That seemed to cause the dynamic template to work, which is actually what I was trying to do. The weatherbeat I wrote is just for practice. I am trying to write a beat to talk to a Hadoop cluster and gather JMX metrics, which come back in many different formats, so trying to define a template would be very tedious. The dynamic part is working now though, so thanks!

4

This topic was automatically closed after 21 days. New replies are no longer allowed.