# Elasticsearch Error: java.lang.IllegalArgumentException

**URL:** <https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 30, 2018, 6:16pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265 "2018-03-30T18:16:23Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![vopsec](https://avatars.discourse-cdn.com/v4/letter/v/5f9b8f/32.png) [@vopsec](https://discuss.elastic.co/u/vopsec)\
**Post date:** [March 30, 2018, 6:16pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/1 "2018-03-30T18:16:23Z")

</div>

Hi,

Several days ago, my filebeat agent started hiccuping with the following error:

```auto
2018-03-30T14:02:55.027-0400 DEBUG [elasticsearch] elasticsearch/client.go:507 Bulk item insert failed (i=0, status=500
): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [destination-ip]
not present as part of path [json.destination-ip]","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalA
rgumentException: field [destination-ip] not present as part of path [json.destination-ip]","caused_by":{"type":"illegal_argumen
t_exception","reason":"field [destination-ip] not present as part of path [json.destination-ip]"}},"header":{"processor_type":"g
eoip"}}

```

It had been working fine for about a month until then. I don't remember changing anything either so I'm at a loss as to what would cause such an error.

I've tried going into `/etc/filebeat/filebeat.yml` and lowering the `bulk_max_size` to 10.  
I've also tried setting `json.keys_under_root: true`.  
I've tried disabling the `pipeline: geoip` pipeline as well.

None of the above worked, unfortunately.

Does anyone happen to have any insight as to what causes this error and perhaps some troubleshooting methods to fix it?

I'd like to keep all of the data already shipped during the past month in the index but if it must be that I have to create a new index or scrap the data, then it must be.

My `/etc/filebeat/filebeat.yml` is below:

```auto
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- type: log

  enabled: true
  paths:
    - /var/log/data/data-out/*
  json.add_error_key: true
  json.message_key: log
  #json.keys_under_root: true

  pipeline: geoip

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.enabled: false
setup.template.name: "my-filebeat"
setup.template.pattern: "my-filebeat-*"

#================================ General =====================================

name: my-filebeat

#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["10.10.10.5:9200"]

  protocol: "http"

  index: "u2elk"

  pipeline: geoip

  bulk_max_size: 10

#================================ Logging =====================================

logging.level: debug

```

Example data being shipped:  
`{"dport-icode": 1000, "vlan-id": null, "destination-ip": "10.255.12.21", "source-ip": "8.8.4.4", "sensor-id": "5, "event-second": 1519907325, "data": "TEST DATA", "data-printable": "test data", "msg": "Firewall Client long host entry", "event-microsecond": 341757}`

Example PUT:

```auto
  "prospector": {
    "type": "log"
  }
}
2018-03-30T14:20:51.457-0400 DEBUG [event] common/event.go:55 Dropped nil value from event where key=json.vlan-id
2018-03-30T14:20:51.457-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {
  "@timestamp": "2018-03-30T18:20:51.455Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.3",
    "pipeline": "geoip"
  },
  "beat": {
    "name": "my-filebeat",
    "hostname": "test-VM",
    "version": "6.2.3"
  },
  "source": "/var/log/data/data-out/events.json-20183827",
  "offset": 1612599,
  "json": {
    "classification": "Attempted User Privilege Gain",
    "event-microsecond": 341757,
    "dport-icode": 1000,
    "error": {
      "type": "json",
      "message": "Key 'log' not found"
    },
    "msg": "Firewall Client long host entry",
    "source-ip": "8.8.4.4",
    "data": "TEST DATA",
    "sensor-id": "5",
    "log": "",
    "destination-ip": "10.255.15.27",
    "event-second": 1519907325,
    "data-printable": "test data"
  },
  "prospector": {
    "type": "log"
  }
}

```

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 6:55pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/2 "2018-03-30T18:55:58Z")

</div>

Hello @vopsec, I think that you are using a custom ingest pipeline, from what I see this error is coming from the `ingest node` and the [geoip processor](https://www.elastic.co/guide/en/elasticsearch/plugins/master/using-ingest-geoip.html), maybe some of your data doesn't have the `destination-ip` and by default the processor will do a hard fail.

You can either make sure your data always have that field if possible or ignore the error by configuring the options `ignore_missing` and set it to **true**

---

<div class="post-metadata">

**Author:** ![vopsec](https://avatars.discourse-cdn.com/v4/letter/v/5f9b8f/32.png) [@vopsec](https://discuss.elastic.co/u/vopsec)\
**Post date:** [March 30, 2018, 8:35pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/3 "2018-03-30T20:35:09Z")

</div>

Bless you, pierhugues! That did the trick! Thank you for the quick reply! I appreciate it!

I'm still trying to learn the jargon of Elasticsearch and its inner workings. Would you happen to have a theory as to why this would happen all of a sudden after working fine for a month? All the data now flooding in contains `json.destination-ip`.

Do plugins such as `ingest-geoip` update automatically?

For posterity, I did:  
`curl -H "Content-Type: application/json" -XPUT 'http://10.10.10.5:9200/_ingest/pipeline/geoip?pretty' -d @new-geoip-plugin.json`  
with `new-geoip-plugin.json` containing:

```auto
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "json.destination-ip",
        "target_field" : "geoip.destination-ip",
        "ignore_missing": true
      }
    },
    {
      "geoip" : {
        "field" : "json.source-ip",
        "target_field" : "geoip.source-ip",
        "ignore_missing": true
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 9:02pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/4 "2018-03-30T21:02:36Z")

</div>

Good!

I would think the producers had changed format? If nothing was changed on elasticsearch or filebeat. This is the only thing I could think of.

Plugins have to be updated when you update elasticsearch, new version of the plugins are released at the same time of es.

---

<div class="post-metadata">

**Author:** ![vopsec](https://avatars.discourse-cdn.com/v4/letter/v/5f9b8f/32.png) [@vopsec](https://discuss.elastic.co/u/vopsec)\
**Post date:** [March 30, 2018, 9:08pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/5 "2018-03-30T21:08:24Z")

</div>

How bizarre...

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [March 30, 2018, 9:43pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/6 "2018-03-30T21:43:08Z")

</div>

Did you do an update on filebeat and you started to have problems?

---

<div class="post-metadata">

**Author:** ![vopsec](https://avatars.discourse-cdn.com/v4/letter/v/5f9b8f/32.png) [@vopsec](https://discuss.elastic.co/u/vopsec)\
**Post date:** [March 30, 2018, 10:41pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/7 "2018-03-30T22:41:33Z")

</div>

I did not- at least, not manually. Hmmm...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2018, 10:41pm UTC](https://discuss.elastic.co/t/elasticsearch-error-java-lang-illegalargumentexception/126265/8 "2018-04-27T22:41:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.