# Problem with transfer Filebeat 6.1.3 \> Logstash 6.1.3 \> Elasticsearch 6.1.3

**URL:** https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264
**Category:** Logstash
**Created:** [June 18, 2018, 9:03am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264 "2018-06-18T09:03:15Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![artful](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@artful](https://discuss.elastic.co/u/artful)
#### Post date: [June 18, 2018, 9:03am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/1 "2018-06-18T09:03:16Z")

</div>

Hello!  
Transfer log `Filebeat > Elasticsearch` without Logstash work fine.  
When I add Logstash, I see in log:  
`[2018-06-18T08:42:00,801][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-tmp-pg_postgresql-9.5-main.log-2018.06.18", :_type=>"doc", :_routing=>nil, :pipeline=>"filebeat-6.1.1-postgresql-log-pipeline"}, #<LogStash::Event:0x5c3141ac>], :response=>{"index"=>{"_index"=>"filebeat-tmp-pg_postgresql-9.5-main.log-2018.06.18", "_type"=>"doc", "_id"=>"JbgOEmQBroNl8DFs2kuj", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}}`

In index mapping for field `host`:

```
  "host": {
    "properties": {
      "name": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "architecture": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "id": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "os": {
        "properties": {
          "family": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "platform": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  }

```

I don't understand how resolve this problem 😥. Please, help me!

---

<div class="post-metadata">

### Author: ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [June 18, 2018, 10:48pm UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/2 "2018-06-18T22:48:13Z")

</div>

The error message comes from Elasticsearch, and is occurring because something is attempting to index a document whose `host` value is a concrete value (likely a string), when the index expects an object (from your mapping, it looks like it should have sub-keys like `name`, `architecture`, `id`, and `os`).

What does your Logstash pipeline configuration look like? If you include a `stdout` output plugin using the (poorly named) `rubydebug` codec, what do your events look like?

```auto
output {
  elasticsearch {
    # ...
  }
  stdout {
    codec => rubydebug
  }
}

```

---

<div class="post-metadata">

### Author: ![artful](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@artful](https://discuss.elastic.co/u/artful)
#### Post date: [June 20, 2018, 1:57pm UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/3 "2018-06-20T13:57:34Z")

</div>

Maybe I can disable sending field `host`?  
My logstash config: /etc/logstash/conf.d/filebeat.conf

```
input {
  beats {
    port => 5044
  }
 }

output {
    elasticsearch {
        hosts => ["elastic1.server.com:9200" , "elastic2.server.com:9200" , "elastic3.server.com:9200"]
        index => "filebeat-tmp-pg_postgresql-9.5-main.log-%{+YYYY.MM.dd}"
        pipeline => "filebeat-6.1.1-postgresql-log-pipeline"
    }
}
```

---

<div class="post-metadata">

### Author: ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [June 21, 2018, 5:43pm UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/4 "2018-06-21T17:43:51Z")

</div>

It looks like your index was created with a new version of Beats that is implementing the ECS (Elastic Common Schema), but that the Logstash output isn't quite in the same shape 😩

ECS has a `host.name` field, which corresponds to the host name being output from Logstash. By adding a Mutate filter with a `rename` directive, you can move the `host` field to `host.name` with the field-reference syntax as below to align with the schema that you already have in Elasticsearch:

```auto
filter {
  mutate {
    rename {
      "[host]" => "[host][name]"
    }
  }
}

```

---

<div class="post-metadata">

### Author: ![artful](https://avatars.discourse-cdn.com/v4/letter/a/c57346/32.png) [@artful](https://discuss.elastic.co/u/artful)
#### Post date: [June 23, 2018, 10:42am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/5 "2018-06-23T10:42:25Z")

</div>

Can I update my Logstash to version, which support field `host.name`?

---

<div class="post-metadata">

### Author: ![mpucholblasco](https://avatars.discourse-cdn.com/v4/letter/m/cc9497/32.png) [@mpucholblasco](https://discuss.elastic.co/u/mpucholblasco)
#### Post date: [July 3, 2018, 8:29am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/6 "2018-07-03T08:29:40Z")

</div>

Same problem here.

I'm using FileBeat (6.1.3 & 6.3.0) -\> Logstash 6.3.0 -\> ElasticSearch 6.3.0.

FileBeat 6.1.3 sends host information as string -\> e.g. `"host":"ip-10-1-100-16"`  
meanwhile FileBeat 6.3.0 sends host information as object -\> e.g. `"host": { "name": "ip-10-1-100-230" }`

In my case, as I also have beats information `( "beat": { "hostname": "ip-10-1-100-230" } }`, I can ignore host information via:

```
filter {
  mutate {
    remove_field => ["host"]
  }
}

```

However, if you want to convert previous content into new one, you can do something similar to:

```
filter {
  if [beat][version] < "6.3" {
    mutate {
      replace => { "host" => { "name" => "%{host}" } }
    }
    json_encode {
      source => "host"
    }
  }
}
```

---

<div class="post-metadata">

### Author: ![pavan.idm](https://avatars.discourse-cdn.com/v4/letter/p/8c91f0/32.png) [@pavan.idm](https://discuss.elastic.co/u/pavan.idm)
#### Post date: [July 19, 2018, 3:31am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/7 "2018-07-19T03:31:42Z")

</div>

I also have the issue and following are the details of the issue.

```
We have upgraded filebeat and logstash to 6.3 as part of patching. 
#logstash -V
logstash 6.3.0

#filebeat version
filebeat version 6.3.0 (amd64), libbeat 6.3.0

we noticed that the format of host through logstash has changed as follows:
e.g. "host":"xxx.ood.ops" to "host": { "name": "xxx.ood.ops" }

After doing the upgrade we are getting the following errors
 In logstash:
 Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"3320728454", :_index=>"logstash-2018.07.18", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x4e8bfcd8>], :response=>{"index"=>{"_index"=>"logstash-2018.07.18", "_type"=>"doc", "_id"=>"3320728454", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}}

 In elasticsearch:
 [2018-07-18T17:01:04,903][DEBUG][o.e.a.b.TransportShardBulkAction] [logstash-2018.07.18][1] failed to execute bulk item (index) BulkShardRequest [[logstash-2018.07.18][1]] containing [11] requests

```

org.elasticsearch.index.mapper.MapperParsingException: object mapping for [host] tried to parse field [host] as object, but found a concrete value

```
 The json format as seen in kibana:
 {

```

"\_index": "logstash-2018.07.18",  
"\_type": "doc",  
"\_id": "2114191840",  
"\_version": 102181,  
"\_score": null,  
"\_source": {  
"prospector": {  
"type": "log"  
},  
"host": {  
"name": "xxx.ood.ops"  
},  
"source": "/var/log/sample.out",  
"beat": {  
"name": "xxx.ood.ops",  
"hostname": "xxx.ood.ops",  
"version": "6.3.0"  
},  
"input": {  
"type": "log"  
}

I tried the solution of renaming and following is the error:

[2018-07-18T16:01:10,186][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>"4218739081", :\_index=\>"logstash-2018.07.18", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x14e8812e], :response=\>{"index"=\>{"\_index"=\>"logstash-2018.07.18", "\_type"=\>"doc", "\_id"=\>"4218739081", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse [host.name]", "caused\_by"=\>{"type"=\>"illegal\_state\_exception", "reason"=\>"Can't get text on a START\_OBJECT at 1:57"}}}}}

Can you please help ?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 19, 2018, 8:37pm UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/8 "2018-07-19T20:37:51Z")

</div>

What error do you see in the elasticsearch log when you see that error in the logstash log?

---

<div class="post-metadata">

### Author: ![pavan.idm](https://avatars.discourse-cdn.com/v4/letter/p/8c91f0/32.png) [@pavan.idm](https://discuss.elastic.co/u/pavan.idm)
#### Post date: [July 20, 2018, 3:03am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/9 "2018-07-20T03:03:55Z")

</div>

hi,  
The elasticsaearch error is also right below the logstash error in my previous post.

As a work around we renamed host.name to hostname for now as the indexing is failing on [host] field.

mutate {  
rename =\> { "[host][name]" =\> "[hostname]" }  
}

Let me know if there is any other way.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 17, 2018, 3:04am UTC](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/10 "2018-08-17T03:04:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
