Failed to parse field [host]

Elastic Stack Logstash
1

I am getting below error. May I know the reason and resolve the issue.
Below is the logstash config.
++++++++++++++++++++++++++===
[2020-05-25T09:35:25,844][WARN ][logstash.outputs.elasticsearch][main][17dee1c8bb9821d81ac5742c7b7489e6b8d06baae3ae3e3e1d8382fd50e0d21f] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{[@metadata][target_index]}", :routing=>nil, :_type=>"_doc"}, #LogStash::Event:0x4b7a6a4e], :response=>{"index"=>{"_index"=>"%{[@metadata][target_index]}", "_type"=>"_doc", "_id"=>"JSMBSnIBvfTbv1fvtlfK", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'JSMBSnIBvfTbv1fvtlfK'. Preview of field's value: '{hostname=INHY-PAPP-ELK01, os={kernel=4.18.0-147.8.1.el8_1.x86_64, codename=Core, name=CentOS Linux, family=redhat, version=8 (Core), platform=centos}, containerized=false, ip=[10.252.10.75, fe80::d4ee:d927:5185:8d0], name=INHY-PAPP-ELK01, id=e1cebd3d12bc4510bdecafd61726096c, mac=[00:15:5d:10:0b:62], architecture=x86_64}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:182"}}}}}

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
input {
  beats {
    port => 5045
  }
}
filter {
geoip {
      source => "destination.ip"
    }
}
output {
  elasticsearch {
    hosts => ["http://10.252.10.76:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"
  }
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2

That does not look like a hostname.

However if you are using Beats to send to Logstash, and all you use ls for is geoip, then why not use an ingest pipeline in Elasticsearch? It'll save on complexity.

3

filebeat will use ECS convention by default , so you will have host information nested under host such as [host][hostname]

logstash generates a host field with a text datatype by default.

if you use filebeat index pattern, the target template will be using ecs compliant mapping, where host is an object, not a text. hence the error.

if you still want to use logstash, you will need to change the host field into object type.

4

How do i convert host filed to Object. I didnt found "object" option in filter->mutate-covert.

5

usually just rename the field

mutate { rename => { "host" => "[host][name]" } }

however, looking at this

your beat already provides [host][name] which matches to the hostname of the beat. what do you want the value of [host][name] to be? also can you provide the logstash stdout as an example?

as @warkolm pointed out, if all you need is geoip, this can be done easily with ingest node on elasticsearch, so you don't really need Logstash.

1 Like
6

Architecture requires logstash. Mean collecting logs & metrics from different locations to main location also using multiple custom indexes. So logstash is required. If i use elasticsearch goip processor can i get all the IPs process trough elasticsearch get the geo information?

7

you can enrich geoip with elasticsearch ingest nodes.
but if you have to use logstash, can you show example output using logstash stdout ? i suspect you will have two host fields, one object (from filebeat) and one text (from logstash)

8

Below is the output. Filebeat installed on xx.xx.ELK01 and logstash installed on xx.xx.ELK02. now i am not mutating any field while generating below output.

{
          "input" => {
        "type" => "netflow"
    },
           "flow" => {
              "id" => "I10gpM6xKdE",
        "locality" => "public"
    },
        "network" => {
           "transport" => "tcp",
           "direction" => "unknown",
               "bytes" => 1400,
        "community_id" => "1:blWSqK8gPDQqDqGMZtcdt4DRrC8=",
             "packets" => 12,
         "iana_number" => 6
    },
       "@version" => "1",
           "host" => {
             "hostname" => "INHY-PAPP-ELK01",
        "containerized" => false,
                   "id" => "e1cebd3d12bc4510bdecafd61726096c",
                   "ip" => [
            [0] "10.252.10.75",
            [1] "fe80::d4ee:d927:5185:8d0"
        ],
                   "os" => {
             "version" => "8 (Core)",
            "platform" => "centos",
              "family" => "redhat",
              "kernel" => "4.18.0-147.8.1.el8_1.x86_64",
            "codename" => "Core",
                "name" => "CentOS Linux"
        },
                  "mac" => [
            [0] "00:15:5d:10:0b:62"
        ],
                 "name" => "INHY-PAPP-ELK01",
         "architecture" => "x86_64"
    },
           "tags" => [
        [0] "ISSQFILE",
        [1] "INHY",
        [2] "beats_input_raw_event",
        [3] "_geoip_lookup_failure"
    ],
         "source" => {
             "geo" => {
            "continent_code" => "NA",
              "country_name" => "United States",
                  "timezone" => "America/Los_Angeles",
             "country_code3" => "US",
                 "longitude" => -122.0748,
             "country_code2" => "US",
                 "city_name" => "Mountain View",
               "region_code" => "CA",
               "region_name" => "California",
                  "latitude" => 37.4043,
                  "dma_code" => 807,
                  "location" => {
                "lon" => -122.0748,
                "lat" => 37.4043
            },
                        "ip" => "216.58.203.35",
               "postal_code" => "94043"
        },
        "locality" => "public",
            "port" => 443,
           "bytes" => 1400,
              "ip" => "216.58.203.35",
         "packets" => 12
    },
        "fileset" => {
        "name" => "log"
    },
       "observer" => {
        "ip" => "192.168.252.13"
    },
          "agent" => {
        "ephemeral_id" => "f1cc8358-cde0-44eb-a6cd-336cc4d33392",
                  "id" => "ee22c382-4c82-4026-b738-70164e05b6cf",
             "version" => "7.7.0",
                "type" => "filebeat",
            "hostname" => "INHY-PAPP-ELK01"
    },
            "ecs" => {
        "version" => "1.5.0"
    },
     "@timestamp" => 2020-05-25T07:01:47.000Z,
        "netflow" => {
                         "protocol_identifier" => 6,
                         "source_ipv4_address" => "216.58.203.35",
                           "forwarding_status" => 64,
                          "packet_delta_count" => 12,
                           "octet_delta_count" => 1400,
                        "flow_end_sys_up_time" => 1211607664,
                              "application_id" => [
            [0] 20,
            [1] 0,
            [2] 0,
            [3] 48,
            [4] 68,
            [5] 0,
            [6] 0,
            [7] 0,
            [8] 0
        ],
                       "source_transport_port" => 443,
           "post_nat_destination_ipv4_address" => "103.44.2.78",
                "post_nat_source_ipv4_address" => "0.0.0.0",
             "post_napt_source_transport_port" => 0,
        "post_napt_destination_transport_port" => 51622,
                             "flow_end_reason" => 3,
                    "destination_ipv4_address" => "10.252.242.11",
                                    "exporter" => {
                  "version" => 9,
                "timestamp" => "2020-05-25T07:01:47.000Z",
            "uptime_millis" => 1211613904,
                  "address" => "192.168.252.13:3873",
                "source_id" => 4
        },
                      "flow_start_sys_up_time" => 1211322514,
                            "egress_interface" => 59,
                "post_ip_diff_serv_code_point" => 255,
                           "ingress_interface" => 3,
                                        "type" => "netflow_flow",
                     "post_packet_delta_count" => 12,
                      "post_octet_delta_count" => 1400,
                  "destination_transport_port" => 51622
    },
    "destination" => {
              "ip" => "10.252.242.11",
             "geo" => {},
        "locality" => "private",
            "port" => 51622
    },
        "service" => {
        "type" => "netflow"
    },
          "event" => {
          "action" => "netflow_flow",
             "end" => "2020-05-25T07:01:40.760Z",
           "start" => "2020-05-25T06:56:55.610Z",
        "category" => "network_traffic",
         "created" => "2020-05-25T07:01:47.000Z",
          "module" => "netflow",
        "duration" => 285150000000,
            "kind" => "event",
         "dataset" => "netflow.log"
    }
}
```````````````````````````````````````````
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.