Trouble ingesting through Filebeat

[tom.p (tom)]

Hello all,

I'm trying to ingest data from a "data" directory into Elasticsearch using Filebeat. When I manually upload these data files through Filebeat I have no problems.

If I use that same mapping for my .conf file and try reading in the file I get this error. I've searched through the forum and found an issue very similar, but their solution didn't work for my problem.

I have the error below. Even though it says that it can not index the event, Filebeat was able to with a manual upload through the gui before. I thought maybe this is a false positive error flag.

Any thoughts would be greatly appreciated.

Thank you

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfb703e4f430f60d, ext:38571866, loc:(*time.Location)(0x5b230e0)}

2020-06-30T15:52:52.894-0400 WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfb703e4f430f60d, ext:38571866, loc:(*time.Location)(0x5b230e0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"e95130a7-ca0e-4de4-9130-f69b61cac698","hostname":"localhost","id":"bsb875c1-6a4b-4bb1-bb8a-f4da88f0245c","name":"localhost","type":"filebeat","version":"7.8.0"},"ecs":{"version":"1.5.0"},"host":{"name":"localhost"},"input":{"type":"log"},"log":{"file":{"path":"/data/ingest/test/data6-test4.json"},"offset":0},"message":"{\"flow_version\":12,\"ip_version\":4,\"tuple\":{\"src_ip\":\"10.104.8.16\",\"dst_ip\":\"10.104.11.255\",\"protocol\":17,\"src_port\":138,\"dst_port\":138},\"continued\":false,\"continuation\":false,\"direction\":\"Internal\",\"bidirectional\":false,\"vlan\":1010,\"scan\":false,\"start_time_usec\":292632,\"end_time_sec\":1591123480,\"end_time_usec\":292632,\"packets\":1,\"bytes\":247,\"payload_bytes\":201,\"src_role\":\"Unknown\",\"metadata\":{\"src_host_name\":[\"taxmd02-veeam-v\"],\"src_host_names\":[{\"domain_name\":\"taxmd02-veeam-v\",\"active_provenance\":[900,1591122621],\"passive_provenance\":null}]}}"}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000684340), Source:"/data/ingest/test/data6-test4.json", Offset:550, Timestamp:time.Time{wall:0xbfb703e4f42dae06, ext:38356819, loc:(*time.Location)(0x5b230e0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x2600b0, Device:0x802}}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"field [end_time_sec] not present as part of path [end_time_sec]"}

[shaunak (Shaunak Kashyap)]

Hi @tom.p, welcome to the Elastic community forums!

A few questions to help us get going here:

1. What version of Filebeat are you using?

2. Can you share your Filebeat configuration file, filebeat.yml here?

3. If I use that same mapping for my .conf file

   What .conf file are you referring to here? Can you share it as well?

4. The error mentions the field end_time_sec not being present as part of some path. Have you enabled any Filebeat modules? If so, could you share the relevant module's configuration file as well?

Thanks,

Shaunak

[tom.p (tom)]

Hello Shaunak,

I really appreciate your feedback and help!

Filebeat version 7.8.0

No filebeat modules are enabled or being used

Filebeat.yml

filebeat.inputs:
- type: log
  paths:
  - '/data/ingest/dns/*.json'

output.elasticsearch:
  hosts: ["10.104.8.107:9200"]
  index: "data6-2_dns184508"
  pipeline: "data6-2_dns184508-pipeline"

setup:
  template.enabled: false
  ilm.enabled: false

The ".conf" file is the mapping copied from the filebeat manual upload through the GUI that we're trying to use as the mapping for the ingestion.

{
  "mappings": {
    "_doc": {
      "_meta": {
        "created_by": "ml-file-data-visualizer"
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "bidirectional": {
          "type": "boolean"
        },
        "bytes": {
          "type": "long"
        },
        "continuation": {
          "type": "boolean"
        },
        "continued": {
          "type": "boolean"
        },
        "direction": {
          "type": "keyword"
        },
        "end_time_sec": {
          "type": "date",
          "format": "epoch_second"
        },
        "end_time_usec": {
          "type": "long"
        },
        "first_tcp_flags": {
          "type": "long"
        },
        "first_tcp_seq": {
          "type": "long"
        },
        "flow_version": {
          "type": "long"
        },
        "ip_version": {
          "type": "long"
        },
        "last_tcp_seq": {
          "type": "long"
        },
        "metadata": {
          "properties": {
            "dst_host_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "dst_host_names": {
              "properties": {
                "active_provenance": {
                  "type": "long"
                },
                "domain_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "passive_provenance": {
                  "type": "long"
                }
              }
            },
            "http_host": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "rr_dns_id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "server_banner_v4": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "src_host_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "src_host_names": {
              "properties": {
                "active_provenance": {
                  "type": "long"
                },
                "domain_name": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "passive_provenance": {
                  "type": "long"
                }
              }
            },
            "tls_cipher": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "tls_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "tls_organization_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "tls_version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "packets": {
          "type": "long"
        },
        "payload_bytes": {
          "type": "long"
        },
        "rest_tcp_flags": {
          "type": "long"
        },
        "scan": {
          "type": "boolean"
        },
        "src_role": {
          "type": "keyword"
        },
        "start_time_sec": {
          "type": "date",
          "format": "epoch_second"
        },
        "start_time_usec": {
          "type": "long"
        },
        "tuple": {
          "properties": {
            "dst_ip": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "dst_port": {
              "type": "long"
            },
            "icmp_code": {
              "type": "long"
            },
            "icmp_inode": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "icmp_type": {
              "type": "long"
            },
            "protocol": {
              "type": "long"
            },
            "src_ip": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "src_port": {
              "type": "long"
            }
          }
        },
        "vlan": {
          "type": "long"
        }
      }
    }
  }
}

[shaunak (Shaunak Kashyap)]

Thanks for the answers. Looking at the Filebeat configuration and index mapping, I suspect the error in your logs is coming from the Elasticsearch ingest pipeline, data6-2_dns184508-pipeline. Could you share that pipeline definition please?

Thanks,

Shaunak

[tom.p (tom)]

Hello Shaunak,

I can't seem to find this file. Is it possible that it doesn't exist or is protected in some way?

[tom.p (tom)]

Ok I was half right, our pipeline is empty.

I'm going to follow the instructions to create the pipeline, but do you have some advice or could check this pipeline file once I've completed?

I'm very sorry for taking so much of your time, but thank you,
Tom

[shaunak (Shaunak Kashyap)]

The pipeline you've referenced via output.elasticsearch.pipeline is an Elasticsearch Ingest Node Pipeline. When Filebeat sends documents to Elasticsearch, they will be passed through this pipeline. before they are indexed. This gives the pipeline the opportunity to transform the events as desired right before indexing them.

Given that your pipeline is named data6-2_dns184508-pipeline, you can create it by calling the PUT _ingest/pipeline/data6-2_dns184508-pipeline Elasticsearch API with the appropriate pipeline definition in the body of the API request. Once the API call succeeds, you can check if your pipeline was indeed created in Elasticsearch using the GET _ingest/pipeline/data6-2_dns184508-pipeline Elasticsearch API. Finally, you might also be interested in testing your pipeline with the POST _ingest/pipeline/data6-2_dns184508-pipeline/_simulate Elasticsearch API.

Hope that helps,

Shaunak