Filebeat monitoring metrics are "dropped" when a GEOIP pipeline is used

Elastic Stack Beats
,
21

but, now we have a new problem - if for some reason the event does NOT have the remote_ip field - the event fails to be sent to ES ....

I just tried to send an event:

{
   "event_uuid":"m_id_1025_8",
   "logstash_id":"m_id_1025_8",
   "cid":"12345",
   "event_timestamp_millis":"1666707272000",
   "activity_date":"2022-10-25",
   "user_agent":"Mozilla/5.0 (X11; CrOS aarch64 13421.102.0) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/86.0.4240.199 Safari/537.36",
   "referer":"https://www.my.site1.com/",
   "ref_param":"https://www.nyt.com",
   "request_status":"500",
   "request_method":"POST",
   "request_size":"52",
   "response_size":"124",
   "latency":"1.3"
}

and see this error:
(status=400): {"type":"illegal_argument_exception","reason":"field [remote_ip] not present as part of path [message.remote_ip]"}, dropping event!"

in filebeat logs:

{"log.level":"info","@timestamp":"2022-10-25T16:52:34.011-0400","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(elasticsearch(http://localhost:9200)) established","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-10-25T16:57:18.934-0400","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.October, 25, 20, 57, 17, 585000000, time.UTC), Meta:{\"_id\":\"m_id_1025_8\",\"pipeline\":\"geoip-no-missing\"}, Fields:{\"agent\":{\"ephemeral_id\":\"83c397dd-ec26-4570-bad6-7a7600671487\",\"id\":\"e0b4f8e6-d0c6-4c38-a62d-ac6ff81a555a\",\"name\":\"dhcp-10-250-50-96.harvard.edu\",\"type\":\"filebeat\",\"version\":\"8.4.3\"},\"ecs\":{\"version\":\"8.0.0\"},\"event\":{\"created\":\"2022-10-25T20:57:17.761Z\",\"id\":\"59279bf715-5532496295095422\"},\"host\":{\"architecture\":\"x86_64\",\"hostname\":\"dhcp-10-250-50-96.harvard.edu\",\"id\":\"xxx443\",\"ip\":[\"fe80::aede:48ff:fe00:1122\",\"fe8022\"],\"name\":\"dhcp-10-250-50-96.harvard.edu\",\"os\":{\"build\":\"21E258\",\"family\":\"darwin\",\"kernel\":\"21.4.0\",\"name\":\"macOS\",\"platform\":\"darwin\",\"type\":\"macos\",\"version\":\"12.3.1\"}},\"input\":{\"type\":\"gcp-pubsub\"},\"message\":{\"activity_date\":\"2022-10-25\",\"cid\":\"12345\",\"event_timestamp_millis\":\"1666707272000\",\"latency\":\"1.3\",\"logstash_id\":\"m_id_1025_8\",\"ref_param\":\"https://www.nyt.com\",\"referer\":\"https://www.my.site1.com/\",\"request_method\":\"POST\",\"request_size\":\"52\",\"request_status\":\"500\",\"response_size\":\"124\",\"user_agent\":\"Mozilla/5.0 (X11; CrOS aarch64 13421.102.0) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/86.0.4240.199 Safari/537.36\"}}, Private:(*pubsub.Message)(0xc0003d21c0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"field [remote_ip] not present as part of path [message.remote_ip]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}
22

ok - makes sense!

So, as a workaround - I will move the pipeline back into elastic output under parameters.pipeline,
and will open a bug.

Where should I open a bug?

Thanks for all your help!!!

23

You can add this

ignore_failure: true

to the processor...then it will go through....

You can also write a condition to check for that field exists as well... which seems to be what ignore_missing is trying to do... but not doing well. I can check that tomorrow I would start with the ignore_failure

24

Ok, so I've added the "ignore_failure" to my pipeline:

{
  "geoip-no-missing": {
    "description": "Add geoip info",
    "processors": [
      {
        "geoip": {
          "field": "message.remote_ip",
          "target_field": "message.remote_ip_geo",
          "ignore_failure": true
        }
      }
    ]
  }
}

making sure this is the pipleine I use in filebeat:

# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: gcp-pubsub
  enabled: true
  project_id: ${PROJECT_ID}
  topic: ${PUBSUB_INPUT_TOPIC}
  subscription.name: ${SUBSCRIPTION_NAME}
  fields_under_root: true
  pipeline: "geoip-no-missing"

processed one event WITHOUT 'remote_ip' field - and see no GEO info added and no failures!

{
  "took": 97,
  "timed_out": false,
  "_shards": {
    "total": 12,
    "successful": 12,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "ibc-parsed-logs-2022.10.25-000003",
        "_id": "m_id_1025_9",
        "_score": 0.2876821,
        "_source": {
          "input": {
            "type": "gcp-pubsub"
          },
          "agent": {
            "name": "dhcp-10-250-50-96.harvard.edu",
            "id": "e0b4f8e6-d0c6-4c38-a62d-ac6ff81a555a",
            "type": "filebeat",
            "ephemeral_id": "de9db623-2d80-4455-b9a6-d59c2e43aeae",
            "version": "8.4.3"
          },
          "@timestamp": "2022-10-25T21:35:51.419Z",
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "hostname": "dhcp-10-250-50-96.harvard.edu",
            "os": {
              "build": "21E258",
              "kernel": "21.4.0",
              "name": "macOS",
              "family": "darwin",
              "type": "macos",
              "version": "12.3.1",
              "platform": "darwin"
            },
            "ip": [
              "fe80"
            ],
            "name": "dhcp-10-250-50-96.harvard.edu",
            "id": "xxx443",
            "mac": [
              "5e"
            ],
            "architecture": "x86_64"
          },
          "event": {
            "created": "2022-10-25T21:35:52.482Z",
            "id": "59279bf715-6095478736584954"
          },
          "message": {
            "request_status": "500",
            "referer": "https://www.my.site1.com/",
            "ref_param": "https://www.nyt.com",
            "event_timestamp_millis": "1666707272000",
            "latency": "1.3",
            "activity_date": "2022-10-25",
            "logstash_id": "m_id_1025_9",
            "request_method": "POST",
            "request_size": "52",
            "user_agent": "Mozilla/5.0 (X11; CrOS aarch64 13421.102.0) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/86.0.4240.199 Safari/537.36",
            "response_size": "124",
            "cid": "12345"
          }
        }
      }
    ]
  }
}

processed one event WITH remote_id field - GEO info is added correctly:

{
  "took": 1022,
  "timed_out": false,
  "_shards": {
    "total": 12,
    "successful": 12,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "ibc-parsed-logs-2022.10.26-000004",
        "_id": "m_id_1026_1",
        "_score": 0.2876821,
        "_source": {
          "input": {
            "type": "gcp-pubsub"
          },
          "agent": {
            "name": "mac-lt2-mpopova.fios-router.home",
            "id": "e0b4f8e6-d0c6-4c38-a62d-ac6ff81a555a",
            "type": "filebeat",
            "ephemeral_id": "3909273a-3f14-42ad-a5b7-a6c15691a912",
            "version": "8.4.3"
          },
          "@timestamp": "2022-10-26T12:57:05.428Z",
          "ecs": {
            "version": "8.0.0"
          },
          "host": {
            "hostname": "mac-lt2-mpopova.fios-router.home",
            "os": {
              "build": "21E258",
              "kernel": "21.4.0",
              "name": "macOS",
              "type": "macos",
              "family": "darwin",
              "version": "12.3.1",
              "platform": "darwin"
            },
            "ip": [
              "fe80"
            ],
            "name": "mac-lt2-mpopova.fios-router.home",
            "id": "xxx443",
            "mac": [
              "7e"
            ],
            "architecture": "x86_64"
          },
          "event": {
            "created": "2022-10-26T12:57:06.629Z",
            "id": "59279bf715-6098619314182042"
          },
          "message": {
            "request_status": "500",
            "ref_param": "https://www.nyt.com",
            "referer": "https://www.my.site1.com/",
            "remote_ip_geo": {
              "continent_name": "North America",
              "region_iso_code": "US-NY",
              "city_name": "The Bronx",
              "country_iso_code": "US",
              "country_name": "United States",
              "region_name": "New York",
              "location": {
                "lon": -73.8616,
                "lat": 40.847
              }
            },
            "latency": "1.3",
            "activity_date": "2022-10-26",
            "logstash_id": "m_id_1026_1",
            "request_method": "POST",
            "response_size": "124",
            "remote_ip": "165.155.130.139",
            "event_timestamp_millis": "1666788992000",
            "request_size": "52",
            "user_agent": "Mozilla/5.0 (X11; CrOS aarch64 13421.102.0) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/86.0.4240.199 Safari/537.36",
            "cid": "12345"
          }
        }
      }
    ]
  }
}

So the "ignore_failure" field works as expected!

thanks!
Marina

25

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.