Filebeat >5.4: Apply geoip only if a IP field exists


(olivier hodac) #1

I listen to my app with a filebeat that watches the application logs. To parse my custom log lines, I have a pipeline in my elasticsearch _ingest.

Some of the lines have a IP address, some others have not. I want to geoip the IP address. but it seems to fail. How can I run the geoip only if there is a IP field? I have tried:

any solution? I cannot imagine I have to use logstash?

Here is my pipeline

    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "%{TIMESTAMP_ISO8601:logdate},... %{LOGLEVEL:level} +~ %{EMAILLOCALPART:who}@%{HOSTNAME:company} ..%{IPORHOST:ip}..: .....%{URIPATHPARAM:route} performed in %{INT:req_ms:int} ms",
            "%{TIMESTAMP_ISO8601:logdate},... %{LOGLEVEL:level} +~ perfutils layout#%{INT:decode_layout:int} msg#%{INT:decode_msg:int} decode:%{INT:decode_ms:int}ms IFTs:%{INT:ifts_ms:int}ms",
            "%{TIMESTAMP_ISO8601:logdate},... %{LOGLEVEL:level} +~ %{GREEDYDATA:text}"
          ]
        }
      },
      {
        "geoip" : {
          "field" : "ip"
        }
      }
    ]
  

(Carlos PĂ©rez Aradros) #2

Hi @dao,

It looks like you could use the ignore_missing setting from the geoip processor? https://www.elastic.co/guide/en/elasticsearch/plugins/current/using-ingest-geoip.html

Best regards


(olivier hodac) #3

great, I missed this one!

Olivier Dao Hodac
FlightWatching

+33 6 76 81 54 11

The information in this e-mail is confidential. The contents may not be
disclosed or used by anyone other than the addressee. Access to this e-mail
by anyone else is not authorised.
If you are not the intended recipient, please notify FlightWatching SAS
immediately and delete this e-mail.
FlightWatching SAS cannot accept any responsibility for the accuracy or
completeness of this e-mail as it has been sent over public networks. If
you have any concerns over the content of this message or its Accuracy or
Integrity, please contact FlightWatching SAS immediately.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.