Conditions, Pipelines and GeoIP

[Ghanlow]

Hello, i'm new in Elastic community, i'm actually French IT Student, i'm not native english so sorry for my mistakes.

I've enjoyed installing ELK Stack.

Here i am, i'm actually using Packetbeat 5.3, sending output to Elasticsearch 5.3 and using Kibana 5.3 as UI.

I would like to know how to create an Elastic output with multiples pipelines,
i've actually investigated around i've seen that packetbeat (for me) can detect flow, icmp, dns and http.
when it's flow, icmp, dns the Geoip pipeline works !
But when it's flow Geoip pipline didn't work.

So i'm arrived with a conclusion that i should create another pipeline to use the specific field..
Because Geoip pipeline is like this,

 "geoip-info": {
    "description": "Add geoip info",
    "processors": [
      {
        "geoip": {
          "field": "client_ip",
          "target_field": "client_geoip",
          "properties": [
            "location"
          ],
          "ignore_failure": true
        }
      }
    ]
  }

In this pipeline we have the field "client_ip", that create an index with geoip informations, client_ip.location.
So logically i'm saying myself if i use the field "source.ip" instead of "client_ip" it should works.
Here it is, ( fsip = for source ip )

"geoip-infofsip": {
    "description": "Add geoip info for source.ip",
    "processors": [
      {
        "geoip": {
          "field": "source.ip",
          "target_field": "source.ip",
          "properties": [
            "location"
          ],
          "ignore_failure": true
        }
      }
    ]
  },

After this i'm adding with Kibana Console with,
PUT _ingest/pipeline/<my-pipeline-name>

So i've created a Config file with this,
Here is a part of my config file config.yml multiple pipeline selector.

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipelines:
    - pipeline: geoip-infofsip
      when.equals:
        type: "flow"
    - pipeline: geoip-info
      when.equals:
        type: ["icmp", "dns", "http"]

After that i've checked with ./packetbeat -configtest -e options it say "Config OK" so the syntax was good
Now let's launch the packetbeat agent
After few sec something show me this,

WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [source.ip]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:86"}}
WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [source.ip]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:83"}}

I don't understand the error
Could you please help me by saying to me where i've failed something ?
And say to me where i could find documentation about Conditions,
i'm using here
when.equals:
I didn't found where something explaining there is something like :

• if.equals:
• while.equals:
• if.greaterthan:
  something like condition.comparaison: who explain all possibility ?

[ruflin]

It seems the error is on the elasticsearch side with the ingest pipeline and not the beats side. Perhaps you could take one of the documents which is sent to elasticsearch and use the simulate API to see what exactly is happening? https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html Do you see any errors in the elasticsearch log?

[steffens (Steffen Siering)]

Error seems to occur in Elasticsearch, when the pipeline is applied.

Do you know which pipeline is failing? And for what kind of traffic? Unfortunately the error message is not helping much. Is it from packetbeat or elasticsearch logs? Maybe Elasitcsearch logs do contain some more information.

Is packetbeat reporting IPv4 or IPv6 addresses? Checking the documentation for exported fields, source.ip is only used for IPv4. IPv6 uses source.ipv6.