How to get internal ip address from filebeat?

Elastic Stack Beats
1

I want to get internal ip address in as a field value in filebeat. currently only global ips indexed into elastic. why is that ?

indexed json i got from elastic as below

{
  "_index": "filebeat-6.4.3-2019.02.25",
  "_type": "doc",
  "_id": "kfC3I2kBtCJwrx4ApvIL",
  "_score": 1,
  "_source": {
    "source": "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex190225.log",
    "message": "2019-02-25 08:13:27 10.100.116.56 GET /Citrix/Roaming/accounts - 443 - 10.100.4.227 libwww-perl/6.05 - 200 0 0 0",
    "offset": 15215747,
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "@version": "1",
    "prospector": {
      "type": "log"
    },
    "beat": {
      "hostname": "HAMXSF",
      "name": "HAMXSF",
      "version": "6.4.3"
    },
    "host": {
      "name": "HAMXSF"
    },
    "fileset": {
      "name": "access",
      "module": "iis"
    },
    "input": {
      "type": "log"
    },
    "@timestamp": "2019-02-25T08:13:50.886Z"
  },
  "fields": {
    "@timestamp": [
      "2019-02-25T08:13:50.886Z"
    ]
  }
}

in the original log there is internal ip data, but in elastic field it's absent. how to fix it ?

2

Could you please share your configuration formatted using </>? Have you uploaded the pipeline provided by the module?

3

filebeat config ?

4

Yes.

5

filebeat config as below

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log
  document_type: iis

  enabled: false

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 1
  
setup.template.name: "filebeat-%{[beat.version]}-*"
setup.template.fields: "fields.yml"
setup.template.pattern: "filebeat-%{[beat.version]}-*"
setup.template.overwrite: true

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["localhost:5044"]

#================================ Procesors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================

logging.level: info
logging.to_files: true
logging.files:
  path: ${path.config}/logs
  name: filebeat
  keepfiles: 10
  permissions: 0644
6

In your configuration there is nothing which sets parsing IIS events. You could use the iis/access filebeat module: https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-module-iis.html

7

but i activated iis module, i thought filebeat run the iis from module config

8

what's the output of ./filebeat modules list?

9

You have set "enabled" to "false". Kindly re-verify it by setting it to "true"

10

If someone enables modules in Filebeat, there is no need to enable inputs in the configuration.

11

give
Enabled IIS
Disabled everything else in module folder

12

why internal ip not indexing ?

13

Have you loaded the pipeline to ES?

14

yes. if the ip is external, i am getting ip details and geoip information.

but if it is internal like 10.?.?.? , i am not getting ip address in the field

15

my iis.yml file is as below

- module: iis
  access:
    enabled: true
    var.paths: ["C:/inetpub/logs/LogFiles/*/*.log"]

  error:
    enabled: true
    var.paths: ["C:/Windows/System32/LogFiles/HTTPERR/*.log"]
16

any idea ?

17

Have you uploaded the pipelines of each fileset?

18

i am using filebeat iis default pipeline

{
  "description": "Pipeline for parsing IIS access logs. Requires the geoip and user_agent plugins.",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
        "%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.referrer} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.request_time_ms}",
        "%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:iis.access.referrer} %{NOTSPACE:iis.access.hostname} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.body_sent.bytes} %{NUMBER:iis.access.body_received.bytes} %{NUMBER:iis.access.request_time_ms}",
        "%{TIMESTAMP_ISO8601:iis.access.time} %{NOTSPACE:iis.access.site_name} %{NOTSPACE:iis.access.server_name} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} HTTP/%{NUMBER:iis.access.http_version} %{NOTSPACE:iis.access.agent} %{NOTSPACE:iis.access.cookie} %{NOTSPACE:iis.access.referrer} %{NOTSPACE:iis.access.hostname} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.body_sent.bytes} %{NUMBER:iis.access.body_received.bytes} %{NUMBER:iis.access.request_time_ms}",
        "%{TIMESTAMP_ISO8601:iis.access.time} %{IPORHOST:iis.access.server_ip} %{WORD:iis.access.method} %{URIPATH:iis.access.url} %{NOTSPACE:iis.access.query_string} %{NUMBER:iis.access.port} %{NOTSPACE:iis.access.user_name} %{IPORHOST:iis.access.remote_ip} %{NOTSPACE:iis.access.agent} %{NUMBER:iis.access.response_code} %{NUMBER:iis.access.sub_status} %{NUMBER:iis.access.win32_status} %{NUMBER:iis.access.request_time_ms}"
        ],
      "ignore_missing": true
    }
  }, {
    "remove":{
      "field": "message"
    }
  }, {
    "rename": {
      "field": "@timestamp",
      "target_field": "read_timestamp"
    }
  }, {
    "date": {
      "field": "iis.access.time",
      "target_field": "@timestamp",
      "formats": ["yyyy-MM-dd HH:mm:ss"]
    }
  }, {
    "remove": {
      "field": "iis.access.time"
    }
  }, {
    "user_agent": {
      "field": "iis.access.agent",
      "target_field": "iis.access.user_agent"
    }
  }, {
    "rename": {
      "field": "iis.access.agent",
      "target_field": "iis.access.user_agent.original"
    }
  }, {
    "geoip": {
      "field": "iis.access.remote_ip",
      "target_field": "iis.access.geoip"
    }
  }],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}
19

please help

20

what to do now? how to debug ?