NMAP Codec Plugin - Logstash Configuration and Index Template Issues

Elastic Stack Logstash
1

Hello, I have been trying to take an Nmap scan's XML output into Elasticsearch and I have come across a few issues.

Below is my Logstash configuration file.

input {
 file {
   mode => "tail"
   path => "/usr/share/logstash/ingest_data/*"
 }
 http {
    host => "10.0.30.135"
    port => 8000
    codec => nmap
    ssl => false
 }
}


filter {
}


output {
 elasticsearch {
   index => "logstash-%{+YYYY.MM.dd}"
   hosts=> "${ELASTIC_HOSTS}"
   
   user=> "${ELASTIC_USER}"
   password=> "${ELASTIC_PASSWORD}"
   cacert=> "certs/ca/ca.crt"
 }
}

I know that I need a template to properly map the files, and I made an API request to create an index template and a component template:

Component:

PUT _component_template/nmap_host
{
  "template": {
    "mappings": {
      "properties" : {
          "addresses" : {
            "properties" : {
              "address" : {
                "type": "text"
              },
              "type" : {
                "type" : "text"
              }
            }
          },
          "ip" : {
            "type" : "text"
            }
          }
        }
      }
    }
  }
}

Index:

PUT _index_template/nmap_index
{
    "index_patterns": ["nmap-*"],
    "template" : {
        "settings" : {
            "number_of_shards" : 1
        },
        "mappings": {
            "_source": {
                "enabled": true
            }
        },
        "properties": {
            "host_name" : {
                "type": "keyword"
            },
            "created_at" : {
                "type" : "date"
            }
        }
    },
    "aliases" : {
        "testdata" : { }
    },
"priority": 500,
"composed_of": ["nmap_host"],
"version": 0
}

The component PUT request goes through, but my Index request returns the following:

{
  "error": {
    "root_cause": [
      {
        "type": "x_content_parse_exception",
        "reason": "[1:226] [template] unknown field [properties]"
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[1:240] [index_template] failed to parse field [template]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[1:226] [template] unknown field [properties]"
    }
  },
  "status": 400
}

Thanks in advance for any assistance!

2

Your template is wrong, the template object can have 3 nested objects, settings, mappings and aliases, in your configuration it has a object named properties that should be inside mappings, you are closing the mappings before.

It needs to be something like this:

{
    "index_patterns": ["nmap-*"],
    "template" : {
        "settings" : {
            "number_of_shards" : 1
        },
        "mappings": {
            "_source": {
                "enabled": true
            },
            "properties": {
                "host_name" : { "type": "keyword" },
                "created_at" : { "type" : "date" }
            }
        },
        "aliases" : {
            "testdata" : { }
        }
    },
    "priority": 500,
    "composed_of": ["nmap_host"],
    "version": 0
}
2 Likes
3

Thank you for the help! this ended up working!