Elasticsearch index not matching stdout test

Elastic Stack Logstash
1

Good morning. I'm currently vexed by a situation with my elastic stack (7.13) and how things are parsing. I am parsing some exported IIS logs, and if I test my config with stdout, things look smashing. When I change my output to elasticsearch, however, only some of my fields show up in my index in kibana. for example, my geoip doesn't show up at all, and @timestamp is auto-generated for the current time instead of using the timestamp parsed by the date plugin.

I suspect that it's got something to do with auto-generation of index templates, but I'm not quite sure how to pursue the answer. I would be grateful if someone can point me in the right direction. Thank you!

My stdin/stdout config is below:

input {
      stdin {
              type => "iis"
                }
}

filter {
 if [type] == "iis" {
            csv {
                columns => ["logdate","time","destination_ip","http_method","cs_uri_stem","cs_uri_query","destination_port","user","source_ip","user_agent","Referer","status","substatus","win32_status","time_taken"]
                    separator => " "
            }
        mutate {
            add_field => {
                "timestamp" => "%{logdate} %{time}"
                         }
        }
        date {
            match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ]
        }
        mutate {
           remove_field => [ "logdate","time","timestamp" ]
        }
        geoip {
           source => "source_ip"
           target => "geoip"
           add_tag => [ "iis_geoip" ]
              }
              }
}

output {
        if [type] == "iis" {
                    stdout { codec => rubydebug }
                        }

}

Sample log line:

2021-07-02 00:00:07 172.16.101.4 GET /pool/boxes/b417 - 80 - 8.8.8.8 Mozilla/5.0+(compatible;+MJ12bot/v1.4.8;+http://mj12bot.com/) - 302 0 0 468
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.