Getting 'Badly formatted index, after interpolation still contains placeholder' error when trying to ingest AWS WAF logs

[feo13]

Hi there,

I'm trying to ingest AWS WAF logs with logstash-8.9.0 and send them to my local ELK stack but am getting a 'Badly formatted index, after interpolation still contains placeholder' error. Here's my logstash config:

input { 
  s3 { 
    "access_key_id" => "foo"
    "secret_access_key" => "bar"
    "region" => "us-east-1" 
    "bucket" => "aws-waf-logs" 
    "type" => "waf-log" 
    "interval" => "300" 
    "sincedb_path" => "/tmp/.waf-log_since.db" 
  } 
} 

filter { 
  if [type] == "waf-log" { 
   json { 
        source => "message" 
  } 
  date { 
        match => [ "[timestamp]", "UNIX_MS" ] 
  } 
  geoip { 
        source => [ "[httpRequest][clientIp]" ] 
        target => geoip 
  } 
  ruby { 
    code => ' 
      event.get("[httpRequest][headers]").each { |kv| 
        event.set(name = kv["name"], value = kv["value"])} 
        ' 
  } 
} 
} 

output { 
  elasticsearch { 
    hosts => ["http://127.0.0.1:9200/"] 
    index => "%{[type]}%-{+YYYY.MM.dd}"
    ilm_enabled => false 
    data_stream => false
    action => "create"
  } 
}

I read Could not index event to Elasticsearch DataStream but changing index to 'index => "%{[@metadata][beat]}-%{[@metadata][version]}"' returns the same 'Badly formatted index' error.

Can anyone point me in the right direction?

Thanks!

[stephenb (Stephen Brown)]

Hi @feo13 Welcome to the community.

@feo13

index => "%{[type]}%-{+YYYY.MM.dd}"

Look like at least a Syntax error got an % in the wrong place

index => "%{[type]}-%{+YYYY.MM.dd}"
...................^

[feo13]

Thank you SO much! It looks like that resolved the 'Badly formatted index' error but now I'm getting this:

[2023-08-16T03:42:25,231][WARN ][logstash.outputs.elasticsearch][main]
[ca2bc6153105dde6379bf39666d95cd4b608ad24418e29aca41368c5e05372ef] Could 
not index event to Elasticsearch. {:status=>400, :action=>["create", 
{:_id=>nil, :_index=>"waf-log-2023.05.12", :routing=>nil}, 
{"terminatingRuleType"=>"REGULAR", "@timestamp"=>2023-05-12T18:43:26.325Z,

From what I've read, that can happen when there's an old index template and the mappings aren't aligned. But when I go into Dev Console and do a 'GET _template', I just see '.monitoring-es' so I'm not sure what exactly to delete.

Thanks again for your response -- I really appreciate it!

[feo13]

OK, I got it. I just deleted all pre-existing indicies and index templates that I'd created for WAF, restarted logstash, and recreated the template.

Thank you again for the help -- I really appreciate it!