Trouble with string interpolation in datastream naming

Elastic Stack Logstash
1

Hello all,
Hoping someone more knowledgeable than I can suggest what I am missing. Attempting to use the below conf file to ingest results and am getting the (also) below error. Is this something that can only be done with an index name or did I miss a beat?

conf file:

input {

  http {
    port => 9050
    additional_codecs => { "application/json" => "es_bulk" }
    response_headers => {
        "Access-Control-Allow-Origin" => "*"
        "Content-Type" => "application/json"
        "Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"
    }
    ecs_compatibility => "disabled"
  }
}

  output {
  if [headers][http_host]=="192.168.10.7:9050"{
    elasticsearch {
      hosts=> "${ELASTIC_HOSTS}"
      user=> "${ELASTIC_USER}"
      password=> "${ELASTIC_PASSWORD}"
      ssl_certificate_authorities => "${CA_CERT}"
      ssl_certificate => "${LOGSTASH_CERT}"
      ssl_key => "${LOGSTASH_KEY}"
      ssl_enabled => true
      data_stream => true
      data_stream_namespace => "velociraptor.%{[Artifact]}"
    }  
  }
}

error msg:

[WARN ][logstash.outputs.elasticsearch][main][c4a621e8c999b41ab0e3e1ae76fc2094a4bc8ddcf5150e628bfee42b11da8276] Badly formatted index, after interpolation still contains placeholder: [logs-generic-velociraptor.%{[Artifact]}]; event: `{"headers"=>{"http_accept"=>nil, "content_type"=>"application/json", "request_path"=>"/_bulk", "accept_encoding"=>"gzip", "http_version"=>"HTTP/1.1", "request_method"=>"POST", "http_host"=>"192.168.10.7:9050", "content_length"=>"3688", "http_user_agent"=>"go-elasticsearch/7.3.1-SNAPSHOT (linux amd64; Go 1.21.5)"}, "@metadata"=>{"codec"=>{"es_bulk"=>{"_index"=>"artifact_generic_client_stats", "action"=>"index", "_id"=>"200677282548208615"}}}, "Hostname"=>"DevBox", "CPU"=>0.845e2, "Timestamp"=>0.17096655315234282e10, "Artifact"=>"Generic.Client.Stats", "@timestamp"=>2024-03-05T19:05:51.267667640Z, "RSS"=>25624576, "@version"=>"1", "host"=>"172.23.0.1", "ClientId"=>"C.dac74a666b848935", "CPUPercent"=>0, "timestamp"=>"2024-03-05T19:05:50Z", "_ts"=>1709665550}`

Thanks for the help!

2

The data_stream_namespace option is never interpolated. It is used when the pipeline is first configured to set the default index name. There are no events at that point, so it cannot reference fields on an event.

If you want to use data from the event to set the datastream name then enable the data_stream_auto_routing option and populate the required fields in the [data_stream] field on the event.

1 Like
3

Ah, that would do it, thanks! I'll give this a try and report back.

4

Thank you sir/ma'am,
I'm getting other errors, but look like generic mapping hiccups. This part at least appears to be working as hoped.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.