How to transform fields but dont insert them into elasticsearch

Elastic Stack Logstash
1

I would like to insert the document name into de index value via variable but i need to lowercase it, so i have to create a new field, also i would like to change the @timestamp to get %{+YYYY.MM.dd} variable. The problem is that i don't want to insert "@timestamp" and "indexName" fields into elastic so i try to delete them but doesn't work. Sorry for my english.

input {
   file { 
     path => "/etc/logstash/data/data*.json"
     start_position => "beginning"
     sincedb_path => "/dev/null"
     codec => "json"
     file_completed_action => "delete"
   }
}

filter {
  mutate {
    copy => { "name" => "indexName" }
  }
  mutate {
    lowercase => [ "indexName" ]
  }
  date {
    match => [ "serverTimestamp","ISO8601"]
    target => "@timestamp"
  }
  mutate {
    remove_field => [ "path", "host","@version", "@timestamp", "indexName" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "data_%{indexName}_%{+YYYY.MM.dd}"
  }
}

Error :

ERROR] 2019-07-09 11:28:24.609 [[main]>worker5] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"data_iot_%{indexName}_", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x786acfe3>], :response=>{"index"=>{"_index"=>"data_%{indexName}_", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [data_%{indexName}_], must be lowercase", "index_uuid"=>"_na_", "index"=>"data_%{indexName}_"}}}}

2

There is a special field @metadata that is not send to elasticsearch from the output.
So you can add your addtional fields below that one and they won't be send to elasticsearch.
e.g.

...
filter {
  mutate {
    copy => { "name" => "[@metadata][indexName]" }
  }
  mutate {
    lowercase => [ "[@metadata][indexName]" ]
  }
  date {
    match => [ "serverTimestamp","ISO8601"]
    target => "[@metadata][@timestamp]"
  }
  mutate {
    remove_field => [ "path", "host","@version", "@timestamp" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "data_%{[@metadata][indexName]}_%{+YYYY.MM.dd}"
  }
}

i will have to search myself if you can format another field as date (will come back to you later, if someone else knows how, feel free to elaborate)

1 Like
3

Shaoranlaos
Thank you very much, I will try it this way, at least it will work for now.

4

You are deleting [indexName] before the event gets to the output. So the sprintf reference to %{indexName} does not get substituted.

Also, you are using %{+YYYY.MM.dd} in the index name, which is a reference to [@timestamp,] but you have also deleted that, so that does not get substituted.

Lastly, if I recall correctly, [@timestamp] is not optional for elasticsearch. I believe it will fail to index events that do not have that field.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.