Logstash mutate add field not showing the added field

Elastic Stack Logstash
1

Hello guys.

I am testing some functionalities of logstash but I am a bit confused about the process of implementing a filter mutate.

Just for testing purpose I have a kiwi syslog generating random logs. Those logs have 2 fields, as follow:

timestamp
message

What I am trying to do is to add an extra field which is the result of the merge of timestamp + message

Sorry for this basic test, but I am trying to understand how this work before I can move to more complex stuff.

So I set a logstash.conf as follow

input {
  tcp {
    port => 514
}
}
filter {
      prune {
        whitelist_names =>["timestamp","message","newfield","@metadata" ]
      }
      mutate {
        add_field => {"newfield" => "%{@timestamp}%{message}"}
      }
    }
output {
  kusto {
    path => "/tmp/kusto/%{+YYYY-MM-dd-HH-mm-ss}.txt"
    ingest_url => "link"
    app_id => "ID"
    app_key => "Key"
    app_tenant => "tenant"
    database => "database"
    table => "table"
    json_mapping => "basicmsg"
 } 
  stdout {
    codec => rubydebug
 }
}

When I run the configuration, the output Is correct and I can see this:

{
    "@timestamp" => 2021-07-22T13:05:04.992Z,
       "message" => "<176>Original Address=203.25.150.43 This is a test message generated by Kiwi SyslogGen\r",
      "newfield" => "2021-07-22T13:05:04.992Z<176>Original Address=203.25.150.43 This is a test message generated by Kiwi SyslogGen\r"
}

Which is perfect because is exactly what I was looking for.

But when I head to gusto (azure data explorer) and query the database. I don't see the new field, but only the timestamp and message

Can please somebody be so kind and explain me what I am doing wrong or what am I missing please?

2

Since Logstash is generating the correct output this may not be a Logstash problem.
Does kusto need to have it's database schema/json mapping refreshed?
In Elasticsearch when you add a new field to an index you need to refresh the index pattern in Kibana before you can get full functionality of that field in the UI. Does Kusto have a similar concept?

3

Oh hello mate. I was testing the code you suggested me yesterday about the filter-prune. This is the testing environment. The filter prune works that is a charm. Later I will implement it into production.
Now I was testing the mutate filter. I looked into Kusto and I don't find any json refresh as in kibana. I will look deeper. Thank you very much, you are so helpful

1 Like
4

You were right. It was about the refreshing json mapping schema. I tried the same code, but in the output a configured elastic search and grafana, and I could see the new field without problem. thank you very much for your help mate

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.