How to change the data type values

Elastic Stack Logstash
1

i have data like this is json file
{\"dataType\":\"System.Single\",\"host\":\"ssss\",\"name\":\"123/xxx\",\"123234\":\"333\",\"time\":\"2018/5/3 7:49:35\",\"value\":\"116.3047\"}

question:
if the dataType is System.Single need set the value is number 。
if the dataType is System.String nedd set the value is string.
it can add field in the output.
how configure the .conf file
thanks.

2

When you index data into Elasticsearch, each field must have a single mapping within each index. Are you planning on sending these different types of data to different indices?

3

in one index. it can add field about it .
like in es data:
"message": "{"dataType":"System.Single","host":"xxx","name":"xx/xxx","123":"222","time":"2018/5/3 8:03:22","value":"-6.5625"}",
if dataType is System.Single i add a filed as value_number the value is -6.5625
if datatype is System.String i add a filed as value_string the value is "stringtest"

4

Yes, if you split it into different fields you can do that.

5

how can configure the filter file
i don't know how to configure the .conf file
can help me?

6

Use a json filter to parse the message. Then use conditionals on the dataType field to copy over data from value to the appropriate field name. Use the mutate filter to convert the type to float where appropriate. You can then remove the value field if you want to.

7

i can't open use give url can give me a complete example as my ask.
thanks so much.. @Christian_Dahlqvist
:joy::joy::joy::joy::joy::joy::joy:

8

the filter is error can't start logstash @Christian_Dahlqvist

filter 
{ 
  json {  
    source => "message"
    id => "json_train"
    if [message][dataType] == "System.Single"
     {
    add_field => { "value_number" => "%{value}"
     }
  }
  date {  
    match => [ "time", "YYYY/M/dd HH:mm:ss" ]
    timezone => "Asia/Shanghai"
    locale => "cn" 
    target => "report_time"
  }

}
9

You can not have conditionals within a filter, so the conditionals need to be outside the json filter.

filter { 
  json {  
    source => "message"
    id => "json_train"
  }

  if [dataType] == "System.Single" {
    mutate {  
      add_field => { "value_number" => "%{value}"}
    }
  }
}
10

is like this
also can't start :joy::joy::joy::joy::joy: can help me to slove the error?

filter
{
    if [message][dataType] == "System.Single"{
  json {
    source => "message"
    id => "json_train"
    add_field => { "value_number" => "%{value}"
     }
     } else {
  json {
    source => "message"
    id => "json_train1"
     add_field => { "value_string" => "%{value}"
     }
   }
  }
  date {
    match => [ "time", "YYYY/M/dd HH:mm:ss" ]
    timezone => "Asia/Shanghai"
    locale => "cn"
    target => "report_time"
  }

}
11

You will not have access to the dataType field until you have run the json filter, which is why your version does not work. I spotted an error in my example and have updated it. Try that and continue to build out the logic within the conditionals until you have what you are looking for.

12

use your new filter is also can't start with

[2018-05-03T17:16:26,761][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 35, column 17 (byte 545) after filter \n{ \n json { \n source => \"message\"\n id => \"json_train\"\n }\n\n if [dataType] == \"System.Single\" {\n mutate { \n add_field => { \"value_number\" => \"%{value}\"}\n }\n \n date {\n match => [ \"time\", \"YYYY/M/dd HH:mm:ss\" ]\n timezone => \"Asia/Shanghai\"\n locale => \"cn\"\n target => \"report_time\"\n }\n\n}\n\noutput {\n elasticsearch ", :backtrace=>["/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:169:ininitialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:315:inblock in converge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:312:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:299:inconverge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/opt/ES/logstash-6.2.3-es/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`

13

use your give filter
also can't start
with

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 26, column 17 (byte 410) after filter { \n json { \n source => \"message\"\n id => \"json_train\"\n }\n\n if [dataType] == \"System.Single\" {\n mutate { \n add_field => { \"value_number\" => \"%{value}\"}\n }\n}\n\noutput {\n elasticsearch ", :backtrace=>["/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:90:in execute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/opt/ES/logstash-6.2.3-es/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in block in initialize'"]}`

14

Think I missed a curly brace. Have added it.

15

as my full filter can't start can slove me where is error? :joy::joy::joy::joy::joy::joy::joy:
thanks @Christian_Dahlqvist

filter { 
  json {  
    source => "message"
    id => "json_train"
  }

  if [dataType] == "System.Single" {
    mutate {  
      add_field => { "value_number" => "%{value}"}
    }else if [dataType] == "System.Double" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    }else if [dataType] == "System.Int16" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    }else if [dataType] == "System.Int32" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    } else if [dataType] == "System.Int64" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    } else if [dataType] == "System.Boolean" {
    mutate {
      add_field => { "value_Boolean" => "%{value}"}
    } else  {
    mutate {
      add_field => { "value_string" => "%{value}"}
    }
  }
}
16

You are missing a number of curly braces before the else statements.

17

thanks so much . slove it .
thanks so.
:elasticheart::elasticheart::elasticheart::elasticheart::elasticheart::elasticheart::elasticheart:

18

the date filed value_Boolean is True or False , i wan't change to 1 or 0 in es .
how can do it ?
@Christian_Dahlqvist

19 
filter { 
  json {  
    source => "message"
    id => "json_train"
  }

  if [dataType] == "System.Single" {
    mutate {  
      add_field => { "value_number" => "%{value}"}
      } 
    } else if [dataType] == "System.Double" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Int16" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Int32" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Int64" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Boolean" {
    mutate {
      add_field => { "value_Boolean" => "%{value}"}
      }
    } else  {
    mutate {
      add_field => { "value_string" => "%{value}"}
        }
    }

  date {
#    match => [ "time", "YYYY/M/dd HH:mm:ss +08:00" ]
    match => [ "time", "YYYY/M/dd HH:mm:ss" ]
    timezone => "Asia/Shanghai"
    locale => "cn"
    target => "report_time"
  }


}
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.