How to change the data type values


(andy_zhou) #1

i have data like this is json file
{\"dataType\":\"System.Single\",\"host\":\"ssss\",\"name\":\"123/xxx\",\"123234\":\"333\",\"time\":\"2018/5/3 7:49:35\",\"value\":\"116.3047\"}

question:
if the dataType is System.Single need set the value is number
if the dataType is System.String nedd set the value is string.
it can add field in the output.
how configure the .conf file
thanks.


(Christian Dahlqvist) #2

When you index data into Elasticsearch, each field must have a single mapping within each index. Are you planning on sending these different types of data to different indices?


(andy_zhou) #3

in one index. it can add field about it .
like in es data:
"message": "{"dataType":"System.Single","host":"xxx","name":"xx/xxx","123":"222","time":"2018/5/3 8:03:22","value":"-6.5625"}",
if dataType is System.Single i add a filed as value_number the value is -6.5625
if datatype is System.String i add a filed as value_string the value is "stringtest"


(Christian Dahlqvist) #4

Yes, if you split it into different fields you can do that.


(andy_zhou) #5

how can configure the filter file
i don't know how to configure the .conf file
can help me?


(Christian Dahlqvist) #6

Use a json filter to parse the message. Then use conditionals on the dataType field to copy over data from value to the appropriate field name. Use the mutate filter to convert the type to float where appropriate. You can then remove the value field if you want to.


(andy_zhou) #7

i can't open use give url can give me a complete example as my ask.
thanks so much.. @Christian_Dahlqvist
:joy::joy::joy::joy::joy::joy::joy:


(andy_zhou) #8

the filter is error can't start logstash @Christian_Dahlqvist

filter 
{ 
  json {  
    source => "message"
    id => "json_train"
    if [message][dataType] == "System.Single"
     {
    add_field => { "value_number" => "%{value}"
     }
  }
  date {  
    match => [ "time", "YYYY/M/dd HH:mm:ss" ]
    timezone => "Asia/Shanghai"
    locale => "cn" 
    target => "report_time"
  }

}

(Christian Dahlqvist) #9

You can not have conditionals within a filter, so the conditionals need to be outside the json filter.

filter { 
  json {  
    source => "message"
    id => "json_train"
  }

  if [dataType] == "System.Single" {
    mutate {  
      add_field => { "value_number" => "%{value}"}
    }
  }
}

(andy_zhou) #10

is like this
also can't start :joy::joy::joy::joy::joy: can help me to slove the error?

filter
{
    if [message][dataType] == "System.Single"{
  json {
    source => "message"
    id => "json_train"
    add_field => { "value_number" => "%{value}"
     }
     } else {
  json {
    source => "message"
    id => "json_train1"
     add_field => { "value_string" => "%{value}"
     }
   }
  }
  date {
    match => [ "time", "YYYY/M/dd HH:mm:ss" ]
    timezone => "Asia/Shanghai"
    locale => "cn"
    target => "report_time"
  }

}

(Christian Dahlqvist) #11

You will not have access to the dataType field until you have run the json filter, which is why your version does not work. I spotted an error in my example and have updated it. Try that and continue to build out the logic within the conditionals until you have what you are looking for.


(andy_zhou) #12

use your new filter is also can't start with

[2018-05-03T17:16:26,761][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 35, column 17 (byte 545) after filter \n{ \n json { \n source => \"message\"\n id => \"json_train\"\n }\n\n if [dataType] == \"System.Single\" {\n mutate { \n add_field => { \"value_number\" => \"%{value}\"}\n }\n \n date {\n match => [ \"time\", \"YYYY/M/dd HH:mm:ss\" ]\n timezone => \"Asia/Shanghai\"\n locale => \"cn\"\n target => \"report_time\"\n }\n\n}\n\noutput {\n elasticsearch ", :backtrace=>["/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:169:ininitialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:315:inblock in converge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:312:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:299:inconverge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/opt/ES/logstash-6.2.3-es/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`


(andy_zhou) #13

use your give filter
also can't start
with

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 26, column 17 (byte 410) after filter { \n json { \n source => \"message\"\n id => \"json_train\"\n }\n\n if [dataType] == \"System.Single\" {\n mutate { \n add_field => { \"value_number\" => \"%{value}\"}\n }\n}\n\noutput {\n elasticsearch ", :backtrace=>["/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:42:incompile_imperative'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:50:in compile_graph'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:12:inblock in compile_sources'", "org/jruby/RubyArray.java:2486:in map'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/compiler.rb:11:incompile_sources'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:51:in initialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline.rb:169:ininitialize'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:315:inblock in converge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:312:inblock in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:299:inconverge_state'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/agent.rb:90:inexecute'", "/opt/ES/logstash-6.2.3-es/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/opt/ES/logstash-6.2.3-es/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}`


(Christian Dahlqvist) #14

Think I missed a curly brace. Have added it.


(andy_zhou) #15

as my full filter can't start can slove me where is error? :joy::joy::joy::joy::joy::joy::joy:
thanks @Christian_Dahlqvist

filter { 
  json {  
    source => "message"
    id => "json_train"
  }

  if [dataType] == "System.Single" {
    mutate {  
      add_field => { "value_number" => "%{value}"}
    }else if [dataType] == "System.Double" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    }else if [dataType] == "System.Int16" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    }else if [dataType] == "System.Int32" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    } else if [dataType] == "System.Int64" {
    mutate {
      add_field => { "value_number" => "%{value}"}
    } else if [dataType] == "System.Boolean" {
    mutate {
      add_field => { "value_Boolean" => "%{value}"}
    } else  {
    mutate {
      add_field => { "value_string" => "%{value}"}
    }
  }
}

(Christian Dahlqvist) #16

You are missing a number of curly braces before the else statements.


(andy_zhou) #17

thanks so much . slove it .
thanks so.
:elasticheart::elasticheart::elasticheart::elasticheart::elasticheart::elasticheart::elasticheart:


(andy_zhou) #18

the date filed value_Boolean is True or False , i wan't change to 1 or 0 in es .
how can do it ?
@Christian_Dahlqvist


(andy_zhou) #19
filter { 
  json {  
    source => "message"
    id => "json_train"
  }

  if [dataType] == "System.Single" {
    mutate {  
      add_field => { "value_number" => "%{value}"}
      } 
    } else if [dataType] == "System.Double" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Int16" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Int32" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Int64" {
    mutate {
      add_field => { "value_number" => "%{value}"}
      }
    } else if [dataType] == "System.Boolean" {
    mutate {
      add_field => { "value_Boolean" => "%{value}"}
      }
    } else  {
    mutate {
      add_field => { "value_string" => "%{value}"}
        }
    }

  date {
#    match => [ "time", "YYYY/M/dd HH:mm:ss +08:00" ]
    match => [ "time", "YYYY/M/dd HH:mm:ss" ]
    timezone => "Asia/Shanghai"
    locale => "cn"
    target => "report_time"
  }


}

(system) #20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.