Not able to parse the incoming msg to json structure

Elastic Stack Logstash
1

Hello,

Below is the message content:

# HELP Embargo_Index_seconds Time taken for Embargo Index\n# TYPE Embargo_Index_seconds summary\nEmbargo_Index_seconds_count{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",class=\"com.e2open.embargo.EmbargoIndex\",exception=\"none\",host=\"in7lin059\",method=\"createEmbargoIndex\"} 1\nEmbargo_Index_seconds_sum{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",class=\"com.e2open.embargo.EmbargoIndex\",exception=\"none\",host=\"in7lin059\",method=\"createEmbargoIndex\"} 1.762137559

Kindly let me know how can I split with \n character and ignore lines starting with #.

Thanks

2

You can split a field on a newline using mutate+split. This does not use a regexp, so you need a literal newline in the configuration.

To iterate over an array and ignore lines that start with # you will need to use ruby.

This includes some other stuff you might want....

    mutate { split => { "someField" => "
" } }
    ruby {
        code => '
            a = event.get("someField")
            if a ; event.set("someField", a.reject { |x| x =~ /^#/ }) ; end
        '
    }
    split { field => "someField" }
    dissect { mapping => { "someField" => "%{type}{%{[@metadata][stuff]}} %{someNumber}" } }
    mutate { convert => { "someNumber" => "float" } }
    kv { source => "[@metadata][stuff]" target => "foo" field_split => "," trim_value => '\\"' }
}

which will create events like

       "foo" => {
         "CUST" => "RPS",
    "exception" => "none",
          "ENV" => "DEV",
       "method" => "createEmbargoIndex",
      "PRODUCT" => "GTM",
        "class" => "com.e2open.embargo.EmbargoIndex",
         "host" => "in7lin059"
},
 "someField" => "Embargo_Index_seconds_count{CUST=\\\"RPS\\\",ENV=\\\"DEV\\\",PRODUCT=\\\"GTM\\\",class=\\\"com.e2open.embargo.EmbargoIndex\\\",exception=\\\"none\\\",host=\\\"in7lin059\\\",method=\\\"createEmbargoIndex\\\"} 1",
"someNumber" => 1.0,
      "type" => "Embargo_Index_seconds_count"
3

Pls let me know if i can get events like below:

	"Embargo_Index_seconds_count" => {
         "CUST" => "RPS",
    "exception" => "none",
          "ENV" => "DEV",
       "method" => "createEmbargoIndex",
      "PRODUCT" => "GTM",
        "class" => "com.e2open.embargo.EmbargoIndex",
         "host" => "in7lin059"
		 "value" => "1"
}

	"Embargo_Index_seconds_sum" => {
         "CUST" => "RPS",
    "exception" => "none",
          "ENV" => "DEV",
       "method" => "createEmbargoIndex",
      "PRODUCT" => "GTM",
        "class" => "com.e2open.embargo.EmbargoIndex",
         "host" => "in7lin059"
		 "value" => "1.762137559"
}
4

You could use

    mutate { split => { "someField" => "
" } }
    ruby {
        code => '
            a = event.get("someField")
            if a ; event.set("someField", a.reject { |x| x =~ /^#/ }) ; end
        '
    }
    split { field => "someField" }
    dissect { mapping => { "someField" => "%{[@metadata][type]}{%{[@metadata][stuff]}} %{[@metadata][value]}" } }
    # mutate { convert => { "[@metadata][value]" => "float" } }
    kv { source => "[@metadata][stuff]" target => "[@metadata][data]" field_split => "," trim_value => '\\"' }
    mutate { rename => { "[@metadata][value]" => "[@metadata][data][value]" } }
    mutate { rename => { "[@metadata][data]" => "%{[@metadata][type]}" } }
5

I get 100's of lines from http poller input, but mentioned below is the 4 lines input which is coming into logstash.

# HELP tasks_scheduled_execution_seconds  
# TYPE tasks_scheduled_execution_seconds summary
tasks_scheduled_execution_seconds_sum{CUST="RPS",ENV="DEV",PRODUCT="GTM",code_function="clearCacheConfigurationByScheduler",code_namespace="com.e2open.warningword.WarningwordService",error="none",exception="none",host="in7lin059",outcome="SUCCESS"} 0.037626831
# HELP tasks_scheduled_execution_seconds_max  
# TYPE tasks_scheduled_execution_seconds_max gauge
tasks_scheduled_execution_seconds_max{CUST="RPS",ENV="DEV",PRODUCT="GTM",code_function="clearCacheConfigurationByScheduler",code_namespace="com.e2open.warningword.WarningwordService",error="none",exception="none",host="in7lin059",outcome="SUCCESS"} 1.9311E-5

Below is the output, all the values are coming fine but for every event there is a json parse failure tag. pls let me know the issue

{
                                     "tags" => [
        [0] "_jsonparsefailure"
    ],
    "tasks_scheduled_execution_seconds_sum" => {
        "code_namespace" => "com.mdi.engine.service.impl.PenaltyConfigServiceInterfaceImplCache",
         "code_function" => "clearCacheConfigurationByScheduler",
                  "CUST" => "RPS",
                   "ENV" => "DEV",
               "PRODUCT" => "GTM",
             "exception" => "none",
                 "value" => "0.037626831",
                 "error" => "none",
                  "host" => "in7lin059",
               "outcome" => "SUCCESS"
    },
                                 "@version" => "1",
                               "@timestamp" => 2025-02-13T20:58:00.863226433Z,
                                  "message" => "tasks_scheduled_execution_seconds_sum{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",code_function=\"clearCacheConfigurationByScheduler\",code_namespace=\"com.mdi.engine.service.impl.PenaltyConfigServiceInterfaceImplCache\",error=\"none\",exception=\"none\",host=\"in7lin059\",outcome=\"SUCCESS\"} 0.037626831"
},
{
                                     "tags" => [
        [0] "_jsonparsefailure"
    ],
    "tasks_scheduled_execution_seconds_max" => {
        "code_namespace" => "com.mdi.engine.service.impl.CommonWordConfigInterfaceImplCache",
         "code_function" => "clearCacheConfigurationByScheduler",
                  "CUST" => "RPS",
                   "ENV" => "DEV",
               "PRODUCT" => "GTM",
             "exception" => "none",
                 "value" => "1.9311E-5",
                 "error" => "none",
                  "host" => "in7lin059",
               "outcome" => "SUCCESS"
    },
                                 "@version" => "1",
                               "@timestamp" => 2025-02-13T20:58:00.863226433Z,
                                  "message" => "tasks_scheduled_execution_seconds_max{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",code_function=\"clearCacheConfigurationByScheduler\",code_namespace=\"com.mdi.engine.service.impl.CommonWordConfigInterfaceImplCache\",error=\"none\",exception=\"none\",host=\"in7lin059\",outcome=\"SUCCESS\"} 0.0"
}
6

You haven't shown your logstash configuration, but using my psychic powers I see a json codec on an input. You should remove that.

7

@Badger Pls find my input below:

input {
   http_poller {
      urls => 
		{
         rps => 
		 {
			 url => "http://Testserver.dev.com:42115/rpsservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51"
			headers => 
			{
				"Authorization" => "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjU5NjQ0NTM4LWYzNmMtLtzuG_9EBeoFgdgySaDG1k7jsM7lAbbOXqZUEUzYdoKu9vy9nePdnq7V9-t7WmoSKP2EFxaSeVoXSHygVA-aFqg3ColmQJ6lyELVa9PMpq740H5cnus3HxrtnGHSvcwg2_LSLWX8Xo-jW_7MaBM4N9Sbn-hBxjeduAlvkqq-rlGR_yY4EVYGackg8SSmYBUwwi9Q6LZDw"
			}
		}
      }
	  keepalive => true
      automatic_retries => 1
      schedule => { cron => "* * * * * UTC"}
	  #metadata_target => "http_poller_metadata"
   }
     
}
8

The default for an http_poller input is to use a json codec, and that will give you that parse error.

It would be nice if the documentation was more explicit about the input expecting the API that it is calling to return (by default) JSON.

Try adding codec => plain or codec => line to your http_poller.

If you use a line codec you may find that it creates a separate event for each line of the http_poller output, in which case you will no longer need the mutate+split, or the split, and the ruby can be replaced by

if [message] =~ /^#/ { cancel {} }

which will nicely simplify your configuration!

1 Like
9

Thanks a lot for your time and patience @Badger My bad didn't provide proper documentation about my input config. I Will TC next time.