# Not able to parse the incoming msg to json structure

**URL:** <https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504>\
**Category:** Logstash\
**Created:** [February 13, 2025, 1:25pm UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504 "2025-02-13T13:25:49Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 13, 2025, 1:25pm UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/1 "2025-02-13T13:25:49Z")

</div>

Hello,

Below is the message content:

```auto
# HELP Embargo_Index_seconds Time taken for Embargo Index\n# TYPE Embargo_Index_seconds summary\nEmbargo_Index_seconds_count{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",class=\"com.e2open.embargo.EmbargoIndex\",exception=\"none\",host=\"in7lin059\",method=\"createEmbargoIndex\"} 1\nEmbargo_Index_seconds_sum{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",class=\"com.e2open.embargo.EmbargoIndex\",exception=\"none\",host=\"in7lin059\",method=\"createEmbargoIndex\"} 1.762137559

```

Kindly let me know how can I split with \n character and ignore lines starting with #.

Thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 13, 2025, 3:18pm UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/2 "2025-02-13T15:18:58Z")

</div>

You can split a field on a newline using mutate+split. This does not use a regexp, so you need a literal newline in the configuration.

To iterate over an array and ignore lines that start with # you will need to use ruby.

This includes some other stuff you might want....

```
    mutate { split => { "someField" => "
" } }
    ruby {
        code => '
            a = event.get("someField")
            if a ; event.set("someField", a.reject { |x| x =~ /^#/ }) ; end
        '
    }
    split { field => "someField" }
    dissect { mapping => { "someField" => "%{type}{%{[@metadata][stuff]}} %{someNumber}" } }
    mutate { convert => { "someNumber" => "float" } }
    kv { source => "[@metadata][stuff]" target => "foo" field_split => "," trim_value => '\\"' }
}

```

which will create events like

```
       "foo" => {
         "CUST" => "RPS",
    "exception" => "none",
          "ENV" => "DEV",
       "method" => "createEmbargoIndex",
      "PRODUCT" => "GTM",
        "class" => "com.e2open.embargo.EmbargoIndex",
         "host" => "in7lin059"
},
 "someField" => "Embargo_Index_seconds_count{CUST=\\\"RPS\\\",ENV=\\\"DEV\\\",PRODUCT=\\\"GTM\\\",class=\\\"com.e2open.embargo.EmbargoIndex\\\",exception=\\\"none\\\",host=\\\"in7lin059\\\",method=\\\"createEmbargoIndex\\\"} 1",
"someNumber" => 1.0,
      "type" => "Embargo_Index_seconds_count"

```

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 13, 2025, 4:14pm UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/3 "2025-02-13T16:14:19Z")

</div>

> [@Badger](#):
>
> `Embargo_Index_seconds_count`

Pls let me know if i can get events like below:

```auto
	"Embargo_Index_seconds_count" => {
         "CUST" => "RPS",
    "exception" => "none",
          "ENV" => "DEV",
       "method" => "createEmbargoIndex",
      "PRODUCT" => "GTM",
        "class" => "com.e2open.embargo.EmbargoIndex",
         "host" => "in7lin059"
		 "value" => "1"
}

	"Embargo_Index_seconds_sum" => {
         "CUST" => "RPS",
    "exception" => "none",
          "ENV" => "DEV",
       "method" => "createEmbargoIndex",
      "PRODUCT" => "GTM",
        "class" => "com.e2open.embargo.EmbargoIndex",
         "host" => "in7lin059"
		 "value" => "1.762137559"
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 13, 2025, 6:04pm UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/4 "2025-02-13T18:04:57Z")

</div>

You could use

```
    mutate { split => { "someField" => "
" } }
    ruby {
        code => '
            a = event.get("someField")
            if a ; event.set("someField", a.reject { |x| x =~ /^#/ }) ; end
        '
    }
    split { field => "someField" }
    dissect { mapping => { "someField" => "%{[@metadata][type]}{%{[@metadata][stuff]}} %{[@metadata][value]}" } }
    # mutate { convert => { "[@metadata][value]" => "float" } }
    kv { source => "[@metadata][stuff]" target => "[@metadata][data]" field_split => "," trim_value => '\\"' }
    mutate { rename => { "[@metadata][value]" => "[@metadata][data][value]" } }
    mutate { rename => { "[@metadata][data]" => "%{[@metadata][type]}" } }

```

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 14, 2025, 2:03am UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/5 "2025-02-14T02:03:40Z")

</div>

> [@Badger](#):
>
> Below Logstash filter i used as per your suggestion:
> 
> ```auto
> filter
> {
> mutate { 
> split => { "message" => "
> " } 
> }
> ruby {
> code => '
> a = event.get("message")
> if a ; event.set("message", a.reject { |x| x =~ /^#/ }) ; end
> '
> }
> 	
> split { field => "message" }
> 	
> dissect { 
> mapping => 
> { 
> "message" => "%{[@metadata][type]}{%{[@metadata][stuff]}} %{[@metadata][value]}" 
> } 
> }
>    
> # mutate { convert => { "[@metadata][value]" => "float" } }
> 
> kv { 
> source => "[@metadata][stuff]" target => "[@metadata][data]" field_split => "," trim_value => '\\"' 
> }
>     
> mutate { rename => { "[@metadata][value]" => "[@metadata][data][value]" } }
>     
> mutate { rename => { "[@metadata][data]" => "%{[@metadata][type]}" } }
> 	
> }
> 
> ```

I get 100's of lines from http poller input, but mentioned below is the 4 lines input which is coming into logstash.

```auto
# HELP tasks_scheduled_execution_seconds  
# TYPE tasks_scheduled_execution_seconds summary
tasks_scheduled_execution_seconds_sum{CUST="RPS",ENV="DEV",PRODUCT="GTM",code_function="clearCacheConfigurationByScheduler",code_namespace="com.e2open.warningword.WarningwordService",error="none",exception="none",host="in7lin059",outcome="SUCCESS"} 0.037626831
# HELP tasks_scheduled_execution_seconds_max  
# TYPE tasks_scheduled_execution_seconds_max gauge
tasks_scheduled_execution_seconds_max{CUST="RPS",ENV="DEV",PRODUCT="GTM",code_function="clearCacheConfigurationByScheduler",code_namespace="com.e2open.warningword.WarningwordService",error="none",exception="none",host="in7lin059",outcome="SUCCESS"} 1.9311E-5

```

Below is the output, all the values are coming fine but for every event there is a json parse failure tag. pls let me know the issue

```auto
{
                                     "tags" => [
        [0] "_jsonparsefailure"
    ],
    "tasks_scheduled_execution_seconds_sum" => {
        "code_namespace" => "com.mdi.engine.service.impl.PenaltyConfigServiceInterfaceImplCache",
         "code_function" => "clearCacheConfigurationByScheduler",
                  "CUST" => "RPS",
                   "ENV" => "DEV",
               "PRODUCT" => "GTM",
             "exception" => "none",
                 "value" => "0.037626831",
                 "error" => "none",
                  "host" => "in7lin059",
               "outcome" => "SUCCESS"
    },
                                 "@version" => "1",
                               "@timestamp" => 2025-02-13T20:58:00.863226433Z,
                                  "message" => "tasks_scheduled_execution_seconds_sum{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",code_function=\"clearCacheConfigurationByScheduler\",code_namespace=\"com.mdi.engine.service.impl.PenaltyConfigServiceInterfaceImplCache\",error=\"none\",exception=\"none\",host=\"in7lin059\",outcome=\"SUCCESS\"} 0.037626831"
},
{
                                     "tags" => [
        [0] "_jsonparsefailure"
    ],
    "tasks_scheduled_execution_seconds_max" => {
        "code_namespace" => "com.mdi.engine.service.impl.CommonWordConfigInterfaceImplCache",
         "code_function" => "clearCacheConfigurationByScheduler",
                  "CUST" => "RPS",
                   "ENV" => "DEV",
               "PRODUCT" => "GTM",
             "exception" => "none",
                 "value" => "1.9311E-5",
                 "error" => "none",
                  "host" => "in7lin059",
               "outcome" => "SUCCESS"
    },
                                 "@version" => "1",
                               "@timestamp" => 2025-02-13T20:58:00.863226433Z,
                                  "message" => "tasks_scheduled_execution_seconds_max{CUST=\"RPS\",ENV=\"DEV\",PRODUCT=\"GTM\",code_function=\"clearCacheConfigurationByScheduler\",code_namespace=\"com.mdi.engine.service.impl.CommonWordConfigInterfaceImplCache\",error=\"none\",exception=\"none\",host=\"in7lin059\",outcome=\"SUCCESS\"} 0.0"
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 14, 2025, 2:20am UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/6 "2025-02-14T02:20:04Z")

</div>

> [@Bhanu\_Praveen](#):
>
> all the values are coming fine but for every event there is a json parse failure tag. pls let me know the issue

You haven't shown your logstash configuration, but using my psychic powers I see a json codec on an input. You should remove that.

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 14, 2025, 2:21am UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/7 "2025-02-14T02:21:33Z")

</div>

@Badger Pls find my input below:

```auto
input {
   http_poller {
      urls => 
		{
         rps => 
		 {
			 url => "http://Testserver.dev.com:42115/rpsservice/actuator/prometheus?tId=e5bf6fc7-457a-406f-b2df-1a38be998f51"
			headers => 
			{
				"Authorization" => "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IjU5NjQ0NTM4LWYzNmMtLtzuG_9EBeoFgdgySaDG1k7jsM7lAbbOXqZUEUzYdoKu9vy9nePdnq7V9-t7WmoSKP2EFxaSeVoXSHygVA-aFqg3ColmQJ6lyELVa9PMpq740H5cnus3HxrtnGHSvcwg2_LSLWX8Xo-jW_7MaBM4N9Sbn-hBxjeduAlvkqq-rlGR_yY4EVYGackg8SSmYBUwwi9Q6LZDw"
			}
		}
      }
	  keepalive => true
      automatic_retries => 1
      schedule => { cron => "* * * * * UTC"}
	  #metadata_target => "http_poller_metadata"
   }
     
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 14, 2025, 2:47am UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/8 "2025-02-14T02:47:11Z")

</div>

The default for an http\_poller input is to use a json codec, and that will give you that parse error.

It would be nice if the documentation was more explicit about the input expecting the API that it is calling to return (by default) JSON.

Try adding `codec => plain` or `codec => line` to your http\_poller.

If you use a line codec you may find that it creates a separate event for each line of the http\_poller output, in which case you will no longer need the mutate+split, or the split, and the ruby can be replaced by

```
if [message] =~ /^#/ { cancel {} }

```

which will nicely simplify your configuration!

---

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [February 15, 2025, 2:43am UTC](https://discuss.elastic.co/t/not-able-to-parse-the-incoming-msg-to-json-structure/374504/9 "2025-02-15T02:43:23Z")

</div>

Thanks a lot for your time and patience @Badger My bad didn't provide proper documentation about my input config. I Will TC next time.