Json Data not getting parsed when sent to Elasticsearch

Didjit · August 24, 2014, 1:54pm

Hi,

The following is a debug from Logstash:

{
"message" =>
"{"EventTime":"2014-08-24T09:44:46-0400","URI":"http://ME/rest/venue/ME/hours/2014-08-24","uri_payload":{"value":[{"open":"2014-08-24T13:00:00.000+0000","close":"2014-08-24T23:00:00.000+0000","isOpen":true,"date":"2014-08-24"}],"Count":1}}\r",
"@version" => "1",
"@timestamp" => "2014-08-24T13:44:48.036Z",
"host" => "127.0.0.1:60778",
"type" => "MY_Detail",
"EventTime" => "2014-08-24T09:44:46-0400",
"URI" => "http://ME/rest/venue/ME//hours/2014-08-24",
"uri_payload" => {
"value" => [
[0] {
"open" => "2014-08-24T13:00:00.000+0000",
"close" => "2014-08-24T23:00:00.000+0000",
"isOpen" => true,
"date" => "2014-08-24"
}
],
"Count" => 1,
"0" => {}
},
"MYId" => "ME"
}

When i look into Elasticsearch, the fields under URI Payload are not
parsed. It shows:

uri_payload.value as the field with "
{"open":"2014-08-21T13:00:00.000+0000","close":"2014-08-21T23:00:00.000+0000","isOpen":true,"date":"2014-08-21"}"

How can I get all the parsed values as fields in elasticsearch? In my
example, fields Open, Close, IsOpen. Initially I thought Logstash was not
parsing all the json, but looking at the debug it is.

Thank you,

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/af983e1d-a67b-4d9f-afa1-2a8c99c4d897%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

moshe_zada · August 24, 2014, 2:11pm

what is your logstash configuration?
did you tried the json codec http://logstash.net/docs/1.4.2/codecs/json?

On Sunday, August 24, 2014 4:54:08 PM UTC+3, Didjit wrote:

Hi,

The following is a debug from Logstash:

{
"message" =>
"{"EventTime":"2014-08-24T09:44:46-0400","URI":"
http://ME/rest/venue/ME/hours/2014-08-24\
","uri_payload":{"value":[{"open":"2014-08-24T13:00:00.000+0000","close":"2014-08-24T23:00:00.000+0000","isOpen":true,"date":"2014-08-24"}],"Count":1}}\r",
"@version" => "1",
"@timestamp" => "2014-08-24T13:44:48.036Z",
"host" => "127.0.0.1:60778",
"type" => "MY_Detail",
"EventTime" => "2014-08-24T09:44:46-0400",
"URI" => "http://ME/rest/venue/ME//hours/2014-08-24",
"uri_payload" => {
"value" => [
[0] {
"open" => "2014-08-24T13:00:00.000+0000",
"close" => "2014-08-24T23:00:00.000+0000",
"isOpen" => true,
"date" => "2014-08-24"
}
],
"Count" => 1,
"0" => {}
},
"MYId" => "ME"
}

When i look into Elasticsearch, the fields under URI Payload are not
parsed. It shows:

uri_payload.value as the field with "
{"open":"2014-08-21T13:00:00.000+0000","close":"2014-08-21T23:00:00.000+0000","isOpen":true,"date":"2014-08-21"}"

How can I get all the parsed values as fields in elasticsearch? In my
example, fields Open, Close, IsOpen. Initially I thought Logstash was not
parsing all the json, but looking at the debug it is.

Thank you,

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fe60df4d-cd36-43c9-a08c-7213abc2dd18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Didjit · August 24, 2014, 2:32pm

Pretty simple (below). . I just added to json codec and tried again and
received the same results. Thank you!

elasticsearch {
host => localhost
cluster => cjceswin
node_name => cjcnode
codec => json
index => "logstash-dwhse-%{+YYYY.MM.dd}"
workers => 3
}

}

On Sunday, August 24, 2014 10:11:44 AM UTC-4, moshe zada wrote:

what is your logstash configuration?
did you tried the json codec http://logstash.net/docs/1.4.2/codecs/json?

On Sunday, August 24, 2014 4:54:08 PM UTC+3, Didjit wrote:

Hi,

The following is a debug from Logstash:

{
"message" =>
"{"EventTime":"2014-08-24T09:44:46-0400","URI":"
http://ME/rest/venue/ME/hours/2014-08-24\
","uri_payload":{"value":[{"open":"2014-08-24T13:00:00.000+0000","close":"2014-08-24T23:00:00.000+0000","isOpen":true,"date":"2014-08-24"}],"Count":1}}\r",
"@version" => "1",
"@timestamp" => "2014-08-24T13:44:48.036Z",
"host" => "127.0.0.1:60778",
"type" => "MY_Detail",
"EventTime" => "2014-08-24T09:44:46-0400",
"URI" => "http://ME/rest/venue/ME//hours/2014-08-24",
"uri_payload" => {
"value" => [
[0] {
"open" => "2014-08-24T13:00:00.000+0000",
"close" => "2014-08-24T23:00:00.000+0000",
"isOpen" => true,
"date" => "2014-08-24"
}
],
"Count" => 1,
"0" => {}
},
"MYId" => "ME"
}

When i look into Elasticsearch, the fields under URI Payload are not
parsed. It shows:

uri_payload.value as the field with "
{"open":"2014-08-21T13:00:00.000+0000","close":"2014-08-21T23:00:00.000+0000","isOpen":true,"date":"2014-08-21"}"

How can I get all the parsed values as fields in elasticsearch? In my
example, fields Open, Close, IsOpen. Initially I thought Logstash was not
parsing all the json, but looking at the debug it is.

Thank you,

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0afd4105-a521-487a-8889-4bcabee419b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Didjit · August 25, 2014, 4:31pm

bump. Anyone?
Thank you,
Chris

On Sunday, August 24, 2014 10:32:23 AM UTC-4, Didjit wrote:

Pretty simple (below). . I just added to json codec and tried again and
received the same results. Thank you!

elasticsearch {
host => localhost
cluster => cjceswin
node_name => cjcnode
codec => json
index => "logstash-dwhse-%{+YYYY.MM.dd}"
workers => 3
}

}

On Sunday, August 24, 2014 10:11:44 AM UTC-4, moshe zada wrote:

what is your logstash configuration?
did you tried the json codec http://logstash.net/docs/1.4.2/codecs/json
?

On Sunday, August 24, 2014 4:54:08 PM UTC+3, Didjit wrote:

Hi,

The following is a debug from Logstash:

{
"message" =>
"{"EventTime":"2014-08-24T09:44:46-0400","URI":"
http://ME/rest/venue/ME/hours/2014-08-24\
","uri_payload":{"value":[{"open":"2014-08-24T13:00:00.000+0000","close":"2014-08-24T23:00:00.000+0000","isOpen":true,"date":"2014-08-24"}],"Count":1}}\r",
"@version" => "1",
"@timestamp" => "2014-08-24T13:44:48.036Z",
"host" => "127.0.0.1:60778",
"type" => "MY_Detail",
"EventTime" => "2014-08-24T09:44:46-0400",
"URI" => "http://ME/rest/venue/ME//hours/2014-08-24",
"uri_payload" => {
"value" => [
[0] {
"open" => "2014-08-24T13:00:00.000+0000",
"close" => "2014-08-24T23:00:00.000+0000",
"isOpen" => true,
"date" => "2014-08-24"
}
],
"Count" => 1,
"0" => {}
},
"MYId" => "ME"
}

When i look into Elasticsearch, the fields under URI Payload are not
parsed. It shows:

uri_payload.value as the field with "
{"open":"2014-08-21T13:00:00.000+0000","close":"2014-08-21T23:00:00.000+0000","isOpen":true,"date":"2014-08-21"}"

How can I get all the parsed values as fields in elasticsearch? In my
example, fields Open, Close, IsOpen. Initially I thought Logstash was not
parsing all the json, but looking at the debug it is.

Thank you,

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/860ab9c6-1867-43d0-b5da-12660ac7eab0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Didjit · August 27, 2014, 5:12pm

FYI, I used this fine persons chunk of code(last post) in my Logstash
filter.

Sharing,

Thank you

Chris

On Monday, August 25, 2014 12:31:24 PM UTC-4, Didjit wrote:

bump. Anyone?
Thank you,
Chris

On Sunday, August 24, 2014 10:32:23 AM UTC-4, Didjit wrote:

Pretty simple (below). . I just added to json codec and tried again and
received the same results. Thank you!

elasticsearch {
host => localhost
cluster => cjceswin
node_name => cjcnode
codec => json
index => "logstash-dwhse-%{+YYYY.MM.dd}"
workers => 3
}

}

On Sunday, August 24, 2014 10:11:44 AM UTC-4, moshe zada wrote:

what is your logstash configuration?
did you tried the json codec
http://logstash.net/docs/1.4.2/codecs/json?

On Sunday, August 24, 2014 4:54:08 PM UTC+3, Didjit wrote:

Hi,

The following is a debug from Logstash:

{
"message" =>
"{"EventTime":"2014-08-24T09:44:46-0400","URI":"
http://ME/rest/venue/ME/hours/2014-08-24\
","uri_payload":{"value":[{"open":"2014-08-24T13:00:00.000+0000","close":"2014-08-24T23:00:00.000+0000","isOpen":true,"date":"2014-08-24"}],"Count":1}}\r",
"@version" => "1",
"@timestamp" => "2014-08-24T13:44:48.036Z",
"host" => "127.0.0.1:60778",
"type" => "MY_Detail",
"EventTime" => "2014-08-24T09:44:46-0400",
"URI" => "http://ME/rest/venue/ME//hours/2014-08-24",
"uri_payload" => {
"value" => [
[0] {
"open" => "2014-08-24T13:00:00.000+0000",
"close" => "2014-08-24T23:00:00.000+0000",
"isOpen" => true,
"date" => "2014-08-24"
}
],
"Count" => 1,
"0" => {}
},
"MYId" => "ME"
}

When i look into Elasticsearch, the fields under URI Payload are not
parsed. It shows:

uri_payload.value as the field with "
{"open":"2014-08-21T13:00:00.000+0000","close":"2014-08-21T23:00:00.000+0000","isOpen":true,"date":"2014-08-21"}"

How can I get all the parsed values as fields in elasticsearch? In my
example, fields Open, Close, IsOpen. Initially I thought Logstash was not
parsing all the json, but looking at the debug it is.

Thank you,

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1ec009ee-5914-466a-a6e5-378ec0801625%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.