Need help retrieving field from ES

Hi All

I am new to ElasticSearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ee76e0af-4e64-430e-89f7-463dea1acfed%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

keepalive field is stored in _source field (if you want to store it
separately you have to add "store" : true to mapping)
hard to tell more based on your example,
also maybe you disabled _source field completely?

On Monday, December 23, 2013 8:40:17 PM UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ddd4df2-b2f6-4a3f-8ff5-5a0196f389d7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Karol

Thanks for the reply. We have been left this ES setup by a previous member
of staff.

We create new Indexes every hour using the following the following perl
statement

Are you saying I have to add store => yes to keepalive?
We don't do that for the other as you can see?

create_index(
index => $index,
settings => {
_timestamp => { enabled => 1, store
=> 1 },
number_of_shards => 3,
number_of_replicas => 1,
},
mappings => {
varnish => {
_timestamp => { enabled => 1,
store => 1 },
properties => {
content_length => { type =>
'integer' },
age => { type => 'integer' },
keepalive => { type =>
'integer' },
resp_time => { type =>
'float' },
host => { type =>
'string', index => 'not_analyzed' },
time => { type =>
'string', store => 'yes' },

location => { type =>
'string', index => 'not_analyzed' },
addr => {
fields => {
ip => { type =>
'ip' },
addr => { type =>
'string', index => 'not_analyzed' },
}
},
}
}
},
);

I would no if the _source field has been disabled, how do I check? Does
this help more:

{

_index: 2013122312

_type: log

_id: Juh_YQJaT4GQ8Pjwk1bnqw

_score: 1

_source: {

protocol: HTTP/1.0

cdn: -

vary: Accept-Encoding,ETag

browser: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.1)
GeckaSeka/20090911 Firefox/3.5.1

encoding: -

location: -

geo: US

ref: -

origin: -

cookie: -

uri: /

cache_control: -

content_length: 54053

userid: 0

age: 11556

resp_time: 0.000110149

method: GET

accept: -

ssl: 0

response_code: 200

accept_language: -

varnstat: hit

_src: log

addr: 41.5.97.6

}

}

Do you need anymore information to help?

Thanks again

Nick

On Tue, Dec 24, 2013 at 2:27 AM, Karol Gwaj karol@gwaj.me wrote:

keepalive field is stored in _source field (if you want to store it
separately you have to add "store" : true to mapping)
hard to tell more based on your example,
also maybe you disabled _source field completely?

On Monday, December 23, 2013 8:40:17 PM UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/roVCeLImQxs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4ddd4df2-b2f6-4a3f-8ff5-5a0196f389d7%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMWL0yPqpG6H5gC58UWZHFodwOV14WM4P_4LpovA_mTB%3DDkG2A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Can anyone help me please?

Many thanks

On Monday, 23 December 2013 20:40:17 UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/be2f3251-283c-4f7b-b5f6-09e807f7fe84%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I think you will get more help if you follow what is explained here: Elasticsearch Platform — Find real-time answers at scale | Elastic

Hard to help without more information.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 janv. 2014 à 18:44, Nick Toseland nick.toseland@gmail.com a écrit :

Can anyone help me please?

Many thanks

On Monday, 23 December 2013 20:40:17 UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/be2f3251-283c-4f7b-b5f6-09e807f7fe84%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B21110F6-14E0-4301-AD38-0113086FC471%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for the pointers David.

I am using the Elasticsearch under Perl, to retrieve data based on facets,
however the keepalive field is not returned. I guess this is because it is
not in the _source. However I can’t seem to get that field to appear.

We create new indexes every hour, the last 24hours form an alias.

I have created a gist with the search and the mapping from the index; here

Thanks again

Nick

On Thursday, 2 January 2014 19:32:22 UTC, David Pilato wrote:

I think you will get more help if you follow what is explained here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Hard to help without more information.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 janv. 2014 à 18:44, Nick Toseland <nick.t...@gmail.com <javascript:>>
a écrit :

Can anyone help me please?

Many thanks

On Monday, 23 December 2013 20:40:17 UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/be2f3251-283c-4f7b-b5f6-09e807f7fe84%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/617940e3-1196-4bc5-b9df-dc97886e7001%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Judging by the one sample document, the keepalive field is not there. You
can use the missing filter to see if any documents do have that field. For
example:

curl -XPOST localhost:9200/2014010119/_count/ -d '
{
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "keepalive"
}
}
}
}
}
}
'

--
Ivan

On Thu, Jan 2, 2014 at 12:35 PM, Nick Toseland nick.toseland@gmail.comwrote:

Thanks for the pointers David.

I am using the Elasticsearch under Perl, to retrieve data based on
facets, however the keepalive field is not returned. I guess this is
because it is not in the _source. However I can’t seem to get that field to
appear.

We create new indexes every hour, the last 24hours form an alias.

I have created a gist with the search and the mapping from the index; here
ElasticSearch Field not returned/stored · GitHub

Thanks again

Nick

On Thursday, 2 January 2014 19:32:22 UTC, David Pilato wrote:

I think you will get more help if you follow what is explained here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Hard to help without more information.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 janv. 2014 à 18:44, Nick Toseland nick.t...@gmail.com a écrit :

Can anyone help me please?

Many thanks

On Monday, 23 December 2013 20:40:17 UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/be2f3251-283c-4f7b-b5f6-09e807f7fe84%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/617940e3-1196-4bc5-b9df-dc97886e7001%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDMK%2BUuKW5XbRQMoT0XG_VixhU%2BJAnFrcem_dCvL4%3D5Cw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan,

That command confirms that no docs have the keepalive.

My next silly question, is how do i get the field to appear? The mapping is
there, are the options correct?
Will the field appear when ES is loaded with data?

Excuse my stupid questions, but I thought the field would be created if an
index was created that had a mapping for keepalive.

Thanks

Nick

On Thursday, 2 January 2014 20:45:46 UTC, Ivan Brusic wrote:

Judging by the one sample document, the keepalive field is not there. You
can use the missing filter to see if any documents do have that field. For
example:

curl -XPOST localhost:9200/2014010119/_count/ -d '
{
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "keepalive"
}
}
}
}
}
}
'

--
Ivan

On Thu, Jan 2, 2014 at 12:35 PM, Nick Toseland <nick.t...@gmail.com<javascript:>

wrote:

Thanks for the pointers David.

I am using the Elasticsearch under Perl, to retrieve data based on
facets, however the keepalive field is not returned. I guess this is
because it is not in the _source. However I can’t seem to get that field to
appear.

We create new indexes every hour, the last 24hours form an alias.

I have created a gist with the search and the mapping from the index;
here ElasticSearch Field not returned/stored · GitHub

Thanks again

Nick

On Thursday, 2 January 2014 19:32:22 UTC, David Pilato wrote:

I think you will get more help if you follow what is explained here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Hard to help without more information.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 janv. 2014 à 18:44, Nick Toseland nick.t...@gmail.com a écrit :

Can anyone help me please?

Many thanks

On Monday, 23 December 2013 20:40:17 UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/be2f3251-283c-4f7b-b5f6-09e807f7fe84%
40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/617940e3-1196-4bc5-b9df-dc97886e7001%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/24223a0b-7cb6-4a2b-8240-e633f6b80266%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Lucene (and therefore elasticsearch) is schemaless, so every document in
the same index does not need to have the same structure. Just because the
mapping contains the keepalive does not mean the document does. You can
enforce that every field in the document must exist in the mapping [1], but
not the inverse AFAIK.

Something in your indexing process is not adding the keepalive field. You
should double check that process first. Debug what is sent to elasticsearch
during indexing.

[1]

Cheers,

Ivan

On Thu, Jan 2, 2014 at 1:28 PM, Nick Toseland nick.toseland@gmail.comwrote:

Ivan,

That command confirms that no docs have the keepalive.

My next silly question, is how do i get the field to appear? The mapping
is there, are the options correct?
Will the field appear when ES is loaded with data?

Excuse my stupid questions, but I thought the field would be created if an
index was created that had a mapping for keepalive.

Thanks

Nick

On Thursday, 2 January 2014 20:45:46 UTC, Ivan Brusic wrote:

Judging by the one sample document, the keepalive field is not there. You
can use the missing filter to see if any documents do have that field. For
example:

curl -XPOST localhost:9200/2014010119/_count/ -d '
{
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "keepalive"
}
}
}
}
}
}
'

--
Ivan

On Thu, Jan 2, 2014 at 12:35 PM, Nick Toseland nick.t...@gmail.comwrote:

Thanks for the pointers David.

I am using the Elasticsearch under Perl, to retrieve data based on
facets, however the keepalive field is not returned. I guess this is
because it is not in the _source. However I can’t seem to get that field to
appear.

We create new indexes every hour, the last 24hours form an alias.

I have created a gist with the search and the mapping from the index;
here ElasticSearch Field not returned/stored · GitHub

Thanks again

Nick

On Thursday, 2 January 2014 19:32:22 UTC, David Pilato wrote:

I think you will get more help if you follow what is explained here:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Hard to help without more information.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 janv. 2014 à 18:44, Nick Toseland nick.t...@gmail.com a écrit :

Can anyone help me please?

Many thanks

On Monday, 23 December 2013 20:40:17 UTC, Nick Toseland wrote:

Hi All

I am new to Elasticsearch, please forgive my stupidity.

I cant seem to get the keepalive field out of ES.

{
"_index" : "lj-2013122320",
"_type" : "varnish",
"_id" : "Y1M18ZItTDaap_rOAS5YOA",
"_score" : 1.0
}

I can get other field out of it cdn:

{
"_index" : "2013122320",
"_type" : "log",
"_id" : "2neLlVNKQCmXq6etTE6Kcw",
"_score" : 1.0,
"fields" : {
"cdn" : "-"
}
}

The mapping is there:

{"log":{"_timestamp":{"enabled":true,"store":true},"properties":
{"keepalive":{"type":"integer"}}}}

Any help is much appreciated.

Thanks in advance

Nick

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/ms
gid/elasticsearch/be2f3251-283c-4f7b-b5f6-09e807f7fe84%40goo
glegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/617940e3-1196-4bc5-b9df-dc97886e7001%
40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/24223a0b-7cb6-4a2b-8240-e633f6b80266%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDWeinrMcY3%3Djm04xbqkyOZKx2zmSHduCb2GgKFQ2rpMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.