Disabling _all-Field but keep Netflow-Events searchable


(Horst Birne) #1

Hey guys,

I really want to disable the _all-Field in the ES-Indices to save some
disk-space on our system.

Normally its not the problem - adjust template in ES, and set
"message"-Field to the new default query field, that is normally available
in any event.

The problem is that we also have many netflow-events with the netflow-codec
that have the following form:

https://lh4.googleusercontent.com/-CDQQs5e5a7o/U6lUvjikncI/AAAAAAAAACo/LHpMXlYLMWw/s1600/netflow.PNG

As you might notice there isnt any "message"-field so the Kibana lucene
query would run into an error.

My question is - how do i manage it to make this work (disabling _all-Field
but search in the netflow-events)?

Thanks for response.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9ab09bba-392f-4f77-8937-aa518c22292f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Alexander Reelsen) #2

Hey,

you can set the index.query.default_field in the mapping to circumvent
this, see
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html#mapping-all-field

--Alex

On Tue, Jun 24, 2014 at 12:39 PM, horst knete baduncle23@hotmail.de wrote:

Hey guys,

I really want to disable the _all-Field in the ES-Indices to save some
disk-space on our system.

Normally its not the problem - adjust template in ES, and set
"message"-Field to the new default query field, that is normally available
in any event.

The problem is that we also have many netflow-events with the
netflow-codec that have the following form:

https://lh4.googleusercontent.com/-CDQQs5e5a7o/U6lUvjikncI/AAAAAAAAACo/LHpMXlYLMWw/s1600/netflow.PNG

As you might notice there isnt any "message"-field so the Kibana lucene
query would run into an error.

My question is - how do i manage it to make this work (disabling
_all-Field but search in the netflow-events)?

Thanks for response.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9ab09bba-392f-4f77-8937-aa518c22292f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9ab09bba-392f-4f77-8937-aa518c22292f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9N1nEPEZi%3D%3DqA4_FDDTjV_twrAcNQZA57Hi1XvxW2OLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Horst Birne) #3

Hey,

thx for your response.

As i already mentioned i tried setting the index.query.default_field to
"message".

{
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s",
"index.number_of_shards" : 3,
"index.number_of_replicas" : 0,
"index.refresh_interval" : "30s",
"index.store.compress.stored" : true,
"index.store.compress.tv" : true,
"index.query.default_field" : "message",
"analysis" : {
"analyzer" : {
"default" : {
"type" : "standard",
"stopwords" : "none"
}
}
}
},
"mappings" : {
"default" : {
"_all" : {"enabled" : false},
"_source": { "compress": true },
"dynamic_templates" : [ {
"string_fields" : {
"match" : "
",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string",
"fields" : {
"{name}" : {"type": "string", "index" : "not_analyzed"}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"@timestamp" : { "type" : "date", "index" : "not_analyzed" },
"tags": { "type": "string", "index" : "not_analyzed" }
}
}
}
}

That was the template that i used. This is working fine for all Events
except the Netflow ones, because they dont have a "message"-field for
Kibana to search in. Thats what my mess is.

Is it possible to adjust the template/mapping per type of event?

Cheers

Am Montag, 30. Juni 2014 09:08:22 UTC+2 schrieb Alexander Reelsen:

Hey,

you can set the index.query.default_field in the mapping to circumvent
this, see
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html#mapping-all-field

--Alex

On Tue, Jun 24, 2014 at 12:39 PM, horst knete <badun...@hotmail.de
<javascript:>> wrote:

Hey guys,

I really want to disable the _all-Field in the ES-Indices to save some
disk-space on our system.

Normally its not the problem - adjust template in ES, and set
"message"-Field to the new default query field, that is normally available
in any event.

The problem is that we also have many netflow-events with the
netflow-codec that have the following form:

https://lh4.googleusercontent.com/-CDQQs5e5a7o/U6lUvjikncI/AAAAAAAAACo/LHpMXlYLMWw/s1600/netflow.PNG

As you might notice there isnt any "message"-field so the Kibana lucene
query would run into an error.

My question is - how do i manage it to make this work (disabling
_all-Field but search in the netflow-events)?

Thanks for response.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9ab09bba-392f-4f77-8937-aa518c22292f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9ab09bba-392f-4f77-8937-aa518c22292f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9f406910-1608-4866-8c9c-42a23f6d8f11%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Horst Birne) #4

Anyone got an idea how to realize that? I think that there are a few uses
which got Netflow AND other types of events inserting into Elasticsearch
and for those a disabled _all Field would save much hard disk space

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2b19c405-46b0-483b-9d22-73fa0c6dca5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #5

This technically sounds like a Kibana question, so you might have better
luck with the Logstash mailing list.

Can't you simply prepend the field name in the query instead of relying on
the default field? You can also change field names in Logstash.

Another option is the copy-to-field. Similar to _all, but with more
flexibility.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#copy-to

Cheers,

Ivan

On Mon, Jul 14, 2014 at 4:12 AM, horst knete baduncle23@hotmail.de wrote:

Anyone got an idea how to realize that? I think that there are a few uses
which got Netflow AND other types of events inserting into Elasticsearch
and for those a disabled _all Field would save much hard disk space

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2b19c405-46b0-483b-9d22-73fa0c6dca5b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2b19c405-46b0-483b-9d22-73fa0c6dca5b%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQC2woJNr301UbpbzA6fh6SAh2SJf8NE9NKtr9dh-M8_3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6