ELK 6 and Future 7 and single type indexes

Elastic Stack Elasticsearch
1

I am trying to figure out the details on a single type index as I am on Elk 5 and upgrading to ELK6 and will upgrade to elk7 as soon as it is in beta.

My question is, since the _type field is getting removed, but I am heavily dependant on it for searches, dashboards monitoring etc. Is there a way I can just "Disable" the field of having any special meaning? in ELK6 ( is _type the same as type ?)

I know I could create different indexes for each type, but that would be over 300 different indexes each day. I probably could just set the same type for them all as I am just doing log aggregation, but then I would have to change every kibana dashboard, nagios query string, and any other developer process, as well as re-educate more user base to use a different field.

2

Did you read yet: https://www.elastic.co/blog/removal-of-mapping-types-elasticsearch ?

3

Yes, i have read through it well actually https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html and was going to read it again but it covers so many different pieces that makes it vague in my mind.

I get creating a custom type field but in the past the _type and type have the same value in prior versions. As I never created a type field but yet it exists and always seems to be the same value as _type.

I guess the confusion is not the actual field in the index that I have a problem with, but how to deal with it with Logstash. maybe this should have been under logstash as a question instead

Since I set the "type" => "sometype" in each of the file inputs. I am not sure what I need to do and instead of just experimenting thought I would ask.

input{ file { path => "/var/log/message" type => "somevalue" }}

do I need to do something like
'input{ file {path => "/var/log/message" type=> "log", add_field => {"type","somvalue"}}}

I guess it could also be mutated in a filter too but, I am trying to figure how to transition my existing rules with the least amount of effort.

I was just thinking about doing as I suggested by doing a mutate to "modify" _type to a common value which will allow others to keep using "type" field without an issue.

4

ok, found the solution to my logstash delema

just setting the
output{
elasticsearch {
...
...
document_type =>
}
}

5

Just use the tags directive in Logstash, it'll add a similar thing.

6

How will adding a "tags" directive fix having multiple different _type values in one index?

7

Because you then don't define document_type and let the default apply.

8

I don't understand, but I will give it a try and see the results. (And read up on the details)

That is the best part of Dev cluster, I get to try anything! :slight_smile:

9

_type is just a lucene field, it was just used in a way in Elasticsearch that wasn't the best.

So all you are doing is moving the values from _type into tags.

10

Ah I got yah,

Oh I am not worrying about losing the _type value, "type" is what everyone
here uses but in the past _type and type got set to the same value. So
while I am using Logstash 2.3 on my remote systems, Kafka .09 and Logstash
Indexer 5.6 talking to ES 6. I have to manipulate the _type so I can
keep the same functionality.

This will "supper" hybrid environment won't last as I am upgrading
everything to 6 but I need some transition time so just trying to what is
easy to get from point A to point Z with the least amount of effort.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.