Index Mapping/Routing Help

Elastic Stack Elasticsearch
1

Hello,

Currently I have the following setup.

Syslog --> Logstash --> ElasticSearch --> Kibana

Logstash is creating a daily index
"/etc/elasticsearch/data/test-elasticsearch/nodes/0/indices/logstash-2014.02.04"
and I'm viewing all of the logs through Kibana. We want to set up some user
based access control using the kibana-authentication-proxy setup due to it
supporting

  • Per-user kibana index supported. now you can use index
    kibana-int-userA for user A and kibana-int-userB for user B

I'd like to make it where all logs coming in from logstash with a location
of "/var/log/UNIX/*.log" get sent to a new index of unix-2014.02.04 instead
of the logstash one. That way I can use the Kibana auth proxy to give my
UNIX users access only to their logs. I've read a little about creating the
mappings but wasn't sure how to tie it all together. I saw you could do
various things with API calls but was curious if I could set all of this up
in the elasticsearch.yml file from the start.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/56e2fc09-c179-4839-a23f-67a805f563ce%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

You should be able to use the input type to direct log events to specific
indexes. For example:

input {
file {
type => "unixlogs"
path => "/var/log/UNIX/*.log"
}
}

output {
if [type] == "unixlogs" {
elasticsearch {
host => "localhost"
index => "unix-%{+YYYY.MM.dd}"
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5107e623-4bf5-46d3-8d9c-83de77f95bd5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks for that information. When I'm looking in Kibana now, it's showing
the correct type but it still shows the index as the original
logstash-2014-02-18. Not sure why it isn't showing the unix-date index. If
I look at Elasticsearch, I can see that it did create the new index I told
it to though.

On Tuesday, February 18, 2014 12:53:22 PM UTC-5, Binh Ly wrote:

You should be able to use the input type to direct log events to specific
indexes. For example:

input {
file {
type => "unixlogs"
path => "/var/log/UNIX/*.log"
}
}

output {
if [type] == "unixlogs" {
elasticsearch {
host => "localhost"
index => "unix-%{+YYYY.MM.dd}"
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5f034ce7-68c9-4f56-918b-bc4c887f74fb%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Yup, you will need to go into your Kibana dashboard - top right corner -
Configure Dashboard | Index and change the settings there to point to your
new index(es) instead of the default logstash-* indexes.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/457a1b21-0768-44a9-a1c3-f8766683ec66%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi Eric

I believe you might find the following blog post handy

Regards

Ronny

On 18 February 2014 22:52, Binh Ly binh@hibalo.com wrote:

Yup, you will need to go into your Kibana dashboard - top right corner -
Configure Dashboard | Index and change the settings there to point to your
new index(es) instead of the default logstash-* indexes.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/457a1b21-0768-44a9-a1c3-f8766683ec66%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACA5U5kFfwHOnB3B79M2HF%2BLoYACP3fH4dYUXMmKPmHp1UxjyA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Thanks! What was throwing me off is that I'm still having the UNIX logs
also write to logstash-date as well so I was seeing that information in my
main dashboard. I wasn't thinking about it writing 2 different times.
Thanks again.

On Tuesday, February 18, 2014 4:52:38 PM UTC-5, Binh Ly wrote:

Yup, you will need to go into your Kibana dashboard - top right corner -
Configure Dashboard | Index and change the settings there to point to your
new index(es) instead of the default logstash-* indexes.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aa721942-fe0b-4c7c-a2dc-8e95a1abcc02%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.