Multiple indexes or a single index

[geertn444]

Hello,

I am analyzing syslogs with Elastic. They all arrive on a single port and this can't be changed. However, the kind of syslog messages are very diverse, lets call it 4 categories: dns, dhcp, switch, httpd
Each of these categories has several specific search fields, lets say 30 specific search fields.

My question is:

1. does it make sense to create 4x 30 = 120 search fields in a single index, while each message only uses 1/4 of these fields (it is only 1 of the 4 categories) ? In other words, does an "empty" field take up space in an index ?
2. wouldn't it be better to create 4 seperate indexes, each with only the relevant fields ?
   a) would i still be possible to link the indexes together or do queries across the indexes ?
   for example, link a dns.ip field in dns index with a httpd.ip field in httpd index ?
   b) is this even possible in logstash for example ?
   take a single input stream and split it into 4 different output streams to 4 different indexes with different fields ? Or do i need 4 input streams also in this case ?

[Sjaak01 (Sjaak)]

Hi Geert

Logstash can drop empty fields so it is possible to index only the actual data. I don't think having 120 fields in one index is a problem.

You can use conditionals to have Logstash send different data to different indexes so if your syslogs contains some data that makes it easy to identify the type of logs, you can use something like a if statement to send it to a different index.

I believe Elasticsearch can search through different indexces but if you plan on using Kibana to visualize some of the data, its probably best to have it all in one index as most visualizations are limited to a single index and can't access data in a different index.