Can I define a "NULLable" multi-field?

Elastic Stack Elasticsearch
1

I am trying to store DNS reply data in ES (5.4) for a passive DNS (pDNS) implementation. I have source data in JSON format.

I want the IP addresses in the DNS replies (A, AAAA queries) to be indexed as - well - IP addresses (IP data type in ES [1]).

The problem is that the source data has (can have) multiple reply fields per query. For example if DNSSEC is in use the source data contains something like the following (this is an excerpt):

   "answers" : [
      {
         "ttl" : 3600,
         "c" : 1,
         "t" : 1,
         "d" : "2xx.1yy.2zz.130",
         "n" : "barracuda.example.com."
      },
      {
         "t" : 46,
         "c" : 1,
         "ttl" : 3600,
         "n" : "barracuda.example.com.",
         "d" : "A 8 3 3600 20170602021315 20170503021315 63894 example.com. YpZyP8zytWDgZvxs-andmuchmorerandomstuff="
      }
   ]

The first element of the array contains the IP address of the "barracuda.example.com." host; the second one is a DNSSEC RRSIG record.

I can set "answers.d" to be an "IP type" field in ES, but it will fail to hold the RRSIG data of course.

So I thought I'd try a multi-field [2], indexing "answers.d" as "keyword" AND "IP" data types. But that fails when "answers.d" is not an IP.

What I am looking for is a way to only index "answers.d" as IP iff it is an IP. Otherwise just index it as a "keyword". Is this possible?

Or am I doing this conceptually wrong? I am using logstash to load the data and was thinking that maybe I have to have multiple types in my index, representing documents that do or do not have an IP. But then, I have documents (like the one above) that have both...

Any hint would be greatly appreciated.

[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/ip.html
[2] https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html

2

I took a different path now: I created two indices, one with "answers.d" of type "IP" and one with "answers.d" of type "keyword". I funnel the source data into the correct index using logstash (I use a "split" filter on the "answers" field in logstash to duplicate an entry like the above and then redirect the copy with the IP into one index and the other into the other index).

That seems to work, although I have two indices now that I might have to query. That does not seem to be a problem for ES though.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.