# Best way to store 2 data sources with 1 linked field (document\_type OR new index)?
**URL:** https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661
**Category:** Elasticsearch
**Created:** [November 28, 2018, 10:57pm UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661 "2018-11-28T22:57:54Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [November 28, 2018, 10:57pm UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/1 "2018-11-28T22:57:54Z")
Hi, I've read the blog post on [Index Vs. Type](https://www.elastic.co/blog/index-vs-type) but I find it hard to understand what's better in my case.
- I have a certain **application's log files** and a great majority of the log lines mention a few remote IP addresses that my server communicates with on a regular basis
- I also have a second data set from network packet analysis, which includes source/destination IP addresses along with network statistics.
The two data sets are **linked** by the `destination IP address` field.
The second data set (network stats) has much higher data frequency and volume.
1. Is the relationship b/w the datasets one that can be called a **parent/child** relationship?
2. Should I be storing those as **separate indexes** or on the **same index, different `document_type`**?
3. If on the same index, should I use the **same field name** for both IP addresses fields?
Thanks in advance.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 28, 2018, 11:56pm UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/2 "2018-11-28T23:56:37Z")
1. You could, but it doesn't really make sense in this use case because which one is the parent?
2. Are the structures very different, similar or the same?
3. If you want to be able to easily query, it makes sense.
---
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [November 29, 2018, 12:07am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/3 "2018-11-29T00:07:24Z")
Thanks @warkolm for the quick reply.
The parent should be the application log and the child would be the network statistics under each IP address mentioned in the application's log.
The data structures are totally different with the only common field being the IP address.
The causal link is that the remote IP that interacts with the application/server doing some action a (mentioned in the app's log files) generates network traffic and therefore network statistics which are brought in with this second data set.
I definitely want to be able to query as this is the reason for bringing these two related data sets together.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 29, 2018, 12:23am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/4 "2018-11-29T00:23:20Z")
> [@nandrik](#):
>
> The parent should be the application log and the child would be the network statistics under each IP address mentioned in the application's log.
But if you have multiple application logs, how do you figure out which networks statistics are related to that log entry?
---
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [November 29, 2018, 1:03am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/5 "2018-11-29T01:03:52Z")
I only have 1 application and all network statistics with matching remote IP addresses should be related.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 29, 2018, 1:33am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/6 "2018-11-29T01:33:09Z")
Are you saying that there is only ever one application log per IP, and all network logs for that IP relate to that log? Is that per minute/hour/day, or forever?
---
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [November 29, 2018, 1:52am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/7 "2018-11-29T01:52:15Z")
That’s right. The network stats logs are per second as the interactions at the application layer happen.
---
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [November 29, 2018, 4:52am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/8 "2018-11-29T04:52:09Z")
There’s a causal relationship. Each reported event at the application layer should have corresponding network statistics logs.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 29, 2018, 5:51am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/9 "2018-11-29T05:51:37Z")
But what I am getting at is that it doesn't seem plausible that each unique application use IP should have _all_ the networks logs against that IP associated with it.
I mean, I use a VPN pretty much 24/7, so would others. What if you used a CDN, or a load balancer? Hard linking one application event to _all_ network events from that IP means you would have hundreds or thousands over time.
If you are only linking them per second, what if the events go past that specific second.
It just seems a little strict and unforgiving.
---
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [November 29, 2018, 6:03am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/10 "2018-11-29T06:03:06Z")
The IPs in the network stats mention a specific port which is linked to that app. So with 100% certainty I can attribute the network stats to the application log events involving that IP.
As you mentioned, the app interactions with that IP will keep repeating and so will the network stats. And the network stats would be dozens of log lines of packet analysis per app interaction.
I would care only about network stats likely immediately before each app log event , involving that specific IP.
So, ideally, I would want all stats for that IP within a certain time window tied to specific app log events. This would provide me with a sense of network performance related to the app-level interaction.
- Is that possible?
- @warkolm, what's the right structure for this?
- Is there a way of defining that parent-child relationship within a certain time window?
---
### Author:  [@nandrik](https://discuss.elastic.co/u/nandrik)
#### Post date: [November 30, 2018, 3:37am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/11 "2018-11-30T03:37:38Z")
The recommendation from an ELK expert outside this forum was to **use two separate indeces sharing a common field name** for the IP address.
This should allow faster filtering using just the common (IP) field because of the index separation.
Do you concur @warkolm ?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [November 30, 2018, 7:25am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/12 "2018-11-30T07:25:47Z")
That would make sense.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 28, 2018, 7:25am UTC](https://discuss.elastic.co/t/best-way-to-store-2-data-sources-with-1-linked-field-document-type-or-new-index/158661/13 "2018-12-28T07:25:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.