# Added pipeline all fields disappear **URL:** \ **Category:** Kibana\ **Created:** [May 8, 2018, 12:59pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008 "2018-05-08T12:59:28Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 12:59pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/1 "2018-05-08T12:59:28Z")
Hi, I had my kibana working great! I added an index to pull in data from another source and lost all visualizations. When I click on the tabs in the overview dashboard they all say "No results displayed because all values equal 0." I have tried searching on specific indexes as well as ones that were displaying previously and nothing. My visualizations are set to query on elastiflow\*. My index names are elastiflow-asi-asa%date% and elastiflow-cvh-asa%date% I am getting data from logstash as my indices are increasing. Here is an example of the data pulling to show I am getting something. { "\_index": "elastiflow-cvh-asa2018.05.08", "\_type": "doc", "\_id": "jzvQP2MBEabB04r\_a4Wr", "\_version": 1, "\_score": null, "\_source": { "@version": "1", "netflow": { "event\_time\_msec": 1525784080181, "icmp\_code": 0, "rev\_flow\_delta\_bytes": 204, "xlate\_dst\_addr\_ipv4": "8.8.8.8", "responderPackets": 1, "l4\_src\_port": 9688, "icmp\_type": 0, "xlate\_src\_addr\_ipv4": "96.10.27.74", "input\_snmp": 3, "xlate\_src\_port": 9688, "flow\_seq\_num": 93920, "l4\_dst\_port": 53, "conn\_id": 51531889, "ipv4\_dst\_addr": "8.8.8.8", "xlate\_dst\_port": 53, "fw\_event": 2, "initiatorPackets": 1, "protocol": 17, "fw\_ext\_event": 2020, "version": 9, "output\_snmp": 2, "flow\_start\_msec": 1525784080141, "fwd\_flow\_delta\_bytes": 30, "flowset\_id": 263, "ipv4\_src\_addr": "192.168.1.106" }, "tags": [ "port\_9597" ], "node": { "ipaddr": "10.10.1.254", "hostname": "10.10.1.254" }, "event": { "host": "10.10.1.254", "type": "netflow v9" }, "flow": { "geoip": { "autonomous\_system": "private" } }, "@timestamp": "2018-05-08T12:54:41.000Z" }, "fields": { "netflow.flow\_start\_msec": [ "2018-05-08T12:54:40.141Z" ], "netflow.event\_time\_msec": [ "2018-05-08T12:54:40.181Z" ], "@timestamp": [ "2018-05-08T12:54:41.000Z" ] }, "sort": [ 1525784081000 ] } and { "\_index": "elastiflow-asi-asa-2018.05.08", "\_type": "doc", "\_id": "3z3VP2MBEabB04r\_X2m6", "\_version": 1, "\_score": null, "\_source": { "@version": "1", "netflow": { "event\_time\_msec": 1525784115490, "icmp\_code": 0, "rev\_flow\_delta\_bytes": 436, "xlate\_dst\_addr\_ipv4": "40.97.147.194", "l4\_src\_port": 55679, "icmp\_type": 0, "xlate\_src\_addr\_ipv4": "64.246.214.189", "input\_snmp": 4, "xlate\_src\_port": 31633, "flow\_seq\_num": 7358466, "l4\_dst\_port": 443, "conn\_id": 388635648, "ipv4\_dst\_addr": "40.97.147.194", "xlate\_dst\_port": 443, "fw\_event": 5, "protocol": 6, "fw\_ext\_event": 2031, "version": 9, "output\_snmp": 3, "flow\_start\_msec": 1525784096470, "fwd\_flow\_delta\_bytes": 0, "flowset\_id": 263, "ipv4\_src\_addr": "10.15.41.25" }, "tags": [ "port\_9596" ], "node": { "ipaddr": "192.168.200.1", "hostname": "192.168.200.1" }, "event": { "host": "192.168.200.1", "type": "netflow v9" }, "flow": { "geoip": { "autonomous\_system": "private" } }, "@timestamp": "2018-05-08T12:55:15.000Z" }, "fields": { "netflow.flow\_start\_msec": [ "2018-05-08T12:54:56.470Z" ], "netflow.event\_time\_msec": [ "2018-05-08T12:55:15.490Z" ], "@timestamp": [ "2018-05-08T12:55:15.000Z" ] }, "sort": [ 1525784115000 ] } Any ideas???? ---
**Author:** ![Stacey\_Gammon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stacey_gammon/32/14025_2.png) [@Stacey\_Gammon](https://discuss.elastic.co/u/Stacey_Gammon)\ **Post date:** [May 8, 2018, 1:36pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/2 "2018-05-08T13:36:59Z")
Hmm, strange, the fields look the same. How do the fields compare between the two indexes in management? Maybe in one index they are slightly different? ![56%20AM](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a08fc3ac270eddf96b625b7e368c2e75d45b5c55.png) Can you try re-creating the same visualization using each index specifically? Maybe one works and one doesn't and it might give you some ideas as to why they aren't being combined into a single index pattern properly. ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 2:59pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/3 "2018-05-08T14:59:56Z")
When I try to make a visualization, I tried a graph chart. When I choose 24 hours, there is data. When I change that to 4 hours there is nothing. ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/5/557c0ebdf01cd142293f27d0768c5ff3f19ca8b3.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/9/29c706ea53fdaa7ba6d3787ca519a9163e7ebc1d.png) ---
**Author:** ![Stacey\_Gammon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stacey_gammon/32/14025_2.png) [@Stacey\_Gammon](https://discuss.elastic.co/u/Stacey_Gammon)\ **Post date:** [May 8, 2018, 4:26pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/4 "2018-05-08T16:26:49Z")
Are you sure data actually exists within the last 4 hours? The discover app may help you see when your data is getting ingested and indexed. ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 5:14pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/5 "2018-05-08T17:14:56Z")
Stacey, my indices is growing and I do have data in the discover tab. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/4889a97fd61d15f691f5f2a0960144e96928e26e.png) ---
**Author:** ![Stacey\_Gammon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stacey_gammon/32/14025_2.png) [@Stacey\_Gammon](https://discuss.elastic.co/u/Stacey_Gammon)\ **Post date:** [May 8, 2018, 6:28pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/6 "2018-05-08T18:28:50Z")
Can you show data for the last 4 hours but include the columns `flow.geoip.city_name` and `flow.bytes`? Could also check out the request and response on the spy panel to see if data is coming back from elastic search. ![09%20PM](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b6d4bc77d44a5def1a5ce0783646ec82128262d1.png) ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 8:13pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/7 "2018-05-08T20:13:41Z")
Stacey, Where is the spy panel? I do not have that. ---
**Author:** ![Stacey\_Gammon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stacey_gammon/32/14025_2.png) [@Stacey\_Gammon](https://discuss.elastic.co/u/Stacey_Gammon)\ **Post date:** [May 8, 2018, 8:36pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/8 "2018-05-08T20:36:38Z")
This little arrow opens up the spy panel: ![35%20PM](https://us1.discourse-cdn.com/elastic/original/3X/5/8/58c19c842d51cb1acb84e72071d8ad83d59aa17c.jpg) ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 8:41pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/9 "2018-05-08T20:41:41Z")
I have found some traffic. But using netflow\_fwd and netflow\_rev. I did go back a couple of days and did not have a flow.bytes field even though it was working. ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/d/bdead650aca21fd3f27dbeedb2f2308fb5c92bb0.png) ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 8:42pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/10 "2018-05-08T20:42:36Z")
Stacey, Here is the request { "size": 0, "\_source": { "excludes": [] }, "aggs": { "2": { "terms": { "field": "flow.client\_hostname", "size": 50, "order": { "1": "desc" } }, "aggs": { "1": { "sum": { "field": "flow.bytes" } } } } }, "version": true, "stored\_fields": [ "_" ], "script\_fields": {}, "docvalue\_fields": [ "@timestamp", "netflow.event\_time\_msec", "netflow.flow\_start\_msec" ], "query": { "bool": { "must": [ { "query\_string": { "query": "_", "analyze\_wildcard": true, "default\_field": "_" } }, { "query\_string": { "analyze\_wildcard": true, "query": "_", "default\_field": "_" } }, { "range": { "@timestamp": { "gte": 1525768914449, "lte": 1525812114449, "format": "epoch\_millis" } } } ], "filter": [], "should": [], "must\_not": [] } }, "highlight": { "pre\_tags": [ "@kibana-highlighted-field@" ], "post\_tags": [ "@/kibana-highlighted-field@" ], "fields": { "_": {} }, "fragment\_size": 2147483647 } } And the reply { "took": 19378, "timed\_out": false, "\_shards": { "total": 18, "successful": 18, "skipped": 0, "failed": 0 }, "hits": { "total": 17900325, "max\_score": 0, "hits": [] }, "aggregations": { "2": { "doc\_count\_error\_upper\_bound": 0, "sum\_other\_doc\_count": 0, "buckets": [] } }, "status": 200 } ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 8:50pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/11 "2018-05-08T20:50:40Z")
Stacey, I have also tried a pie graph with the slices being source and destination IP address. Here is the code in my .conf file. It should be converting the fwd\_flow and rev\_flow # Populate normalized ElastiFlow fields with bytes transferred in the flow. ``` if [netflow][in_bytes] { mutate { id => "netflow_9_in_bytes" add_field => { "[flow][bytes]" => "%{[netflow][in_bytes]}" } } } else if [netflow][out_bytes] { mutate { id => "netflow_9_out_bytes" add_field => { "[flow][bytes]" => "%{[netflow][out_bytes]}" } } } else if [netflow][in_permanent_bytes] { mutate { id => "netflow_9_in_permanent_bytes" add_field => { "[flow][bytes]" => "%{[netflow][in_permanent_bytes]}" } } } else if [netflow][fwd_flow_delta_bytes] or [netflow][rev_flow_delta_bytes] { ruby { id => "netflow_9_normalize_bytes_from_fwd_rev_bytes" code => " event.set( '[flow][bytes]', event.get('[netflow][fwd_flow_delta_bytes]').to_i + event.get('[netflow][rev_flow_delta_bytes]').to_i ) " } } if [flow][bytes] { mutate { id => "netflow_9_convert_bytes" convert => { "[flow][bytes]" => "integer" } } } ``` ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/a/ca6b1bfb9852e6b00240a4a0ab4c41c241ce8d4b.png) ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 8:58pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/12 "2018-05-08T20:58:03Z")
Stacey, Here is a request and response from yesterday. I can pull up that data in a graph { "size": 0, "\_source": { "excludes": [] }, "aggs": { "2": { "terms": { "field": "flow.client\_hostname", "size": 50, "order": { "1": "desc" } }, "aggs": { "1": { "sum": { "field": "flow.bytes" } } } } }, "version": true, "stored\_fields": [ "_" ], "script\_fields": {}, "docvalue\_fields": [ "@timestamp", "netflow.event\_time\_msec", "netflow.flow\_start\_msec" ], "query": { "bool": { "must": [ { "query\_string": { "query": "_", "analyze\_wildcard": true, "default\_field": "_" } }, { "query\_string": { "analyze\_wildcard": true, "default\_field": "_", "query": "_" } }, { "range": { "@timestamp": { "gte": 1525665600000, "lte": 1525812912210, "format": "epoch\_millis" } } } ], "filter": [], "should": [], "must\_not": [] } }, "highlight": { "pre\_tags": [ "@kibana-highlighted-field@" ], "post\_tags": [ "@/kibana-highlighted-field@" ], "fields": { "_": {} }, "fragment\_size": 2147483647 } } Response { "took": 15000, "timed\_out": false, "\_shards": { "total": 18, "successful": 18, "skipped": 6, "failed": 0 }, "hits": { "total": 52419268, "max\_score": 0, "hits": [] }, "aggregations": { "2": { "doc\_count\_error\_upper\_bound": -1, "sum\_other\_doc\_count": 3957447, "buckets": [ { "1": { "value": 117663411070 }, "key": "10.15.20.44", "doc\_count": 22838 }, { "1": { "value": 16970322328 }, "key": "10.15.43.2", "doc\_count": 398011 }, { "1": { "value": 6137660341 }, "key": "71.71.102.134", "doc\_count": 649 }, { "1": { "value": 3922909619 }, "key": "52.114.188.27", "doc\_count": 86 }, { "1": { "value": 3178533278 }, "key": "10.15.43.93", "doc\_count": 25069 }, { "1": { "value": 2842703973 }, "key": "66.76.190.205", "doc\_count": 36824 }, { "1": { "value": 2262909881 }, "key": "10.15.100.97", "doc\_count": 15740 }, { "1": { "value": 2051331626 }, "key": "10.15.41.10", "doc\_count": 2359618 }, { "1": { "value": 2008112993 }, "key": "10.1.1.12", "doc\_count": 58502 }, { "1": { "value": 2002242041 }, "key": "24.143.223.90", "doc\_count": 82817 }, { "1": { "value": 1617100964 }, "key": "74.202.227.226", "doc\_count": 77556 }, { "1": { "value": 1523385499 }, "key": "172.17.200.55", "doc\_count": 9744 }, { "1": { "value": 1479248292 }, "key": "10.15.41.9", "doc\_count": 1179 }, { "1": { "value": 1471095687 }, "key": "10.15.10.53", "doc\_count": 16883 }, { "1": { "value": 1438655810 }, "key": "10.15.41.2", "doc\_count": 23904 }, { "1": { "value": 1376762321 }, "key": "204.111.84.66", "doc\_count": 122360 }, { "1": { "value": 1326359430 }, "key": "216.201.215.154", "doc\_count": 75463 }, { "1": { "value": 1212821578 }, "key": "10.15.20.55", "doc\_count": 2715 }, { "1": { "value": 1199431785 }, "key": "10.15.10.52", "doc\_count": 13582 }, { "1": { "value": 1170899214 }, "key": "108.169.151.106", "doc\_count": 157234 }, { "1": { "value": 1154793312 }, "key": "10.15.41.23", "doc\_count": 11104 }, { "1": { "value": 1138719373 }, "key": "10.15.41.37", "doc\_count": 3016 }, { "1": { "value": 1029625910 }, "key": "10.15.41.6", "doc\_count": 11431 }, { "1": { "value": 1010658626 }, "key": "64.246.214.189", "doc\_count": 16366 }, { "1": { "value": 966816900 }, "key": "10.15.10.46", "doc\_count": 16745 }, { "1": { "value": 853682208 }, "key": "50.58.80.182", "doc\_count": 33767 }, { "1": { "value": 789523579 }, "key": "10.15.10.38", "doc\_count": 20064 }, { "1": { "value": 788419290 }, "key": "10.15.41.49", "doc\_count": 13932 }, { "1": { "value": 773352856 }, "key": "10.15.43.90", "doc\_count": 30405 }, { "1": { "value": 754616693 }, "key": "208.45.245.194", "doc\_count": 71347 }, { "1": { "value": 747842952 }, "key": "10.15.43.92", "doc\_count": 25674 }, }, "status": 200 } ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 8, 2018, 9:06pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/13 "2018-05-08T21:06:35Z")
Here is a sample flow from yesterday { "\_index": "elastiflow-asi-asa-2018.05.07", "\_type": "doc", "\_id": "akhqPGMBxenACWdiGiog", "\_version": 1, "\_score": null, "\_source": { "event": { "host": "192.168.200.1", "type": "netflow v9" }, "tags": [ "port\_9596" ], "netflow": { "version": 9, "protocol": 6, "flowset\_id": 263, "fw\_event": 5, "xlate\_dst\_port": 88, "fw\_ext\_event": 2031, "xlate\_src\_addr\_ipv4": "10.15.43.1", "ipv4\_src\_addr": "10.15.43.1", "input\_snmp": 4, "l4\_src\_port": 64996, "output\_snmp": 3, "l4\_dst\_port": 88, "conn\_id": 384495818, "flow\_seq\_num": 6520957, "icmp\_type": 0, "xlate\_dst\_addr\_ipv4": "10.77.25.10", "event\_time\_msec": 1525726848977, "rev\_flow\_delta\_bytes": 152, "fwd\_flow\_delta\_bytes": 301, "flow\_start\_msec": 1525726848577, "ipv4\_dst\_addr": "10.77.25.10", "icmp\_code": 0, "xlate\_src\_port": 64996 }, "node": { "ipaddr": "192.168.200.1", "hostname": "192.168.200.1" }, "flow": { "geoip": { "autonomous\_system": "private" } }, "@version": "1", "@timestamp": "2018-05-07T21:00:49.000Z" }, "fields": { "netflow.flow\_start\_msec": [ "2018-05-07T21:00:48.577Z" ], "netflow.event\_time\_msec": [ "2018-05-07T21:00:48.977Z" ], "@timestamp": [ "2018-05-07T21:00:49.000Z" ] }, "sort": [ 1525726849000 ] } ---
**Author:** ![dalamar666](https://avatars.discourse-cdn.com/v4/letter/d/e9bcb4/32.png) [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Post date:** [May 11, 2018, 2:34pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/14 "2018-05-11T14:34:59Z")
Stacey, Any ideas? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 8, 2018, 2:35pm UTC](https://discuss.elastic.co/t/added-pipeline-all-fields-disappear/131008/15 "2018-06-08T14:35:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.