Kibana Visualization - JSON Input Example

[Laker99 (KJ)]

Hello Everyone Good day!
May I please take a few minutes of your time?

I have documents in ElasticSearch with fields like 'server_name','switch_name','op_type','conn_id'. I want to create visualizations (charts, vertical bars) in Kibana where op_type = 'Established' and server_name = 'ABC' and 'switch_name ' S123'.

1. Is there a way I can specify these in JSON Input? If so, how?
2. Can I use the "Filters"? If so, how?

I tried scripted fields, but the amount of data is really huge - app. 3 million documents a day.

Right now, I have been able to create a simple search using ES, but that gives us only a list view, which does not really help.

Can someone please explain how to achieve this using a visualization? Thanks in advance for your time and cooperation

Take care and have a great day.

[Christian_Dahlqvist]

What does a document look like once it has been indexed into Elasticsearch?

[Laker99 (KJ)]

Hello Christian Thanks a lot for your response. Here's how the document looks,

op_type Established
server_name zlssABC
switch_name S123
beat.hostname HOST123

Each of these are fields in the document. If I create a scripted field using these fields, I am able to calculate the desired value, but it breaks due to the really large number of documents.

Can you please help me with an example for the JSON input?

Thanks again! Have a great day.

[Christian_Dahlqvist]

Can you show the JSON structure of the document as well as the desired value you are scripting?

[Laker99 (KJ)]

Hello Christian, thanks for your response Here's how my JSON structure looks for that document,

{
"_index": "<my_index>-2017-07-11",
"_type": "genesys_tserver",
"_id": "2b49029d4f1c3744_53154_000003",
"_version": 1,
"_score": null,
"_source": {
"server_name": "server_123",
"offset": 35482434,
"input_type": "log",
"dn": "53154",
"source": "<path_to_log file>",
"message": "@00:00:03.4983 [gctmi] Party [2b49029d4f1c3744:53154,sa,tDN,rDST,lINT] processEstablished",
"type": "log",
"log_hh": "00",
"log_mm": "00",
"log_ss": "03",
"tags": [],
"@timestamp": "2017-07-11T05:00:03.682Z",
"callflow_step": "TServer",
"@version": "1",
"beat": {
"hostname": "Host123",
"name": "HostName123",
"version": "5.4.0"
},
"host": "Host123",
"connid": "2b49029d4f1c3744",
"cctp_doc_id": "2b49029d4f1c3744_53154_000003",
"op_type": "Established",
"log_msec": "4983",
"event_time": "00:00:03"
},
"fields": {
"@timestamp": [
1499749203682
]
},
"highlight": {
"op_type": [
"@kibana-highlighted-field@Established@/kibana-highlighted-field@"
],
"message": [
"@00:00:03.4983 [gctmi] Party [2b49029d4f1c3744:53154,sa,tDN,rDST,lINT] @kibana-highlighted-field@Established@/kibana-highlighted-field@"
],
"op_type.keyword": [
"@kibana-highlighted-field@Established@/kibana-highlighted-field@"
]
},
"sort": [
1499749203682
]
}

Now, I tried the following,
{ "_source": { "op_type": "Established" } }
{ "op_type": "Established" }
none of them yield any results.

What am I missing? Please advise.

As always, thanks a bunch for your patience and cooperation

[Christian_Dahlqvist]

@Laker99

Now, I tried the following,
{ "source": { "optype": "Established" } }
{ "op_type": "Established" }
none of them yield any results.

I do not understand what you are trying to do. What are you using this for? What is it you were trying to use a scripted field for?

[Laker99 (KJ)]

Hello Christian, thanks for your response
Basically, I am trying to filter documents in a Kibana visualization where op_type = "Established" and "server_name" = "server_123". I do not have any existing field in the document, for e.g. isMyField that satisfies the above condition. So, I am looking to calculate this run-time in a visualization. Now, I tried creating a Kibana scripted field, but I get the message - "too many script compilations reached...", due to too many documents. Hence, I am looking for something in JSON Input.

Is that even possible? What exactly is the JSON Input used for?

I hope I have explained it better now. If you still have any questions, please let me know. Thank you once again for your patience and cooperation

[Christian_Dahlqvist]

Which version of Kibana are you using?

[Laker99 (KJ)]

Hello Christian,
I am using v5.4 for all ELK components - Filebeat, Logstash, Elasticsearch and Kibana. Please let me know if you need more details. Thanks again for your time and cooperation Much appreciated.

[Christian_Dahlqvist]

If you are looking to add a permanent filter to the data in a visualisation, you can view the index pattern in Discover mode, add the filters you want, either by selecting fields on the left or through Lucene expressions in the search bar at the top. Once you have filtered out the correct set of data you can save this as a saved search, which you can then base a Visualisation on. Does this match what you are trying to do? If not, can you show some screen shots to explain?

[Laker99 (KJ)]

Hello Christian, thanks for your quick response.

1. Yes, I can add multiple filters and create an Elastic search (i.e. saved search), but this shall display the results only as a table. The only visualization I can have is by simply importing this table. This does not help, as I want to be able to generate charts and graphs.

2. I am simply looking for a filter of sorts in the JSON Input field in the Kibana visualization. For e.g. whenever I start constructing a vertical graph, I choose a Y-Axis, where I select an aggregation (for e.g. Unique Count) and specify the field (for e.g. connid.keyword). Under "Advanced" there is a JSON Input field. That's the one I am most interested in.

a. What exactly does this "JSON Input" do?
b. Is there any way I specify any filter criteria on one/more fields in my document, for e.g.
op_type Established
server_name zlssABC
switch_name S123
beat.hostname HOST123

That way, when I execute the graph, the unique count of connid.keyword is automatically filtered as above. Without the filter, I will get all the calls (for e.g. 1,00,000), but with the above filter in the JSON Input, I will only get say, 300,000.

Hope I have explained my situation better. I am unable to send screenshots through this message board. Perhaps, I can email them across if that's fine? Please let me know.

Thanks again for your time and cooperation Really appreciate it.

Take care.