How can I use Elastic/Logstash/Kibana to explore JSON data?

Elastic Stack Elasticsearch
1

My company has a large amount of XML data that we want to explore for its possible value. We can convert the data to JSON in order to get it to import into ElasticSearch, but have been slowed by trying to learn enough about Logstash and Kibana to import it...hence my question here.

I am currently stuck at the point where I need to "Configure an index pattern" in Kibana...so this has been like pulling teeth!

So I though I'd describe my situation and ask for advice...

The data are semi-structured, in that there a lot of similar elements in the document schemas, but no two records are going to be the same. I was hoping to be able to dump this JSON data into Elastic, and use it to explore the JSON docs, such as how many have similar elements, the geographic distribution (it has time and location data), etc.

Can Elastic be used for this? I would assume so...but how to use Elastic to explore JSON docs where I don't have a consistent schema? Is this possible?

Thanks,
Rick Casey

2

Do you have the data in ES yet? If so then it's just the name of that index.

Yep!

What sort of exploration do you want to do?

3

Mark,

Thanks for the reply. At this point, the exploration consists of discovering the types of XML elements in the data, and the geographic distribution of the data (which is global). The exploration would get more refined as I learn better to use Elastic and Kibana.

I am still getting up to speed on these tools so any helpful hints are appreciated...

--Rick

4

I forgot to mention that the XML data has been converted to JSON, since that is what Elastic seems to need.

But the problem is the JSON structure is somewhat random; here's an example:

{"?xml":{"@version":"1.0","@encoding":"utf-8"},"HomerProblemRecord":{"@xmlns:i":"http://www.w3.org/2001/XMLSchema-instance","Problem":{"@z:Id":"i1","@xmlns:z":"http://schemas.microsoft.com/2003/10/Serialization/","Batteries":{"Battery":{"@z:Id":"i2","Abbreviation":"1kWh LA","CapitalCostMultiplierSensitivity":{"@z:Id":"i3","decimal":"1"},"CostMatrix":{"Cost":{"@z:Id":"i4","Initial":"300","Operating":"10","Quantity":"1","Replacement":"300"}},"Footprint":"0.05","IsAc":"false","LogoLarge":{"@i:nil":"true"},"LogoSmall":....

and so on.

I have no idea how to write an "index pattern" for this, which is what the documentation refers to as essential to getting data imported. I am hoping others have encountered this same problem, but learned how to overcome it.

thanks for any help,
Rick

5

Index pattern refers to the name of the index in ES, is the logical bucket you inserted the data into. Have a look at https://www.elastic.co/guide/en/elasticsearch/guide/2.x/_indexing_employee_documents.html

6

Thanks...good article...should help.

Rick