How to parse json messages in elasticstack

Elastic Stack Logstash
1

Hi Team,

I am receiving messages in JSON format and wanted to know how do I directly send those in elastic? Can I directly store them in elastic without sending them through logstash or has to route it through logstash?
Here are some

./eve.json:{"timestamp":"2018-03-22T14:49:40.014495+0530","flow_id":1201901318977695,"in_iface":"eno16777736","event_type":"alert","src_ip":"192.168.1.9","src_port":65416,"dest_ip":"192.168.1.42","dest_port":53,"proto":"UDP","alert":{"action":"allowed","gid":1,"signature_id":5700011,"rev":1,"signature":"CleanDNS_Phase1: Malicious domain 57g7spgrzlojinas.onion","category":"","severity":3},"app_proto":"dns","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":82,"bytes_toclient":0,"start":"2018-03-22T14:49:40.014495+0530"}}

./eve.json:{"timestamp":"2018-03-22T14:49:40.014495+0530","flow_id":1201901318977695,"in_iface":"eno16777736","event_type":"dns","src_ip":"192.168.1.9","src_port":65416,"dest_ip":"192.168.1.42","dest_port":53,"proto":"UDP","dns":{"type":"query","id":35,"rrname":"57g7spgrzlojinas.onion","rrtype":"A","tx_id":0}}

./eve.json:{"timestamp":"2018-03-22T14:49:40.014710+0530","flow_id":1201901318977695,"in_iface":"eno16777736","event_type":"dns","src_ip":"192.168.1.42","src_port":53,"dest_ip":"192.168.1.9","dest_port":65416,"proto":"UDP","dns":{"type":"answer","id":35,"rcode":"NXDOMAIN","rrname":"57g7spgrzlojinas.onion"}}

2

You can store them yourself if you like, just use the Elasticsearch's REST API. You can also use Logstash. Your call.

3

What I mean here is, do I need to write parsers to ingest json based data like any other text data? or can directly be ingested and elasticsearch will understand fields and mappings?

4

Elasticsearch is a JSON document database so it understands JSON perfectly well. No pre-parsing is necessary.

5

awesome!! that is what I wanted to know. Thanks!!

6

I believe sending logs through filebeat would be much easier since the file is growing consistently and I need a logs in real-time. what say?

7

Sure, although Logstash provides the same functionality.

8

Yeah thanks I was able to achieve that. Thanks for the support

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.