Trouble importing json log file to ELK via Elasticsearch (or other method)

Elastic Stack Elasticsearch
1

I've been fighting this for a while now and may have run into a bit of a roadblock.

PROBLEM: I'm trying to import 5GB worth of Zeek/Bro logs (see sample log below)

WHAT I TRIED:

  • I've tried to use CURL to send the logs over, that didn't seem to go, I might be doing something wrong.
  • I tried to use jsonpyes but that was having trouble, it looks like there might be a bug in the current code around UTF-8.
  • I set up filebeat and put the logfile.json into a folder, enabled the zeek/bro module and that imported the file and all the entries but pre-pended everything with filebeat import messages rather than only the json that's included below.

WHAT I AM ASKING:
Can anyone point me in the right direction to accomplish getting entries like those below into an ELK stack through elasticsearch? If there's a better direction to go with importing the data please let me know I'm happy to try other directions.

SAMPLE LINES FROM FILE

{"_path":"weird","_system_name":"sensorname","_write_ts":"2019-07-02T15:49:59.204752Z","ts":"2019-07-02T15:49:59.204752Z","id.orig_h":"1.1.1.1","id.orig_p":0,"id.resp_h":"2.2.2.2","id.resp_p":0,"name":"non_ip_packet_in_encap","notice":false}
{"_path":"weird","_system_name":"sensorname","_write_ts":"2019-07-01T15:22:15.770209Z","ts":"2019-07-01T15:22:15.770209Z","uid":"CR9lXabCKxjmoLhxFg","id.orig_h":"1.2.3.4","id.orig_p":60463,"id.resp_h":"12.34.56.78","id.resp_p":5355,"name":"dns_unmatched_msg","notice":false}
{"_path":"dns","_system_name":"sensorname","_write_ts":"2019-07-01T15:22:15.770209Z","ts":"2019-07-01T15:22:05.770203Z","uid":"CR9lXabCKxjmoLhxFg","id.orig_h":"1.2.3.4","id.orig_p":60463,"id.resp_h":"12.34.56.78","id.resp_p":5355,"proto":"udp","trans_id":36335,"query":"12.34.56.78.in-addr.arpa","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false}
2

You can use Logstash to import file into Elasticsearch.

input {
file {
type => "json"
path => "/opt/samplejson.json"
start_position => "beginning"
}
}

filter {
json {
source => "message"
}
}

output {
stdout {

}
}

3

Thank you for the reply.

I am a little new to this so I apologize. You mean that I can:

  1. create a logstash config with that in it.
  2. run logstash from command line, specifying the config file that was created.
  3. it will then run and parse each of the json entries without any more activity.

Is this the correct understanding?

4

Yes. More, you should prepare the index template before indexing if you don't want to use default index mapping

5

Okay, that is helpful. :smiley:

I do not know what you mean by "index template". Can you share an example of what it could look like?

6

When you put a document into a non-exist index, Elasticsearch will create index automatically and apply default mapping(schema) on it. You can manage mapping if you don't want to use default mapping. Index template is convenient to define template which will be applied when new index is created.
define a template:

PUT _template/template_1
{
  "index_patterns": ["te*", "bar*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_source": {
      "enabled": false
    },
    "properties": {
      "host_name": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date",
        "format": "EEE MMM dd HH:mm:ss Z yyyy"
      }
    }
  }
}

above template will be applied when te* or "bar* pattern indices are created.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.