# Trouble importing json log file to ELK via Elasticsearch (or other method) **URL:** \ **Category:** Elasticsearch\ **Created:** [August 21, 2019, 11:41pm UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210 "2019-08-21T23:41:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![0x00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/0x00/32/52748_2.png) [@0x00](https://discuss.elastic.co/u/0x00)\ **Post date:** [August 21, 2019, 11:41pm UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210/1 "2019-08-21T23:41:29Z")
I've been fighting this for a while now and may have run into a bit of a roadblock. **PROBLEM** : I'm trying to import 5GB worth of Zeek/Bro logs (see sample log below) **WHAT I TRIED** : - I've tried to use _CURL_ to send the logs over, that didn't seem to go, _I might be doing something wrong_. - I tried to use _jsonpyes_ but that was having trouble, _it looks like there might be a bug in the current code around UTF-8_. - I set up filebeat and put the logfile.json into a folder, enabled the zeek/bro module and that imported the file and all the entries but pre-pended everything with filebeat import messages rather than _only_ the json that's included below. **WHAT I AM ASKING** : Can anyone point me in the right direction to accomplish getting entries like those below into an ELK stack through elasticsearch? If there's a better direction to go with importing the data please let me know I'm happy to try other directions. **SAMPLE LINES FROM FILE** ``` {"_path":"weird","_system_name":"sensorname","_write_ts":"2019-07-02T15:49:59.204752Z","ts":"2019-07-02T15:49:59.204752Z","id.orig_h":"1.1.1.1","id.orig_p":0,"id.resp_h":"2.2.2.2","id.resp_p":0,"name":"non_ip_packet_in_encap","notice":false} {"_path":"weird","_system_name":"sensorname","_write_ts":"2019-07-01T15:22:15.770209Z","ts":"2019-07-01T15:22:15.770209Z","uid":"CR9lXabCKxjmoLhxFg","id.orig_h":"1.2.3.4","id.orig_p":60463,"id.resp_h":"12.34.56.78","id.resp_p":5355,"name":"dns_unmatched_msg","notice":false} {"_path":"dns","_system_name":"sensorname","_write_ts":"2019-07-01T15:22:15.770209Z","ts":"2019-07-01T15:22:05.770203Z","uid":"CR9lXabCKxjmoLhxFg","id.orig_h":"1.2.3.4","id.orig_p":60463,"id.resp_h":"12.34.56.78","id.resp_p":5355,"proto":"udp","trans_id":36335,"query":"12.34.56.78.in-addr.arpa","qclass":1,"qclass_name":"C_INTERNET","qtype":12,"qtype_name":"PTR","AA":false,"TC":false,"RD":false,"RA":false,"Z":0,"rejected":false} ``` ---
**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\ **Post date:** [August 22, 2019, 2:23am UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210/2 "2019-08-22T02:23:21Z")
You can use Logstash to import file into Elasticsearch. > input { > file { > type =\> "json" > path =\> "/opt/samplejson.json" > start\_position =\> "beginning" > } > } > > filter { > json { > source =\> "message" > } > } > > output { > stdout { > > } > } ---
**Author:** ![0x00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/0x00/32/52748_2.png) [@0x00](https://discuss.elastic.co/u/0x00)\ **Post date:** [August 22, 2019, 2:28am UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210/3 "2019-08-22T02:28:10Z")
Thank you for the reply. I am a little new to this so I apologize. You mean that I can: 1. create a logstash config with that in it. 2. run logstash from command line, specifying the config file that was created. 3. it will then run and parse each of the json entries without any more activity. Is this the correct understanding? ---
**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\ **Post date:** [August 22, 2019, 2:41am UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210/4 "2019-08-22T02:41:46Z")
Yes. More, you should prepare the index template before indexing if you don't want to use default index mapping ---
**Author:** ![0x00](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/0x00/32/52748_2.png) [@0x00](https://discuss.elastic.co/u/0x00)\ **Post date:** [August 22, 2019, 2:53am UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210/5 "2019-08-22T02:53:18Z")
Okay, that is helpful. 😃 I do not know what you mean by "index template". Can you share an example of what it could look like? ---
**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\ **Post date:** [August 22, 2019, 5:17am UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210/6 "2019-08-22T05:17:08Z")
When you put a document into a non-exist index, Elasticsearch will create index automatically and apply default mapping(schema) on it. You can manage mapping if you don't want to use default mapping. Index template is convenient to define template which will be applied when new index is created. define a template: > ``` > PUT _template/template_1 > { > "index_patterns": ["te*", "bar*"], > "settings": { > "number_of_shards": 1 > }, > "mappings": { > "_source": { > "enabled": false > }, > "properties": { > "host_name": { > "type": "keyword" > }, > "created_at": { > "type": "date", > "format": "EEE MMM dd HH:mm:ss Z yyyy" > } > } > } > } > > ``` above template will be applied when te\* or "bar\* pattern indices are created. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 19, 2019, 5:17am UTC](https://discuss.elastic.co/t/trouble-importing-json-log-file-to-elk-via-elasticsearch-or-other-method/196210/7 "2019-09-19T05:17:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.