# Elastic Json parse into logstash **URL:** \ **Category:** Logstash\ **Created:** [October 10, 2018, 11:58am UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840 "2018-10-10T11:58:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 10, 2018, 11:58am UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/1 "2018-10-10T11:58:43Z")
Hello, I am retrieving indices from an elasticsearch containing syslogs messages through a curl commande then they are read by a logstash. and want to extract 4 fields from this messages (datetime\_received,"ip\_host\_pkt","source\_msg, "\_type") and renamed them to date,host, message, type. original message from the extract. { "\_id": "AWUbrXRdjZ-987654akh", "\_index": "index-2018-08-09", "\_score": 0.0, "\_source": { "datetime\_log": "2018-08-09T00:34:36.051+02:00", "datetime\_receive": "2018-08-09T00:34:36.051+02:00", "group": "DEFAUT", "ip\_host": "22.33.44.55", "ip\_host\_pkt": "22.33.44.55", "source\_msg": "22.33.44.55: -Trashback= XXXXXXXXX XXXXXXXXXX XXXXXXXXX 8002DAD8 YYYYYYYY ZZZZZZZZZ UUUUUUUU IIIIIIIIII", "unix\_level": "local7", "unix\_priority": "crit" }, "\_type": "logs\_2" }, Tryed to remove fields I don't want to keep but without success. below is my logstash conf. I am stuck, if anyone have a clue. Logstash Config input { file { path =\> "/data/\*.log" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" codec =\> "json" } } filter { json { source =\> "message" } mutate { remove\_field =\> ["[message][0][\_source][unix\_priority]" ] remove\_field =\> ["[message][0][\_source][\_id]" ] remove\_field =\> ["[message][0][\_source][\_index]" ] } thnaks ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 23, 2018, 11:31am UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/2 "2018-10-23T11:31:46Z")
Hi @moughrom > [@moughrom](#): > > I am retrieving indices from an elasticsearch containing syslogs messages through a curl commande then they are read by a logstash. The simplest way to retrieve documents from Elasticsearch and read these into Logstash is by using the Logstash Elasticsearch input plugin, see: [Elasticsearch input plugin | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch) ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 12:27pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/3 "2018-10-23T12:27:52Z")
the elastic source is secured and not Under my authority, the only way to access it is through the proxy api and only with a POST commande to retrieve an authentication token to launch curl GET command. also I have a full conf logstash ready but to receive syslog in csv format. But now I am getting json file as input, so I want to read the multiline json and extract 4 fields (date, msg, host, type) and transform then as csv to process them with my ready logstash csv conf. ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 23, 2018, 12:46pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/4 "2018-10-23T12:46:10Z")
Hi @moughrom If all you need to do is to rename and remove fields, you can do this inside Elasticsearch without extracting the documents, using Ingest operations, here is an example based on your example data: > ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 1:04pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/5 "2018-10-23T13:04:17Z")
My problem is that the format of the data source has change from csv to json and I want to lean on the dev done in logstash csv conf to enrich data, geo,... So we can't bypass the logstash. I have syslogs in json (elastic extract) format and need to retrieve 4 fields convert those as csv and process the csv with all the logstash filters already set. ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 1:16pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/6 "2018-10-23T13:16:03Z")
1 - Extract fields from a json multiline 2 - transform those extract as csv. "\_id": "AWUbrXRdjZ-987654akh", "\_index": "index-2018-08-09", "\_score": 0.0, "\_source": { "datetime\_log": "2018-08-09T00:34:36.051+02:00", **"datetime\_receive":**"2018-08-09T00:34:36.051+02:00", "group": "DEFAUT", "ip\_host": "22.33.44.55", **"ip\_host\_pkt": "22.33.44.55**", **"source\_msg":**"22.33.44.55: -Trashback= XXXXXXXXX XXXXXXXXXX XXXXXXXXX 8002DAD8 YYYYYYYY ZZZZZZZZZ UUUUUUUU IIIIIIIIII", "unix\_level": "local7", "unix\_priority": "crit" }, **"\_type":**"logs\_2" }, ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 23, 2018, 2:44pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/7 "2018-10-23T14:44:45Z")
Hi @moughrom > [@moughrom](#): > > I have syslogs in json (elastic extract) format and need to retrieve 4 fields convert those as csv and process the csv with all the logstash filters already set. What is "(elastic extract) format"? i.e. what request or technology are you using to extract the JSON? ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 3:31pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/8 "2018-10-23T15:31:31Z")
this is the json format extracted from elasticsearch. "\_id": "AWUbrXRdjZ-987654akh", "\_index": "index-2018-08-09", "\_score": 0.0, "\_source": { "datetime\_log": "2018-08-09T00:34:36.051+02:00", "datetime\_receive": "2018-08-09T00:34:36.051+02:00", "group": "DEFAUT", "ip\_host": "22.33.44.55", "ip\_host\_pkt": "22.33.44.55", "source\_msg": "22.33.44.55: -Trashback= XXXXXXXXX XXXXXXXXXX XXXXXXXXX 8002DAD8 YYYYYYYY ZZZZZZZZZ UUUUUUUU IIIIIIIIII", "unix\_level": "local7", "unix\_priority": "crit" }, "\_type": "logs\_2" }, ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 3:33pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/9 "2018-10-23T15:33:09Z")
but only interested with 4 fields "\_id": "AWUbrXRdjZ-987654akh", "\_index": "index-2018-08-09", "\_score": 0.0, "\_source": { "datetime\_log": "2018-08-09T00:34:36.051+02:00", "datetime\_receive": "2018-08-09T00:34:36.051+02:00", "group": "DEFAUT", "ip\_host": "22.33.44.55", "ip\_host\_pkt": "22.33.44.55", "source\_msg": "22.33.44.55: -Trashback= XXXXXXXXX XXXXXXXXXX XXXXXXXXX 8002DAD8 YYYYYYYY ZZZZZZZZZ UUUUUUUU IIIIIIIIII", "unix\_level": "local7", "unix\_priority": "crit" }, "\_type": "logs\_2" }, ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 23, 2018, 3:33pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/10 "2018-10-23T15:33:55Z")
Thanks @moughrom my question is _how_ is the JSON format extracted from Elasticsearch? I understand curl is involved. I do not understand what the request is and this makes a difference as there are many options. Please could you give an example of the request (with the auth token redacted). Thanks ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 3:35pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/11 "2018-10-23T15:35:08Z")
two curl command. first one to get an authentification token and the second one to extract data. ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 23, 2018, 3:36pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/12 "2018-10-23T15:36:22Z")
Hi @moughrom please could you give an example of the curl command used to extract data, this makes a difference, thankyou ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 3:41pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/13 "2018-10-23T15:41:35Z")
this is the one used to retrieve data. curl -k -silent -H "Authorization: Bearer $token" -X GET [https://mydomain/api/v1/proxy/myindex-2018-06-06/logs\_unix/\_search](https://mydomain/api/v1/proxy/myindex-2018-06-06/logs_unix/_search) -d'{"from" : 0, "size" : 3000}' ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 23, 2018, 5:26pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/14 "2018-10-23T17:26:33Z")
OK that makes sense. Are the curl commands issued by a bash script &/or is there any other scripted processing before the raw elasticsearch query output is read by logstash? ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 23, 2018, 7:19pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/15 "2018-10-23T19:19:38Z")
It will be executed at the end by a bash script and it has been executed as a script. it is the only script run to retrieve a json log files. ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 24, 2018, 8:11am UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/16 "2018-10-24T08:11:25Z")
Hey Dominic, here are the answers to your questions. Are the curl commands issued by a bash script \>\>\> YES is there any other scripted processing before the raw elasticsearch query output is read by logstash\>\>\> NO ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 24, 2018, 11:24am UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/17 "2018-10-24T11:24:32Z")
Hi @moughrom That's great. Logstash has codecs to deal with multiline messages and with JSON, however, chaining codecs is not supported (the reasons are documented in Github). As you are using bash script this gives us an opportunity to process the extracted JSON, which contains multiple documents as well as metadata for the query, into a file containing newline-delimited JSON with one line per document. `curl -X GET http://0.0.0.0:9200/index-2018-08-09/_search | sed -e "s/^.*_source\"://" | grep -v "}]}}" | awk '/^}/ {print (NR==1?"":RS)$0;next} {printf "%s",$0}' | grep -v "^}$" | sed -e "s/$/\}/" > /Users/Shared/logs/test.json` You can add your additional curl parameters and Elasticsearch instance. Then, you can use Logstash to ingest this using the JSON codec. Use the JSON not json\_line. ```auto input { # Read all documents from Elasticsearch matching the given query file { path => "/Users/Shared/logs/*.json" codec => json } } ``` Logstash filters can then rename, remove etc. You can add any extra filters you already have here. ```auto filter { mutate { rename => {"datetime_receive" => "date"} rename => {"datetime_log" => "datetime_log"} rename => {"group" => "group"} rename => {"ip_host_pkt" => "host"} rename => {"source_msg" => "message"} remove_field => ["unix_priority","@version","tags","@timestamp"] } } ``` note that some fields, e.g. `datetime_log` appear to be renamed to themselves, this is to prevent them getting stored in a `_source` subfield. Note that the `_type` field can no longer be modified from 6.0 onwards so I have left that as is. The full code is in this GIST [https://gist.github.com/djp-search/52175df2262072a41252d7761b8e72bc](https://gist.github.com/djp-search/52175df2262072a41252d7761b8e72bc) Hope this helps! ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 24, 2018, 8:09pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/18 "2018-10-24T20:09:43Z")
Hey Dominic, I am only getting one syslog message in the json output file, I have change \> to \>\> to concatanate, but still getting one msg at max I am getting 3 msg when I use -d '{"from":"zero","size":3000}' curl -X GET [http://0.0.0.0:9200/index-2018-08-09/\_search](http://0.0.0.0:9200/index-2018-08-09/_search) | sed -e "s/^.\*\_source"://" | grep -v "}]}}" | awk '/^}/ {print (NR==1?"":RS)$0;next} {printf "%s",$0}' | grep -v "^}$" | sed -e "s/$/}/" **\>\>** /Users/Shared/logs/test.json please advice. ---
**Author:** ![moughrom](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@moughrom](https://discuss.elastic.co/u/moughrom)\ **Post date:** [October 24, 2018, 8:42pm UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/19 "2018-10-24T20:42:55Z")
here it is the query: curl -H "Authorization: Bearer $token" -X GET [https://localhost/api/v1/proxy/index-2018-06-11/logs\_unix/\_search](https://localhost/api/v1/proxy/index-2018-06-11/logs_unix/_search) -d'{"size":1000}' --insecure | sed -e "s/^.\*\_source"://" | grep -v "}]}}" | awk '/^}/ {print (NR==1?"":RS)$0;next} {printf "%s",$0}' | grep -v "^}$" | sed -e "s/$/}/" \> /data/log.json and the result, only one line: {"datetime\_log":"2018-06-11T02:55:19.933+02:00","datetime\_receive":"2018-06-11T02:55:19.933+02:00","group":"DEFAUT","ip\_host":"192.168.33.12","ip\_host\_pkt":"192.168.33.12","source\_msg":"mdd-04935DDDD1 : Jun 11 02:55:19 #NOTIFY# IF: GigabitEthernet 0/2 is DOWN","unix\_level":"local12","unix\_priority":"notice"},"\_type":"logs\_unix"}],"max\_score":0.0,"total":200603},"timed\_out":false,"took":101328}} ---
**Author:** ![Dominic\_Page](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dominic_page/32/48318_2.png) [@Dominic\_Page](https://discuss.elastic.co/u/Dominic_Page)\ **Post date:** [October 25, 2018, 8:42am UTC](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840/20 "2018-10-25T08:42:37Z")
Hi @moughrom I don't have the resources to fully replicate your process offline. Here are some basic pointers: The simplest way to "fix" Elasticsearch data is inside the cluster using painless. You could do this using appropriate requests and the AUTH tokens, with the limitation that you would not have the added value of the logstash geoip lookup. However, all the other field operations in your requirements could be performed simply and reliably inside the cluster without extracting the data. If you _do_ need to pull the data out, you may need to do some parsing before resubmitting to Elasticsearch. I would choose the technology that you are most comfortable with to do this. From your replies above ^^ , it looks like AWK is not matching your pattern. You could fix this by looking more into AWK, however, I would contra-indicate the append operation "\>\>" as you don't want to write the same document multiple times to the file. The reason I suggested AWK was to avoid adding more technology, however, there are many options of third-party open-source scripting languages that you could use to process the data to a format that is ready for ingest via logstash. I would suggest to go with whichever you are most experienced in. [Next page](https://discuss.elastic.co/t/elastic-json-parse-into-logstash/151840.md?page=2)