Hey Dominic,
I have not a big experience on tools like AWK or others.
please advice tools that I may use instead of AWK.
Best and thanks a lot for your support.
Mohamed
Hi @moughrom
Your choice of technology to manipulate JSON is up to you. If Logstash were connecting to Elasticsearch via Elastic Security, then the Elasticsearch input plugin for Logstash would already be receiving JSON, off-the-shelf, and you would not have this issue.
As you have a different security model, you are extracting the documents as JSON documents within a larger JSON document. You need a tool that will allow you to parse those documents into a format that can be easily read by Logstash, for example a file containing multiple, single JSON documents, each on a separate line. As well as the standard command line tools, there are many scripting languages and open-source tools available, I hope you can find one that suits your need.
elastic security is install on my cluster (6.4)
but got issue with json multiline.
can you name few tools you are talking about.
Hi @moughrom
If you have Elastic Security Installed, your Logstash should be able to connect directly to Elasticsearch. Then Logstash can read the documents directly and there will be no need for parsing JSON. Logstash can make the necessary updates and additions and then index the amended data to a new index in Elasticsearch, again securely.
You can find guidance on connecting Logstash to Elastic Security here:
Hope this helps
Thanks Dominic, I have googled "json tools linux command line" and got a bunch of github projects.
My issue at the end was the grep/awk/sed was getting out after the first event and not processing next events.
I have started learning awk/grep/sed to understand, but as an expert cans you see why we got this behavior, from my experience in powershell when a pipeline end is that a part of the pipeline stop sending to the next pipeline and display on the screen instead and breaking the chains.
please advice.
curl -X GET http://0.0.0.0:9200/index-2018-08-09/_search | sed -e "s/^.*_source"://" | grep -v "}]}}" | awk '/^}/ {print (NR==1?"":RS)$0;next} {printf "%s",$0}' | grep -v "^}$" | sed -e "s/$/}/" >> /Users/Shared/logs/test.json
I gave a bash example to show that in principle logstash could be used to ingest the data once this had been prepared. As mentioned above I do not have the resources to replicate your full environment, so it will not be possible for me to progress your JSON parsing issue further. I recommend you seek a local resource who can help you with your parsing issue hands-on.
Many Thanks for your answers.
it has really help.
Best.
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.