How to create .raw field through logstash


(Fredrick Yessaian) #1

Hello,

I have been using ELK to see our application logs.
I achieved this by feeding application generated logs as JSON string through Logstash --> Elasticsearch --> Kibana.
All the objects in my JSON stings are analyzed and I'm able to search using that, Which I need all objects to be analyzed.
Everything is fine. But I want to see the data through visualize (Pie Chart) also, that I'm not able to because all the objects are analyzed.
As per the blog, If we have objects as non analyzed or indexed then it is possible. But I need both .
I want to copy & convert a filed as .raw type then visualization could be achieved, so that I will have that field (Object) for search and also for the aggregation.
I don't want to reindex existing data instead I should be able to do it while data is processed in logstash.

I want to give a try , I want to copy & convert the type of one of the objects in JSON string as .raw type.
How can I achieve through either filter or output plugins in logstash

my application generates log output like this

{"loggingTime":"Wed 7 Feb 2018 10-02-38 672 EST","applicationName":"JPA Server","hostName":"server_name","loglevel":"INFO","logMessage":"Request execution time warning threshold is set to 1000ms"}

This will be fed to logstash.
I want to add or create loglevel as .raw field along with existing fields. so that I can have logleve.raw field for aggregation.

Can I get a help?

Thanks in Advance.
Fredrick.


(Magnus B├Ąck) #2

This is more of an Elasticsearch question. You should look into Elasticsearch index templates and how to make subfields of string fields non-analyzed (ES pre-5.0) or keyword fields (ES post-5.0). You can use the index templates bundled with Logstash as inspiration.

Once you've written an index template file you can configure Logstash to upload it to ES (but you can also manage the template outside of Logstash).


(Fredrick Yessaian) #3

Thanks Magnus.. I will take a look


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.