# Logstash Filter for JSON data

**URL:** <https://discuss.elastic.co/t/logstash-filter-for-json-data/72574>\
**Category:** Logstash\
**Created:** [January 24, 2017, 7:09am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574 "2017-01-24T07:09:44Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 24, 2017, 7:09am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574/1 "2017-01-24T07:09:44Z")

</div>

hello,

i have the same issue with this post.

> <https://stackoverflow.com/questions/38869886/sending-json-format-log-to-kibana-using-filebeat-logstash-and-elasticsearch>

the difference is he use filebeat to send data to logstash, then the logstash will send data to elasticsearch.

i use logstash to load my log file (written in json), then i export it to elasticsearch.  
but the problem is my json not separated to different object in elasticsearch.  
its like they merged into one object, named message.

> "message": "{ \n "user\_id":0,\n "cart\_id":"222",\n "error\_status":"",\n "error\_message":"",\n "data":{ \n "ord\_id":2233,\n "cart":{ \n "cart\_id":"222",\n "session\_id":"7afaaf7fac9bb934de4c6ecb2567461a6178495e",\n "ip\_address":"180.254.65.91",\n "usr\_id":"0",\n "created\_date":"2017-01-19 17:29:20",\n "updated\_date":"2017-01-19 18:08:35"\n },\n "cart\_data":{ \n "coupon\_code":"",\n "ord\_email":"gutasaputra@gmail.com",\n "ord\_firstname":"guta"}"

what i want is like this :

"message": "{  
user\_id:0,  
cart\_id:3486,  
[ord\_email:gutasaputra@gmail.com](mailto:ord_email:gutasaputra@gmail.com)  
}"

how can i do that via logstash?

ohy, here is my logstash conf :

```
input
{
    file
    {
        codec => multiline
        {
            pattern => '^\{'
            negate => true
            what => previous
        }

        path => ["/usr/local/Cellar/logstash/5.1.1/test_payment.json"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter {
    if [message] =~ /^{.*}$/
    {
        json { source => message }
    }

}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    user => elastic
    password => techno2013
    index => "test_payment"
 }
  stdout { codec => rubydebug }
}

```

thank you

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 24, 2017, 7:15am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574/2 "2017-01-24T07:15:00Z")

</div>

The correct solution is to use a json filter just like you're doing now. If it doesn't seem to be working, check that there's nothing wrong with your conditional (i.e. try commenting it) and that the JSON parsing isn't failing (check the Logstash log).

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 24, 2017, 7:47am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574/3 "2017-01-24T07:47:05Z")

</div>

here is my logstash log file.

> [2017-01-24T13:55:11,592][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::Elasticsearch", :hosts=\>["localhost:9200"]}  
> [2017-01-24T13:55:11,596][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500}  
> [2017-01-24T13:55:11,637][INFO][logstash.pipeline] Pipeline main started  
> [2017-01-24T13:55:11,700][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

the only error msg that i had in my log file is this :

> [2017-01-24T13:53:02,894][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
> "org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:159)", "org.jruby.runtime.CompiledBlock19.call(CompiledBlock19.java:87)", "org.jruby.runtime.Block.call(Block.java:101)", "org.jruby.RubyProc.call(RubyProc.java:300)", "org.jruby.RubyProc.call19(RubyProc.java:281)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "rubyjit.LogStash::Util::WrappedSynchronousQueue::ReadBatch$$each\_8ee46fd8c62d1155253b7a084f2e67ff25953d7e1956725890.block\_0$RUBY$ **file** (/usr/local/Cellar/logstash/5.1.1/libexec/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:192)", "rubyjit$LogStash::Util::WrappedSynchronousQueue::ReadBatch$$each\_8ee46fd8c62d1155253b7a084f2e67ff25953d7e1956725890$block\_0$RUBY$ **file**.call(rubyjit$LogStash::Util::WrappedSynchronousQueue::ReadBatch$$each\_8ee46fd8c62d1155253b7a084f2e67ff25953d7e1956725890$block\_0$RUBY$ **file** )", "org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)", "org.jruby.runtime.Block.yield(Block.java:142)", "org.jruby.RubyHash$13.visit(RubyHash.java:1355)", "org.jruby.RubyHash.visitLimited(RubyHash.java:648)", "org.jruby.RubyHash.visitAll(RubyHash.java:634)", "org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1306)", "org.jruby.RubyHash.each\_pairCommon(RubyHash.java:1351)", "org.jruby.RubyHash.each19(RubyHash.java:1342)", "org.jruby.RubyHash$INVOKER$i$0$0$each19.call(RubyHash$INVOKER$i$0$0$each19.gen)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)", "rubyjit.LogStash::Util::WrappedSynchronousQueue::ReadBatch$$each\_8ee46fd8c62d1155253b7a084f2e67ff25953d7e1956725890. **file** (/usr/local/Cellar/logstash/5.1.1/libexec/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:191)", "rubyjit.LogStash::Util::WrappedSynchronousQueue::ReadBatch$$each\_8ee46fd8c62d1155253b7a084f2e67ff25953d7e1956725890. **file** (/usr/local/Cellar/logstash/5.1.1/libexec/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:161)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)", "rubyjit.LogStash::Pipeline$$filter\_batch\_390eee8140cc612b1771fb72e9904822f536cabf1956725890.chained\_0\_rescue\_1$RUBY$SYNTHETIC\_\_file\_\_(/usr/local/Cellar/logstash/5.1.1/libexec/logstash-core/lib/logstash/pipeline.rb:294)", "rubyjit.LogStash::Pipeline$$filter\_batch\_390eee8140cc612b1771fb72e9904822f536cabf1956725890. **file** (/usr/local/Cellar/logstash/5.1.1/libexec/logstash-core/lib/logstash/pipeline.rb)", "rubyjit.LogStash::Pipeline$$filter\_batch\_390eee8140cc612b1771fb72e9904822f536cabf1956725890. **file** (/usr/local/Cellar/logstash/5.1.1/libexec/logstash-core/lib/logstash/pipeline.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)", "org.jruby.ast.WhileNode.interpret(WhileNode.java:131)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)", "org.jruby.evaluator.ASTInterpreter.INTERPRET\_METHOD(ASTInterpreter.java:74)", "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)", "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)", "org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)", "org.jruby.evaluator.ASTInterpreter.INTERPRET\_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)", "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)", "org.jruby.runtime.Block.call(Block.java:101)", "org.jruby.RubyProc.call(RubyProc.java:300)", "org.jruby.RubyProc.call(RubyProc.java:230)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)", "java.lang.Thread.run(Thread.java:745)"]}

i think thats because of my filter configuration before.  
i try to put this filter :

> filter {  
> if [tags][json] {  
> json {  
> source =\> "message"  
> }  
> }  
> }

then the error comes.

so, what should i do sir?  
i search all over the web, but still have no solutions.  
can you pls help me.

thank you.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 24, 2017, 8:06am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574/4 "2017-01-24T08:06:43Z")

</div>

> ```
> if [tags][json] {
> 
> ```

I don't know what this is supposed to mean. For now drop the conditional and focus on getting things working.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 24, 2017, 8:08am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574/5 "2017-01-24T08:08:20Z")

</div>

its my old configuration sir.

my new one is this :  
filter {  
if [message] =~ /^{.\*}$/  
{  
json { source =\> message }  
}

}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 24, 2017, 8:11am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574/6 "2017-01-24T08:11:03Z")

</div>

I repeat: Drop the conditional and focus on getting things working.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2017, 8:11am UTC](https://discuss.elastic.co/t/logstash-filter-for-json-data/72574/7 "2017-02-21T08:11:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.