Hi,
I am new to logstash and want to know how to parse a file containing multiple format such as :---
{"name":"travel.train","hostname":"mktldstickets01","pid":4241,"meta":{"route":"/search","type":"search","date":"2016-07-10","destination":"chennai","source":"guntur"},"level":30,"msg":"Search request received","time":"2016-06-11T13:28:12.470Z","v":0}
2016-06-11T13:28:12.508Z GET /v2/cities/*?id=123245&deviceIdentifier=LENOVO-LenovoA6000-868087025001742&deviceManufacturer=LENOVO&deviceName=Lenovo_A6000&client=windows&version=5.1.1&playStore=true&lat=435.34225912&long=2341.2397352&language=en&imei=868012340986&osVersion=5.0.2 Dalvik/2.1.0 (Linux; U; Android 5.0.2; Lenovo A6000 Build/LRX22G) 200 981 6.425 ms
The json object here is oneevent and text that is line starting with time stamp format is another event. I want to write a filter for logstash where it can parse both lines but as separate event. Please help.
I have written one filter but that is actaully giving me _grokparsefailure when json is parsed and _jsonfailure when text is parsed.
filter {
multiline {
#this one will look for any line starting with whitespace and join it to the previous line
what => "previous"
pattern => "^\s" }
grok{
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:method} %{URIPATH:request}%{URIPARAM:querystring}? %{GREEDYDATA:agent} %{NUMBER:status} %{NUMBER:bytes} %{NUMBER:duration}" }
}
kv {
field_split => "*?&"
include_keys => [ "sso_token", "deviceIdentifier","order_id"]
}
multiline {
#this one will look for any line starting with and join it to the previous line
what => "previous"
pattern => "^{"
}
json {
source => message
}
}
How about using a conditional to only parse it as JSON if it looks like JSON?
filter {
if [message] =~ /^{"/ {
json {
source => "message"
}
} else {
# do other stuff
}
}
Hi magnusbaeck,
I already did that thanks and it worked somehow I am not able to parse, nested json it is giving me an error saying.
[2016-06-15 02:10:36,631][WARN ][index.engine ] [mktpsdwdes07] [ftickets-app-logs-2016.06.14][0] failed engine [indices:data/write/bulk[s] failed on replica]
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [ftickets-app-logs] tried to parse field [removed_buses] as object, but got EOF, has a concrete value been provided to it?
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:495)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:466)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:566)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:250)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:229)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Json parsed:---
"name":"travel.bus","hostname":"ftickets01","pid":3237,"meta":{"route":"/search","type":"search","orderBy":0,"sortBy":"fare","date":"2016-06-17","destination":"vijayawada"
,"source":"hyderabad"},"level":30,"removed_buses":{"bus":5,"travl_yi":117,"ahi_bus":27,"ila":15,"tktgoe":5},"msg":"Buses removed from search results due to duplication
s","time":"2016-06-17T09:14:44.675Z","v":0}
It is somewhat related to mapping issue. Can you please make me understand? I am new to ELK