I am trying to deploy this pipeline on Kibana/Logstash pipelines:
input {
rabbitmq {
host => "xxxxxxxxxx"
vhost => "/"
queue => "outrights"
port => 5672
heartbeat => 30
durable => true
password => "xxxxxxxx!"
user => "xxxxxxxx"
}
}
filter {
dissect {
mapping => { "message" => "%{permission},%{sourceID},%{type},%{sessionIndicator},%{ccyPair},%{priceType},%{tenor},%{bidPrice},%{askPrice},%{source},%{region},%{city},%{regionID},%{cityID},%{currentPrice},%{tradeOpen},%{tradeHigh},%{tradeLow},%{trend},%{direction},%{change},%{percentChange},%{tradePrice},%{yesterdayTradeClose},%{tradeDateTime},%{quoteDateTime},%{sentDateTime},%{fractionalIndicator},%{previousTradeDate},%{desktopEligibilityIndicator},%{midPrice}" }
}
mutate
{
remove_field => ["permission","desktopEligibilityIndicator","message","regionID","cityID","tags","quoteDateTime","tradeDateTime","trend","direction","sessionIndicator","yesterdayTradeClose","tradeHigh","tradeLow","tradeOpen","tradePrice","currentPrice","previousTradeDate","change","percentChange","type" ]
}
}
output {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
user => "elastic"
password => "elastic"
index => "outrights"
template_overwrite => "true"
manage_template => "false"
}
stdout{codec=> json}
}
This is the ERROR:
[2018-07-03T14:53:19,209][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character (',' (code 44)): Expected space separating root-level values
at [Source: (String)"1321,1321,X,,USDBRL,O,12M,3.97110,3.97620,HSBC,NAM,NYC,160115,163103,,,,,-+-++-++,-,-0.01080,-0.27000,3.97110,,2018-06-28T19:58:28.548Z,2018-06-28T19:58:28.548Z,2018-06-28T19:58:28.548Z,,,1,3.97365,280=4"; line: 1, column: 6]>, :data=>"1321,1321,X,,USDBRL,O,12M,3.97110,3.97620,HSBC,NAM,NYC,160115,163103,,,,,-+-++-++,-,-0.01080,-0.27000,3.97110,,2018-06-28T19:58:28.548Z,2018-06-28T19:58:28.548Z,2018-06-28T19:58:28.548Z,,,1,3.97365,280=4"}
This is my logstash yml:
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://127.0.0.1:9200
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: elastic
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true
xpack.monitoring.elasticsearch.sniffing: false
#
# X-Pack Management
# https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html
xpack.management.enabled: true
xpack.management.elasticsearch.url: http://127.0.0.1:9200/
xpack.management.elasticsearch.username: elastic
xpack.management.elasticsearch.password: elastic
xpack.management.logstash.poll_interval: 5s
xpack.management.pipeline.id: outrights
Please can you help?