I am using BELK stack for Log Analytics.
versions:
ES-5.1.1 is on AWS service
Logstash-5.3.0, filebeat-5.3.0 and kibana-5.1.1
filebeat is running on the server where cdrs are generated.
Logstash is running on separate server and sends data to date based rolling index in elasticsearch.
We are using java logback.xml to write single line json events in .log file which rotates basis size and time logic
${CDR_NEW_LOG_PATH}//cdr.log ${CDR_NEW_LOG_PATH}//cdr-${IP_ADD}.%d{yyyy-MM-dd-HH-mm}.%i.log ${CDR_NEW_LOG_MAX_FILE_SIZE}
In filebeat.yml,
filebeat.prospectors:
- input_type: log
paths:- /root/cdrs/-.log
json.keys_under_root: true
- /root/cdrs/-.log
json.add_error_key: true
ignore_older: 24h
close_inactive: 12h
scan_frequency: 30s
clean_inactive: 48h
clean_removed: true
close_removed: true
close_eof: true
output.logstash:
The Logstash hosts
hosts: ["VALID IP:5043"]
fields_under_root: false
Logstash Config
input {
beats {
port => 5043
client_inactivity_timeout => 86400
}
}
filter
{
grok {
match => { "g2uEvent" => "%{TIMESTAMP_ISO8601:g2uEventTime}"}
}
date {
match => ["g2uEventTime", "ISO8601"]
target => "@timestamp"
}
}
output {
amazon_es {
hosts => ["VALID ELASTIC SEARCH END POINT"]
index => "d2c-%{+YYYY-MM-dd}"
}
stdout { codec => rubydebug }
}
Issues come randomly when there is traffic. If the same file is sent to filebeat again by, all data is correctly send to ES.
Common errors that I get in filebeat:
ERR Error decoding JSON: invalid character '}' looking for beginning of value
ERR Error decoding JSON: json: cannot unmarshal string into Go value of type map[string]interface {}
ERR Error decoding JSON: EOF
ERR Error decoding JSON: invalid character '\n' in string literal
ERR Error decoding JSON: invalid character 'm' in literal true (expecting 'r')
Sample CDR events
{"g2uEvent":{"g2uEventName":"CAMPAIGN_USER_LOGIN_MOBILE_DATA_EVENT","g2uEventTime":"2017-11-10T05:22:32.270Z","g2uEventDate":"2017-11-10"},"g2uCountry":{"g2uCountryIsoname":"IN","g2u_analytic_country_name":"IND"},"g2uOperatorInfo":{"g2uOperatorName":"Idea","g2uOperatorAnalyticsname":"IDEA"},"g2uLanguage":"en","g2uUser":{"g2uUserMsisdn":"91xxxxxxxx","g2uUserChannel":"mobileData","g2uUserStatus":"STATUS_EXPIRED"},"g2uGames":{"g2uGamesId":"","g2uGamesName":"","g2uGamesCategory":""},"g2uGamesVendor":{"g2uGamesVendorName":""},"g2uGamesUtm":{"g2uUtmSource":"test","g2uUtmMedium":"18333108","g2uUtmCampaign":"ADS"}}
{"g2uEvent":{"g2uEventName":"CAMPAIGN_USER_ADVERTISER_CALLBACK_BLOCKED_EVENT","g2uEventTime":"2017-11-09T14:21:45.918Z","g2uEventDate":"2017-11-09"},"g2uCountry":{"g2uCountryIsoname":"MY","g2uAnalyticCountryName":""},"g2uOperatorInfo":{"g2uOperatorName":"Maxis","g2uOperatorAnalyticsname":""},"g2uUser":{"g2uUserMsisdn":"60xxxxxxx","g2uUserChannel":"MobileData","g2uUserStatus":"STATUS_SUBSCRIBED"},"g2uGames":{"g2uGamesId":"","g2uGamesName":"","g2uGamesCategory":""},"g2uGamesVendor":{"g2uGamesVendorName":""},"g2uGamesUtm":{"g2uUtmSource":"test1","g2uUtmMedium":"18290630","g2uUtmCampaign":"ADS"}}
Please help me to resolve the same.