nyet
July 30, 2019, 2:54pm
1
I have a log source that generates json log files with excess line feeds.
I can get around this (partially) by adding exclude_lines: ['^$'] as suggested here:
i have a json log file, where each entry is separated by a blank line
{"@timestamp":1456544565,"valid_domain":true,"ip":"8.8.8.8"}
{"@timestamp":2464564543,"valid_domain":false,"ip":"1.2.3.4"}
{"@timestamp":3454678735,"valid_domain":false,"ip":"9.8.7.6"}
filebeat refused to parse it, giving the error
2018-08-08T10:08:43.626-0300 ERROR reader/json.go:33 Error decoding JSON: EOF
so i tried excluding empty lines with
exclude_lines : ['^$']
and it worked! until... i restarted the service. th…
This solves the problem on the logstash/ES side, but syslog where file beats is running is completely filled with
2019-07-30T14:45:41.319Z#011ERROR#011readjson/json.go:52#011Error decoding JSON: EOF
See also:
opened 08:11AM - 10 Nov 17 UTC
closed 10:16PM - 12 Nov 17 UTC
I am using BELK stack for Log Analytics.
--------------------------------------… ----------------------------------------------------------
versions:
ES-5.1.1 is on AWS service
Logstash-5.3.0, filebeat-5.3.0 and kibana-5.1.1
------------------------------------------------------------------------------------------------
filebeat is running on the server where cdrs are generated.
Logstash is running on separate server and sends data to date based rolling index in elasticsearch.
We are using java logback.xml to write single line json events in .log file which rotates basis size and time logic
<appender name="SIZE_AND_TIME_BASED_LOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${CDR_NEW_LOG_PATH}//cdr.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${CDR_NEW_LOG_PATH}//cdr-${IP_ADD}.%d{yyyy-MM-dd-HH-mm}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>${CDR_NEW_LOG_MAX_FILE_SIZE}</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
</rollingPolicy>
------------------------------------------------------------------------------------------------
In filebeat.yml,
filebeat.prospectors:
- input_type: log
paths:
- /opt/cdrs/*-*.log
json.keys_under_root: true
json.add_error_key: true
ignore_older: 24h
close_inactive: 12h
scan_frequency: 30s
clean_inactive: 48h
clean_removed: true
close_removed: true
close_eof: true
output.logstash:
hosts: ["VALID IP:5043"]
fields_under_root: false
------------------------------------------------------------------------------------------------
Logstash Config
input {
beats {
port => 5043
client_inactivity_timeout => 86400
}
}
filter
{
grok {
match => { "g2uEvent" => "%{TIMESTAMP_ISO8601:g2uEventTime}"}
}
date {
match => ["g2uEventTime", "ISO8601"]
target => "@timestamp"
}
}
output {
amazon_es {
hosts => ["VALID ELASTIC SEARCH END POINT"]
index => "d2c-%{+YYYY-MM-dd}"
}
stdout { codec => rubydebug }
}
------------------------------------------------------------------------------------------------
Issues come randomly when there is traffic. If the same file is sent to filebeat, all data is correctly send to ES.
Common erros in filebeat:
ERR Error decoding JSON: invalid character '}' looking for beginning of value
ERR Error decoding JSON: json: cannot unmarshal string into Go value of type map[string]interface {}
ERR Error decoding JSON: EOF
ERR Error decoding JSON: invalid character '\n' in string literal
ERR Error decoding JSON: invalid character 'm' in literal true (expecting 'r')
------------------------------------------------------------------------------------------------
Sample CDR
{"g2uEvent":{"g2uEventName":"CAMPAIGN_USER_LOGIN_MOBILE_DATA_EVENT","g2uEventTime":"2017-11-10T05:22:32.270Z","g2uEventDate":"2017-11-10"},"g2uCountry":{"g2uCountryIsoname":"IN","g2u_analytic_country_name":"IND"},"g2uOperatorInfo":{"g2uOperatorName":"Idea","g2uOperatorAnalyticsname":"IDEA"},"g2uLanguage":"en","g2uUser":{"g2uUserMsisdn":"91xxxxxxxx","g2uUserChannel":"mobileData","g2uUserStatus":"STATUS_EXPIRED"},"g2uGames":{"g2uGamesId":"","g2uGamesName":"","g2uGamesCategory":""},"g2uGamesVendor":{"g2uGamesVendorName":""},"g2uGamesUtm":{"g2uUtmSource":"test","g2uUtmMedium":"18333108","g2uUtmCampaign":"ADS"}}
{"g2uEvent":{"g2uEventName":"CAMPAIGN_USER_ADVERTISER_CALLBACK_BLOCKED_EVENT","g2uEventTime":"2017-11-09T14:21:45.918Z","g2uEventDate":"2017-11-09"},"g2uCountry":{"g2uCountryIsoname":"MY","g2uAnalyticCountryName":""},"g2uOperatorInfo":{"g2uOperatorName":"Maxis","g2uOperatorAnalyticsname":""},"g2uUser":{"g2uUserMsisdn":"60xxxxxxx","g2uUserChannel":"MobileData","g2uUserStatus":"STATUS_SUBSCRIBED"},"g2uGames":{"g2uGamesId":"","g2uGamesName":"","g2uGamesCategory":""},"g2uGamesVendor":{"g2uGamesVendorName":""},"g2uGamesUtm":{"g2uUtmSource":"test1","g2uUtmMedium":"18290630","g2uUtmCampaign":"ADS"}}
system
August 27, 2019, 3:03pm
2
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.