Hi,
I am using filebeat to push logs to elastic search. The logs is mainly a json which is being pushed by different celery workers all in one single file. However I am noticing that these logs are getting merged for few cases.
Errors in filebeat error log
ERR Rotate error: rename /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry.new /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry: no such file or directory
2019/11/14 09:26:51.198936 registrar.go:217: ERR Writing of registry returned error: rename /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry.new /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry: no such file or directory. Continuing...
2019/11/14 09:27:01.312579 helper_other.go:14: ERR Rotate error: rename /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry.new /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry: no such file or directory
2019/11/14 09:27:01.312611 registrar.go:217: ERR Writing of registry returned error: rename /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry.new /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry: no such file or directory. Continuing...
2019/11/14 09:27:05.328919 helper_other.go:14: ERR Rotate error: rename /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry.new /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry: no such file or directory
2019/11/14 09:27:05.328949 registrar.go:217: ERR Writing of registry returned error: rename /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry.new /etc/filebeat_6/filebeat-6.0.0-linux-x86_64/data/registry: no such file or directory. Continuing...
Adding a sample log message
{
"primary_id": "3f72d0fd-7f05-4f2a-9e5c-a5a090407cbf__0",
"ch": "android",
"ad": ["APA91bGRgCheCfghPyC_4BZuI-cXx3W3-xOd5HeX-93ZM1VfzYxslQPvoxR1FPGJ4zZF6wl7c01NL7l4BKKL2tCX5Gq5LlZ6QD7Rq3V2IM5OQekLjmDSrcFlU6egcvADhZxpqUBLfMBk", "APA91bHbKPa6Ae0ff0-YttkeFUi33tRjK0MyUlWdWeZQR5C8oWE4Ie1N6QIvHlmfmxv5hjYRXhYrCD49E-sc1blpwjH5s2-PlMaMsD0nXNM1OMFEMRCS6l6ROO7RZnYMAMzpORQ0Ah1d", "APA91bE8vcxuzQ8_ItHaw5qaZYyiDo62PoImh0kygmeuNWjgYwb6KQ3Zrin54ZK7sloOB3QgIyd1Y5Cbi-tWdBaphenAyM4TwQ89VD14CSYHDZlAFV1z3IdCgGw4tBMGTJC-xtNX-bgw", "APA91bHJXjBkEzV9A5hfv6h-yv-hUm9ixOEhr-lZp7mgWothmJAIzeiNXyX7lF0Xsk8H46MF77oWroE9GErFwtg2927DSaCHvdgVcFkLqy3sc1drbnnraVnx7QA44q5qCCci7mZ6KNzk", "APA91bE7UESyiIAF_58lgpd9mEETxu9q7XPlXGFpWopbwojlGCzQBbvWLAvH2VV_dIA36dIj9aoT_SfvLvpulmXpmivGRD_Eeq6kCJIe69NShfBt-0YQ9jyxUOzv18B3Zzcy8_nfqauu", "APA91bETfPnCygDyxMKDAFJhvytRDb9xltvT1ASnycRP3guAAYy6f3_-6SJ0snGkJrwvBLXesAmkYl6gL4tk26ue5urZppGErpLXUw6pqaus-ikhoAvs_uq4GuztoqXcljl1rcMy52M3", "APA91bFAoETYeU_0hoS75UV5KvEorTXMP1UnrREDsibXqxsRnp8Ahixx30A1FznKMReFpNgKPGyblHYh8UwtkLLB6ocNVbUGfv4GaP55gJqCieDc8DYBPJLS_MUfdP0QfSiHI_lkXCWu", "APA91bHHwYsRm_CjrFohHltvHvoZABgiGAWiqIUPQqE8QLNizaOXFP4CPV5KyvTINJfd6VORDqD4xQIJgU9c8BWiyNgjkrwUiDEkpc1w1h5vxn3OYPSLdh67CIee7OzPF1a9ekNVZAPy"],
"md": "2019-11-13 14:14:24",
"cb": "",
"app": "mmt",
"message_identifiers": {},
"nid": "3f72d0fd-7f05-4f2a-9e5c-a5a090407cbf",
"cd": "2019-11-13 14:14:24",
"tn": "mmt_template",
"tag": [{
"value": "aakash.rockstar2009@gmail.com",
"key": "link"
}, {
"value": "917895406243",
"key": "link"
}, {
"value": "APA91bGRgCheCfghPyC_4BZuI-cXx3W3-xOd5HeX-93ZM1VfzYxslQPvoxR1FPGJ4zZF6wl7c01NL7l4BKKL2tCX5Gq5LlZ6QD7Rq3V2IM5OQekLjmDSrcFlU6egcvADhZxpqUBLfMBk",
"key": "pntoken"
}, {
"value": "APA91bHbKPa6Ae0ff0-YttkeFUi33tRjK0MyUlWdWeZQR5C8oWE4Ie1N6QIvHlmfmxv5hjYRXhYrCD49E-sc1blpwjH5s2-PlMaMsD0nXNM1OMFEMRCS6l6ROO7RZnYMAMzpORQ0Ah1d",
"key": "pntoken"
}, {
"value": "APA91bE8vcxuzQ8_ItHaw5qaZYyiDo62PoImh0kygmeuNWjgYwb6KQ3Zrin54ZK7sloOB3QgIyd1Y5Cbi-tWdBaphenAyM4TwQ89VD14CSYHDZlAFV1z3IdCgGw4tBMGTJC-xtNX-bgw",
"key": "pntoken"
}, {
"value": "APA91bHJXjBkEzV9A5hfv6h-yv-hUm9ixOEhr-lZp7mgWothmJAIzeiNXyX7lF0Xsk8H46MF77oWroE9GErFwtg2927DSaCHvdgVcFkLqy3sc1drbnnraVnx7QA44q5qCCci7mZ6KNzk",
"key": "pntoken"
}, {
"value": "APA91bE7UESyiIAF_58lgpd9mEETxu9q7XPlXGFpWopbwojlGCzQBbvWLAvH2VV_dIA36dIj9aoT_SfvLvpulmXpmivGRD_Eeq6kCJIe69NShfBt-0YQ9jyxUOzv18B3Zzcy8_nfqauu",
"key": "pntoken"
}, {
"value": "APA91bETfPnCygDyxMKDAFJhvytRDb9xltvT1ASnycRP3guAAYy6f3_-6SJ0snGkJrwvBLXesAmkYl6gL4tk26ue5urZppGErpLXUw6pqaus-ikhoAvs_uq4GuztoqXcljl1rcMy52M3",
"key": "pntoken"
}, {
"value": "APA91bFAoETYeU_0hoS75UV5KvEorTXMP1UnrREDsibXqxsRnp8Ahixx30A1FznKMReFpNgKPGyblHYh8UwtkLLB6ocNVbUGfv4GaP55gJqCieDc8DYBPJLS_MUfdP0QfSiHI_lkXCWu",
"key": "pntoken"
}, {
"value": "APA91bHHwYsRm_CjrFohHltvHvoZABgiGAWiqIUPQqE8QLNizaOXFP4CPV5KyvTINJfd6VORDqD4xQIJgU9c8BWiyNgjkrwUiDEkpc1w1h5vxn3OYPSLdh67CIee7OzPF1a9ekNVZAPy",
"key": "pntoken"
}],
"fd": null,
"time": "2019-11-13T08:44:24.605837",
"dt": {
"primary_id": "3f72d0fd-7f05-4f2a-9e5c-a5a090407cbf__0",
"token": ["APA91bGRgCheCfghPyC_4BZuI-cXx3W3-xOd5HeX-93ZM1VfzYxslQPvoxR1FPGJ4zZF6wl7c01NL7l4BKKL2tCX5Gq5LlZ6QD7Rq3V2IM5OQekLjmDSrcFlU6egcvADhZxpqUBLfMBk", "APA91bHbKPa6Ae0ff0-YttkeFUi33tRjK0MyUlWdWeZQR5C8oWE4Ie1N6QIvHlmfmxv5hjYRXhYrCD49E-sc1blpwjH5s2-PlMaMsD0nXNM1OMFEMRCS6l6ROO7RZnYMAMzpORQ0Ah1d", "APA91bE8vcxuzQ8_ItHaw5qaZYyiDo62PoImh0kygmeuNWjgYwb6KQ3Zrin54ZK7sloOB3QgIyd1Y5Cbi-tWdBaphenAyM4TwQ89VD14CSYHDZlAFV1z3IdCgGw4tBMGTJC-xtNX-bgw", "APA91bHJXjBkEzV9A5hfv6h-yv-hUm9ixOEhr-lZp7mgWothmJAIzeiNXyX7lF0Xsk8H46MF77oWroE9GErFwtg2927DSaCHvdgVcFkLqy3sc1drbnnraVnx7QA44q5qCCci7mZ6KNzk", "APA91bE7UESyiIAF_58lgpd9mEETxu9q7XPlXGFpWopbwojlGCzQBbvWLAvH2VV_dIA36dIj9aoT_SfvLvpulmXpmivGRD_Eeq6kCJIe69NShfBt-0YQ9jyxUOzv18B3Zzcy8_nfqauu", "APA91bETfPnCygDyxMKDAFJhvytRDb9xltvT1ASnycRP3guAAYy6f3_-6SJ0snGkJrwvBLXesAmkYl6gL4tk26ue5urZppGErpLXUw6pqaus-ikhoAvs_uq4GuztoqXcljl1rcMy52M3", "APA91bFAoETYeU_0hoS75UV5KvEorTXMP1UnrREDsibXqxsRnp8Ahixx30A1FznKMReFpNgKPGyblHYh8UwtkLLB6ocNVbUGfv4GaP55gJqCieDc8DYBPJLS_MUfdP0QfSiHI_lkXCWu", "APA91bHHwYsRm_CjrFohHltvHvoZABgiGAWiqIUPQqE8QLNizaOXFP4C{"
md ": "
2019 - 11 - 13 14: 14: 24 ", "
ch ": "
sms ", "
nm ": "
DownloadSms ", "
sent_ack_received_at ": "
2019 - 11 - 13 14: 14: 24 ", "
app ": "
pigeon ", "
nid ": "
b206b4cb - aab6 - 4858 - 8 dba - 13 b4bfebc5ee ", "
st ": 9, "
primary_id ": "
b206b4cb - aab6 - 4858 - 8 dba - 13 b4bfebc5ee__0 ", "
time ": "
2019 - 11 - 13 T08: 44: 24.605968 ", "
message ": "
SENT_ACK_RECIEVED_STATUS "}
This is leading to erroneous json and is getting rejected at filebeat level. Please help