Hello, I need help about sending CSV from filebeat into logstash.
The error is "Invalid FieldReference: []"
I tried this solution but it doesn't work. (the thread is closed so I ma sorry for opening new topic)
This is my logstash.conf
input {
beats {
port => 5044
ssl_enabled => false
}
}
## Add your filters / logstash plugins configuration here
filter {
mutate {
gsub => ["message", "[\[\]]", " "]
}
}
filter {
csv {
separator => ","
columns => ["",
"deviceName","description","status","alerts","ip","ipv6Address","txRx","noise24G","noise5G","noise6G","airtime24G","airtime5G","airtime6G","latency24G","latency50G","latency6G","capacity","capacity24G","capacity50G","capacity6G","connectionFailure","model","apMac","channel24G","channel5G","channel6G","channel24gValue","channel50gValue","channel6gValue","meshRole","meshMode","zoneName","zoneAffinityProfileName","apGroupName","extIp","extPort","firmwareVersion","serial","retry24G","retry5G","retry6G","configurationStatus","lastSeen","numClients","numClients24G","numClients5G","numClients6G","tx","tx24G","tx50G","tx6G","rx","rx24G","rx50G","rx6G","txRx24G","txRx50G","txRx6G","location","wlanGroup24Id","wlanGroup50Id","wlanGroup6gId","wlanGroup24Name","wlanGroup50Name","wlanGroup6gName","enabledBonjourGateway","controlBladeName","lbsStatus","administrativeState","registrationState","provisionMethod","provisionStage","registrationTime","managementVlan","configOverride","indoorMapId","apGroupId","indoorMapXy","indoorMapName","indoorMapLocation","deviceGps","connectionStatus","zoneId","zoneFirmwareVersion","domainId","domainName","partnerDomainId","dpIp","controlBladeId","isCriticalAp","crashDump","cableModemSupported","cableModemResetSupported","swapInMac","swapOutMac","isOverallHealthStatusFlagged","isLatency24GFlagged","isLatency50GFlagged","isLatency6GFlagged","isCapacity24GFlagged","isCapacity50GFlagged","isCapacity6GFlagged","isConnectionFailure24GFlagged","isConnectionFailure50GFlagged","isConnectionFailure6GFlagged","isConnectionTotalCountFlagged","isConnectionFailureFlagged","isAirtimeUtilization24GFlagged","isAirtimeUtilization50GFlagged","isAirtimeUtilization6GFlagged","uptime","eirp24G","eirp50G","eirp6G","supportFips","ipsecSessionTime","ipsecTxPkts","ipsecRxPkts","ipsecTxBytes","ipsecRxBytes","ipsecTxDropPkts","ipsecRxDropPkts","ipsecTxIdleTime","ipsecRxIdleTime","ipType","ipv6Type","packetCaptureState","cellularWanInterface","cellularConnectionStatus","cellularSignalStrength","cellularIMSISIM0","cellularIMSISIM1","cellularICCIDSIM0","cellularICCIDSIM1","cellularIsSIM0Present","cellularIsSIM1Present","cellularTxBytesSIM0","cellularTxBytesSIM1","cellularRxBytesSIM0","cellularRxBytesSIM1","cellularActiveSim","cellularIPaddress","cellularSubnetMask","cellularDefaultGateway","cellularOperator","cellular3G4GChannel","cellularCountry","cellularRadioUptime","cellularGpsHistory","fipsEnabled","medianTxRadioMCSRate24G","medianTxRadioMCSRate50G","medianTxRadioMCSRate6G","medianRxRadioMCSRate24G","medianRxRadioMCSRate50G","medianRxRadioMCSRate6G","monitoringEnabled","txPowerOffset24G","txPowerOffset5G","txPowerOffset6G","rxDesense24G","rxDesense5G","rxDesense6G","poePortStatus","cumulativeTx24G","cumulativeTx5G","cumulativeTx6G","cumulativeRx24G","cumulativeRx5G","cumulativeRx6G","cumulativeTxRx24G","cumulativeTxRx5G","cumulativeTxRx6G","isDual5gMode","timestamp","name"]
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "logstash_internal"
password => "${LOGSTASH_INTERNAL_PASSWORD}"
}
}
And here is my filebeat.yml
- type: log
id: csvinput
index: csvdata
enabled: true
- /home/debian/statsapf1filkom.csv
tags: ["ruckusap"]
output.logstash:
# The Logstash hosts
hosts: ["192.168.1.10:5044"]
and this is the error message I got from logstash:
[2024-08-29T11:28:36,911][WARN ][logstash.filters.csv ][main][b9a883182c812d54fd6821163c5e7d58d78a342d6d1cd18a463659a4b227788e] Error parsing csv {:field=>"message", :source=>"9,A1-Tengah,,Online,2,10.34.11.56,2405:8740:6314:3400:82bc:37ff:fe2d:1da0,59045821,-92,-93,,38,1,,0,0,,24,17,7,,3.0,R350,80:BC:37:2D:1D:A0,1 (20MHz),157 (20MHz),,1,157,,DISABLED,Auto,14-FILKOMv6,DP2-10G-v6,Lantai 1,10.34.11.56,52225,6.1.0.0.9240,422273003040,9701657,254691,,Up-to-date,1724930815000,1,1,0,0,53620923,53433721,187202,0,5424898,5287054,137844,0,58720775,325046,0,FILKOM Semua Gedung lantai 1,a4cf71b1-9f07-11ee-879b-4a886de8381d,a4cf71b1-9f07-11ee-879b-4a886de8381d,,Event with Default,Event with Default,,False,wlc-ruckus-tik,Disable,Unlocked,Approved,Discovered,,1703074934185,1,False,,ab3c0f7f-854c-46a9-a744-4bea9ac9bc1b,,,,\"-7.953851,112.615004\",Connect,a996c892-45b6-4147-aa6a-1087dc5dd58b,6.1.0.0.9240,c7e6e74f-8c4f-43cc-ba81-f67afc9f81f8,14-FILKOM,c7e6e74f-8c4f-43cc-ba81-f67afc9f81f8, 10.1.1.33 :23233,e80ddc19-0902-4c20-8f58-9ea5a3c34ae8,False,1,False,False,,,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,460872,15,6,,,0,0,0,0,0,0,0,0,0,IPV4_IPV6,Autoconfig,Idle,,,,,,,,,,,,,,,,,,,,,,,,137600,137600,,0,0,,False,0,0,,0,0,,1000Mbps,35805201859,304837589,,1717498964,31251792,,37522700823,336089381,,False,2024-08-29 18:28:02.155144,0.35", :exception=>#<RuntimeError: Invalid FieldReference: `[]`>}
I use docker to deploy ELK Stack, and send csv from different host using filebeat.
Are there anything I missed?