# Invalid FieldReference: \`\[\]\` for CSV somehow persists even after using mutate gsub **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [August 29, 2024, 11:42am UTC](https://discuss.elastic.co/t/invalid-fieldreference-for-csv-somehow-persists-even-after-using-mutate-gsub/365763 "2024-08-29T11:42:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![canifer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/canifer/32/116909_2.png) [@canifer](https://discuss.elastic.co/u/canifer)\ **Post date:** [August 29, 2024, 11:42am UTC](https://discuss.elastic.co/t/invalid-fieldreference-for-csv-somehow-persists-even-after-using-mutate-gsub/365763/1 "2024-08-29T11:42:10Z")
Hello, I need help about sending CSV from filebeat into logstash. The error is "Invalid FieldReference: `[]`" I [tried this solution](https://discuss.elastic.co/t/invalid-fieldreference/273834) but it doesn't work. (the thread is closed so I ma sorry for opening new topic) This is my logstash.conf ```auto input { beats { port => 5044 ssl_enabled => false } } ## Add your filters / logstash plugins configuration here filter { mutate { gsub => ["message", "[\[\]]", " "] } } filter { csv { separator => "," columns => ["", "deviceName","description","status","alerts","ip","ipv6Address","txRx","noise24G","noise5G","noise6G","airtime24G","airtime5G","airtime6G","latency24G","latency50G","latency6G","capacity","capacity24G","capacity50G","capacity6G","connectionFailure","model","apMac","channel24G","channel5G","channel6G","channel24gValue","channel50gValue","channel6gValue","meshRole","meshMode","zoneName","zoneAffinityProfileName","apGroupName","extIp","extPort","firmwareVersion","serial","retry24G","retry5G","retry6G","configurationStatus","lastSeen","numClients","numClients24G","numClients5G","numClients6G","tx","tx24G","tx50G","tx6G","rx","rx24G","rx50G","rx6G","txRx24G","txRx50G","txRx6G","location","wlanGroup24Id","wlanGroup50Id","wlanGroup6gId","wlanGroup24Name","wlanGroup50Name","wlanGroup6gName","enabledBonjourGateway","controlBladeName","lbsStatus","administrativeState","registrationState","provisionMethod","provisionStage","registrationTime","managementVlan","configOverride","indoorMapId","apGroupId","indoorMapXy","indoorMapName","indoorMapLocation","deviceGps","connectionStatus","zoneId","zoneFirmwareVersion","domainId","domainName","partnerDomainId","dpIp","controlBladeId","isCriticalAp","crashDump","cableModemSupported","cableModemResetSupported","swapInMac","swapOutMac","isOverallHealthStatusFlagged","isLatency24GFlagged","isLatency50GFlagged","isLatency6GFlagged","isCapacity24GFlagged","isCapacity50GFlagged","isCapacity6GFlagged","isConnectionFailure24GFlagged","isConnectionFailure50GFlagged","isConnectionFailure6GFlagged","isConnectionTotalCountFlagged","isConnectionFailureFlagged","isAirtimeUtilization24GFlagged","isAirtimeUtilization50GFlagged","isAirtimeUtilization6GFlagged","uptime","eirp24G","eirp50G","eirp6G","supportFips","ipsecSessionTime","ipsecTxPkts","ipsecRxPkts","ipsecTxBytes","ipsecRxBytes","ipsecTxDropPkts","ipsecRxDropPkts","ipsecTxIdleTime","ipsecRxIdleTime","ipType","ipv6Type","packetCaptureState","cellularWanInterface","cellularConnectionStatus","cellularSignalStrength","cellularIMSISIM0","cellularIMSISIM1","cellularICCIDSIM0","cellularICCIDSIM1","cellularIsSIM0Present","cellularIsSIM1Present","cellularTxBytesSIM0","cellularTxBytesSIM1","cellularRxBytesSIM0","cellularRxBytesSIM1","cellularActiveSim","cellularIPaddress","cellularSubnetMask","cellularDefaultGateway","cellularOperator","cellular3G4GChannel","cellularCountry","cellularRadioUptime","cellularGpsHistory","fipsEnabled","medianTxRadioMCSRate24G","medianTxRadioMCSRate50G","medianTxRadioMCSRate6G","medianRxRadioMCSRate24G","medianRxRadioMCSRate50G","medianRxRadioMCSRate6G","monitoringEnabled","txPowerOffset24G","txPowerOffset5G","txPowerOffset6G","rxDesense24G","rxDesense5G","rxDesense6G","poePortStatus","cumulativeTx24G","cumulativeTx5G","cumulativeTx6G","cumulativeRx24G","cumulativeRx5G","cumulativeRx6G","cumulativeTxRx24G","cumulativeTxRx5G","cumulativeTxRx6G","isDual5gMode","timestamp","name"] } } output { elasticsearch { hosts => "elasticsearch:9200" user => "logstash_internal" password => "${LOGSTASH_INTERNAL_PASSWORD}" } } ``` And here is my filebeat.yml ```auto - type: log id: csvinput index: csvdata enabled: true - /home/debian/statsapf1filkom.csv tags: ["ruckusap"] output.logstash: # The Logstash hosts hosts: ["192.168.1.10:5044"] ``` and this is the error message I got from logstash: ```auto [2024-08-29T11:28:36,911][WARN][logstash.filters.csv][main][b9a883182c812d54fd6821163c5e7d58d78a342d6d1cd18a463659a4b227788e] Error parsing csv {:field=>"message", :source=>"9,A1-Tengah,,Online,2,10.34.11.56,2405:8740:6314:3400:82bc:37ff:fe2d:1da0,59045821,-92,-93,,38,1,,0,0,,24,17,7,,3.0,R350,80:BC:37:2D:1D:A0,1 (20MHz),157 (20MHz),,1,157,,DISABLED,Auto,14-FILKOMv6,DP2-10G-v6,Lantai 1,10.34.11.56,52225,6.1.0.0.9240,422273003040,9701657,254691,,Up-to-date,1724930815000,1,1,0,0,53620923,53433721,187202,0,5424898,5287054,137844,0,58720775,325046,0,FILKOM Semua Gedung lantai 1,a4cf71b1-9f07-11ee-879b-4a886de8381d,a4cf71b1-9f07-11ee-879b-4a886de8381d,,Event with Default,Event with Default,,False,wlc-ruckus-tik,Disable,Unlocked,Approved,Discovered,,1703074934185,1,False,,ab3c0f7f-854c-46a9-a744-4bea9ac9bc1b,,,,\"-7.953851,112.615004\",Connect,a996c892-45b6-4147-aa6a-1087dc5dd58b,6.1.0.0.9240,c7e6e74f-8c4f-43cc-ba81-f67afc9f81f8,14-FILKOM,c7e6e74f-8c4f-43cc-ba81-f67afc9f81f8, 10.1.1.33 :23233,e80ddc19-0902-4c20-8f58-9ea5a3c34ae8,False,1,False,False,,,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,460872,15,6,,,0,0,0,0,0,0,0,0,0,IPV4_IPV6,Autoconfig,Idle,,,,,,,,,,,,,,,,,,,,,,,,137600,137600,,0,0,,False,0,0,,0,0,,1000Mbps,35805201859,304837589,,1717498964,31251792,,37522700823,336089381,,False,2024-08-29 18:28:02.155144,0.35", :exception=>#} ``` I use docker to deploy ELK Stack, and send csv from different host using filebeat. Are there anything I missed? ---
**Author:** ![Musab\_Dogan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/musab_dogan/32/70691_2.png) [@Musab\_Dogan](https://discuss.elastic.co/u/Musab_Dogan)\ **Post date:** [August 29, 2024, 11:46am UTC](https://discuss.elastic.co/t/invalid-fieldreference-for-csv-somehow-persists-even-after-using-mutate-gsub/365763/2 "2024-08-29T11:46:52Z")
Looks like the `filter` part is duplicated. Can you remove one of them and try again? You can put `mutate` and `csv` filters in the same filter. ---
**Author:** ![canifer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/canifer/32/116909_2.png) [@canifer](https://discuss.elastic.co/u/canifer)\ **Post date:** [August 29, 2024, 11:54am UTC](https://discuss.elastic.co/t/invalid-fieldreference-for-csv-somehow-persists-even-after-using-mutate-gsub/365763/3 "2024-08-29T11:54:03Z")
Hi! @Musab_Dogan thanks for your reply, I just did that you suggested ```auto filter { mutate { gsub => ["message", "[\[\]]", " "] } csv { separator => "," columns => ["", "deviceName","description","status","alerts","ip","ipv6Address","txRx","noise24G","noise5G","noise6G","airtime24G","airtime5G","airtime6G","latency24G","latency50G","latency6G","capacity","capacity24G","capacity50G","capacity6G","connectionFailure","model","apMac","channel24G","channel5G","channel6G","channel24gValue","channel50gValue","channel6gValue","meshRole","meshMode","zoneName","zoneAffinityProfileName","apGroupName","extIp","extPort","firmwareVersion","serial","retry24G","retry5G","retry6G","configurationStatus","lastSeen","numClients","numClients24G","numClients5G","numClients6G","tx","tx24G","tx50G","tx6G","rx","rx24G","rx50G","rx6G","txRx24G","txRx50G","txRx6G","location","wlanGroup24Id","wlanGroup50Id","wlanGroup6gId","wlanGroup24Name","wlanGroup50Name","wlanGroup6gName","enabledBonjourGateway","controlBladeName","lbsStatus","administrativeState","registrationState","provisionMethod","provisionStage","registrationTime","managementVlan","configOverride","indoorMapId","apGroupId","indoorMapXy","indoorMapName","indoorMapLocation","deviceGps","connectionStatus","zoneId","zoneFirmwareVersion","domainId","domainName","partnerDomainId","dpIp","controlBladeId","isCriticalAp","crashDump","cableModemSupported","cableModemResetSupported","swapInMac","swapOutMac","isOverallHealthStatusFlagged","isLatency24GFlagged","isLatency50GFlagged","isLatency6GFlagged","isCapacity24GFlagged","isCapacity50GFlagged","isCapacity6GFlagged","isConnectionFailure24GFlagged","isConnectionFailure50GFlagged","isConnectionFailure6GFlagged","isConnectionTotalCountFlagged","isConnectionFailureFlagged","isAirtimeUtilization24GFlagged","isAirtimeUtilization50GFlagged","isAirtimeUtilization6GFlagged","uptime","eirp24G","eirp50G","eirp6G","supportFips","ipsecSessionTime","ipsecTxPkts","ipsecRxPkts","ipsecTxBytes","ipsecRxBytes","ipsecTxDropPkts","ipsecRxDropPkts","ipsecTxIdleTime","ipsecRxIdleTime","ipType","ipv6Type","packetCaptureState","cellularWanInterface","cellularConnectionStatus","cellularSignalStrength","cellularIMSISIM0","cellularIMSISIM1","cellularICCIDSIM0","cellularICCIDSIM1","cellularIsSIM0Present","cellularIsSIM1Present","cellularTxBytesSIM0","cellularTxBytesSIM1","cellularRxBytesSIM0","cellularRxBytesSIM1","cellularActiveSim","cellularIPaddress","cellularSubnetMask","cellularDefaultGateway","cellularOperator","cellular3G4GChannel","cellularCountry","cellularRadioUptime","cellularGpsHistory","fipsEnabled","medianTxRadioMCSRate24G","medianTxRadioMCSRate50G","medianTxRadioMCSRate6G","medianRxRadioMCSRate24G","medianRxRadioMCSRate50G","medianRxRadioMCSRate6G","monitoringEnabled","txPowerOffset24G","txPowerOffset5G","txPowerOffset6G","rxDesense24G","rxDesense5G","rxDesense6G","poePortStatus","cumulativeTx24G","cumulativeTx5G","cumulativeTx6G","cumulativeRx24G","cumulativeRx5G","cumulativeRx6G","cumulativeTxRx24G","cumulativeTxRx5G","cumulativeTxRx6G","isDual5gMode","timestamp","name"] } } ``` Then I reloaded the logstash service. But the problem still persists ```auto [2024-08-29T11:51:31,818][WARN][logstash.filters.csv][main][b25cae4c16ced09a5a378887182cec6bc1c6b7cfaac1e44765e2698c73aed73a] Error parsing csv {:field=>"message", :source=>"6,F1-Ext-Timur-R750-F8:D0,,Online,0,10.34.11.8,2405:8740:6314:3400:2ab3:71ff:fe2b:f8d0,4794639729,0,-96,,0,16,,0,36148,,66,0,66,,3.0,R750,28:B3:71:2B:F8:D0,Disabled (20MHz),149 (20MHz),,0,149,,DISABLED,Auto,14-FILKOMv6,DP2-10G-v6,Lantai 1,10.34.11.8,51216,6.1.0.0.9240,202002006291,0,40311221,,Up-to-date,1724932100000,11,0,11,0,2882909483,0,2882909483,0,1911730246,0,1911730246,0,0,4794639729,0,FILKOM Semua Gedung lantai 1,a4cf71b1-9f07-11ee-879b-4a886de8381d,a4cf71b1-9f07-11ee-879b-4a886de8381d,,Event with Default,Event with Default,,False,wlc-ruckus-tik,Disable,Unlocked,Approved,Discovered,,1604297583866,1,False,,ab3c0f7f-854c-46a9-a744-4bea9ac9bc1b,,,,\"-7.952504,112.613860\",Connect,a996c892-45b6-4147-aa6a-1087dc5dd58b,6.1.0.0.9240,c7e6e74f-8c4f-43cc-ba81-f67afc9f81f8,14-FILKOM,c7e6e74f-8c4f-43cc-ba81-f67afc9f81f8, 10.1.1.33 :23233,e80ddc19-0902-4c20-8f58-9ea5a3c34ae8,False,1,False,False,,,False,False,False,False,False,False,False,False,False,False,False,False,False,False,False,462050,,19,,,0,0,0,0,0,0,0,0,0,IPV4_IPV6,Autoconfig,Idle,,,,,,,,,,,,,,,,,,,,,,,,0,103200,,0,0,,False,0,0,,0,0,,1000Mbps,0,356086078568,,0,142311039054,,0,498397117622,,False,2024-08-29 18:51:02.154767,0.36", :exception=>#} ``` I deploy Elastic Stack from this repo [GitHub - deviantony/docker-elk: The Elastic stack (ELK) powered by Docker and Compose.](https://github.com/deviantony/docker-elk) with ELASTIC\_VERSION=8.15.0, maybe this information can help to trace the problem Any more suggestions? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 29, 2024, 12:58pm UTC](https://discuss.elastic.co/t/invalid-fieldreference-for-csv-somehow-persists-even-after-using-mutate-gsub/365763/4 "2024-08-29T12:58:34Z")
> [@canifer](#): > > ```auto > csv { > separator => "," > columns => ["", ... > > ``` You cannot have an empty field name. You will get the same exception using ``` input { generator { count => 1 lines => ['a,b'] } } output { stdout { codec => rubydebug { metadata => false } } } filter { csv { separator => "," columns => ["", "bar"] } } ``` ---
**Author:** ![canifer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/canifer/32/116909_2.png) [@canifer](https://discuss.elastic.co/u/canifer)\ **Post date:** [August 29, 2024, 11:02pm UTC](https://discuss.elastic.co/t/invalid-fieldreference-for-csv-somehow-persists-even-after-using-mutate-gsub/365763/5 "2024-08-29T23:02:28Z")
Awesome ! @Badger You've been a great help! Thank you for spotting my mistake.