Hi,
I am using logstash to ingest a pipe delimited file into elasticsearch. I am using the autodetect_column_names in the csv filter. Noticed when I run logstash, I get the following warnings below:
[2021-06-25T16:48:30,488][WARN ][logstash.filters.csv ][main][e3f9b3727f260df15603cdaacf8cfe4d8d3f72cbc0bbfe41b7014266879f70ae] Error parsing csv {:field=>"message", :source=>"3755629048|\"324174503\"|\"\"|\"2021-06-16 23:52:10\"|\"2021-06-17 00:00:00\"|\"122.53.245.0\"|\"0\"|\"0\"|\"4\"|\"Mandaluyong\"|\"Philippines\"|\"PH\"|\"14.5794\"|\"121.0359\"|\"Converge ICT Solutions Inc.\"|\"Philippines\"|\"PH\"|\"Metro Manila\"|\"00\"|\"Manila\"|\"14.596\"|\"120.978\"|\"4.01372\"|\"Philippine Long Distance Telephone\"|\"1\"|\"TM\"|\"515\"|\"2\"|\"2\"|\"\"|\"OPPO\"|\"OP4C72L1\"|\"qcom\"|\"RKQ1.200903.002\"|\"OPPO\"|\"CPH2059\"|\"CPH2059\"|\"\"|\"\"|2|\"Manila, Metro Manila, Philippines\"|\"Manila\"|\"locality\"|\"Philippines\"|\"PH\"|\"Metro Manila\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"4.5.26\"|\"\"|\"02:00:00:00:00:00\"|\"\"|\"-53\"|\"\"|\"\"|2|\"\"|\"\"|\"Asia/Manila\"|\"28800\"|\"1\"|\"30\"|\"arm64-v8a\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"0\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"4\"|\"USER_INTERRUPT_DOWNLOAD\"|\"11.9\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"A92\"|\"Globe\"|\"\"|\"\"|\"0\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"Q_V1_P14,Q_V1_P14\"", :exception=>#<RuntimeError: Invalid FieldReference: `[]`>}
[2021-06-25T16:48:30,490][WARN ][logstash.filters.csv ][main][e3f9b3727f260df15603cdaacf8cfe4d8d3f72cbc0bbfe41b7014266879f70ae] Error parsing csv {:field=>"message", :source=>"id|android_device_id|android_fingerprint|test_date|received_date_a|client_ip_address|download_kbps|upload_kbps|latency|server_name|server_country|server_country_code|server_latitude|server_longitude|server_sponsor_name|client_country|client_country_code|client_region_name|client_region_code|client_city|client_latitude|client_longitude|miles_between|isp_name|is_isp|network_operator_name|mcc|mnc|pre_connection_type|post_connection_type|brand|device|hardware|build_id|manufacturer|model|product|cdma_cell_id|gsm_cell_id|location_type|gmaps_formatted_address|gmaps_name|gmaps_type|gmaps_country|gmaps_country_code|gmaps_region|gmaps_subregion|gmaps_subsubregion|gmaps_postal_code|phone_type_a|device_software_version_a|sim_network_operator_name_a|sim_network_operator_code_a|app_version_a|start_cell_id_a|wifi_bssid_a|wifi_secure_a|wifi_rssi_a|alt_sim_network_operator_name_a|alt_sim_network_operator_code_a|test_method_a|gsm_lac_a|alt_sim_operator_alpha_long_a|timezone_name_a|timezone_offset_seconds_a|data_connection_type_a|android_api_a|architecture_a|signal_cell_type_a|pci_a|tac_a|base_station_id_a|network_id_a|system_id_a|cid_a|lac_a|psc_a|asu_level_a|dbm_a|level_a|timing_advance_a|cdma_dbm_a|cdma_ecio_a|cdma_level_a|evdo_dbm_a|evdo_ecio_a|evdo_level_a|evdo_snr_a|signal_string_a|is_rooted_a|tr_ip_0_a|tr_latency_a|tr_ip_1_a|tr_latency_1_a|wifi_speed_mbps_a|wifi_frequency_mhz_a|error_code_a|error_message_a|simplified_error_code_a|simplified_error_message_a|jitter_a|ploss_sent_a|ploss_recv_a|rsrp_a|rsrq_a|rssnr_a|cqi_a|ookla_device_name_a|ookla_carrier_name_a|download_kb_a|upload_kb_a|is_airplane_mode_a|client_latitude_end|client_longitude_end|location_type_end|download_thread_count_a|download_test_duration_a|download_stop_reason_a|radio_a", :exception=>#<RuntimeError: Invalid FieldReference: `[]`>}
[2021-06-25T16:48:30,489][WARN ][logstash.filters.csv ][main][e3f9b3727f260df15603cdaacf8cfe4d8d3f72cbc0bbfe41b7014266879f70ae] Error parsing csv {:field=>"message", :source=>"3755629085|\"361013554\"|\"\"|\"2021-06-16 23:59:37\"|\"2021-06-17 00:00:02\"|\"136.158.82.0\"|\"0\"|\"0\"|\"0\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"Philippines\"|\"PH\"|\"Calabarzon\"|\"40\"|\"Santiago\"|\"14.349\"|\"120.908\"|\"\"|\"Converge\"|\"1\"|\"GLOBE\"|\"515\"|\"2\"|\"2\"|\"2\"|\"vivo\"|\"1804\"|\"qcom\"|\"QP1A.190711.020\"|\"vivo\"|\"vivo 1804\"|\"1804\"|\"\"|\"86431018\"|1|\"General Trias, Cavite, Philippines\"|\"General Trias\"|\"locality\"|\"Philippines\"|\"PH\"|\"Calabarzon\"|\"Cavite\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"4.5.36\"|\"86431018\"|\"5c:3a:3d:f5:a9:0d\"|\"1\"|\"-58\"|\"\"|\"\"|1|\"3002\"|\"\"|\"Asia/Manila\"|\"28800\"|\"1\"|\"29\"|\"arm64-v8a\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"0\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"\"|\"1\"|\"getaddrinfo failed: no data\"|\"6\"|\"NETWORK_ERROR_CONFIG\"|\"\"|\"\"|\"\"|\"-100\"|\"-10\"|\"\"|\"\"|\"V11\"|\"Globe\"|\"\"|\"\"|\"0\"|\"14.349\"|\"120.908\"|\"1\"|\"\"|\"\"|\"\"|\"660_GEN_PACK-1.272060.2.279491.1,660_GEN_PACK-1.272060.2.279491.1\"", :exception=>#<RuntimeError: Invalid FieldReference: `[]`>}
Noticed that the warning messages in logstash shows that the first line being read is a row value rather than the header line. My source file looks something like the following:
id|android_device_id|android_fingerprint|test_date|received_date_a|client_ip_address|download_kbps|upload_kbps|latency|server_name|server_country|server_country_code|server_latitude|server_longitude|server_sponsor_name|client_country|client_country_code|client_region_name|client_region_code|client_city|client_latitude|client_longitude|miles_between|isp_name|is_isp|network_operator_name|mcc|mnc|pre_connection_type|post_connection_type|brand|device|hardware|build_id|manufacturer|model|product|cdma_cell_id|gsm_cell_id|location_type|gmaps_formatted_address|gmaps_name|gmaps_type|gmaps_country|gmaps_country_code|gmaps_region|gmaps_subregion|gmaps_subsubregion|gmaps_postal_code|phone_type_a|device_software_version_a|sim_network_operator_name_a|sim_network_operator_code_a|app_version_a|start_cell_id_a|wifi_bssid_a|wifi_secure_a|wifi_rssi_a|alt_sim_network_operator_name_a|alt_sim_network_operator_code_a|test_method_a|gsm_lac_a|alt_sim_operator_alpha_long_a|timezone_name_a|timezone_offset_seconds_a|data_connection_type_a|android_api_a|architecture_a|signal_cell_type_a|pci_a|tac_a|base_station_id_a|network_id_a|system_id_a|cid_a|lac_a|psc_a|asu_level_a|dbm_a|level_a|timing_advance_a|cdma_dbm_a|cdma_ecio_a|cdma_level_a|evdo_dbm_a|evdo_ecio_a|evdo_level_a|evdo_snr_a|signal_string_a|is_rooted_a|tr_ip_0_a|tr_latency_a|tr_ip_1_a|tr_latency_1_a|wifi_speed_mbps_a|wifi_frequency_mhz_a|error_code_a|error_message_a|simplified_error_code_a|simplified_error_message_a|jitter_a|ploss_sent_a|ploss_recv_a|rsrp_a|rsrq_a|rssnr_a|cqi_a|ookla_device_name_a|ookla_carrier_name_a|download_kb_a|upload_kb_a|is_airplane_mode_a|client_latitude_end|client_longitude_end|location_type_end|download_thread_count_a|download_test_duration_a|download_stop_reason_a|radio_a
3755629038|"363006346"|""|"2021-06-16 23:56:48"|"2021-06-17 00:00:00"|"110.54.174.0"|"0"|"0"|"38"|"Iloilo"|"Philippines"|"PH"|"11.0000"|"122.6667"|"Globe Telecom"|"Philippines"|"PH"|"Western Visayas"|"06"|"Bacolod City"|"10.668"|"122.959"|"30.3475"|"Globe Telecom"|"1"|"GLOBE"|"515"|"2"|"2"|""|"Redmi"|"merlin"|"mt6769z"|"QP1A.190711.020"|"Xiaomi"|"M2003J15SC"|"merlin_global"|""|""|2|"Bacolod, Negros Occidental, Philippines"|"Bacolod"|"locality"|"Philippines"|"PH"|"Western Visayas"|"Negros Occidental"|""|"6100"|""|""|""|""|"4.5.36"|""|"02:00:00:00:00:00"|""|"-33"|""|""|2|""|""|"Asia/Manila"|"28800"|"1"|"29"|"arm64-v8a"|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|"0"|""|""|""|""|""|""|""|""|"4"|"USER_INTERRUPT_DOWNLOAD"|"7.7"|""|""|""|""|""|""|"Redmi Note 9"|"Globe"|""|""|"0"|""|""|""|""|""|""|"MOLY.LR12A.R3.MP.V98.P83,MOLY.LR12A.R3.MP.V98.P83"
3755629048|"324174503"|""|"2021-06-16 23:52:10"|"2021-06-17 00:00:00"|"122.53.245.0"|"0"|"0"|"4"|"Mandaluyong"|"Philippines"|"PH"|"14.5794"|"121.0359"|"Converge ICT Solutions Inc."|"Philippines"|"PH"|"Metro Manila"|"00"|"Manila"|"14.596"|"120.978"|"4.01372"|"Philippine Long Distance Telephone"|"1"|"TM"|"515"|"2"|"2"|""|"OPPO"|"OP4C72L1"|"qcom"|"RKQ1.200903.002"|"OPPO"|"CPH2059"|"CPH2059"|""|""|2|"Manila, Metro Manila, Philippines"|"Manila"|"locality"|"Philippines"|"PH"|"Metro Manila"|""|""|""|""|""|""|""|"4.5.26"|""|"02:00:00:00:00:00"|""|"-53"|""|""|2|""|""|"Asia/Manila"|"28800"|"1"|"30"|"arm64-v8a"|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|""|"0"|""|""|""|""|""|""|""|""|"4"|"USER_INTERRUPT_DOWNLOAD"|"11.9"|""|""|""|""|""|""|"A92"|"Globe"|""|""|"0"|""|""|""|""|""|""|"Q_V1_P14,Q_V1_P14"
Any ideas why this is happening and how to work around this? I am fairly new to logstash as well. Any help would be appreciated. Thank you.