I have a csv
file with 7543 entries that I'd like to get to cloud SIEM.
My config file is as follows
input {
file
{
path => "/home/<user>/myfile.csv"
start_position => beginning
#sincedb_path => ""
}
}
filter {
csv
{
autodetect_column_names => true
separator => ","
skip_empty_columns => false
}
}
output {
microsoft-logstash-output-azure-loganalytics {
workspace_id => "MyID"
workspace_key => "Mykey"
custom_log_table_name => "<Mytablename>"
key_names => ['name'. 'of', 'my', 'columns']
}
stdout{}
}
The configuration is working perfectly but I have a couple of problems.
-
Not all entries were sent to the SIEM. I run it several times, renaming the file and the first time 4000 something were sent, second time 6000 something was sent, but not all 7543.
-
Second problem is that I had some errors popping out on the standard output and it seems that the Logstash split several entries on two because of what I believe is a char that needs to be escaped or something like that. Check the below entry and pay closer attention on the bolt part
33746,v33746,dali163,IBM-MF-DALI,1350,SuSe,Suse Linux Enterprise Server 12.2,Linux,20,0,2,,zLINUX,RD-Test Server,SIMPANA (30d),ACTIVE,none,10.20.77.163,unknown,null,IF-MF-VM,hkaf,"dow, John",1011310020,"SLES 11_x000d_
Adabas test server",null,thsc,4/28/2014 16:53,EUR\bas,11/4/2021 5:30,UNIX Container Agent,,null,Germany,DAE,Darmstadt (Germany; DAE),V9,IBM z15,15,6,10,11,33,1,22,59963,5205,57,57,11/4/2021 1:36,0.04,null,null,11/4/2021 4:30,11/4/2021 5:30,0.04,,,,,0,,,null,null,
The console output
- The file that Logstash reads can sometimes change some of the values in the same entries, meaning it's not adding new rows but just updating some of the values in the old ones. I did a test, changing one value and Logstash didn't recognize this. The way sincedb works is that its just waiting for new rows but how about old entries with changed values, is there a way I tell Logstash to watch for this too?
Thank you all in advance