Hi everyone, Issue: I have two kinds CSV file, one kind (default) has two columns, and the other one (custom) has about seventeen columns. Default CSV: rosbagTimestamp data 1.56E+18 53829632 1.56E+18 53829632 1.56E+18 53829632 1.56E+18 53829632 1.56E+18 53829632 1.56E+18 538296…

Thank you so much @Badger ! one more question, if you had two kinds of CSV files that lets say, they have +15 columns, what is the best way to apply different filtering patterns to these two CSV kinds? Is counting the number of columns still the best option or can we do better?

You could use the add_field option on the csv filters to add a document_type field. Then make the filtering conditional on that, either if [docment_type] == "oneThing" { #Filters for oneThing } Or possibly using pipeline-to-pipeline communication with a distributor pattern. If the processing i…

[image] Badger: You could use the add_field option on the csv filters to add a document_type field. Then make the filtering conditional on that, either @Badger But both documents are type CSV, I don't understand why I need to check the document_type?

[image] EZprogramming: But both documents are type CSV, I don't understand why I need to check the document_type? If you have two kinds of CSV then you need a field which tells which kind an event is from.

@Badger , so basically the idea is to add a dummy/fake field containing a value that can help me identify the CSV files uniquely?

Yes. I just realized I might have misunderstood the second question. Do you have two types of CSVs that both have 15 columns, or do you have two types of CSVs that have different number of columns. If it is the former you are going to have to find a regular expression that allows you recognize one …

One CSV file has around 17 columns, the other has around 100 columns which some of the column values in the csv are of the form csv key, value pairs. Here is an example of it: rosbagTimestamp header seq stamp secs nsecs frame_id status - level name message hardware_id values - key value - key v…

OK, so count the number of columns using the method I linked to, then save that in a field on the event (possibly a field within the [@metadata] object), then do things conditionally based on that field.

Sure I will look into that, and let you know what happens. If I find the solution, I'll post it.

[image] Badger: Answered here . @Badger , question about your code. Could you please explain what is the purpose of having "[@metadata][fields]" inside event.set(...) in your code down below? The objective is to count the number of columns, so I don't understand why we need to take @metadata …

Discuss the Elastic Stack

CSV files have different columns

Elastic Stack Logstash

Badger June 10, 2019, 9:59pm 2

Answered here.

Topic		Replies	Views
Parse 1st Line of Multiple CSV files and set as Columns Logstash	5	3521	July 6, 2017
CSV with different number of columns Logstash	1	659	July 6, 2017
Process - Reshape a CSV file using logstash Logstash	30	1478	September 17, 2019
Multiple csv files to multiple auto field in kibana Logstash	1	330	November 13, 2019
CSV Filter Logstash	1	510	December 8, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

CSV files have different columns

Related topics