Adding a new field from each field of row from table imported from .csv file

Elastic Stack Logstash
1

Hello everyone,
I imported an .csv file with filebeat and in logstash config file I created this filter:

 filter {
     csv {
	   separator => ","
	   columns => ["ID","Name","Age","Photo","Nationality","Flag","Overall","Potential","Club","Club Logo","Value","Wage","Special","Preferred Foot","International Reputation","Weak Foot","Skill Moves","Work Rate","Body Type","Real Face","Position","Jersey Number","Joined","Loaned From","Contract Valid Until","Height","Weight","LS","ST","RS","LW","LF","CF","RF","RW","LAM","CAM","RAM","LM","LCM","CM","RCM","RM","LWB","LDM","CDM","RDM","RWB","LB","LCB","CB","RCB","RB","Crossing","Finishing","HeadingAccuracy","ShortPassing","Volleys","Dribbling","Curve","FKAccuracy","LongPassing","BallControl","Acceleration","SprintSpeed","Agility","Reactions","Balance","ShotPower","Jumping","Stamina","Strength","LongShots","Aggression","Interceptions","Positioning","Vision","Penalties","Composure","Marking","StandingTackle","SlidingTackle","GKDiving","GKHandling","GKKicking","GKPositioning","GKReflexes","Release Clause"]
	}
   }

My question is: how can I do create for each field of row a separate field
ready to be use directly in an elasticsearch query?

For example:

I have imported this table :

ID - Name - Age
0 - John - 24
1 - Mark- 21
2 - Anthony- 30

Filebeat returned me this JSON:

{
  "_index": "filebeat-6.5.4-2019.01.21",
  "_type": "doc",
  "_id": "ZwK2cGgB2IyobF7dIj_Q",
  "_version": 1,
  "_score": null,
  "_source": {
    "input": {
      "type": "log"
    },
    "@version": "1",
    "message": "0,John,24",
    "offset": 9120600,
    "prospector": {
      "type": "log"
    }
}

and in the "message" field I have all the fields of the first row...
I would like to have in the JSON a field named "age" with value 24, a field named "myid" with value 0and a field named "name" with value John.

{
  "_index": "filebeat-6.5.4-2019.01.21",
  "_type": "doc",
  "_id": "ZwK2cGgB2IyobF7dIj_Q",
  "_version": 1,
  "_score": null,
  "_source": {
    "input": {
      "type": "log"
    },
    "@version": "1",
    "message": "0,John,24",
    "age": 24,
    "myid": 0,
    "name": "John",
    "offset": 9120600,
    "prospector": {
      "type": "log"
    }
}

Thank's in advance for the replies!

2

Hi,

Any chance to get

1- input section of logstash config file
2- snippet of csv file you are trying to ingest.

In meanwhile try this

input {
file {
path => "/path/to/csv file on your system"
}
}
filter {
csv {
"columns" => ["ID","Name","Age","Photo","Nationality","Flag","Overall","Potential","Club","Club Logo","Value","Wage","Special","Preferred Foot","International Reputation","Weak Foot","Skill Moves","Work Rate","Body Type","Real Face","Position","Jersey Number","Joined","Loaned From","Contract Valid Until","Height","Weight","LS","ST","RS","LW","LF","CF","RF","RW","LAM","CAM","RAM","LM","LCM","CM","RCM","RM","LWB","LDM","CDM","RDM","RWB","LB","LCB","CB","RCB","RB","Crossing","Finishing","HeadingAccuracy","ShortPassing","Volleys","Dribbling","Curve","FKAccuracy","LongPassing","BallControl","Acceleration","SprintSpeed","Agility","Reactions","Balance","ShotPower","Jumping","Stamina","Strength","LongShots","Aggression","Interceptions","Positioning","Vision","Penalties","Composure","Marking","StandingTackle","SlidingTackle","GKDiving","GKHandling","GKKicking","GKPositioning","GKReflexes","Release Clause"]
"skip_header" => true
}

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.