Elasticsearch index not getting created when add_field is used

[pranayv66]

Hi,

I'm facing a problem when creating an elasticsearch index. I'm new to this, so please let me know what I could do to fix this.
I'm pasting the two config files which I've used for reference. The one at the top is working, which I used just to see whether a simple load is working or not.

      file {
        path => "C:/Users/pranay/data.csv"
        start_position => "beginning"
      }
    }

    filter {
          csv {
            columns => [ "Desc","time","util","data" ]
         }
         mutate {convert => ["time", "float"] }
         mutate {convert => ["util", "float"] }
        }

    output {
      elasticsearch { 
      hosts => ["localhost:9200"] 
      index => "network"
      }
     }

This seems to be working fine. However when I use the below block, since I need to create a kibana visualization based on certain values from the fields, I'm using this

    input {
    	file {
    		path => "C:\Users\pranay\data.csv"
    		start_position => "beginning"
    	}
    }

    filter {
    	csv {
    		columns => [ "Desc","time","util","data" ]

    	}

    	mutate {convert => ["time", "float"] }
         mutate {convert => ["util", "float"] }

      if [data] == "data_one" {
    		mutate {
    		add_field => [ "Desc", "Snap" ]
    		}
    	}
    	else if [data] == "data_two" {
    		mutate {
    		add_field => [ "Desc", "Snaptwo" ]
    		}
    	}
    	else if [data] == "data_three" {
    		mutate {
    		add_field => [ "Description", "Snapthree" ]
    		}
    	}
    } 
    output {
    	elasticsearch {
    		hosts => ["localhost:9200"]
    		index => "test"
    	}
    }

I've also checked the logstash console, it doesn't give any tracebacks per se.

Any help on this will be appreciated.

Thanks,
Pranay.

[christiancj (Christian)]

looks to me the error is around the else if conditions and the way to set the add field, instead you should use nested else conditions, try this below,

if "data_one" in [data] {
    		mutate {
				add_field => { "Desc" => ["Snap" ] }
    		}
    	}  # end if "data_one"
    	else {
			if "data_two" in [data]  {
				mutate {
					add_field => { "Desc" => ["Snaptwo" ] }
				}
			} # end if "data_two"
			else {
				if "data_three" in [data]  {
					mutate {
						add_field => { "Description" => ["Snapthree" ] }
					}
				} # end if "data_three"		
			} # end 2nd else
		} # end 1st else

also, another similar post

Multiple IF conditions - Logstash

Hi, I am fixing bigger logstash config file where I have custom grok patterns but that is just tip of the iceberg regarding my problems. In Filebeat I have multiple log files and some of them (their log events) are visible in Kibana ok, and some not because their content ends up in the message field. Some log events are not even visible in Kibana when I try to filter it by tags in Kibana. I can fix stuff if I have one IF statement but when multiple IF statements are involved I am not sure wha…

[Badger]

@pranayv66

path => "C:\Users\pranay\data.csv"

Do not use backslash in the path option of a file filter, it is treated as an escape. Use forward slash.

[pranayv66]

@Badger No, that is not a problem here, as I stated above, in both of the codes, I've used a similar path for the input file and I get the index generated for the 1st one, whereas couldn't get that in the second one. So clearly there is something wrong in the later part of it.

Thanks,
Pranay.

[pranayv66]

@christiancj I've tried this and unfortunately this also does not create an index
Could you suggest anything else that I can give a shot?

[Badger]

@pranayv66

@Badger No, that is not a problem here, as I stated above, in both of the codes, I've used a similar path for the input file and I get the index generated for the 1st one, whereas couldn't get that in the second one.

No, you have not use a similar path. In the first one you used

path => "C:/Users/pranay/data.csv"

with forward slash, and it worked. In the second one you used

path => "C:\Users\pranay\data.csv"

with backslash and it did not work. Change the backslashes to forward slash.

[pranayv66]

@Badger I did replace the '' with '/' and it is still stuck there.

[christiancj (Christian)]

@pranayv66 - here is a conf that is working in my end, a sample index created with the if and columns conversion executed.

   input {
    	file {
			path => "C:/Users/pranay/data.csv"
    		start_position => "beginning"
			sincedb_path => "C:/Users/pranay/.since.sample.log"
    	}
    }

    filter {
	
    	csv { 
			columns => [ "Desc","time","util","data" ] 
			convert => {
				"time" => "float"
				"util" => "float"
			}
			
			}

    if "data_one" in [data] {
    		mutate {
				add_field => { "Description" => ["Snap" ] }
    		}
    	}  # end if "data_one"
    else {
			if "data_two" in [data]  {
				mutate {
					add_field => { "Description" => ["Snaptwo" ] }
				}
			} # end if "data_two"
			else {
				if "data_three" in [data]  {
					mutate {
						add_field => { "Description" => ["Snapthree" ] }
					}
				} # end if "data_three"		
			} # end 2nd else
		} # end 1st else
		
	mutate {
		remove_field => [ "column1","column2","column3","column4" ] 
	}
	
    }
	
    output {
   
	  stdout { codec => json }
    
         elasticsearch {
			hosts => ["localhost:9200"]
			index => "test"
		}
	}

sample csv - don't include column/header names in the file, just the data to ingest

abc,20.8,19.75,data_three
xya,34.5,19.5,data_two

as suggestion, don't forget to delete your sincedb path file when you're running testing (reload objects), this file track the last object/row processed.

[pranayv66]

@christiancj Thanks for the solution. This worked perfectly well !!