How to siparate message string to fields?

Elastic Stack Kibana
1

Hi!
I need a help. I imported csv file using filebeat to the kibana. after that created in management -> index pattern. Now I have field message where written all my strings from file. How can I get from message string fields to build visualization?

for example I have message string like this: 59, Macedonia,75.7,77.8,73.5

I need to sipareta this message string to the Fields like this:
rowid=59
Country=Mecadonia
a=75.7
b=77.8
c=73.5

total 180 strings need to siparate.

IMG_0779
IMG_0779.jpeg2851×2138 3.12 MB

2

That separation should have happened at ingest time. It doesn't look like filebeat can parse CSV files directly, but you can use the ingest node feature in Elasticsearch to do that. Effectively, you'd have a setup like this:

  • Filebeat reads the csv file and sends each row as a new document to the Elasticsearch ingest pipeline
  • The pipeline parses the line, turning each of the "column" values in the CSV into a field in an Elasticsearch document

This blog post explains how you might go about setting that up: https://www.elastic.co/blog/indexing-csv-elasticsearch-ingest-node

3

It's interesting blog post, but I'm studying only 3 days elasticsearch and don't understand how I can to use this blog post in my example. Maybe is it possible write to me how it can be look with my csv document?

4

You should be able to register a pipeline like this in Elasticsearch through the Kibana Console (under Dev Tools):

PUT _ingest/pipeline/parse_csv
{
 "description": "Parsing CSV",
 "processors": [
   {
     "grok": {
       "field": "message",
       "patterns": [
         "%{NUMBER:rowid:int},%{DATA:country},%{NUMBER:a:float},%{NUMBER:b:float},%{NUMBER:c:float}"
       ]
     }
   },
   {
     "remove": {
       "field": "message"
     }
   }
 ]
}

You can then reference this pipeline in your Filebeat config and have it parse the data before it is indexed into Elasticsearch.

2 Likes
5

I take result like this

{
"acknowledged" : true
}

6

Yes, It works! Thanks!

7

Only one last question!
To visualise this, I can choose all fields (rowid, a,b,c), but country not. Where could be a problem?

8

It probably has to do with the mapping on that field, and what you are trying to do with that data. If, for example, you are trying to do a term aggregation on country, but the field is being analyzed (searchable), that won't work, you need to aggregate on the raw value, but the "chunked up" parts of the value. I'm not super fluent in how the ingest node formats fields, but it's common to have the analyzed value stored at the raw location (ie. country) and the non-analyzed form stored as a keyword value (ie. country.keyword), which can be aggregated.

Some more details about what you are trying would help us guide you better. Screenshots are usually sufficient, if you can provide them without leaking sensitive information.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.