# Indexing csv with header in elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [January 8, 2020, 12:13pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229 "2020-01-08T12:13:13Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![johnkary](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@johnkary](https://discuss.elastic.co/u/johnkary)\ **Post date:** [January 8, 2020, 12:13pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/1 "2020-01-08T12:13:13Z")
Hi all! I' m using Elasticsearch 6.6 and Kibana 6.4 installed on my Google Cloud account. Here is my issue: I have a folder there where csv logs arrive from some IoT devices and with a Logstash pipeline that i created, i injest these csvs into an index in my Elasticsearch instance. The point is that these csv are gonna be changed and will come with headers inside and i want in someway to include this extra information too and correlate it with the body of the corresponding csv. So, in the next step, when i perform a search with some keywords from the header, i want to be able to get results related to the body. How can i do this? Can anyone help me? Thank you in advance! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 8, 2020, 12:50pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/2 "2020-01-08T12:50:19Z")
Welcome! Here is a (very old) tutorial which might help: [http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/](http://david.pilato.fr/blog/2015/04/28/exploring-capitaine-train-dataset/) Note that there's now a CSV import tool in Kibana as well. That might be useful. ---
**Author:** ![johnkary](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@johnkary](https://discuss.elastic.co/u/johnkary)\ **Post date:** [January 8, 2020, 1:19pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/3 "2020-01-08T13:19:47Z")
Hi dadoonet, and thanks for the quick response! This is not exactly what i want, my fault, i didn't explain it clearly. I want to include the header (not the default one, but the headers that the IoTs produce) of the csv in the injesting process. Below, i have attached a sample file to see the actual header that i mean. The header i mean consists of the first 7 with "#" in front of: "Mode", "Operation", "CameraNo", "CudaStreamCameraNo", "SnapshotCamId", "CenterCamId", "AgronomistDose". ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/8/e859d52fc26d0a824364c3d284a5e6587de87341.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 8, 2020, 1:31pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/4 "2020-01-08T13:31:54Z")
I moved your question to #logstash as the community there could find a better answer hopefully. Please don't post unformatted code, logs, or configuration as it's very hard to read. Instead, paste the text and format it with \ icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer. It would be great if you could update your post to solve this. ---
**Author:** ![oranieri](https://avatars.discourse-cdn.com/v4/letter/o/919ad9/32.png) [@oranieri](https://discuss.elastic.co/u/oranieri)\ **Post date:** [January 8, 2020, 1:49pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/5 "2020-01-08T13:49:27Z")
Hi @johnkary, So your file is not a standard CSV file, only the `#Body` field. For what I understood, with the header values, you want to include them for all the rows from body. I believe you won't be able to use CSV plugin the way it is, perhaps if you ingest as a file and perform grok matches to distinguish if it's a header with # or a standard CSV line. Wouldn't be easier for you to create a script (using python perhaps) to perform some manipulation/normalization on these CSV files before logstash to ingest? ---
**Author:** ![johnkary](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@johnkary](https://discuss.elastic.co/u/johnkary)\ **Post date:** [January 8, 2020, 1:55pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/6 "2020-01-08T13:55:18Z")
Ok, well, suppose i created the pyhon script and split the csv into 2, one for it's body and one for header. Respectivelly, the next step is to use 2 different indexes, one for header and another one for body? And if i do that, how can i correlate these 2 indexes (include header values for all the rows from body) in order to get full results? ---
**Author:** ![oranieri](https://avatars.discourse-cdn.com/v4/letter/o/919ad9/32.png) [@oranieri](https://discuss.elastic.co/u/oranieri)\ **Post date:** [January 8, 2020, 2:04pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/7 "2020-01-08T14:04:57Z")
Why not transforming into a single CSV file with the header information into all the records? E.g. ```auto SystemId, Mode, ... , frameID, ... 00:04:4b:df:35:96, NORMAL, ... , 3, ... 00:04:4b:df:35:96, NORMAL, ... , 11, ... 00:04:4b:df:35:96, NORMAL, ... , 19, ... ``` With that format, you could use [CSV filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html). ---
**Author:** ![johnkary](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@johnkary](https://discuss.elastic.co/u/johnkary)\ **Post date:** [January 8, 2020, 2:11pm UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/8 "2020-01-08T14:11:54Z")
I have already done this and works fine for now, but potentially i'm gonna have trouble with memory as the actual csv that will come will have thousands or millions of records and the header is gonna have some extra fields, such as 'local\_path' which is of 'longtext' type. So, if i put this header into all the records i will have serious redundancy of information. ---
**Author:** ![johnkary](https://avatars.discourse-cdn.com/v4/letter/j/a9adbd/32.png) [@johnkary](https://discuss.elastic.co/u/johnkary)\ **Post date:** [January 9, 2020, 10:41am UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/9 "2020-01-09T10:41:40Z")
I have also seen approaches like [parent-child](https://www.elastic.co/guide/en/elasticsearch/guide/current/parent-child.html) model or [nested-object](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/nested-mapping.html) mapping, or [application-side joins](https://www.elastic.co/guide/en/elasticsearch/guide/current/application-joins.html) and [data denormalization](https://www.elastic.co/guide/en/elasticsearch/guide/current/denormalization.html) but i am a bit confused of figuring out which solution best fits to my case. The crucial question here is: _does any of the above 4 approaches eliminates data redundancy which occurs on @oranieri 's suggestion_ above? Any ideas? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 6, 2020, 10:41am UTC](https://discuss.elastic.co/t/indexing-csv-with-header-in-elasticsearch/214229/10 "2020-02-06T10:41:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.