# Importing multiple large csv and json files into a single index **URL:** \ **Category:** Elasticsearch\ **Created:** [March 1, 2023, 8:42am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738 "2023-03-01T08:42:23Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![mansi\_raval](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mansi_raval/32/117855_2.png) [@mansi\_raval](https://discuss.elastic.co/u/mansi_raval)\ **Post date:** [March 1, 2023, 8:42am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/1 "2023-03-01T08:42:23Z")
I have an folder containing data (20 GB) and this folder contains 26 subfolders that are sorted city-wise. Each of these subfolder contain many more subfolders comprising of csv and json files (The data that is stored in the files have different as well as somewhat similar fields) that I want to upload in bulk on elasticsearch. Apart from this, I also want to be able to specify the index name and mapping during the upload. For the same I have the following queries/requests : 1. Is this possible 2. Can anyone please help out with a detailed explanation as to how this can be done as I'm a beginner in this field. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 1, 2023, 9:05am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/2 "2023-03-01T09:05:42Z")
1. Yes 2. Yes! You can tell Filebeat to look for particular files in it's input and then have those in specific input sections, eg one for `*.csv` and one for`*.json`. On the input you can also tag an event. Then when you send them to Elasticsearch you can tell an output to filter only specific events, so the csv or json ones will go to the index you specifiy in set in each output section, you can also set the mapping there. Take a look at [filestream input | Filebeat Reference [8.6] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html) and [Configure the Elasticsearch output | Filebeat Reference [8.6] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html). ---
**Author:** ![mansi\_raval](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mansi_raval/32/117855_2.png) [@mansi\_raval](https://discuss.elastic.co/u/mansi_raval)\ **Post date:** [March 1, 2023, 9:58am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/3 "2023-03-01T09:58:37Z")
Can you please provide an example of exactly how this can be done along with a step-wise set of instructions for the same? I have referred to multiple documentations related to this but I find the instruction given to be a bit confusing. Sorry for this but I'm new to elasticsearch and took this up as my first project. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 1, 2023, 9:59am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/4 "2023-03-01T09:59:35Z")
How about you share what you have tried and what's not working and we can offer suggestions? That way we can help point out any mistakes and make it easier to learn. ---
**Author:** ![mansi\_raval](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mansi_raval/32/117855_2.png) [@mansi\_raval](https://discuss.elastic.co/u/mansi_raval)\ **Post date:** [March 1, 2023, 10:21am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/5 "2023-03-01T10:21:42Z")
Alright that makes sense. To start with, I'm facing issues with connecting filebeats with elasticsearch and kibana using cloud.auth (I'm unable to find it in the deployments overview-Security Section). ```auto E:\elastic\filebeats\Elastic\Beats\filebeat>filebeat -e -E cloud.id="DETAILS -E cloud.auth="DETAILS" Usage: filebeat [flags] filebeat [command] Available Commands: export Export current config or index template generate Generate Filebeat modules, filesets and fields.yml help Help about any command keystore Manage secrets keystore modules Manage configured modules run Run filebeat setup Setup index template, dashboards and ML jobs test Test config version Show current version info Flags: -E, --E setting=value Configuration overwrite -M, --M setting=value Module configuration overwrite -N, --N Disable actual publishing for testing -c, --c string Configuration file, relative to path.config (default "filebeat.yml") --cpuprofile string Write cpu profile to file -d, --d string Enable certain debug selectors -e, --e Log to stderr and disable syslog/file output --environment environmentVar set environment being ran in (default default) -h, --help help for filebeat --httpprof string Start pprof http server --memprofile string Write memory profile to this file --modules string List of enabled modules (comma separated) --once Run filebeat only once until all harvesters reach EOF --path.config string Configuration path (default "") --path.data string Data path (default "") --path.home string Home path (default "") --path.logs string Logs path (default "") --strict.perms Strict permission checking on config files (default true) -v, --v Log at INFO level Use "filebeat [command] --help" for more information about a command. ``` This is the response i'm getting when I reset the password in the security section of the deployment that I created on elastic cloud. I also made changes in the filebeats.yml file directly, hope that works. Additionally, I'm unable to figure out exactly how to tell Filebeat to look for particular files in it's input. ---
**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\ **Post date:** [March 1, 2023, 2:51pm UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/6 "2023-03-01T14:51:36Z")
Since this is your first elastic project I'd also recommend taking a look at uploading the file through kibana. By default it can handle csv, and json files up to 100MB in size. It will let you get the data in easily and get more familiar with mappings etc without too much overhead. Once you are more comfortable with that you it might help with configuring filebeat as well. You can upload files through Kibana by going to Integrations and then searching for "Upload". ---
**Author:** ![mansi\_raval](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mansi_raval/32/117855_2.png) [@mansi\_raval](https://discuss.elastic.co/u/mansi_raval)\ **Post date:** [March 4, 2023, 7:12am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/7 "2023-03-04T07:12:28Z")
The only question that I have regarding this is that the data that I have with me is really large in number. So via Kibana, I'd have to individually upload every file and create indices for each of those files. In this case, is there any way I can upload multiple files under a common index during the importing process? ---
**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\ **Post date:** [March 4, 2023, 12:19pm UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/8 "2023-03-04T12:19:50Z")
If itโ€™s more than a one off using filebeat (or logstash) would probably be best then. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 5, 2023, 10:40pm UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/9 "2023-03-05T22:40:24Z")
I have edited your post, you really want to avoid posting auth details publicly like that. `filebeat -E cloud.id="DETAILS -E cloud.auth="DETAILS"` is what you should need to use. ---
**Author:** ![mansi\_raval](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mansi_raval/32/117855_2.png) [@mansi\_raval](https://discuss.elastic.co/u/mansi_raval)\ **Post date:** [March 7, 2023, 6:25am UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/10 "2023-03-07T06:25:40Z")
`Access is denied.` This is the error that's getting displayed when I run the above command. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 7, 2023, 9:29pm UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/11 "2023-03-07T21:29:17Z")
Please make sure you share the full command you are running and the error, it helps us help you ๐Ÿ™‚ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 4, 2023, 9:29pm UTC](https://discuss.elastic.co/t/importing-multiple-large-csv-and-json-files-into-a-single-index/326738/12 "2023-04-04T21:29:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.