# Can I upload csv IPs to elastic, and then use it in Grafana

**URL:** <https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362>\
**Category:** Elasticsearch\
**Created:** [June 18, 2021, 9:26am UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362 "2021-06-18T09:26:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![unico](https://avatars.discourse-cdn.com/v4/letter/u/f4b2a3/32.png) [@unico](https://discuss.elastic.co/u/unico)\
**Post date:** [June 18, 2021, 9:26am UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362/1 "2021-06-18T09:26:35Z")

</div>

Hello!

I have CSV file with some datasets like that:

```auto
PAIS; IP/PUERTO; CONEXIONES
AT,Austria; 52.97.172.13:143; 6
AT,Austria; 52.97.174.21:143; 1
AT,Austria; 52.97.182.189:143; 7004
BR,Brazil; 179.151.1.251:143; 1
BR,Brazil; 179.168.227.211:143; 1
BR,Brazil; 45.167.36.50:143; 1
CA,Canada; 104.245.157.43:143; 1
CA,Canada; 68.183.198.153:143; 1
CH,Switzerland; 194.12.140.214:143; 1399
CH,Switzerland; 194.12.174.180:143; 1284
CL,Chile; 181.42.46.195:143; 26
CL,Chile; 200.74.55.246:143; 15223
CN,China; 117.146.54.173:143; 1
CN,China; 123.172.112.191:143; 1
CN,China; 58.34.26.178:143; 1

```

and more 4000 lines...

Main aim, to add these IPs to elastic. Then connect DB Elasticsearch to Grafana to see all these IPs on the map. I tied to solve this issue almost one week, and can find correct solution.

Also I tried to create logstash-geoip.conf with GROK, but this config does not work ☹

Could you please help me with advice? Thank you

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 18, 2021, 3:04pm UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362/2 "2021-06-18T15:04:04Z")

</div>

Hello @unico Welcome to the community.

Apologies this seemed so hard. Your CSV need a little parsing.  
Plus with geo\_point data you need to define a mapping (data type) first.

In Kibana -\> Dev Tools

Create a mapping important because geo\_point needs to be defined ahead of time.

```auto
PUT my-discuss-connexions/
{
  "mappings": {
    "properties": {
      "connexion": {
        "type": "long"
      },
      "port": {
        "type": "long"
      },
      "ip": {
        "type": "ip"
      },
      "pais": {
        "type": "keyword"
      },
      "pais_code": {
        "type": "keyword"
      },
      "geoip": { <!--- This is the important part 
        "properties": {
          "location": {
            "type": "geo_point"
          }
        }
      }
    }
  }
}

```

Logstash

Your IPs would probably not GEOIP because the port was attached.

```auto
# discuss-csv-ip-data.csv
input {
  file {
    path => "/Users/sbrown/workspace/sample-data/discuss/csv-ip-data.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
    dissect {
      mapping => {
        "message" => "%{pais_code},%{pais}; %{ip}:%{port}; %{connexion}"
      }
    }
    geoip {
      source => "ip"
    }
}

output {
  stdout {codec => rubydebug}
  
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "my-discuss-connexions"
  }
}

```

`$ sudo /Users/sbrown/workspace/elastic-install/7.13.0/logstash-7.13.0/bin/logstash -r -f ./discuss-csv-ip-data.conf`

Sample output

```auto
{
      "@version" => "1",
       "message" => "CH,Switzerland; 194.12.140.214:143; 1399",
     "pais_code" => "CH",
          "path" => "/Users/sbrown/workspace/sample-data/discuss/csv-ip-data.csv",
         "geoip" => {
         "country_code3" => "CH",
        "continent_code" => "EU",
              "location" => {
            "lon" => 8.1551,
            "lat" => 47.1449
        },
                    "ip" => "194.12.140.214",
              "latitude" => 47.1449,
              "timezone" => "Europe/Zurich",
             "longitude" => 8.1551,
         "country_code2" => "CH",
          "country_name" => "Switzerland"
    },
    "@timestamp" => 2021-06-18T14:58:25.682Z,
     "connexion" => "1399",
          "port" => "143",
          "host" => "ceres",
          "pais" => "Switzerland",
            "ip" => "194.12.140.214"
}

```

Create Index Pattern If you choose to use the time fields you will need to be aware of that on the map...

 ![Screen Shot 2021-06-18 at 8.00.39 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/7/27150abc8b01a6f57a85b4299185b9beda765a6a.png)

Add to Map in Kibana (Sorry I am not a Grafana Expert)

 ![Screen Shot 2021-06-18 at 8.02.48 AM](https://us1.discourse-cdn.com/elastic/original/3X/a/c/aca3becea7da312fe3bac7c32b0f3e3e68472a94.jpeg)

 ![Screen Shot 2021-06-18 at 8.02.57 AM](https://us1.discourse-cdn.com/elastic/original/3X/f/7/f780e0e34ac33938894f4a5ce22a71ceb09fe586.jpeg)

---

<div class="post-metadata">

**Author:** ![unico](https://avatars.discourse-cdn.com/v4/letter/u/f4b2a3/32.png) [@unico](https://discuss.elastic.co/u/unico)\
**Post date:** [June 23, 2021, 7:15am UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362/3 "2021-06-23T07:15:40Z")

</div>

Hello!

Thank yo so much for your answer! This is really works!!!

One more questions, when I came to work at this morning, my Kibana does not work, but yesterday was all good and worked.  
Now I have error:

```auto
root@geoip4g:/usr/share/logstash/bin# ./logstash -r -f /etc/logstash/conf.d/logstash-geoip.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO] 2021-06-23 10:00:42.433 [main] runner - Starting Logstash {"logstash.version"=>"7.13.2", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [linux-x86_64]"}
[WARN] 2021-06-23 10:00:43.275 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[FATAL] 2021-06-23 10:00:43.300 [LogStash::Runner] runner - Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[FATAL] 2021-06-23 10:00:43.306 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

```

Also, I have an error in browser:

 ![KInaba](https://us1.discourse-cdn.com/elastic/original/3X/c/8/c8bdd0bf6a69a5df6d33547826d47ce03b1b2c41.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 23, 2021, 1:43pm UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362/4 "2021-06-23T13:43:24Z")

</div>

Hi @unico

Good to hear the Geo IP works.

1. You are in the `bin` directory you need to run Logstash from it's base directory.

`root@geoip4g:/usr/share/logstash#`

Also it looks like trying to run more than 1 instance.

1. The info above is about logstash not Kibana so I do not know why Kibana is failing.

2. if Kibana is failing please open a separate thread on that with a specific subject and all pertinent info. Kibana.yml, startup command startup logs

3. If Logstash keeps failing do the same open a new thread.

---

<div class="post-metadata">

**Author:** ![unico](https://avatars.discourse-cdn.com/v4/letter/u/f4b2a3/32.png) [@unico](https://discuss.elastic.co/u/unico)\
**Post date:** [June 24, 2021, 7:10am UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362/5 "2021-06-24T07:10:02Z")

</div>

Ok, I see.

I need a little bit more of your professional help. What should I do, if I have not only one file, but for example 10? I should to create \*.conf file in folder /etc/logstash/conf.d for each file with IPs?

C:\Users\soporte\Downloads\informe\_21-04-21\_26-04-21\ **ips**

 ![2021-06-24_9-01-18](https://us1.discourse-cdn.com/elastic/original/3X/4/4/44e74125200af14d76d8529aac014e31fdac2700.png)

C:\Users\soporte\Downloads\informe\_21-04-21\_26-04-21\ **total**

 ![2021-06-24_9-03-37](https://us1.discourse-cdn.com/elastic/original/3X/c/d/cd3a67b797e0fc24a45e87d7a93e739609df7939.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 24, 2021, 12:54pm UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362/6 "2021-06-24T12:54:58Z")

</div>

Take a look at the docs [here](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-path)

For all the files of the same type you can use path like.

`"path" => "/Users/sbrown/workspace/sample-data/**/*.csv`

You will probably need to make different confs for different file types / structure, that would make more sense.

Also be aware of [this setting](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path)

` sincedb_path => "/dev/null"`

That means do not keep track of which files I have loaded, once you are ready you would take that line out and then it will only load each file once, and new ones when the arrive.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 22, 2021, 12:54pm UTC](https://discuss.elastic.co/t/can-i-upload-csv-ips-to-elastic-and-then-use-it-in-grafana/276362/7 "2021-07-22T12:54:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.