# Loading astronomical data with Logstash

**URL:** https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959
**Category:** Logstash
**Created:** [April 29, 2018, 2:13pm UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959 "2018-04-29T14:13:51Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![Marcel-Jan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcel-jan/32/31171_2.png) [@Marcel-Jan](https://discuss.elastic.co/u/Marcel-Jan)
#### Post date: [April 29, 2018, 2:13pm UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/1 "2018-04-29T14:13:51Z")

</div>

Hi,  
I've only just done a course on Elasticsearch, Kibana and Logstash. I have a Ubuntu 64-bit in VirtualBox with Elasticsearch 6.2.3 and Logstash And I thought it would be a good idea to do a little project to see what I can do with the ELK stack now. So I decided to load the newly released second dataset of the Gaia telescope ([https://gea.esac.esa.int/archive/](https://gea.esac.esa.int/archive/)).

To start off, I just used one 40 MB csv file with 14,000+ stars ([http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia\_source/csv/GaiaSource\_1000172165251650944\_1000424567594791808.csv.gz](http://cdn.gea.esac.esa.int/Gaia/gdr2/gaia_source/csv/GaiaSource_1000172165251650944_1000424567594791808.csv.gz)) instead of the 1.6 billion available. Just to see how things go. And I created this conf file:

```
input {
    file {
        path => ["/home/marcel-jan/gaia/GaiaSource_1000172165251650944_1000424567594791808.csv"]
        start_position => "beginning"
        sincedb_path => "/null"
        type => "data"
    }
}

filter {
    csv {
        separator => ","
        columns => [
            "solution_id",
            "designation",
            "source_id",
            "random_index",
            <etc...>
        ]
	}
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "gaiadr2"
    }
}

```

And this works. So hurray. But unfortunately all the columns are of the string type and that's not so useful. So I added a convert clause in the filter:  
filter {  
csv {  
separator =\> ","  
columns =\> [  
"solution\_id",  
"designation",  
"source\_id",  
"random\_index",  
\<etc...\>  
]  
}  
mutate {  
convert =\> {  
"solution\_id" =\> "integer"  
"designation" =\> "integer"  
"source\_id" =\> "integer"  
"ref\_epoch" =\> "float"  
"ra" =\> "float"  
"ra\_error" =\> "float"  
\<etc..\>  
}  
}  
}

Now this works when I try only the integer ones. Some of the data are, according to the documentation ([https://gea.esac.esa.int/archive/documentation/GDR2/Gaia\_archive/chap\_datamodel/sec\_dm\_main\_tables/ssec\_dm\_gaia\_source.html](https://gea.esac.esa.int/archive/documentation/GDR2/Gaia_archive/chap_datamodel/sec_dm_main_tables/ssec_dm_gaia_source.html)) of the dataset of the double type. But I have already found out Logstash doesn't support the double data type. I've tried float instead. (Here are some examples of the double data: 103.4475289523685, 0.04109941963375859, 56.02202543042615. It seems it should fit in a float.)

And when I do that, Logstash simply hangs silently, not importing any data.

Where am I going wrong? Is it the data type? Is there a way around this?

---

<div class="post-metadata">

### Author: ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)
#### Post date: [April 29, 2018, 8:02pm UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/2 "2018-04-29T20:02:08Z")

</div>

I recommend defining mapping in Elasticsearch rather than Logstash.  
You can define much more types in Elasticsearch. See eg. [numeric datatypes](https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html).

---

<div class="post-metadata">

### Author: ![Marcel-Jan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcel-jan/32/31171_2.png) [@Marcel-Jan](https://discuss.elastic.co/u/Marcel-Jan)
#### Post date: [April 30, 2018, 9:31am UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/3 "2018-04-30T09:31:15Z")

</div>

Thanks @atira.

I've made a mapping. Tried running Logstash without the mutate and.. it hangs silently. It didn't read the CSV file. From stdout I can't see why. No errors. Is there any place where I can find out why Logstash isn't importing the data?

The mapping:  
curlh -XPUT 127.0.0.1:9200/gaiadr2 -d '  
{  
"mappings": {  
"gaiadr2": {  
"properties": {  
"solution\_id": { "type": "long" },  
"designation": { "type": "text" },  
"source\_id": { "type": "long" },  
"random\_index": { "type": "long" },  
"ref\_epoch": { "type": "double" },  
"ra": { "type": "double" },  
"ra\_error": { "type": "double" },  
"dec": { "type": "double" },  
"dec\_error": { "type": "double" },  
"parallax": { "type": "double" },  
"parallax\_error": { "type": "double" },  
"parallax\_over\_error": { "type": "float" },  
"pmra": { "type": "double" },  
"pmra\_error": { "type": "double" },  
"pmdec": { "type": "double" },  
"pmdec\_error": { "type": "double" },  
"ra\_dec\_corr": { "type": "float" },  
"ra\_parallax\_corr": { "type": "float" },  
"ra\_pmra\_corr": { "type": "float" },  
"ra\_pmdec\_corr": { "type": "float" },  
"dec\_parallax\_corr": { "type": "float" },  
"dec\_pmra\_corr": { "type": "float" },  
"dec\_pmdec\_corr": { "type": "float" },  
"parallax\_pmra\_corr": { "type": "float" },  
"parallax\_pmdec\_corr": { "type": "float" },  
"pmra\_pmdec\_corr": { "type": "float" },  
"astrometric\_n\_obs\_al": { "type": "integer" },  
"astrometric\_n\_obs\_ac": { "type": "integer" },  
"astrometric\_n\_good\_obs\_al": { "type": "integer" },  
"astrometric\_n\_bad\_obs\_al": { "type": "integer" },  
"astrometric\_gof\_al": { "type": "float" },  
"astrometric\_chi2\_al": { "type": "float" },  
"astrometric\_excess\_noise": { "type": "double" },  
"astrometric\_excess\_noise\_sig": { "type": "double" },  
"astrometric\_params\_solved": { "type": "byte" },  
"astrometric\_primary\_flag": { "type": "boolean" },  
"astrometric\_weight\_al": { "type": "float" },  
"astrometric\_pseudo\_colour": { "type": "double" },  
"astrometric\_pseudo\_colour\_error": { "type": "double" },  
"mean\_varpi\_factor\_al": { "type": "float" },  
"astrometric\_matched\_observations": { "type": "short" },  
"visibility\_periods\_used": { "type": "short" },  
"astrometric\_sigma5d\_max": { "type": "float" },  
"frame\_rotator\_object\_type": { "type": "integer" },  
"matched\_observations": { "type": "short" },  
"duplicated\_source": { "type": "boolean" },  
"phot\_g\_n\_obs": { "type": "integer" },  
"phot\_g\_mean\_flux": { "type": "double" },  
"phot\_g\_mean\_flux\_error": { "type": "double" },  
"phot\_g\_mean\_flux\_over\_error": { "type": "double" },  
"phot\_g\_mean\_mag": { "type": "float" },  
"phot\_bp\_n\_obs": { "type": "integer" },  
"phot\_bp\_mean\_flux": { "type": "double" },  
"phot\_bp\_mean\_flux\_error": { "type": "double" },  
"phot\_bp\_mean\_flux\_over\_error": { "type": "float" },  
"phot\_bp\_mean\_mag": { "type": "float" },  
"phot\_rp\_n\_obs": { "type": "integer" },  
"phot\_rp\_mean\_flux": { "type": "double" },  
"phot\_rp\_mean\_flux\_error": { "type": "double" },  
"phot\_rp\_mean\_flux\_over\_error": { "type": "float" },  
"phot\_rp\_mean\_mag": { "type": "float" },  
"phot\_bp\_rp\_excess\_factor": { "type": "float" },  
"phot\_proc\_mode": { "type": "byte" },  
"bp\_rp": { "type": "float" },  
"bp\_g": { "type": "float" },  
"g\_rp": { "type": "float" },  
"radial\_velocity": { "type": "double" },  
"radial\_velocity\_error": { "type": "double" },  
"rv\_nb\_transits": { "type": "integer" },  
"rv\_template\_teff": { "type": "float" },  
"rv\_template\_logg": { "type": "float" },  
"rv\_template\_fe\_h": { "type": "float" },  
"phot\_variable\_flag": { "type": "text" },  
"l": { "type": "double" },  
"b": { "type": "double" },  
"ecl\_lon": { "type": "double" },  
"ecl\_lat": { "type": "double" },  
"priam\_flags": { "type": "long" },  
"teff\_val": { "type": "float" },  
"teff\_percentile\_lower": { "type": "float" },  
"teff\_percentile\_upper": { "type": "float" },  
"a\_g\_val": { "type": "float" },  
"a\_g\_percentile\_lower": { "type": "float" },  
"a\_g\_percentile\_upper": { "type": "float" },  
"e\_bp\_min\_rp\_val": { "type": "float" },  
"e\_bp\_min\_rp\_percentile\_lower": { "type": "float" },  
"e\_bp\_min\_rp\_percentile\_upper": { "type": "float" },  
"flame\_flags": { "type": "long" },  
"radius\_val": { "type": "float" },  
"radius\_percentile\_lower": { "type": "float" },  
"radius\_percentile\_upper": { "type": "float" },  
"lum\_val": { "type": "float" },  
"lum\_percentile\_lower": { "type": "float" },  
"lum\_percentile\_upper": { "type": "float" }  
}  
}  
}  
}'

---

<div class="post-metadata">

### Author: ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)
#### Post date: [April 30, 2018, 10:50am UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/4 "2018-04-30T10:50:21Z")

</div>

ES would throw away documents where there are mismatching mapping types. But it would log it.  
You could go around it with the [ignore\_malformed](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html) option.

It's weird that there are no log entries about that. Are you sure you're not getting anything in either logs? Did you try lowering the log level?

Anyway, I find this one disturbing:

`sincedb_path => "/null"`

Is that similar to /dev/null? Logstash needs sincedb to keep track which files/lines it already read.

---

<div class="post-metadata">

### Author: ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [April 30, 2018, 11:50am UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/5 "2018-04-30T11:50:00Z")

</div>

I agree with @atira.

When you specified `sincedb_path => "/null"` Logstash actually created a file called null in the hdd device root directory and its using it to track the position the file was read up to.

When you use `"/dev/null"` the position tracking option is disabled.

---

<div class="post-metadata">

### Author: ![Marcel-Jan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcel-jan/32/31171_2.png) [@Marcel-Jan](https://discuss.elastic.co/u/Marcel-Jan)
#### Post date: [April 30, 2018, 12:41pm UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/6 "2018-04-30T12:41:54Z")

</div>

Thanks for the quick responses, @atira and @guybroertje

I have changed the mapping, adding ignore\_malformed: false.

I have gotten errors now. This looks like something I can troubleshoot.  
`[WARN] 2018-04-30 14:29:37.374 [Ruby-0-Thread-9@[main]>worker0: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"gaiadr2", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x55b1fe04>], :response=>{"index"=>{"_index"=>"gaiadr2", "_type"=>"doc", "_id"=>"7hmHFmMBa2YHi_IUn7af", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [gaiadr2] as the final mapping would have more than 1 type: [doc, gaiadr2]"}}}}`

Is that problem something that has to do with this part?

```
curlh -XPUT 127.0.0.1:9200/gaiadr2 -d '
{
  "settings": {
    "index.mapping.ignore_malformed": false 
  },
  "mappings": {
    "gaiadr2": { <-----
      "properties": {
        "solution_id": { "type": "long" },
```

---

<div class="post-metadata">

### Author: ![Marcel-Jan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcel-jan/32/31171_2.png) [@Marcel-Jan](https://discuss.elastic.co/u/Marcel-Jan)
#### Post date: [April 30, 2018, 3:58pm UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/7 "2018-04-30T15:58:20Z")

</div>

Just changed that type to "doc" and all the data has loaded.

And I can now see the data in Kibana with the right data types. I can make graphs of the data. This is pretty cool.

Thanks for the help! In a while I'm planning to do a short Youtube video on my first experiences with Elasticsearch

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 21, 2018, 10:02pm UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/8 "2018-05-21T22:02:47Z")

</div>

I used this to grab all 5200 or so files to process, pity they didn't offer a torrent of them!

```auto
for i in `curl http://cdn.gea.esac.esa.int/Gaia/gdr1/gaia_source/csv/ |grep csv.gz|cut -d\" -f2`; do wget http://cdn.gea.esac.esa.int/Gaia/gdr1/gaia_source/csv/$i; done

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 18, 2018, 10:03pm UTC](https://discuss.elastic.co/t/loading-astronomical-data-with-logstash/129959/9 "2018-06-18T22:03:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.