# HELP ! analyze unstructured data to be able to visualize them

**URL:** <https://discuss.elastic.co/t/help-analyze-unstructured-data-to-be-able-to-visualize-them/140373>\
**Category:** Logstash\
**Created:** [July 17, 2018, 5:39pm UTC](https://discuss.elastic.co/t/help-analyze-unstructured-data-to-be-able-to-visualize-them/140373 "2018-07-17T17:39:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Irving\_Hernandez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/irving_hernandez/32/43340_2.png) [@Irving\_Hernandez](https://discuss.elastic.co/u/Irving_Hernandez)\
**Post date:** [July 17, 2018, 5:39pm UTC](https://discuss.elastic.co/t/help-analyze-unstructured-data-to-be-able-to-visualize-them/140373/1 "2018-07-17T17:39:33Z")

</div>

I want to analyze the file java.log are unstructured data which I would like to be able to obtain specific fields such as: error, java, idCatCatalog and ect. Intenete realizing a template loading it to Logstash but you do not appear to me the fields that add. There is a way of realizing using curl, api java, api python, etc.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [July 18, 2018, 12:54am UTC](https://discuss.elastic.co/t/help-analyze-unstructured-data-to-be-able-to-visualize-them/140373/2 "2018-07-18T00:54:55Z")

</div>

If you plan to read these files with the [File Input Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html), you'll likely need to use the [multiline codec](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html) (default behaviour is to interpret log files as one event per line, but it's clear each of your events span multiple lines). Since your events begin with a square-bracketed funky-format date, you'll need to specify a pattern that could reliably match any date that is output.

```auto
input {
  file {
    path => "/var/log/*.log"
    codec => multiline {
      pattern => "\[\d{1,2}/\d{1,2}/\d{2} \d{1,2}:\d{2}:\d{2}:\d{3} [A-Z]{3,4}\]"
      negate => true
    }
  }
}

```

From there, you may want to paste a few lines into the [Grok Constructor](http://grokconstructor.appspot.com/do/construction), which is a helpful tool for building patterns for extracting information from semistructured data.

The bit between the `------` delimiters looks pretty well-structured key-value to me, so using the [KV Filter Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) could prove useful, perhaps something like:

```auto
filter {
  kv {
    field_split_pattern => "\n"
    value_split_pattern => ": "
  }
}

```

While you're developing your pipeline, the [STDOUT output plugin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html) combined with the [Rubydebug Codec](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-rubydebug.html) can be very helpful in enabling you to see the structure that you're successfully extracting from the flat events:

```auto
output {
  stdout {
    codec => rubydebug
  }
}

```

Once you're fairly confident that you're extracting the useful bits, you may want to start using the [Elasticsearch Output Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) to get the events into Elasticsearch, where you can use a visualisation tool like Kibana.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 15, 2018, 12:54am UTC](https://discuss.elastic.co/t/help-analyze-unstructured-data-to-be-able-to-visualize-them/140373/3 "2018-08-15T00:54:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.