# Bad charset encoding in field names - III

**URL:** <https://discuss.elastic.co/t/bad-charset-encoding-in-field-names-iii/136812>\
**Category:** Elasticsearch\
**Created:** [June 21, 2018, 9:09am UTC](https://discuss.elastic.co/t/bad-charset-encoding-in-field-names-iii/136812 "2018-06-21T09:09:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![maxirmx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxirmx/32/32589_2.png) [@maxirmx](https://discuss.elastic.co/u/maxirmx)\
**Post date:** [June 21, 2018, 9:09am UTC](https://discuss.elastic.co/t/bad-charset-encoding-in-field-names-iii/136812/1 "2018-06-21T09:09:23Z")

</div>

There is a bug in logstash that causes problems mentioned [here](https://discuss.elastic.co/t/bad-charset-encoding-in-field-names/107947) and [here](https://discuss.elastic.co/t/bad-charset-encoding-in-field-names-ii/129715)

Access to fields with no-ASCII names fails in any environment that uses Ruby event API.  
Plugins that use Java API directly work. For example, dissect plugin works OK with non-ASCII field name.

Config:

> input { stdin { } }
> 
> filter {  
> mutate {  
> add\_field =\> { "Русское название" =\> "Content of the field" }  
> }  
> date {  
> match =\> ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]  
> }  
> }
> 
> output {  
> stdout { codec =\> rubydebug }  
> }

Output:

> [2018-06-22T18:15:11,760][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"C:/maxirmx/logstash/modules/fb\_apache/configuration"}  
> [2018-06-22T18:15:11,791][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"C:/maxirmx/logstash/modules/netflow/configuration"}  
> [2018-06-22T18:15:12,094][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
> [2018-06-22T18:15:13,047][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.2.4"}  
> [2018-06-22T18:15:13,797][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
> [2018-06-22T18:15:19,885][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
> [2018-06-22T18:15:20,119][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x2e14da1c run\>"}  
> The stdin plugin is now waiting for input:  
> [2018-06-22T18:15:20,260][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]}  
> Some data  
> {  
> "message" =\> "Some data\r",  
> "@version" =\> "1",  
> "@timestamp" =\> 2018-06-22T15:15:28.309Z,  
> "host" =\> "NSC181278",  
> "? ?\u0083?\u0081?\u0081?????? ???°?·???°??????" =\> "Content of the field"  
> }

---

<div class="post-metadata">

**Author:** ![maxirmx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxirmx/32/32589_2.png) [@maxirmx](https://discuss.elastic.co/u/maxirmx)\
**Post date:** [June 22, 2018, 3:45pm UTC](https://discuss.elastic.co/t/bad-charset-encoding-in-field-names-iii/136812/2 "2018-06-22T15:45:14Z")

</div>

It looks like the problem is here

**JrubyEventExtLibrary.java**

> ```
> @JRubyMethod(name = "get", required = 1)
> public IRubyObject ruby_get_field(ThreadContext context, RubyString reference)
> {
> return Rubyfier.deep(
> context.runtime,
> this.event.getUnconvertedField(FieldReference.from(reference.getByteList()))
> );
> }
> 
> @JRubyMethod(name = "set", required = 2)
> public IRubyObject ruby_set_field(ThreadContext context, RubyString reference, IRubyObject value)
> {
> final FieldReference r = FieldReference.from(reference.getByteList());
> if (r.equals(FieldReference.TIMESTAMP_REFERENCE)) {
> if (!(value instanceof JrubyTimestampExtLibrary.RubyTimestamp)) {
> throw context.runtime.newTypeError("wrong argument type " + value.getMetaClass() + " (expected LogStash::Timestamp)");
> }
> this.event.setTimestamp(((JrubyTimestampExtLibrary.RubyTimestamp)value).getTimestamp());
> } else {
> this.event.setField(r, Valuefier.convert(value));
> }
> return value;
> }
> 
> ```

**reference.getByteList()** does not look correct above. It shall rather be **reference.getValue()** IMHO

---

<div class="post-metadata">

**Author:** ![maxirmx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxirmx/32/32589_2.png) [@maxirmx](https://discuss.elastic.co/u/maxirmx)\
**Post date:** [June 26, 2018, 9:16am UTC](https://discuss.elastic.co/t/bad-charset-encoding-in-field-names-iii/136812/3 "2018-06-26T09:16:05Z")

</div>

> <https://github.com/elastic/logstash/issues/9789>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 24, 2018, 9:16am UTC](https://discuss.elastic.co/t/bad-charset-encoding-in-field-names-iii/136812/4 "2018-07-24T09:16:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.