[BUG] UTF-8 YAML files with accents in version 1.9.0 raise incompatible character encodings: UTF-8 and ASCII-8BIT

[kivanio]

For long time I have a code like this:

STATES = I18n.t('states').with_indifferent_access.freeze

The yaml file is in UTF-8 and it has accents in some words:

pt-BR:
  states:
    Acre: AC
    Amapá: AP
    Ceará: CE
    Piauí: PI
    Paraná: PR

with new release 1.9.0
It starts failing in our CI in a lot of places:

ActionView::Template::Error incompatible character encodings: UTF-8 and ASCII-8BIT
Failure/Error: = f.select(:state, City::STATES,

ActionView::Template::Error:
  incompatible character encodings: UTF-8 and ASCII-8BIT
./app/views/customers/_form.html.slim:86:in `block in 

Downgrade to 1.8.11 make everything works again.

With 1.9.0 use rails console it loads like:

rails c
Running via Spring preloader in process 4964
Loading development environment (Rails 6.1.4.4)
[1] pry(main)> I18n.t('states').with_indifferent_access.freeze
=> {"Acre"=>"AC",
 "Alagoas"=>"AL",
 "Amazonas"=>"AM",
 "Amap\xC3\xA1"=>"AP",
 "Bahia"=>"BA",
 "Cear\xC3\xA1"=>"CE",
 "Distrito Federal"=>"DF",
 "Esp\xC3\xADrito Santo"=>"ES",
 "Goi\xC3\xA1s"=>"GO",
 "Maranh\xC3\xA3o"=>"MA",
 "Minas Gerais"=>"MG",
 "Mato Grosso do Sul"=>"MS",
 "Mato Grosso"=>"MT",
 "Par\xC3\xA1"=>"PA",
 "Para\xC3\xADba"=>"PB",
 "Pernambuco"=>"PE",
 "Piau\xC3\xAD"=>"PI",
 "Paran\xC3\xA1"=>"PR",
 "Rio de Janeiro"=>"RJ",
 "Rio Grande do Norte"=>"RN",
 "Rond\xC3\xB4nia"=>"RO",
 "Roraima"=>"RR",
 "Rio Grande do Sul"=>"RS",
 "Santa Catarina"=>"SC",
 "Sergipe"=>"SE",
 "S\xC3\xA3o Paulo"=>"SP",
 "Tocantins"=>"TO"}

With 1.8.11 use rails console it loads like:

rails c
Running via Spring preloader in process 3411
Loading development environment (Rails 6.1.4.4)
[1] pry(main)> I18n.t('states').with_indifferent_access.freeze
=> {"Acre"=>"AC",
 "Alagoas"=>"AL",
 "Amazonas"=>"AM",
 "Amap\xC3\xA1"=>"AP",
 "Bahia"=>"BA",
 "Cear\xC3\xA1"=>"CE",
 "Distrito Federal"=>"DF",
 "Esp\xC3\xADrito Santo"=>"ES",
 "Goi\xC3\xA1s"=>"GO",
 "Maranh\xC3\xA3o"=>"MA",
 "Minas Gerais"=>"MG",
 "Mato Grosso do Sul"=>"MS",
 "Mato Grosso"=>"MT",
 "Par\xC3\xA1"=>"PA",
 "Para\xC3\xADba"=>"PB",
 "Pernambuco"=>"PE",
 "Piau\xC3\xAD"=>"PI",
 "Paran\xC3\xA1"=>"PR",
 "Rio de Janeiro"=>"RJ",
 "Rio Grande do Norte"=>"RN",
 "Rond\xC3\xB4nia"=>"RO",
 "Roraima"=>"RR",
 "Rio Grande do Sul"=>"RS",
 "Santa Catarina"=>"SC",
 "Sergipe"=>"SE",
 "S\xC3\xA3o Paulo"=>"SP",
 "Tocantins"=>"TO"}

The output seems exactly the same.
Is that a bug in new version?

Probably something between new version and load in rails.
This is just a sample
I have others files with accents and all of them are raising same exception.

Versions of i18n, rails, and anything else you think is necessary

ruby: 3.0.3
i18n: 1.9.0
rails: 6.1.4.3
rspec-rails: 5.1.0
rspec: 3.10.0

and # frozen_string_literal: true in ruby files.

[radar]

Hello, thank you for the detailed reproduction steps.

I was unable to reproduce this issue using:

• Ruby 3.0.0
• i18n 1.9.1
• Rails 6.1.4.3

(The RSpec versions are highly likely to be irrelevant to this issue)

I will try with Ruby 3.0.3 now.

[radar]

I am unable to reproduce this issue with 3.0.3. Could you please put a Rails app that does reproduce this issue on GitHub so that I can clone it down and investigate?

[joergschiller]

We have a similar issue. I've created this repo to demonstrate the difference.

Using i18n 1.8.11 the keys are symbolized with UTF-8 encoding. With 1.9.1 it's ASCII.

gem 'i18n', "< 1.9.0"
I18n.t("foobar").keys => [:ö]
I18n.t("foobar").keys.first.encoding => #<Encoding:UTF-8>

I18n.t("foobar").values.first.encoding => #<Encoding:UTF-8>

gem 'i18n', "> 1.9.0"
I18n.t("foobar").keys => [:"\xC3\xB6"]
I18n.t("foobar").keys.first.encoding => #<Encoding:ASCII-8BIT>

I18n.t("foobar").values.first.encoding => #<Encoding:UTF-8>

https://github.com/joergschiller/i18n_issue_606/blob/293513b938310239e06bf4062be975e0ec772fc0/Gemfile#L6-L12

(With Ruby 3.0.3)

[kivanio]

Thank you @joergschiller
I am not alone 🙏🏻
I was going to make a repo and you saved me.
I was believing that It was something desired in new version but now I think we have a bug.

[radar]

Thank you @joergschiller. I can reproduce this issue now with your repository. I'll find the commit that broke it.

[radar]

Commit that breaks this behaviour is 0fda789, as discovered through a git bisect:

 ~/code/gems/i18n   v1.9.0~7^2 (bisect)  bad
0fda789ea745cd462658a8948ee085201aba5c6f is the first bad commit
commit 0fda789ea745cd462658a8948ee085201aba5c6f
Author: Paarth Madan <paarth.madan@shopify.com>
Date:   Wed Nov 3 12:33:12 2021 -0400

    Symbolize and freeze keys when loading from YAML

 lib/i18n/backend/base.rb    | 2 +-
 test/backend/simple_test.rb | 7 ++++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

[radar]

@paarthmadan Do you have any time today to investigate this one?

[radar]

Looked into where the unsafe_load_file method was coming from bootsnap. cc @casperisfine.

/Users/ryan.bigg/.asdf/installs/ruby/3.0.3/lib/ruby/gems/3.0.0/gems/bootsnap-1.10.2/lib/bootsnap/compile_cache/yaml.rb:203:in `unsafe_load_file': wrong number of arguments (given 0, expected 1+) (ArgumentError)

If I remove bootsnap from this test application, the issue goes away:

irb(main):001:0> I18n.t(:foobar).keys[0]
=> :ö

@casperisfine: Would you like me to create an issue on the Bootsnap repo page for this one?

[casperisfine]

👀

[casperisfine]

Ah damn it, I know what the problem is. It's because Bootsnap uses msgpack to accelarate YAML parsing, and MessagePack use an API that doesn't preserve symbols encoding properly. See an issue I opened a while ago msgpack/msgpack-ruby#211

Let me go over my old research see how we could sidestep this in bootsnap. I'll update here ASAP.

[casperisfine]

Ok, so the bug is actually in msgpack, I opened a PR here: msgpack/msgpack-ruby#246

You can apply the patch with:

gem 'msgpack', github: 'Shopify/msgpack-ruby', branch: 'symbolize-keys-fix-encoding'

>> I18n.t(:foobar)
=> {:ö=>"ü"}

Alternatively, if you'd rather not run a gem branch, you can disable Bootsnap YAML caching. Sorry for the bug :/

[radar]

Thank you very much for the deep investigation here :)