Encoding::UndefinedConversionError when upgrading to 0.6.2

[jonsmock]

We just upgraded from simplecov 0.5.4 to 0.6.2. I started getting this error (shortened) when running our specs:

simplecov/source_file.rb:81:in `readlines': U+00C6 from UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)

We're using ruby 1.9.2-p290. Here's ruby -v:

ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.0.1]

I see the line was changed here, which looks correct to me (?): https://github.com/colszowka/simplecov/pull/103/files

I see that U+00C6 is http://graphemica.com/%C3%86 . When I run: grep -r Æ . I only get images and vendor/cache/*.gem files.

If I downgrade 0.5.4, simplecov runs fine. If it is something with simplecov, I would like to help fix it, but I don't know what the next step to debug it is. I also don't understand why it's going to ASCII-8BIT in the above error message.

[jonsmock]

Full trace, if it helps:

/Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/source_file.rb:81:in readlines': U+00C6 from UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/source_file.rb:81:inblock in initialize'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/source_file.rb:81:in open' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/source_file.rb:81:ininitialize'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/result.rb:24:in new' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/result.rb:24:inblock in initialize'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/result.rb:23:in each' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/result.rb:23:inmap'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/result.rb:23:in initialize' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov.rb:44:innew'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov.rb:44:in result' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/configuration.rb:133:inblock in at_exit'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/defaults.rb:51:in call' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.2/lib/simplecov/defaults.rb:51:inblock in <top (required)>'

[colszowka]

Not sure how this happened. Another encoding-related issue was fixed with #117 and will be part of 0.6.4, which I'll release today. Could you please check against that version (or, if you're very fast, try the current git master in your Gemfile) and report back?

[jonsmock]

Absolutely - thanks a lot. I should have time later today or early tomorrow to test it out. You rock!

[jonsmock]

Looks like the new release didn't fix our issue. Anything I can do to isolate this better? Is there a way to figure out which file it's choking on?

/Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/source_file.rb:82:in readlines': U+00C6 from UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/source_file.rb:82:inblock in initialize'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/source_file.rb:82:in open' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/source_file.rb:82:ininitialize'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/result.rb:24:in new' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/result.rb:24:inblock in initialize'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/result.rb:23:in each' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/result.rb:23:inmap'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/result.rb:23:in initialize' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov.rb:44:innew'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov.rb:44:in result' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/configuration.rb:133:inblock in at_exit'
from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:in call' from /Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/defaults.rb:51:inblock in <top (required)>'

[jonsmock]

Oops the first line of that backtrace should be (still in source.rb):

/Users/jonsmock/.rvm/gems/ruby-1.9.2-p290/gems/simplecov-0.6.4/lib/simplecov/source_file.rb:82:in `readlines': U+00C6 from UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError)

[colszowka]

@jonsmock Have you been able to figure out what the problem is?

[limhoff-r7]

I'm seeing the same problem. I tried adding simplecov to a project today and I had to back down all the way to 0.5.4 to get around the UndefinedConversionError. I can supply more detailed logs if you need it, but it's in the same readlines method as reported by @jonsmock

[obieq]

I just ran into this issue tonight while working on a new gem. After doing a big of digging around, here's what I found:

I'm working on a Fluentd plugin for Azure Table Storage. There's a gem (waz-storage) that I need in order to write to ATS. Due to my use of the waz-storage gem, simplecov tries at some point to both open and to read the /home/obie/.rvm/rubies/ruby-1.9.3-p327/lib/ruby/1.9.1/rexml/rexml.rb file, which contains the copyright symbol. Simplecov balks with the same error documented by this ticket, which is how I ended up here in the first place.

In my particular instance, the culprit is the fluentd gem. Within its engine.rb file, it sets both the Encoding.default_external and Encoding.default_internal properties to "ASCII-8BIT". It is the setting of the default_internal property that induces heartburn later on if any file is subsequently opened and ready by Simplecov that contains higher unicode characters, such as a copyright symbol.

Ruby provides a mechanism by which we can specify both the default external encoding and the internal encoding for an IO object such as File.

So, with respect to source_file.rb, line 82 could be written as either

File.open(filename, mode: 'r', external_encoding: Encoding::UTF_8, internal_encoding: Encoding::UTF_8) {|f| @src = f.readlines }

or

File.open(filename, "r:UTF-8:UTF-8") {|f| @src = f.readlines }

The latter is preferable, but there's a catch. The most current ruby version (1.9.3-p327) displays a superfluous warning message for each file opened and read by Simplecov. It's a bug that's already been reported to the ruby team and resolved in the ruby trunk (http://bugs.ruby-lang.org/projects/ruby-trunk/repository/revisions/35538).

NOTE: the current incarnation of line 82 ("r:UTF-8") sets/overrides only the Encoding.default_external property.

I hope this helps shed some light on why this issue affects some users and not others. There may be other use cases that throw the same error, but I can't imagine why the two options listed above wouldn't work for them as well.

[colszowka]

The fix for this by @justfalter is shipped as part of the 0.8 prereleases, 0.8.0 official is coming very soon. I assume this resolves this problem. If not, please report back once 0.8.0 is out.