ensure tcl works within virtualenv

[carljm]

Originally reported by Shawn Wheatley at https://bugs.launchpad.net/virtualenv/+bug/449537

Environment:
Tested on Windows XP SP3 and Windows 7
Python 2.6.3
Virtualenv 1.3.4

Within a fresh, just activated VirtualEnv, I get the following error trying to run a simple Tkinter window:

H:\My Documents\pythondev>virtualenv H:\testvirtualenv
New python executable in H:\testvirtualenv\Scripts\python.exe
Installing setuptools...................done.

H:\My Documents\pythondev>H:\testvirtualenv\Scripts\activate.bat
(testvirtualenv) H:\My Documents\pythondev>python
Python 2.6.3 (r263rc1:75186, Oct 2 2009, 20:40:30) [MSC v.1500 32 bit (Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.

import Tkinter
Tkinter._test()
Traceback (most recent call last):
File "", line 1, in
File "C:\Python26\Lib\lib-tk\Tkinter.py", line 3749, in _test
root = Tk()
File "C:\Python26\Lib\lib-tk\Tkinter.py", line 1643, in init
self.tk = _tkinter.create(screenName, baseName, className, interactive, want
objects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories:
C:/Python26/lib/tcl8.5 H:/testvirtualenv/lib/tcl8.5 H:/lib/tcl8.5 H:/testvir
tualenv/library H:/library H:/tcl8.5.2/library H:/tcl8.5.2/library

This probably means that Tcl wasn't installed properly.

Shawn Wheatley wrote on 2009-10-13:

Virtualenv does not copy the Tcl library file(s) (or any core C extensions, that I can see) to the new virtualenv, but instead adds a reference to the PYTHONPATH. If I copy the "tcl" folder from C:\Python26\ over to the root of the new Virtualenv, Tkinter.Tk() shows a new window without throwing an exception.
Jannis Leidel wrote on 2009-10-13: #2

Just suspecting here but could you create a virtualenv on the same drive as the system Python (C:)? I have the feeling it could be related to https://bugs.launchpad.net/virtualenv/+bug/352844.

Shawn Wheatley wrote on 2009-10-14:

Same problem when creating an environment on C:

C:\Temp>virtualenv TestTkinterBug
New python executable in TestTkinterBug\Scripts\python.exe
Installing setuptools...................done.

C:\Temp>cd TestTkinterBug

C:\Temp\TestTkinterBug>Scripts\activate.bat
(TestTkinterBug) C:\Temp\TestTkinterBug>python
Python 2.6.3 (r263rc1:75186, Oct 2 2009, 20:40:30) [MSC v.1500 32 bit (Intel)]
on win32
Type "help", "copyright", "credits" or "license" for more information.

import Tkinter
Tkinter._test()
Traceback (most recent call last):
File "", line 1, in
File "C:\Python26\Lib\lib-tk\Tkinter.py", line 3749, in _test
root = Tk()
File "C:\Python26\Lib\lib-tk\Tkinter.py", line 1643, in init
self.tk = _tkinter.create(screenName, baseName, className, interactive, want
objects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories:
C:/Python26/lib/tcl8.5 C:/Temp/TestTkinterBug/lib/tcl8.5 C:/Temp/lib/tcl8.5
C:/Temp/TestTkinterBug/library C:/Temp/library C:/Temp/tcl8.5.2/library C:/tcl8.
5.2/library

This probably means that Tcl wasn't installed properly.

Shawn Wheatley wrote on 2009-10-15:

I've figured out the problem, but wondering if someone could give me a suggestion on a "proper" solution. I dug into the Python source and found that importing Tkinter on win32 causes an import of a module called FixTk:

http://svn.python.org/view/python/tags/r263/Lib/lib-tk/FixTk.py?revision=75184&view=markup

This file checks to see if the folder "tcl" exists in sys.prefix ("C:\python26\tcl"), or if the folder tcltk\lib exists in the parent of sys.prefix ("C:\tcltk\lib"). Once it finds an appropriate folder, it drills in to find the actual Tcl library folder, and sets appropriate environment variables (TCL_LIBRARY, TK_LIBRARY, TIX_LIBRARY). Since virtualenv doesn't copy the first folder to the new location, and the second folder doesn't exist (at least on my system), the rest of the module exits without completion--without setting any environment variables. You can see an example of this in a virtualenv on win32 by doing the following:

import os
os.environ.keys().index('TCL_LIBRARY')
Traceback (most recent call last):
File "", line 1, in
ValueError: list.index(x): x not in list
import Tkinter
os.environ.keys().index('TCL_LIBRARY')
Traceback (most recent call last):
File "", line 1, in
ValueError: list.index(x): x not in list

Compare this with stock Python on win32:

import os
os.environ.keys().index('TCL_LIBRARY')
Traceback (most recent call last):
File "", line 1, in
ValueError: list.index(x): x not in list
import Tkinter
os.environ.keys().index('TCL_LIBRARY')
26

I see a few ways to solve this problem, some more elegant than others:

1. virtualenv copies the "tcl" folder on win32 to the target folder

Pros: the least impact on both virtualenv and the environment variables; this will continue to allow Tkinter to function as expected in the system Python.
Cons: feels kind of hacky compared to how virtualenv references the rest of the standard library

2. virtualenv performs the functions of FixTk.py and sets the appropriate environment variables in activate.bat

Pros: this change is isolated to a Windows-only file (the batch file) and should also cause minimal "damage" in the case where the *_LIBRARY variables are already set (unless changing these in a batch file affects them system wide, I don't remember how exactly that works)
Cons: depending on how variables are set, this may destroy any system-wide *_LIBRARY variables associated with Tcl.

3. something similar to VirtualEnvWrapper, set a post deploy hook that sets the appropriate variables

Pros: no patch to virtualenv (yay!)
Cons: Tkinter support on win32 under virtualenv is still broken, this is a workaround that would need to be applied separately on every fresh install (although I'm sure avid virtualenv users already have some bootstrap scripts they use in this fashion anyway)

I seem to be the only Tkinter/virtualenv user on Windows :) so I'd be happy to work on a patch, but I'd like some suggestions as to the approach.

Ian Bicking wrote on 2009-10-16: Re: [Bug 449537] Re: Tkinter calls fail in fresh VirtualEnv

On Thu, Oct 15, 2009 at 3:30 PM, Shawn Wheatley swheatley@gmail.com wrote:

2. virtualenv performs the functions of FixTk.py and sets the
   appropriate environment variables in activate.bat

Pros: this change is isolated to a Windows-only file (the batch file) and should also cause minimal "damage" in the case where the *_LIBRARY variables are already set (unless changing these in a batch file affects them system wide, I don't remember how exactly that works)
Cons: depending on how variables are set, this may destroy any system-wide *_LIBRARY variables associated with Tcl.

This seems reasonable to me. I'm not clear if the same problem exists
on other platforms? It doesn't seem to for me.

After thinking a bit, I think the best option to implement this would
be to put the FixTk code into the virtualenv site.py. Then activation
won't be required to get Tkinter to work. The FixTk code would be
modified to use sys.real_prefix in addition to sys.prefix.

[whart222]

Note that this issue blocks the use of MatPlotLib using a virtual Python environment. IMHO, this makes this ticket a big deal.

I like carljm's suggestion of putting the fix in site.py...

[JustinSGray]

it would greatly help me if this was delt with. It is a big problem for using MatplotLib with the standard backend inside a virtual env.

[Wietse]

I solved this by setting the following environment variables in my activate script:
$env:TCL_LIBRARY = "C:\Python32\tcl\tcl8.5"
$env:TK_LIBRARY = "C:\Python32\tcl\tk8.5"

[geographika]

Thanks Wietse. I took a similar approach by modifying my activate.bat script to include:

set TCL_LIBRARY=C:\Python26\ArcGIS10.0\tcl\tcl8.5
set TK_LIBRARY=C:\Python26\ArcGIS10.0\tcl\tk8.5

mapplotlib is now working in the virtualenv.

[thisizkp]

Modified the activate.bat script to include the above commands, and it is working.
I am using virtualenvwrapper-win and I had multiple virtualenvs and should i change the activate.bat in all of them..? or is there any one single specific file that would suffice.?

@kp25 - you could patch virtualenv applying #627, which solves the issue for all virtual envs you will create afterward. Otherwise, until that pull request is accepted you need to change your activate.bat file for every virtual env you create. The problem is that virtualenv does not copy to the virtual env the file that python needs in order to figure out how to import Tkinter.

You could also manually do for every one of your virtual envs what #627 does upon virtual env creation: add a copy of the file <your python install dir>\Lib\lib-tk\FixTk.py to <your virtualenv python dir>\Lib (don't create the lib-tk subdir), with lines 49 and 52 of this copied FixTk.py edited to replace sys.prefix with sys.real_prefix. The advantage of doing so over editing the activate.bat files is that you don't need to specify the correct path for your python install.

[pfmoore]

@citterio I've added some comments to your PR. I don't use Tk at all myself, so I don't have any direct understanding of the situation, but I can understand the problem if this breaks matplotlib.

[rbtcollins]

Seems like FixTK could be fixed in the std lib itself?

for windows
in your dircetory "C:\Users\g2\Envs\DataVizproj\Scripts\activate.bat"
just add
set "TCL_LIBRARY=C:\Python27\tcl\tcl8.5"
set "TK_LIBRARY=C:\Python27\tcl\tk8.5"
and restart your cmd or shell
worked

[jedie]

Any news for a fix with Py2 and Py3 ?!?

EDIT:

set "TCL_LIBRARY=C:\Python27\tcl\tcl8.5"
set "TK_LIBRARY=C:\Python27\tcl\tk8.5"

But this will only work, if the version number are the same ;)

My work-a-round looks like this:

        if sys.version_info[0] == 2:
            virtualprefix = sys.prefix
            sys.prefix = sys.real_prefix
            import FixTk
            if "TCL_LIBRARY" not in os.environ:
                # reload module, so that sys.real_prefix be used
                reload(FixTk)
            sys.prefix = virtualprefix
        else: # Python 3
            virtualprefix = sys.base_prefix
            sys.base_prefix = sys.real_prefix
            from tkinter import _fix
            if "TCL_LIBRARY" not in os.environ:
                # reload module, so that sys.real_prefix be used
                from imp import reload
                reload(_fix)
            sys.base_prefix = virtualprefix

Any better idea?

EDIT: Full work-a-round here: jedie/DragonPy@0c1add8#diff-f11e22674d5a7f1c68650e7bdcea3a50R8