Created by: gwideman, Oct 7, 2013 12:24 am
Revised by: gwideman, Oct 17, 2013 4:34 pm (30 revisions)

2013-10-17: Note: Work in progress -- or more like in a shambles

Overview

This page collects my notes covering understanding what happens when a user launches a Python script, particularly on a machine where multiple versions of Python are installed or available.
Multiple Python installations are a frequent occurrence, primarily because the Python language of the 3.x series of interpreters is not fully compatible with the language of the 2.x series. Despite the improvements of the 3.x series, scripts and libraries based on Python 2.x are still in wide use, so a Python user often needs their machine equipped for both. Additionally, features changed significantly along the progression of 2.x versions, and 3.x versions, again suggesting the need for more installations. Yet more installations may be occasioned in order to install different sets of libraries which are either incompatible, or to reduce complexity of any one installation.
All of this precipitates the need to manage which of multiple installations of Python gets invoked for a particular script.
Topics include handling the different user actions which can invoke a Python script and determining the environment the script will see once running. To do this we need to understand the Windows and Python mechanisms which affect these. In outline:

• User actions which invoke a script

  • Command line: several different command-line ways to launch the script and Python
  • Windows Explorer: Double-clicking on the script, or using the Explorer right-click menu
  • Called from another program
  • Run in IDE debugger

• The environment within which the script operates

  • Version of Python
    • 2.something, 3.something
    • 32 or 64 bit
  • What libraries or modules will be available to the Python script

• The mechanisms which affect the above issues

  • Windows:
    • Environment variables: PATH and PATHEXT
    • File Associations
    • Registry settings: App Paths
  • Python:
    • Invoke either the python.exe interpreter directly, or indirectly via Windows Python Launcher
    • Windows Python Launcher: py.exe and pyw.exe
      • py.ini configuration files and corresponding environment variables
      • Command-line arguments to py.exe
    • Which Pythons have been installed (and corresponding registry entries). Also:
      • Virtual Python environments, if any
      • Libraries and modules that have been installed into particular Python base or virtual installations
      • Python settings and environment variables that affect what libraries/modules the running Python environment can see via sys.path
  • Script:
    • "shebang line": A special "comment" on the first line of a script, which provides instructions requesting a particular Python. On unix, this is a convention interpreted by the shell. On Windows, the shebang line is interpreted by the Python Launcher for Windows (introduced with Python 3.3, and able to select any previous or current version of Python.)
      • Special treatment of "virtual command" shebang-lines

Related issues

• Virtual environments; venv
• Script or library installation
  • What is it that installers do? How does that intersect with the launch variants.

Background

The central task: launching a Python environment that suits a particular script...

The central launch task is to ensure that the invoked Python environment does two things:

• understands the version of the Python language (version X.Y) that the script makes use of. A script typically needs a Python interpreter that is:
  • definitely within the same X family, either the 2 family, or the 3 family.
  • often a version number greater or equal to particular X.Y. For example, '2.6 or above', or '3.2 or above'. an exact version X.Y: "must be 3.3".
  • occasionally a version number greater than X.Y.Z: 'Must have features of 3.2, with the bug fix that was applied at 3.2.2'.
  • occasionally an exact X.Y version, like 'exactly 3.2', perhaps because that's what the developers tested the script with.
  • almost never an exact X.Y.Z, except perhaps as a script developer testing the script with a very specific version of Python.
• includes or provides access to the libraries (packages or other modules) that the script makes use of.

... and to do these two things in the face of Python updates, and possibly replacement, as time goes by. This is the classic problem of "program vs library versions" familiar in many operating system and language environments -- famously "dependency hell" or "DLL hell".

...is not entirely solved

The short story is that Python does not have a mechanism that completely solves this set of problems. Instead, there are a number of different mechanisms in play, each of which addresses part of the problem, sometimes in conflicting ways, and they do not add up to a complete solution. Instead they add up to a requirement that developers, and to an extent users, have to encounter a considerable amount of superfluous complexity, just to get scripts launched reliably.

Key Problems

The key constraining features are:

• Additional libraries generally get installed into a particular Python installation. If a different Python installation is invoked, it does not provide access to those additional libraries.
• Launch methods generally don't select Python version and libraries separately.
• The generic python installer can install multiple Python installations, but just one per X.Y. This is an issue when there's a need to two or more different sets of libraries to support different scripts, perhaps because the libraries are mutually incompatible. This in turn motivates virtual Python environments, which then requires knowing to methods to select one of those during launch.

The Python Launcher for Windows

The Python Launcher for Windows is a feature which gets installed with Python starting with version 3.3. It implements several previously unavailable ways to control the launch of Python, most notably, the ability to understand "shebang" lines. Once installed, Launcher intervenes in the launch for all versions of Python, including pre-3.3 and 2.x series (but not for absolutely all methods of launch). So in what follows, a distinction must be made between what happens on a system in which Launcher has been installed, versus a system on which it is absent.

- - - - - - - - - -

• All interpreters are named "python.exe", conflicts with the PATH facilities: ...so a particular Python must be selected ultimately by its directory path.
  • Having multiple python installations included in PATH is not useful, though this idea is less poorly implemented on Unix where 'python2' or 'python3.2' are on the PATH and are either executables or symbolic links to executables.
  • Setting PATH is a system-wide or user-wide setting, whereas we want per-script selection of Python version and libraries
• Libraries are tied to a particular Python installation: For the most part, additional libraries are installed into the directory structure of a particular Python installation. This has some benefits (libraries are associated with an interpreter that understands them), but the entanglement also raises some problems
• 
  

Entanglement of Python version with Python environment

A Python installation includes a directory tree containing the Python components. This tree also contains a directory ([python_installation]\lib\site-packages\) , which is the most convenient location to place additional packages and modules which can be used by scripts running with this installation of Python.
The mechanism which makes packages and modules available to your scripts is the Python variable sys.path. Here are examples of sys.path, as it appears for a couple of different Python installations.

sys.path for a 2.7 installation	sys.path for a 3.3 installation
[directory of current program] C:\Windows\system32\python27.zip C:\Python\27\DLLs C:\Python\27\lib C:\Python\27\lib\plat-win C:\Python\27\lib\lib-tk C:\Python\27 C:\Python\27\lib\site-packages	[directory of current program] C:\Windows\system32\python33.zip C:\Python\33\DLLs C:\Python\33\lib C:\Python\33 C:\Python\33\lib\site-packages

Note: The installation directory names "C:\Python\27" and "C:\Python\33" are just my convention for organizing Python installs. What is of most interest for the current discussion is the structure below the installation directory. The paths shown here are built in to the Python loader code.
The inclusion of the site-packages directory within the Python directory structure is what organizes additional packages per installation. The inclusion of that path in the default sys.path list is what makes additional installed packages available to your scripts when running that particular Python.
Additional package directories (eg., outside of the Python installation) can be added to sys.path, either by setting PYTHONPATH environment variable before launching Python, or by the script itself adding items to sys.path (obviously before calling those libraries). However, the sys.path shown above is the default and thus most convenient to work with, absent any strong reason not to.

The entanglement has awkward implications

The main implication of this entanglement is that choosing to invoke the python.exe in a particular Python installation also selects the particular set of packages and modules that the script will be able to access conveniently. So, when multiple Python X.Y versions are installed, the following concerns arise:

• Selection of Python version: Obviously you want a Python X.Y version to be invoked which understands the language version that the script uses.
  • The main mechanisms for launching a script and corresponding Python focus on this issue first.
• Availability of libraries when using different Python version: If a script relies on a particular third-party package that you have installed in the Python X.Y installation, and then you invoke the script with some version of Python other than X.Y, it may well be that the needed package has not been installed for that Python installation, and your script will fail.
  • You can install libraries into multiple Python installations.
• Managing multiple incompatible libraries: Different scripts, even though requiring the same language version, may need packages or modules that are mutually incompatible, and should not be installed in the same Python installation. This raises the need to create multiple installations of the same Python X.Y version, which creates a number of wrinkles (and motivates "Python virtual environments").
  • The most immediate problem is that whichever mechanism is used to actually select and launch a particular python.exe must use something more precise than X.Y Python version number to find the right Python installation.

In context

I will shortly address all the means available to deal with these challenges. However, it is worth first noting the baseline: at worst, it is always possible to issue a command line command which explicitly selects a particular python.exe to run (having a particular X.Y language version, and its surrounding directory tree and installed packages), like this:

C:\python\33\python.exe myscript.py

If your script needs access to additional packages or modules, then you can set PYTHONPATH and any other Python environment variables you need, like:

PYTHONPATH=D:\MySpecialPythonPackages
C:\python\33\python.exe myscript.py

And of course these could be placed in a batch file for convenience. (Though this in turn raises a number of questions).
So for the most part, the remaining discussion is about how to get that effect automatically, and for a variety of ways of invoking a script, without you (or your users) having to know these details (or having to type them in or set them up).

=============================

Notes

Plain vs "w"

For each python install, there is a python.exe variant named pythonw.exe. The "w" version suppresses launching a shell window when it starts up. This is useful when a program wants to launch python as a subsidiary process and not have a shell window visibly flash onto the screen. However, it is only suitable or scripts which don't want to accept input from, or direct output to, a command shell. Scripts designed to operate without a shell window can be given the filename extension ".pyw". On a computer with a normal python installation, such a script will cause Windows to invoke Launcher pyw.exe instead of py.exe, or pythonw.exe instead of python.exe. The descriptions on this page focus primarily on the plain py.exe and python.exe variant, with the "w" variant working the same way except for the shell window suppression.

Mode of launch

		Windows		Linux
Invoking environment	Invocation method	No Launcher	With Launcher	(No launcher)
Command-line shell	Windows: C:\Full\path\to\python.exe myscript.py Linux: /full/path/to/python myscript.py	Launcher not involved. Invokes the python at that location	Launcher not involved; see no-launcher case.	Invokes the python at that location
	python myscript.py	Launcher not involved. Shell finds python interpreter using PATH (or possibly ...\App Paths registry key)	Launcher not involved; see no-launcher case .	Shell finds python interpreter using PATH.
	py myscript.py	Error. No py.exe	Launcher py.exe inspects script's shebang line or falls back to launcher's own configuration or algorithm to find a python to launch. [See Shebang processing section]	N/A (No launcher on linux).
	py -3.2 myscript.py py -3 myscript.py	Error. No py.exe	Launcher py.exe tries looks for the highest-numbered installed python version that matches the first argument number(s). (Launcher ignores PATH) OR -- does py look at the PYTHON2 or PYTHON3 variable as part of the process?	N/A
	myscript.py	Whichever python installation is registered in Windows File Associations with '.py' will be invoked.	Launcher is registered as the File Association for '.py', so launcher is invoked. Acts the same as 'py scriptname.py' case.	If myscript has executable permission set, and contains a valid shebang line, the shebang line determines which python is invoked. ("execve" processing.) If there is not shebang line, the shell attempts to execute the file as a shell script. [See Shebang processing section]
	myscript.pyw	Whichever python installation is registered in Windows File Associations with '.pyw' will be invoked.	W version of Launcher is registered as the File Association for '.pyw', so launcher is invoked. Acts the same as 'pyw scriptname.pyw' case.
	myscript	If .py is in PATHEXT, Windows proceeds as with myscript.py (see above).	If .py is in PATHEXT, Windows proceeds as with myscript.py (see above).	Shell just tries to run myscript as a script. If that actual script name is myscript.py, shell does not find it.
	pipe to or from myscript.py (affects byte/unicode setting of stdio)	Whichever python installation is registered in Windows File Associations with '.py' will be invoked. [I think]	???	Same as plain myscript.py case. Shell, via execve, uses shebang line to launch specified python [See Shebang processing section]
Windows Explorer, double-click	myscript.py	Whichever python installation is registered in Windows File Associations with '.py' will be invoked. (Command shell gets invoked???)	Launcher invoked. Acts same as py myscript.py case. (Command shell gets invoked???)	(Double click in linux file browser.) Same as command-line myscript.py case. (Shell invoked???)
	myscript.pyw	Whichever python installation is registered in Windows File Associations with '.pyw' will be invoked. ???	???	???
	Double-click shortcut (could include args)	Windows processes the shortcut's "Target" command according to the variants in this table.	Windows processes the shortcut's "Target" command according to the variants in this table.	(Double-click link in linux file browser). Proceeds as though double-clicking on the link's target file. (I think ???)
Another executable	Invoke process involving either a *.py/*.pyw file, or invoking py.exe or python explicitly	???	???	???
Within an IDE	Run with or without debugger	???

Shebang line processing

In a Python script, the first line may be a "shebang line". After the initial "#!", the shebang line may contain:

• A command-line starting with explicit path to a Python executable (python.exe). Possibly with additional argument(s)
• A command-line that starts with what looks like a common unix-style path to a python executable, but is interpreted by Launcher as a special "virtual shebang-line command". May be followd by arguments.
• A command-line that starts with a command defined by the user: a "custom virtual-shebang-line command". May be followed by arguments.

Variant	Shebang line example (!# omitted for brevity)	Windows Python Launcher action	Linux action
No shebang line	[none]	Launcher proceeds as though the PY_PYTHON=2. (Ie: launches highest-version 2.x python. Or version specified by PY_PYTHON2 default, if any?)	Shell tries to run script as an ordinary shell script (ie: ignores filename extension). This obviously does not have the intended effect. In particular, the first shell-recognizable line in a python script is often 'import...', causing shell to try to run import program on PATH. Commonly this finds 'import' from image magick, leaving the computer seemingly hung, with peculiar cross cursor, waiting for completion of screen capture.
Explicit path	C:\python\3.2\python.exe arg1 ...	Launcher runs program (presumably a python interpreted, but could be any program) at that exact path, otherwise fails. Launcher passes the shebang-line arguments to the program, along with the path to the current script.	Assuming a linux-style path, the shell, as usual, interprets this as a path to an executable it can run. The target of the path could be a link to an executable. The path is normally an absolute path. However, it could be a relative path (interpreted as relative to what???), but this is almost never useful. Note that shell never searches PATH for a target for the command, so the command must be an explicit path to an executable or link.
Virtual shebang-line command	/usr/bin/python /usr/local/bin/python	These two specific paths, commonly used in linux python scripts, are interpreted by Launcher as "virtual shebang-line commands". (Does Launcher skip even checking if these are valid paths?) Launcher interprets these as request for a version of python to be selected according to Launcher's configuration and algorithm	As usual, the shell launches the program at the path supplied. That path may be a link, or even a series of links, ultimately pointing to a python interpreter executable.
Virtual shebang-line command with some constraints	/usr/bin/python2 arg1... /usr/bin/python3.2 /usr/bin/python3.2.1	Variants of the special "virtual shebang-line commands" which tell Launcher to apply its selection process constrained to a version of python within the family indicated, 2.x and 3.2.x in the examples.	As usual, the shell launches the program at the path supplied. Python usually installs several links such as link python2 pointing to actual interpreter executable python2.7.4 and so on, in this way implementing a way for shebang lines to request a python version with more or less specificity.
Custom virtual shebang-line	myvcommand arg1	Causes Launcher to find, from its py.ini file(s), a corresponding [command] definition, which tells what command-line to execute in this case.	Shell will respond with an error, (??? what is the specific error)
	_._._._._._._._._._._._._

Launcher configuration settings

The launcher gets configuration from the following files and environment variables, in order of decreasing precedence: This allows users to override global settings, and shell scripts which launch other python scripts to set environment variables for ad hoc selection of a particular python version or installation.

Configuration scope	Form of configuration	Location	Discussion
Command line	First arg to py.exe if starts with dash	Command line	Example: py -3 myscript.py. Launcher must refer to other configuration information(below in this table) to fully interpret.
Script	Shebang line; explicit path	First line of script	Tells Launcher to run interpreter at an explicit path.
Script, flexible	Shebang line; "virtual command"	First line of script	Gives Launcher a "shebang-line virtual command", to be interpreted according to other configuration information (below in this table).
Ad hoc	Environment variables	Shell session environment variables
Per user	Environment variables	User environment variables
"Global"	Environment variables	System environment variables
Per-user	INI file	%LOCALAPPDATA%\py.ini	py.ini file in User's "Local Application Data" directory. Windows 7: C:\Users\[username]\AppData\Local.
"Global"	INI file	[py.exe's dir]\py.ini	py.ini file in same directory as py.exe. Usually C:\Windows
Launcher internal algorithm	n/a	py.exe source code	If there is no other configuration info selecting a python version, py.exe will select a 2.x version if available before a 3.x version.

To be determined from source code:

• Do global environment variables override per-user ini file? I don't think py.exe looks at whether an environment variable is just in this shell, or User, System, so an environment variable set at any level overrides py.ini?.
• Does per-user ("local") ini file override global one on a per-file basis, or per variable basis?

Launcher INI file format and environment variables

Launcher inspect two py.ini files:

• Global py.ini in the same directory as py.exe; typically C:\Windows\py.ini
• User py.ini in user's %LOCALAPPDATA% directory; typically C:\Users\AppData\Local\py.ini.

The user py.ini file overrides the global py.ini file. (Is this override on a per-file, or per-variable basis?).
In addition, Launcher recognizes environment variables whose names parallel some of the possible keys in the [defaults] section of the INI file. (And no environment variables corresponding to the [commands] section?) The possible lines in the ini files, and corresponding environment variables, are detailed below.

py.ini format

[defaults]
python=2
python2=2.6
python3=3.3
[...]
[commands]
vpython=c:\bin\vpython.exe -foo

(Can the [commands] section define the actual path for non-custom virtual commands?)
The launcher executables py.exe and pyw.exe both read the same set of optional ini files and optional environment variables.

[defaults] section of py.ini

Pattern	Example	Equivalent env't var	Description	Applies to situation
[defaults] section of ini
python=X	python=3	set PY_PYTHON=3	If Launcher must resolve plain "python", this ini setting or environment variable tells Launcher to find the highest-versioned python in the X family; 3.[highest] in the example. NOTE: With no ini or environment variables, py.exe behaves as though PY_PYTHON=2, even though python 3.something is installed.	With or without shebang lines: Command-lines 'py' or 'py myscript.py' 'myscript.py' Explorer double-click
python=X.Y python=X.Y.Z	python=3.2 python=3.2.1	set PY_PYTHON=3.2 set PY_PYTHON=3.2.1	If Launcher must resolve plain "python", then this ini setting or environment variable tells Launcher to find the highest-versioned python in the X.Y, or X.Y.Z range; 3.2.[highest], and 3.2.1.[highest] in the examples.	With or without shebang lines: Command-lines 'py' or 'py myscript.py' 'myscript.py' Explorer double-click
pythonX=X.Y pythonX=X.Y.Z	python3=3.2 python3=3.2.1	set PY_PYTHON3=3.2 set PY_PYTHON3=3.2.1	If Launcher must resolve "pythonX" (where X is 2 or 3), then this ini setting or environment variable tells Launcher to find the highest-versioned python in the X.Y, or X.Y.Z range; 3.2.[highest], and 3.2.1.[highest] in the examples.	'py -3' or 'py -3 myscript.py' command-line With shebang-line like python2 or python3: 'myscript.py' Explorer double-click
pythonX.Y=X.Y.Z	python3.2=3.2.1	set PY_PYTHON3.2=3.2.1	If Launcher must resolve "pythonX.Y", then this ini setting or environment variable tells Launcher to find the highest-versioned python in the X.Y.Z range; 3.2.1.[highest] in the examples.	'py -3.2' or 'py -3.2 myscript.py' command-line With shebang-line like python2.7 or python3.2: 'myscript.py' Explorer double-click
python=X-bb [...] pythonX.Y=X.Y.Z-bb (bb = 32 or 64)	python3.2=3.2.1-64	set PY_PYTHON3.2=3.2.1-64	When Launcher resolves the python request, 'bb' selects 32- or 64-bit version or python.
		_._._._._._._._._._._._._._._._

[commands] section of ini

This section defines shebang-line virtual commands, commands which Launcher will recognize in the shebang line of a script. Each definition's key part defines the command name. Each definition's value part defines the OS command which Launcher should execute, so this is typically a path to a python.exe, along with associated flags, if any.

Pattern	Example	Equivalent env't var	Description
mycmd=c:\path\to\python.exe args	vpython=c:\bin\vpython.exe -foo	None?	Defines a custom command that Launcher will recognize in a shebang line, and the path and args this translates to.
Can this section give definitions for the built-in virtual commands?

py.exe python version search mechanism

• Mostly uses registry, not PATH

Interaction with Python virtual environments

• As search candidates for py.exe
• As explicit path targets
• Activation

What Installers do (what makes a script "installed")

Conclusions

• Recommendations for how to set shebang lines (or not) for "main" scripts.
• How to set shebang lines for modules that are not intended to be executable stand-alone, but which are specific to a version.
• Virtual environments

References

• The #! magic, details about the shebang/hash-bang mechanism on various Unix flavours: www.in-ulm.de/~mascheck/various/shebang
• Screenshot of Mac Python Launcher: thetitleiseverything.com/blog/2013/9/9/download-catch-up-tv