Better handling Python Multi-project import dependencies

In other words, a project needs to be developed during this time, and new projects are dependent on almost all existing projects. Watercress existing several major projects, basically is around the main station Shire rely on, that is, other projects on the Shire of the single dependency, these projects in need of the main station Shire module, without exception, will Shire project path to other projects Sys.path, People familiar with Python import must immediately realize that this is not a good way, because this will cause the Python import search path to produce some uncertainties, that is, import search space pollution, for example, I am in Anduin (Watercress said) project Import main Station (shire) of the project, from Luzong.user Import user, looks like a very magic thing, but if Anduin also has a Luzong directory (of course, this does not), This will result in a search conflict, a serious point where the hermit's wrong import of the other modules is possible. In other words, this is a very uncertain, very dangerous thing, that watercress those big project is how to deal with the danger of this magic, a: Each new project top-level directory is not the same as the existing project top-level directory.

Well, as soon as I heard it was a very painful idea, I asked the Anrs on Skype and it seemed that he was following the rules. However, with the increase in internal engineering, the naming of each project should consider the naming of other projects is very very painful things, in my personal view is very bad, then how to do? In this blog I will gradually introduce the next two days I explored the method, really very magic, very useful:)

First of all, we think of, in order not to pollute the import of imported space, we do not use each project as a top-level module can be imported, so that the content of each project in its own separate namespace, there will be no such magic of the implicit namespace pollution. Well, this is good, add a __init__.py empty file in the top-level directory of each project, and then we'll develop something like import_project:

def import_project (path):
Sys.path.insert (0, Os.path.dirname (path))
Project = __import__ (os.path.basename (path))
Sys.path.remove (Os.path.dirname (path))
Globals () [Os.path.basename (path)] = Project
Return project

Then, we import the dependent project in the initialization file of our project:

conf.__init__.py

Shire = Import_project (Cfg.deps.shire)

When using the

From Conf Import Shire
From Shire.luzong.user Import user

Wow, it is really magic, is not the end of it, the answer is often not so simple, not so as you wish, think of Anduin project, in the watercress server, we developed a lot of tests to deploy the project, such as Shuo-test, Anduin-test, Auduin, if according to the above method, the line certainly does not have any problem, because Shire is Shire, that if I test the project top-level to shire-test, this can not be processed, so there is behind the things, gentlemen and look down:)

After a half-day hard code word, finally out of a still can use things, thought is, the import part of a similar hook hack, let him not these shuo-test, Anduin-test and other similar magic shring have any dependence. Therefore, a function is developed that implements a function similar to normal import, but does not result in the search for space pollution between projects, and does not depend on these similar shuo-test, anduin-test magic strings, I would like to introduce my things, And then give a demo:)

Code word a day developed the following this thing import_helper.py

# Libs.import_helper
#-*-Coding:utf-8-*-
# [email protected]

Import OS
Import Sys

Def import_deps (locals, Path, Imps=none):
From conf import cfg
Pro_name, M_path = path[:p ath.find ('. ')], Path[path.find ('. ') + 1:]
Re_pro_path = Cfg.getbypath (' deps. ' + pro_name)
Re_pro_dir = Os.path.dirname (Re_pro_path)
Re_pro_name = Os.path.basename (Re_pro_path)

# import Project
Sys.path.insert (0,re_pro_dir)
Project = __import__ (Re_pro_name)
Sys.path.remove (Re_pro_dir)
Locals[pro_name] = Project

Imp_module = Re_pro_name + '. ' + M_path
__import__ (Imp_module)
module = GetAttr (Project, M_path)

# Add Imps to locals
If Imps:
For imp in Imps:
If Isinstance (imp, tuple):
LOCALS[IMP[1]] = getattr (module, imp[0])
Else
Locals[imp] = getattr (module, IMP)

The main logic in this file, it is very convenient to use, and can handle from XX import A as B form. Here's a demo to illustrate:

1. Create a test environment

$CD && mkdir-p test/test11 test22 && CD test

$touch test11/__init__.py && echo "age = 1″> test11/app.py

$CD test22

We have created a test catalog here, tests, two engineering test11 and test22, this test11 project name is called Test (although his directory is test11, but like Shuo-test to Anduin:)

2. Create a configuration file (I used a config thing for the sake of simplicity)

$mkdir conf && Emacs config.cfg

Path_prefix: ' os.environ[' HOME '
Deps:
{
Test: $path _prefix + '/test/test11 '
}

$emacs conf/__init__.py

Import OS
Import Posixpath
From config import config

Config_file = Posixpath.abspath (' conf/dimholt.cfg ')
CFG = Config (file (config_file))

OK, our configuration file is configured, here, I configured my project to rely on a project called Test, the directory of the project in my home directory test/test11.

3. Place the import_helper.py (download provided behind the article) in the Libs directory under the test22 directory.
4. Use the Demo

Start the Python interpreter
>>> from Libs.import_helper import import_deps
>>> import_deps (Locals=locals (), ' Test.app ')
This will allow you to access the Test.app in your project if you want to simulate from A import B
>>> import_deps (Locals=locals (), ' Test.app ', [' age '])
This will import a variable called age if you want to simulate import a as B
>>> import_deps (Locals=locals (), ' Test.app ',
[(' Age ', ' g_age ')])
One more full bloom.
>>> import_deps (Locals=locals (), ' Test.app ',
[' Age ', (' name ', ' Screen_name ')])

Well, isn't it magic?

To summarize, after using this method to solve the problem of two inter-project import, one is the import search path pollution, we by the import space of each dependent project is limited to each project name to achieve, the second is the project alias problem, We use this developed function of comparing magic to solve. Personal feeling is still very useful, in addition to the need to enter something more. But the benefits are very obvious and worthwhile.

Ann, Beijing:)

PostScript: (bugfixed)

Say, this would like to take a good look at Emacs today, the results of a sudden thought of a problem, Python locals and globals processing is different, globals () can be modified, and locals () cannot be modified, so the above is not available, Attempts to modify the various methods of locals are not very good, so changed the use of the next project, plus a project import support:

import_helper.py

# Libs.import_helper
#-*-Coding:utf-8-*-
# [email protected]

Import OS
Import Sys

def import_deps (Path, Imps=none):
If '. ' Not in Path:
Pro_name, M_path = path, '
Else
Pro_name, M_path = path[:p ath.find ('. ')], Path[path.find ('. ') + 1:]
From conf import cfg
Re_pro_path = Cfg.getbypath (' deps. ' + pro_name)
Re_pro_dir = Os.path.dirname (Re_pro_path)
Re_pro_name = Os.path.basename (Re_pro_path)

Sys.path.insert (0,re_pro_dir)
Project = __import__ (Re_pro_name)
Sys.path.remove (Re_pro_dir)

If not m_path:
Return project

Imp_module = Re_pro_name + '. ' + M_path
__import__ (Imp_module)
module = GetAttr (Project, M_path)

If Imps:
return [GetAttr (module, IMP) for IMP in Imps]
Return module

When used:

>>> from Libs.import_helper import import_deps

You can import the top-level engineering module directly
>>>shire = import_deps (' Shire ')

You can also import any engineering external module
>>>user = import_deps (' Shire.luzong.user ')

You can specify a method like from import
>>>user = import_deps (' Shire.luzong.user ', [' User '])

Of course if you want to simulate the operation of AS
>>>shireuser = import_deps (' Shire.luzong.user ', [' User '])

The back part can be a list, which means import multiple parameters
>>>get_user_rank, user = Import_deps (' Shire.luzong.user ',

[' Get_user_rank ', ' user '])

Summary, because the locals space can not be modified, so use this method to deal with, if you can determine the use of globals space, that plus globals line, just like the previous method, but pass globals parameters, there is no locals in the module layer, In the module layer to pass locals is actually used globals, remember.

Excerpted from python.cn

Better handling Python Multi-project import dependencies