Proper module organisation for testing - python

I have a project that currently looks like this:
src/
proc.py
adapter.py
etc.
test/
core/test_proc.py
adapters/test_adapter.py
I want to run all the tests beneath test/, most of which import some modules from src. To this end, I have added src/ to my PYTHONPATH, however using 'python3 -m unittest discover -s test/' gives a cannot import name 'Proc' (from proc.py), although 'python3 -m unittest discover -s test/core/' works fine
Is this how I should be doing things (and if so, whats going wrong) or is there a better/more sensible way of organising my source and tests such that I can easily run all my tests at the top level and they themselves can import from src/?

You need to add init.py files to your directories/subdirectories.
project/
src/
proc.py
adapter.py
__init__.py
test/
core/
test_proc.py
__init__.py
adapters/
test_adapter.py
__init__.py
Then you can discover tests from project/

You should use relative imports. For that you should encapsulate your project into a single directory, something like this:
src/
__init__.py
app/
__init__.py
proc.py
adapter.py
etc.
test/
__init__.py
core/
__init__.py
test_proc.py
adapters/
__init__.py
test_adapter.py
Then, inside test_proc.py you can use from ...app.proc import something

Related

Cannot Import Class in Python

I have the following project skeleton.
ex47
bin/
docs/
ex47/
__init__.py
tests/
__init__.py
game_tests.py
game.py
setup.py
Working on Aptana Studio. In game_tests.py I have
from nose.tools import *
from ex47.game import Room
but Aptana is yelling at me for not being able to find Room, which I defined in 'game.py' as a
class. When I run nosetests on command line I got Error: Import Error( no module named game).
What's seems to be wrong?
I see two issues:
topmost ex47 is not a valid package (there is no __init__.py)
topmost ex47 is not on your PYTHONPATH
The first one is obvious. If you want game.py to be importable using ex47.game then ex47 has to be a valid package. So most probably you wanted to put it in the inner ex47 which is a valid package?
When it comes to the second issue, python will look for ex47 on your PYTHONPATH and in the current directory (the one you are in when issuing commands). Probably none of those is the case, hence ex47 can't be found.
Considering the above, if you had the following directory structure:
ex47
bin/
docs/
ex47/
__init__.py
game.py
tests/
__init__.py
game_tests.py
setup.py
and tried to run tests like this:
nosetests tests
while being in the topmost ex47 directory it should work (note that there is no __init__.py inside the topmost ex47).

How to have multi-directory or multi-package python project?

I have project structure like this:
package1/__init__.py
package1/file1.py
package1/file2.py
package2/__init__.py
package2/file1.py
package2/file2.py
__init__.py
script1.py
script2.py
Unfortunately, I found that I can run code only from root directory, for example, from script1.py. If I run say from pakage2/file2.py, all links between files are lost, i.e. all imports of package1 from package2 becomes not found.
What is the correct directory structure in Python, which constraints package structure over all directories?
You either need both package1 and package2 to be inside a package, in which case they can both import from each other:
root_package/
__init__.py
package1/
package2/
Or add the packages to your PYTHONPATH, in which case any python script on your system can import from them:
export PYTHONPATH="$PYTHONPATH:/path/to/package1:/path/to/package2"
Update: you cannot import as part of a package if you are running the scripts directly. What you should do is define classes and functions in your packages as desired, then import them from another script:
root_package/
__init__.py
my_script.py
package1/
package2/
script.py:
from package1 import ...
from package2 import ...

Python / Import and design of program

I’m looking for solution for designing my program.
My program consists of 3 blocks:
Classes
Functions
Other utilities
I want to structure my program this way:
program_folder/
main.py
classes_folder/
class_1.py
class_2.py
functions_folder/
set_of_func_1.py
set_of_func_1.py
utilities_folder/
set_of_utilities_1.py
set_of_utilities_1.py
I want to:
any scripts in «classes_folder» were able to import any of scripts in
«functions_folder».
any scripts in «functions_folder» were able
to import any of scripts in «utilities_folder».
all scripts were
normally used by main.py.
all scripts in «classes_folder»,
«functions_folder» and «utilities_folder» could be tested when worked
as «main» (if __name__ == “__main__”: some tests)
«program_folder»
could be in any place in my computer (there shouldn’t be dependency
on exact path to «program_folder»).
From all the above I thought I have to:
Change import search path for all scripts in «classes_folder»,
«functions_folder» and «utilities_folder».
Set current working
directory to «program_folder» for all scripts?
Is there a way I can do it?
Does my idea look good or have I put there some unexpected problems?
You can create a skeleton project like the following:
/path/to/project/
setup.py
my_project/
__init__.py
a/
   __init__.py
  b/
  __init__.py
==> ./my_project/__init__.py <==
print('my_project/__init__.py')
==> ./my_project/a/__init__.py <==
import my_project
print('my_project/a/__init__.py')
==> ./my_project/b/__init__.py <==
import my_project.a
print('my_project/b/__init__.py')
==> ./setup.py <==
from distutils.core import setup
setup(name='my_project',
version='1.0',
description='my_project',
author='author',
packages=['my_project'])
Then you can install the project locally using pip install -e /path/to/project/ (the project folder is not copied, just gets registered; there's a dependency on the exact path, but this dependency is not hard-coded in project files themselves).
As the result, import my_project, import my_project.a etc. do that they mean:
$ python my_project/b/__init__.py
my_project/__init__.py
my_project/a/__init__.py
my_project/b/__init__.py
A common Python project structure could look like this:
project_name/
setup.py
requirements.txt
project_name/
__main__.py
classes/
__init__.py
class1.py
class2.py
functions/
__init__.py
functions.py
utils/
__init__.py
utils.py
Then, you could modify your imports from absolute to relative and run your package using something like:
$ /path/to/project_name> python -m project_name
Note that setup.py is only required if you want to install your package under some of your interpreters.
Note: see comments below also

Can a Python script in a (sub)module import from upstream in its directory hierarchy?

I realize there are a slew of posts on SO related to Python and imports, but it seems like a fair number of these posts are asking about import rules/procedures with respect to creating an actual Python package (vs just a project with multiple directories and python files). I am very new to Python and just need some more basic clarification on what is and is not possible with regard to access/importing within the context of multiple py files in a project directory.
Let's say you have the following project directory (to be clear, this is not a package that is somewhere on sys.path, but say, on your Desktop):
myProject/
├── __init__.py
├── scriptA.py
└── subfolder
├── __init__.py
└── scriptB.py
└── subsubfolder
├── __init__.py
└── scriptC.py
└── foo.py
Am I correct in understanding that the only way scriptC.py could import and use methods or classes within scriptB.py if scriptC.py is run directly via $ python scriptC.py and from within the subsubfolder directory is if I add the parent directory and path to scriptB.py to the Python path at runtime via sys.path ?
It is possible, however, for scriptC.py to import foo.py or for scriptB.py to import scriptC.py or foo.py without dealing with sys.path, correct? Adjacent py files and py files in subdirectories are accessible just by using relative import paths, you just can't import python scripts that live in parent or sibling directories (without using sys.path) ?
What's Possible
Anything.
No, really. See the imp module, the the imputil module -- take a look at how the zipimport module is written if you want some inspiration.
If you can get a string with your module's code in a variable, you can get a module into sys.modules using the above, and perhaps hack around with its contents using the ast module on the way.
A custom import hook that looks in parent directories? Well within the range of possibilities.
What's Best Practice
What you're proposing isn't actually good practice. The best-practice approach looks more like the following:
myProject/
├── setup.py
└── src/
├── moduleA.py
└── submodule/
├── __init__.py
├── moduleB.py
└── subsubmodule/
├── __init__.py
└── moduleC.py
Here, the top of your project is always in myProject/src. If you use setup.py to configure moduleA:main, submodule.moduleB:main and submodule.subsubmodule.moduleC:main as entry points (perhaps named scriptA, scriptB and scriptC), then the functions named main in each of those modules would be invoked when the user ran the (automatically generated by setuptools) scripts so named.
With this layout (and appropriate setuptools use), your moduleC.py can absolutely import moduleA, or import submodule.moduleB.
Another approach, which doesn't involve entrypoints, to invoke the code in your moduleC.py (while keeping the module's intended hierarchy intact, and assuming you're in a virtualenv where python setup.py develop has been run) like so:
python -m submodule.subsubmodule.moduleC

Importing in __init__.py to a unittest package

I have a package and a test package. According to advice from Where do the Python unit tests go?, the tests should be in a different directory. The project's directory tree is as follows:
project\
kernel\
__init__.py
file1.py
file2.py
tests\
__init__.py
test1.py
test2.py
test3.py
I would like to import the kernel package to the tests package, because that is where file1.py and file2.py are being tested. Also, I would like to use one import statement in the __init__.py instead of importing kernel again and again in each test.
I tried adding the following to the __init__.py file in tests and to test2.py , test2.py (together and separately), with no success (the first does no harm, the second gives a syntax error):
import kernel
import ../kernel
I'm using python2.6. From command line all the above happens. When I use Eclipse PyDev, everything magically works.
The relative imports you're using will only work if the "project" directory is a python package (i.e. it has an __init__.py file in it). Try that first and see if that works for you.
If the kernel directory is acting as the "package" that would be distributed then you could put the tests directory inside of that and do the relative imports that way. So it would look like this:
project/
kernel/
__init__.py
file1.py
file2.py
tests/
__init__.py
test1.py ...
And you would import the kernel modules from the tests directory either as:
from kernel import file1 # if it's installed in the python path/environment
Or:
from .. import file1
# 'import ..file1' might work, but I'm not sure that's syntactically correct

Resources