Thursday, July 19, 2012

How to use sys.meta_path with Python

I was asked on Reddit a short while ago as to how to use sys.meta_path, which is an extremely valuable tool that can be used to implement import hooks. Since it seems there's little documentation as to how to use it properly, I decided to write an example that shows basic usage of sys.meta_path. I posted the example on Reddit in the thread I linked to, but I've put it here for posterity - enjoy. :)

import sys

class VirtualModule(object):

   def hello(self):
      return 'Hello World!'

class CustomImporter(object):

   virtual_name = 'my_virtual_module'

   def find_module(self, fullname, path=None):
      """This method is called by Python if this class
         is on sys.path. fullname is the fully-qualified
         name of the module to look for, and path is either
         __path__ (for submodules and subpackages) or None (for
         a top-level module/package).

         Note that this method will be called every time an import
         statement is detected (or __import__ is called), before
         Python's built-in package/module-finding code kicks in.
         Also note that if this method is called via pkgutil, it is possible
         that path will not be passed as an argument, hence the default value.
         Thanks to Damien Ayers for pointing this out!"""

      if fullname ==  self.virtual_name:
         # As per PEP #302 (which implemented the sys.meta_path protocol),
         # if fullname is the name of a module/package that we want to
         # report as found, then we need to return a loader object.
         # In this simple example, that will just be self.

         return self

      # If we don't provide the requested module, return None, as per
      # PEP #302.

      return None

   def load_module(self, fullname):
      """This method is called by Python if CustomImporter.find_module
         does not return None. fullname is the fully-qualified name
         of the module/package that was requested."""

      if fullname != self.virtual_name:
         # Raise ImportError as per PEP #302 if the requested module/package
         # couldn't be loaded. This should never be reached in this
         # simple example, but it's included here for completeness. :)
         raise ImportError(fullname)

      # PEP#302 says to return the module if the loader object (i.e,
      # this class) successfully loaded the module.
      # Note that a regular class works just fine as a module.
      return VirtualModule()

if __name__ == '__main__':

   # Add our import hook to sys.meta_path

   # Let's use our import hook
   import my_virtual_module
   print my_virtual_module.hello()

Note: The original PEP that defined the sys.meta_path protocol is PEP #302, which you can read here.


  1. Hello I have a question related to this subject:

    I have a test platform written in python and seek to make a kind of "personal
    developement space "- I will call it in the bellow text as PDS.

    Basically it should be a directory on the platform where the user should be able to put the modified platform files .

    The point is not to have a mix of files from an official delivery and the ones from current work. The problem is that those files that the user will change have the same name (I know is bad to do so but it seems that many users change the files as they want) as in the official platform files (even if they are in different folders), so it might have some import issues.

    At first I thought I enter in sys.path as the first director the PDS directory and if I understood well it should work
    But I am not too sure that it is the right method, since someone may add another folder as first one and so on .

    Then I found on the net sys.meta_path version.

    From what I understand from your example python importer will search first using the customimporter class methods .
    I tried to implement a similar hook but I encounter some problems

    I created a module called (see below code) (here again I
    need your help)
    I imported (not sure that I should do so - please if you have other suggestions -?)

    I added the command sys.meta_path
    sys.meta_path.append (CustomImporter ())

    If I understood correctly once you added customImporter in sys.meta_path every time an import "somefile" is done the customImporter methods are called first. Is this right?

    Question: In the method find_module :if the file is found in my PDS what should I return?

    Then in the load_module should I use imp .load_source to import the module?

    Below is the used code:

    class CustomImporter(object):

    global path_TA_dev_dict

    def find_module(self, fullname, path):

    if fullname in path_TA_dev_dict.keys(): #searching if the file is in PDS

    return self

    return None

    def load_module(self, fullname):
    if fullname in path_TA_dev_dict.keys():

    loader = imp.load_source(fullname, path_TA_dev_dict[fullname])

    raise ImportError(fullname)

    return loader

    1. If I understood correctly once you added customImporter in sys.meta_path every time an import "somefile" is done the customImporter methods are called first. Is this right?

      Yes, this is correct.

      Question: In the method find_module :if the file is found in my PDS what should I return?

      According to PEP 302, that method should return a module loader object which implements a "load_module" method. As the PEP suggests (as does my blog post), that load_module method can be in the same class as the find_module method, so simply "return self" should be sufficient.

      Then in the load_module should I use imp .load_source to import the module?

      That should work. Anything that returns a Python object - as that's all modules are as far as Python is concerned - should work, actually.

  2. Hey, it's louisthomas11. I know this is unrelated but, what is going on with OpenBlox?

    1. It's... complicated.

      Pretty sure you already know this, but SourceForge took down the main site's database about the start of this year due to it being bloated with spam. In addition to (obviously) taking the main site down, it also means the wiki and bug tracker cannot be edited, since they rely on that database for user information. Source code and stuff is still online, however.

      On top of that, I have a lot of studies and other things taking up my free time, and SourceForge has inexplicably not responded to my requests to give me the database back, so while not technically dead, OpenBlox is sort of dead in the water for probably the next 3-4 months, at least.

      Thanks for being patient with me - I know it's been frustrating with the site being down for so long like this, but I'll get it up and running as soon as I can.

  3. Thanks for the detailed instructions on using sys.meta_path .

    I've run into a small bug, with an easy fix. The method signature for find_module should be:

    def find_module(self, fullname, path=None)

    With 'path' as an optional argument. There are some code paths in pkgutil in py2.7 (at least) which call 'find_module' with a single argument, and so fall over if 'path' is required.

    1. Firstly, sorry for the late reply - college can be quite time-consuming! I've updated the article with the corrected method signature, and given you credit in the original post.