Python bindings for Xapian (2023)

The Python bindings for Xapian are packaged in the xapian module,and largely follow the C++ API, with the following differences andadditions. Python strings and lists, etc., are converted automaticallyin the bindings, so generally it should just work as expected.

The examples subdirectory contains examples showing how to use thePython bindings based on the simple examples from,,'s also simplematchdecider.pywhich shows how to define a MatchDecider in Python.

The Python bindings come with a test suite, consisting of two test and These are run by the"make check" command, or may be run manually. By default, theywill display the names of any tests which failed, and then display a count oftests which run and which failed. The verbosity may be increased by settingthe "VERBOSE" environment variable: a value of 1 will displaydetailed information about failures, and a value of 2 will display furtherinformation about the progress of tests.


Xapian exceptions are translated into Python exceptions with the same names and inheritance hierarchy as the C++ exception classes. The base class of all Xapian exceptions is the xapian.Error class, and this in turn is a child of the standard python exceptions.Exception class.

This means that programs can trap all xapian exceptions using "except xapian.Error", and can trap all exceptions which don't indicate that the program should terminate using "except Exception".


The xapian Python bindings accept unicode strings as well as simple strings (ie, "str" type strings) at all places in the API which accept string data. Any unicode strings supplied will automatically be translated into UTF-8 simple strings before being passed to the Xapian core. The Xapian core is largely agnostic about character encoding, but in those places where it does process data in a character encoding dependent way it assumes that the data is in UTF-8. The Xapian Python bindings always return string data as simple strings.

Therefore, in order to avoid issues with character encodings, you should always pass text data to Xapian as unicode strings, or UTF-8 encoded simple strings. There is, however, no requirement for simple strings passed into Xapian to be valid UTF-8 encoded strings, unless they are being passed to a text processing routine (such as the query parser, or the stemming algorithms). For example, it is perfectly valid to pass arbitrary binary data in a simple string to the xapian.Document.set_data() method.

(Video) Add a search engine to your application using Xapian

It is often useful to normalise unicode data before passing it to Xapian - Xapian currently has no built-in support for normalising unicode representations of data. The standard python module "unicodedata" provides support for normalising unicode: you probably want the "NFKC" normalisation scheme: in other words, use something like

 unicodedata.normalize('NFKC', u'foo')

to normalise the string "foo" before passing it to Xapian.


The iterator classes in the Xapian C++ API are wrapped in a "Pythonic" style.The following are supported (where marked as default iterator, it means__iter__() does the rightthing so you can for instance use for term in document toiterate over terms in a Document object):

ClassMethodEquivalent toIterator type
MSetdefault iteratorbegin()MSetIter
ESetdefault iteratorbegin()ESetIter
Querydefault iteratorget_terms_begin()TermIter
Databaseallterms() (also as default iterator)allterms_begin()TermIter
Databasepositionlist(docid, tname)positionlist_begin(docid, tname)PositionIter
Documenttermlist() (also as default iterator)termlist_begin()TermIter

The pythonic iterators generally return Python objects, with propertiesavailable as attribute values, with lazy evaluation where appropriate. Anexception is the PositionIter object returned byDatabase.positionlist, which returns an integer.

The lazy evaluation is mainly transparent, but does become visible in one situation: if you keep an object returned by an iterator, without evaluating its properties to force the lazy evaluation to happen, and then move the iterator forward, the object may no longer be able to efficiently perform the lazy evaluation. In this situation, an exception will be raised indicating that the information requested wasn't available. This will only happen for a few of the properties - most are either not evaluated lazily (because the underlying Xapian implementation doesn't evaluate them lazily, so there's no advantage in lazy evaluation), or can be accessed even after the iterator has moved. The simplest work around is simply to evaluate any properties you wish to use which are affected by this before moving the iterator. The complete set of iterator properties affected by this is:

  • Database.allterms (also accessible as Database.__iter__): termfreq
  • Database.termlist: termfreq and positer
  • Document.termlist (also accessible as Document.__iter__): termfreq and positer
  • Database.postlist: positer

In older releases, the pythonic iterators returned lists representing theappropriate item when their next() method was called. These wereremoved in Xapian 1.1.0.

Non-Pythonic Iterators

Before the pythonic iterator wrappers were added, the python bindings providedthin wrappers around the C++ iterators. However, these iterators don't behavelike most iterators do in Python, so the pythonic iterators were implemented toreplace them. The non-pythonic iterators are still available to allow existingcode to continue to work, but they're now deprecated and we plan to remove themin Xapian 1.3.0. The documentation below is provided to aid migration away fromthem.

(Video) Building a Xapian index of Wikipedia in less time than this talk takes

All non-pythonic iterators support next() and equals() methods to move through and test iterators (as for all language bindings). MSetIterator and ESetIterator also support prev(). Python-wrapped iterators also support direct comparison, so something like:

 m=mset.begin() while m!=mset.end(): # do something

C++ iterators are often dereferenced to get information, eg (*it). With Python these are all mapped to named methods, as follows:

IteratorDereferencing method

Other methods, such as MSetIterator.get_document(), are available unchanged.


MSet objects have some additional methods to simplify access (these work using the C++ array dereferencing):

Method nameExplanation
get_hit(index)returns MSetItem at index

Additionally, the MSet has a property, mset.items, which returns alist of tuples representing the MSet. This is now deprecated - please use theproperty API instead (it works in Xapian 1.0.x too). The tuple members and theequivalent property names are as follows:

IndexProperty nameContents
xapian.MSET_DIDdocidDocument id
xapian.MSET_PERCENTpercentPercentage weight
xapian.MSET_DOCUMENTdocumentDocument object (Note: this member of the tuple was never actually set!)

Two MSet objects are equal if they have the same number and maximum possiblenumber of members, and if every document member of the first MSet exists at thesame index in the second MSet, with the same weight.


The ESet has a property, eset.items, which returns a list oftuples representing the ESet. This is now deprecated - please use theproperty API instead (it works in Xapian 1.0.x too). The tuple members and theequivalent property names are as follows:

(Video) Apt_Xapian_Index___Enrico_Zini

IndexProperty nameContents
xapian.ESET_TNAMEtermTerm name

Non-Class Functions

The C++ API contains a few non-class functions (the Database factoryfunctions, and some functions reporting version information), which arewrapped like so for Python:

    • Xapian::version_string() is wrapped as xapian.version_string()
    • Xapian::major_version() is wrapped as xapian.major_version()
    • Xapian::minor_version() is wrapped as xapian.minor_version()
    • Xapian::revision() is wrapped as xapian.revision()
    • Xapian::Auto::open_stub() is wrapped as xapian.open_stub()
    • Xapian::Brass::open() is wrapped as xapian.brass_open()
    • Xapian::Chert::open() is wrapped as xapian.chert_open()
    • Xapian::Flint::open() is wrapped as xapian.flint_open()
    • Xapian::InMemory::open() is wrapped as xapian.inmemory_open()
    • Xapian::Remote::open() is wrapped as xapian.remote_open() (boththe TCP and "program" versions are wrapped - the SWIG wrapper checks the parameter list todecide which to call).
    • Xapian::Remote::open_writable() is wrapped as xapian.remote_open_writable() (boththe TCP and "program" versions are wrapped - the SWIG wrapper checks the parameter list todecide which to call).


In C++ there's a Xapian::Query constructor which takes a query operator and start/end iterators specifying a number of terms or queries, plus an optional parameter. In Python, this is wrapped to accept any Python sequence (for example a list or tuple) to give the terms/queries, and you can specify a mixture of terms and queries if you wish. For example:

 subq = xapian.Query(xapian.Query.OP_AND, "hello", "world") q = xapian.Query(xapian.Query.OP_AND, [subq, "foo", xapian.Query("bar", 2)])

MatchAll and MatchNothing

As of 1.1.1, these are wrapped as xapian.Query.MatchAll andxapian.Query.MatchNothing.


Custom MatchDeciders can be created in Python; simply subclassxapian.MatchDecider, ensure you call the super-constructor, and define a__call__ method that will do the work. The simplest example (which does nothinguseful) would be as follows:

class mymatchdecider(xapian.MatchDecider): def __init__(self): xapian.MatchDecider.__init__(self) def __call__(self, doc): return 1


The ValueRangeProcessor class (and its subclasses) provide an operator() method(which is exposed in python as a __call__() method, making the class instancesinto callables). This method checks whether a beginning and end of a range arein a format understood by the ValueRangeProcessor, and if so, converts thebeginning and end into strings which sort appropriately. ValueRangeProcessorscan be defined in python (and then passed to the QueryParser), or there areseveral default built-in ones which can be used.

Unfortunately, in C++ the operator() method takes two std::string arguments byreference, and returns values by modifying these arguments. This is notpossible in Python, since strings are immutable objects. Instead, in thePython implementation, when the __call__ method is called, the resulting valuesof these arguments are returned as part of a tuple. The operator() method inC++ returns a value number; the return value of __call__ in python consists ofa 3-tuple starting with this value number, followed by the returned "begin"value, followed by the returned "end" value. For example:

 vrp = xapian.NumberValueRangeProcessor(0, '$', True) a = '$10' b = '20' slot, a, b = vrp(a, b)

Additionally, a ValueRangeProcessor may be implemented in Python. The Pythonimplementation should override the __call__() method with its ownimplementation, and, again, since it cannot return values by reference, itshould return a tuple of (value number, begin, end). For example:

(Video) Perl 6 NativeCall Screencast

 class MyVRP(xapian.ValueRangeProcessor): def __init__(self): xapian.ValueRangeProcessor.__init__(self) def __call__(self, begin, end): return (7, "A"+begin, "B"+end)

Apache and mod_python/mod_wsgi

By default, both mod_python and mod_wsgi use a separate sub-interpreter foreach application. However, Python's sub-interpreter support is incompatiblewith the simplified GIL state API which SWIG-generated Python bindings useby default. So to avoid deadlocks, you need to tell mod_python and mod_wsgito run applications which use Xapian in the main interpreter as detailed below.

This restriction could be removed - the details are in Xapian's bugtracker - seeticket #364 for details ofwhat needs doing.


You need to set this option in the Apache configuration section for allmod_python scripts which use Xapian:

PythonInterpreter main_interpreter

You may also need to use Python >= 2.4 (due to problems in Python2.3 with the APIs the code uses).


You need to set theWSGIApplicationGroup option like so:

WSGIApplicationGroup %{GLOBAL}

The mod_wsgi documentation also discusses this issue.

Last updated $Date: 2011-02-21 14:48:22 +0000 (Mon, 21 Feb 2011) $


1. Ubuntu 11.04 "Unable to locate package xapian-core" (3 Solutions!!)
(Roel Van de Paar)
2. EmacsConf 2019 - 10 - notmuch new(s) - David Bremner
(EmacsConf and Emacs hangouts)
3. Python | How to open an image | PIL & Matplotlib
4. SOLVED : ERROR: Could not find a version that satisfies the requirement python-opencv
(Problem Solving Point)
5. How to build and install BLFS 9.1 - Part 20: Security - make-ca dependency: Libxml2 python module
6. 11 Lightning Talks
(DebConf Videos)


Top Articles
Latest Posts
Article information

Author: Kerri Lueilwitz

Last Updated: 08/27/2023

Views: 5717

Rating: 4.7 / 5 (67 voted)

Reviews: 82% of readers found this page helpful

Author information

Name: Kerri Lueilwitz

Birthday: 1992-10-31

Address: Suite 878 3699 Chantelle Roads, Colebury, NC 68599

Phone: +6111989609516

Job: Chief Farming Manager

Hobby: Mycology, Stone skipping, Dowsing, Whittling, Taxidermy, Sand art, Roller skating

Introduction: My name is Kerri Lueilwitz, I am a courageous, gentle, quaint, thankful, outstanding, brave, vast person who loves writing and wants to share my knowledge and understanding with you.