Indexing Pipeline¶
This section is an overview of how to take a collection to a state in which it can be queried. This process is intentionally broken down into several steps, with a bunch of independent tools performing different tasks. This is because we want the flexibility of experimenting with each individual step without recomputing the entire pipeline.
External Resources¶
Raw Collection¶
The raw collection is a dataset containing the documents to index. A
collection is encoded in one of the supported
formats that stores a list of document
contents along with some metadata, such as URL and title. The
parse_collection tool takes a collection as an input and parses it to
a forward index (see Forward Index). See
Parsing for more details.
CIFF Index¶
This is an inverted index in the Common Index File Format. It can be converted to an uncompressed PISA index (more information below) with the ``ciff2pisa` <https://github.com/pisa-engine/ciff>`_ tool.
Forward Index¶
A forward index is the output of the parse_collection tool.
It represents each document as a list of tokens (terms) in the order of their appearance.
To learn more about parsing and the forward index format, see Parsing.
Inverted Index¶
An inverted index is the most fundamental structure in PISA. For each term in the collection, it contains a list of documents the term appears in. PISA distinguishes two types of inverted index.
Uncompressed / Binary Collection¶
The uncompressed index stores document IDs and frequencies as 4-byte integers.
It is an intermediate format between forward index and compressed inverted index.
It is obtained by running invert on a forward index.
To learn more about inverting a forward index, see Inverting.
Optionally, documents can be reordered with reorder-docids to obtain another
instance of uncompressed inverted index with different assignment of IDs to documents.
More on reordering can be found in Document Reordering.
Compressed¶
An uncompressed index is large and therefore before running queries, it must be compressed with one of many available encoding methods. It is this compressed index format that is directly used when issuing queries. See Compress Index to learn more.
WAND Data¶
This is a special metadata file containing additional statistics used during query processing. See Build additional data.
