Welcome to PISA

Performant Indexes and Search for Academia

Description

PISA is a text search engine able to run on large-scale collections of documents. It allows researchers to experiment with state-of-the-art techniques, allowing an ideal environment for rapid development.

Some features of PISA are listed below:

• Written in C++ for performance;
• Indexing & Parsing & Sharding capabilities;
• Many index compression methods implemented;
• Many query processing algorithms implemented;
• Implementation of document reordering;
• Free and open-source with permissive license;

Note

PISA is still in its initial release and many new features are going to come in the next versions. Contributions are also welcome!

Contents:

• Getting Started
  • Building the code
  • Run unit tests
• Parsing
  • Generating mapping files
  • Supported stemmers
  • Supported formats
• Inverting
  • Inverted index format
    • Reading the inverted index using Python
• Sharding
  • partition_fwd_index
  • invert-shards.sh
  • compress-shards.sh
• Compress Index
  • Usage
  • Compression Algorithms
    • Binary Interpolative Coding
    • Elias-Fano
    • MaskedVByte
    • OptPFD
    • Partitioned Elias Fano
    • QMX
    • SIMD-BP128
    • Simple8b
    • Simple16
    • StreamVByte
    • Varint-G8IU
    • VarintGB
• Query an index
  • Usage
  • Build additional data
  • Query algorithms
    • AND
    • OR
    • MaxScore
    • WAND
    • BlockMax WAND
    • BlockMax MaxScore
    • Variable BlockMax WAND
• Document Reordering
  • Document name & URL based ordering
    • 1. Mapping file creation
    • 2. Index remapping
  • Random Ordering
  • Recursive Graph Bisection
    • Description
    • Usage