Plugins¶

foliq is a small engine with sockets. Every socket can be filled by a built-in plugin, your own Python class, or your own Rust code.

Socket	Does	Built in
Source	lists files (with versions) and reads them	folders
Parser	turns bytes into Markdown	Markdown, HTML, DOCX, PPTX, XLSX, PDF
Splitter	cuts a document into chunks	one chunk per section
Store	saves and searches documents and chunks	SQLite with full-text search
Sink	receives every change after a sync	text mirror, JSON Lines

Python plugins work through foliq.Index. The foliq command runs Rust code only, so it uses the built-in plugins and Rust plugins compiled into it.

Python¶

A plugin is any object with the right method. There is nothing to inherit.

import csv
import io

import foliq


class CsvParser:
    name = "csv"
    version = "1"           # change it to re-parse every CSV file
    extensions = ("csv",)   # tried before the built-in parsers

    def parse(self, data: bytes, path: str) -> str:
        rows = list(csv.reader(io.StringIO(data.decode())))
        lines = ["| " + " | ".join(row) + " |" for row in rows]
        lines.insert(1, "|" + " --- |" * len(rows[0]))
        return "\n".join(lines)  # or {"text": ..., "title": ..., "metadata": {...}}


class ParagraphSplitter:
    def split(self, text: str) -> list[str]:
        # Return pieces of the text, in order. foliq adds the lines, pages and headings.
        return [paragraph for paragraph in text.split("\n\n") if paragraph.strip()]


class PrintSink:
    def apply(self, changes: list[dict]) -> None:
        for change in changes:
            # {"type": "upsert", "document": {...}, "chunks": [...]}
            # {"type": "delete", "document_id": ..., "collection": ..., "path": ...}
            print(change["type"], change.get("document", change).get("path"))


index = foliq.Index(parsers=[CsvParser()], splitter=ParagraphSplitter(), sinks=[PrintSink()])

Parsers run in parallel threads; foliq takes the GIL for each call. An exception in a plugin fails only that file, and the sync report says why.

Rust¶

Rust plugins implement a trait from the foliq-core crate and are added with Index::builder:

//! A Rust plugin: a parser for `.csv` files, used by an index.
//!
//! Run it with `cargo run -p foliq --example custom_parser -- <folder>`.

use std::sync::Arc;

use foliq::{Index, Parsed, Parser, Result, SearchOptions, SourceEntry};

/// Turns CSV files into Markdown tables.
struct CsvParser;

impl Parser for CsvParser {
    fn name(&self) -> &str {
        "csv"
    }

    fn extensions(&self) -> Vec<String> {
        vec!["csv".to_string()]
    }

    fn parse(&self, bytes: &[u8], _entry: &SourceEntry) -> Result<Parsed> {
        let text = String::from_utf8_lossy(bytes);
        let mut markdown = String::new();
        for (number, line) in text.lines().enumerate() {
            let cells: Vec<&str> = line.split(',').collect();
            markdown.push_str(&format!("| {} |\n", cells.join(" | ")));
            if number == 0 {
                markdown.push_str(&format!("|{}\n", " --- |".repeat(cells.len())));
            }
        }
        // Return Markdown: foliq finds the headings, tables and pages in it.
        Ok(Parsed::markdown(markdown))
    }
}

fn main() -> Result<()> {
    let folder = std::env::args().nth(1).unwrap_or_else(|| ".".to_string());
    let index = Index::builder(".foliq").parser(Arc::new(CsvParser)).open()?;
    index.add_folder(folder.as_ref(), None, Vec::new())?;
    let report = index.sync()?;
    println!("{} added, {} updated", report.added, report.updated);
    for result in index.search("price", &SearchOptions::default())? {
        println!("{}", result.citation);
    }
    Ok(())
}

The traits are small: see crates/foliq-core/src. Coming next: Rust plugins packaged as Python wheels, so pip install foliq-something adds a fast plugin to foliq.Index.