# Getting started with tantivyr

    library(tantivyr)

`tantivyr` indexes text and searches it with BM25 ranking, structured filters, highlighting and incremental updates — locally, with no server. This vignette walks through the two ways of using it: the one-call convenience wrapper and the explicit schema API.

## The convenience layer: `tnt_index_df()`

Most of the time you have a data frame and want to search some of its columns. `tnt_index_df()` infers a schema, indexes every row and commits in one call.

    news <- data.frame(
      id     = 1:5,
      title  = c(
        "Orçamento público aprovado pelo congresso",
        "Reforma tributária avança no senado",
        "Nova lei de licitações entra em vigor",
        "Congresso debate orçamentos municipais",
        "Tribunal de contas analisa despesas"
      ),
      source = c("A", "B", "A", "C", "B"),
      year   = c(2022L, 2023L, 2024L, 2024L, 2023L)
    )

    idx <- tnt_index_df(
      news,
      text      = title,         # full-text column(s)
      filters   = c(source, year), # columns to filter / order on
      stemmer   = "portuguese",
      stopwords = TRUE
    )
    idx
    #>
    #> ── <tnt_index> (in-memory)
    #> 5 documents · 4 fields
    #> • id: i64
    #> • title: text [tnt_pt_stop]
    #> • source: text [raw]
    #> • year: i64

### Searching

`tnt_search()` returns a tibble with a `score` column followed by every stored field. Because we used the Portuguese stemmer, a search for orçamento also matches orçamentos.

    tnt_search(idx, "orçamento")
    #> # A tibble: 2 × 5
    #>   score    id title                                     source  year
    #>   <dbl> <dbl> <chr>                                     <chr>  <dbl>
    #> 1 0.893     1 Orçamento público aprovado pelo congresso A       2022
    #> 2 0.893     4 Congresso debate orçamentos municipais    C       2024

### Filtering

Filters can be written as ordinary R comparisons. They are combined with the text query.

    tnt_search(idx, "", filter = year >= 2024)
    #> # A tibble: 2 × 5
    #>   score    id title                                  source  year
    #>   <dbl> <dbl> <chr>                                  <chr>  <dbl>
    #> 1     1     3 Nova lei de licitações entra em vigor  A       2024
    #> 2     1     4 Congresso debate orçamentos municipais C       2024

    tnt_search(idx, "congresso", filter = source == "A")
    #> # A tibble: 1 × 5
    #>   score    id title                                     source  year
    #>   <dbl> <dbl> <chr>                                     <chr>  <dbl>
    #> 1  1.77     1 Orçamento público aprovado pelo congresso A       2022

    tnt_search(idx, "", filter = year %in% c(2022, 2024), limit = 10)
    #> # A tibble: 3 × 5
    #>   score    id title                                     source  year
    #>   <dbl> <dbl> <chr>                                     <chr>  <dbl>
    #> 1 1.39      1 Orçamento público aprovado pelo congresso A       2022
    #> 2 0.875     3 Nova lei de licitações entra em vigor     A       2024
    #> 3 0.875     4 Congresso debate orçamentos municipais    C       2024

You can also pass a raw Tantivy query string for anything the helpers do not cover:

    tnt_search(idx, "", filter = "year:[2023 TO *] AND source:B")
    #> # A tibble: 2 × 5
    #>   score    id title                               source  year
    #>   <dbl> <dbl> <chr>                               <chr>  <dbl>
    #> 1 0.875     5 Tribunal de contas analisa despesas B       2023
    #> 2 0.875     2 Reforma tributária avança no senado B       2023

### Highlighting and ordering

    tnt_search(idx, "congresso", highlight = title)$title_snippet
    #> [1] "Orçamento público aprovado pelo <b>congresso</b>"
    #> [2] "<b>Congresso</b> debate orçamentos municipais"

    tnt_search(idx, "", order_by = year, desc = TRUE)[, c("title", "year")]
    #> # A tibble: 5 × 2
    #>   title                                      year
    #>   <chr>                                     <dbl>
    #> 1 Nova lei de licitações entra em vigor      2024
    #> 2 Congresso debate orçamentos municipais     2024
    #> 3 Tribunal de contas analisa despesas        2023
    #> 4 Reforma tributária avança no senado        2023
    #> 5 Orçamento público aprovado pelo congresso  2022

### Counting

`tnt_count()` returns the total number of matches, ignoring any limit.

    tnt_count(idx, "congresso")
    #> [1] 2
    tnt_count(idx, "", filter = year == 2024)
    #> [1] 2

## The explicit layer: schemas and persistence

For full control over how each field is stored, indexed and analysed, build a schema with `tnt_schema()` and the `tnt_*()` field constructors, then manage the index yourself.

    sch <- tnt_schema(
      id    = tnt_i64(),
      slug  = tnt_text(stemmer = "raw"),                 # exact key for updates
      title = tnt_text(stemmer = "portuguese", stored = TRUE),
      body  = tnt_text(stemmer = "portuguese"),
      date  = tnt_date(fast = TRUE)
    )

    path <- tempfile()
    idx <- tnt_index(path, schema = sch)

Add documents and commit to make them searchable. Operations return the index invisibly, so they pipe.

    docs <- data.frame(
      id    = 1:2,
      slug  = c("edital-001", "edital-002"),
      title = c("Edital de licitação 001", "Edital de licitação 002"),
      body  = c("Aquisição de equipamentos de informática.",
                "Contratação de serviços de limpeza."),
      date  = as.Date(c("2024-02-01", "2024-03-15"))
    )

    idx |> tnt_add(docs) |> tnt_commit()
    tnt_num_docs(idx)
    #> [1] 2

### Incremental updates and deletes

`tnt_update()` replaces documents by a key column; `tnt_delete()` removes them. Both need a commit to take effect.

    idx |>
      tnt_update(
        data.frame(id = 1L, slug = "edital-001",
                   title = "Edital de licitação 001 (retificado)",
                   body = "Aquisição de notebooks.",
                   date = as.Date("2024-02-10")),
        by = slug
      ) |>
      tnt_commit()

    tnt_search(idx, "notebooks")[, c("id", "title")]
    #> # A tibble: 1 × 2
    #>      id title
    #>   <dbl> <chr>
    #> 1     1 Edital de licitação 001 (retificado)

    idx |> tnt_delete(slug == "edital-002") |> tnt_commit()
    tnt_num_docs(idx)
    #> [1] 1

### Reopening an index

On-disk indexes survive across sessions. Call `tnt_index()` with just the path to reopen — the schema is restored automatically.

    reopened <- tnt_index(path)
    tnt_num_docs(reopened)
    #> [1] 1

## Where to go next

- `?tnt_search` documents every search option.
- `?tnt_field` lists the field types and their stemming/stop-word options.
- `tnt_stemmers()` returns the supported languages.