Jarvis

Back in 2022 I wanted to have LLMs as first-class citizens in my favourite note app. It was clear that they could make the editing, brainstorming and searching experience immensely better if they were integrated within my personal knowledge base. So I built Jarvis, and expanded it over the following months until it became a powerful assistant that can answer deep questions about my personal and professional life.

The first release went out on 20 November 2022, ten days before ChatGPT, and it was basic. Retrieval over notes came the following spring, and there have been releases every year since. It is the de facto standard AI assistant for Joplin.

Answering from my own notes, scoped here to the whole collection. The numbers are links into the notes it used.

Watch it

The Joplin team made their own tutorial video for Jarvis.

How it finds things

Chatting with your notes runs the question through this:

A block scores as its best match against any one sub-query, not an average, so a block that answers part of a question well is not diluted by the rest. The two rankings merge by position rather than by score, so a keyword hit and a vector hit can be compared without inventing a shared scale.

Reviewing the literature

The other thing I wanted early on was a literature review I could trust. ChatGPT in early 2023 would name real authors working on the right subject and then invent their papers.

So the command never asks a model for a review. It asks for research questions and search queries, runs those against Semantic Scholar, Crossref, Scopus and Springer, and Wikipedia alongside them, has the model judge each result for relevance, and only then writes the review from the papers it actually found, citing them. I wrote the pipeline up at the time.

That shape is a product category now, and the hosted versions of it are better than mine. In April 2023 it was not obvious.

A database that syncs

Joplin has a synced and secure note database, which can also be easily backed up. I wanted to leverage that by storing the vector database inside note metadata and syncing it across devices. This way, low-powered devices do not have to embed the entire notes corpus, and if you are paying an API for embeddings you pay once rather than once per device.

Jarvis runs on anything that exposes an OpenAI-compatible API, from Ollama, LM Studio or Jan on your own machine to OpenAI, Anthropic and Google, and it ships with a built-in embedding model. Several can coexist, each keeping its own vectors, so indexing on a phone with a small model does not disturb a laptop using a larger one. And when settings drift between devices, Jarvis reports how many notes are affected instead of quietly mixing two indexes.

To keep that metadata small, the vectors are quantised to 8 bits per value, a quarter of the size of float32 and in agreement with it to about 99%.

Doing this did expose one problem. Joplin stored a full copy of the metadata in every note revision, so editing a single word duplicated all of that note’s vectors into its history. I fixed that in Joplin itself. Debugging Jarvis also turned up a memory leak in Joplin’s plugin API that had been affecting plugins for years. I traced it to the callback registry, and a maintainer shipped the fix two days later.

Where it goes next

A shared, native, cross-plugin embedding index, built once and built well, would let every feature on top of it improve at once, rather than each one solving retrieval again for itself. I set that case out when Joplin was scoping its AI projects.

The other direction is agents. In 2025 I wanted mine to reach into the same second brain, so I built joplin-mcp, an MCP server that lets any agent read and write Joplin notes through a standard interface. It has become the most widely used MCP for Joplin, and the project is now building one into the core app as part of its new AI infrastructure.