Unobtanium Search
Table of Contents
What is Unobtanium?
- Status
- Live and working
- Language
- Rust
- Sourcecode
- https://codeberg.org/unobtanium/unobtanium
- Started
- 2023-05-27
- Docs
- https://doc.unobtanium.rocks/
- Try it
- https://unobtanium.rocks
A stand alone and independent search engine with its own crawler.
Unobtanium is a stand alone web search engine for making a searchbox for your own little corner of the web. It tries to be easy to self-host and maintain.
It doesn't independent on external services and uses SQLite as its database.
Why SQLite?: SQLite is great because it doesn't require most of the typical database administration chores, it is simply a file that is managed by unobtanium that requires no special tools for backing it up, moving, etc.
Under the hood there is a three step pipeline of crawling, summarising and serving the search frontend. (No feedback loop yet …, though the crawler can remember the last crawl)
Th search algorithm is currently a pretty simple "bag of words" type, ranked using bm25 and a custom score that prefers leaf pages over navigation pages.
For more information please refer to the technical documentation as it is likely more up to date.
unobtanium.rocks
unobtanium.rocks is the "official" instance that is also a scalability experiment, it has an index focused on small websites, technology, the hacking community (CCC, Freifunk, hackspaces, FabLabs, …), independent wikis, and some other topics in between.
The index configuration is public and open to contributions.
I want to selfhost!
Great! Though there are a few things to consider …
While unobtanium tries to be as easy to use as possible, there are some challenges that are inherent to how the modern web works that one should be aware of. For just a few sites that don't change very often (i.e. a blog) these are almost nonexistent, but they grow with your index and how dynamic the websites are.
How can I help?
You want to help? Great!
You want to write rust code?
- Help unobtanium on Codeberg it is made of one big repository and multiple smaller parts that help with various aspects of the search engine, any improvement there can almost directly improve unobtanium as a whole.
- Help with the dependencies! Unobtanium depends on a lot of external crates to do its job. Let me know if you contributed to one because of this 😃.
- Help with code documentation: Software maintainers are Humans too, they sometimes forget or don't have enough time for good documentation, any improvement there is welcome, even if you "just" edited some comments!
Code reading and markdown?
How it Started
Unobtanium started in May of 2023 as a simple web crawler out of curiosity and boredom.
The original plan was to create a dead link checker, once that worked the scope was expanded to a simple search application, first with Meilisearch as a backend (around September of 2023), later with SQLite fts5.