Short version of the situation is that I have an old site I frequent for user written stories. The site is ancient (think early 2000’s), and has terrible tools for sorting and searching the stories. Half of the time, stories disappear from author profiles. Thousands of stories and you can only sort by top, new, and 30-day top.
I’m in the process of programming a scraper tool so I can archive the stories and give myself a library to better find forgotten stories on the site. I’ll be storing tags, dates, authors, etc, as well as the full body of the text.
Concerning the data, there are a few thousand stories- ascii only, and various data points for each story with the body of many stores reaching several pages long.
Currently, I’m using Python to compile the data and would like to know what storage solution is ideal for my situation. I have a little familiarity with SQL, json, and yaml, but not enough to know what might be best. I am also open to any other solutions that work well with Python.
I do like the sound of that.
I’m not too worried about performance, since, once everything is running, most of the operations will only be ran every few weeks or so. Don’t want it slowing to a crawl for sure though.
The text search looks promising. I’ve had the idea of automating “likely tags” that look for keywords (sword = fantasy while spaceship = sci-fi). It’s not perfect, but it could be useful to roughly categorize all the stories that are missing tags.
An alternative could be to use something like postgres with the pgvector extension to do semantic searches instead of just text-based searches. You can generate embeddings for the text content of the story, then do the same for “sci-fi” or something, and see if searching that way gets you most of the way there.
Generating embeddings locally might take some time though if you don’t have hardware suitable for it.
Is there anything Postgres doesn’t do?
I would say run Doom, but I’m not confident in that. At the very least, Skyrim hasn’t been rereleased on it yet.