Short version of the situation is that I have an old site I frequent for user written stories. The site is ancient (think early 2000’s), and has terrible tools for sorting and searching the stories. Half of the time, stories disappear from author profiles. Thousands of stories and you can only sort by top, new, and 30-day top.
I’m in the process of programming a scraper tool so I can archive the stories and give myself a library to better find forgotten stories on the site. I’ll be storing tags, dates, authors, etc, as well as the full body of the text.
Concerning the data, there are a few thousand stories- ascii only, and various data points for each story with the body of many stores reaching several pages long.
Currently, I’m using Python to compile the data and would like to know what storage solution is ideal for my situation. I have a little familiarity with SQL, json, and yaml, but not enough to know what might be best. I am also open to any other solutions that work well with Python.
SQL is designed for querying (it’s a query language lol). If the stories are huge, you can save them to individual files and store the filepath in the database, but otherwise it can hold columns with a fair amount of data if needed.
You can probably get away with using sqlite. A more traditional database would be postgres, but it sounds like you just need the database available locally.
It’s definitely possible to store the stories in columns, but there’s also very little reason to do it. I think filepath in SQL and the stories in separate files in whatever format makes the most sense (html, txt, epub). If you ever want to search the stories for keywords, write a python script to build indexes in SQL, performs much better than doing LIKE on a maxed out varchar column.
I was thinking maybe Elastisearch, but I don’t know how much work that is to set up. For a hobby project, writing your own indexer isn’t too hard and might be more fun and easier to maintain than an industry-grade solution.