been dealing with this at work lately and figured id ask here. we have a few vendor websites where the terms change without notice — pricing pages, SLAs, privacy policies. twice now weve had situations where a vendor changed something and when we pushed back they just said “its always been that way”

started looking into ways to automatically screenshot or archive pages on a schedule. the wayback machine is great but its not on-demand and sometimes misses updates for weeks.

tried a few approaches:

  • wget + cron to save full HTML. works but comparing diffs is painful and you lose the visual layout

  • browser extension that saves pages manually. doesnt scale, forgets to run it

  • https://snapshotarchive.com/ — this one actually does scheduled screenshots and keeps a visual timeline. been testing it for about a month on ~20 pages. the diff view showing whats changed between captures is what sold me honestly

the visual comparison part matters more than I expected. a text diff misses layout changes, moved sections, removed buttons etc. when you screenshot at regular intervals you catch everything

anyone else doing something similar? curious what tools or workflows people use for this. especially interested if theres a good self-hosted option I havent found yet

  • hendrik@palaver.p3x.de
    link
    fedilink
    English
    arrow-up
    20
    ·
    edit-2
    3 days ago

    I have used changedetection.io for a while. It can parse the pages and do notifications, also launch a headless browser and do screenshots. And from what I see they also seem to have added a proper open-source license a while ago. (On the downside they also added a lot of AI stuff which wasn’t there when I used it.) But I also had some minor hiccups. Not sure it’s good enough for important legal stuff.

    • cybermonkey@piefed.zip
      link
      fedilink
      English
      arrow-up
      5
      ·
      3 days ago

      I use that as well. Some websites don’t work well with it and for that i use WebCopy. As for single page capture, i use Singlefile addon for Firefox. There’s also another version which makes it compress to reduce the file’s size by the same developer.

  • wraithcoop@programming.dev
    link
    fedilink
    arrow-up
    11
    ·
    3 days ago

    My company does this, without getting into too much detail we save the page source as it is and also open the page up in a headless chrome and take a screenshot. It sucks but there’s not really a better way to get an accurate visual without just using a browser

  • AnAmericanPotato@programming.dev
    link
    fedilink
    English
    arrow-up
    6
    ·
    3 days ago

    I’ve never found a single tool that works on 100% of pages. The best I’ve found is the SingleFile browser extension. It is very good at preserving what is currently in the DOM (including what’s scrolled off screen), but it can’t handle dynamically loading content that disappears out of the DOM when scrolling.