Apologies if crossposting is against the rules; I’m not entirely sure where the lines are drawn here yet.

I posted this in lemmy.fosscad (fosscad@lemmy?), but realize that may not be the most active venue.

I grabbed archives of fosscad and took a look at the contents of the zst’s. I think I could probably rebuild the contents of the subreddit in some manner or another; the question is scale and hosting. How would we make the posts easily searchable, where would they live, what endpoint can we upload hundreds of thousands of comments into in a reasonable time frame… all that fun stuff.

The archives don’t contain pictures, but contain links to the pictures and the ones I’ve checked are currently still live (meaning the pics are still hosted on reddit). Dunno how long that will remain the case.

I have no idea what the size of the archives would be with pics downloaded; gigs, a TB, no clue. I’m posting this to gauge public interest and I haven’t done much preliminary work (oh, these are json. Yep, dictionaries work. Wingo.)

Is there any interest in making this more publicly available? I’ve run into an issue with a particular build and I’ll be diving through the archives to fix it for my self. It seems like a shame that all this information would be inaccessible to everyone who isn’t able or interested in trawling through their own local archives.

I’m not a programmer by trade, but work in an adjacent space. I can plink along on this if other people are interested (and if anyone is interested enough to help pitch in, even better).

  • hoglegOPEnglish
    5·
    2 months ago

    Reposting here in case it gets lost in the sauce:

    There should be more; they may be split into several differential files. I’m going to work on getting the others right now, but I’ve got limited time before I have to leave for work. If I can’t get them up in time, I’ve been using resources like https://academictorrents.com/details/ba051999301b109eab37d16f027b3f49ade2de13/tech&filelist=1 (if url’s can’t be posted, it’s academictorrents dot com, posts by Watchful1) I could have sworn I grabbed newer data for the fosscad subreddit specifically from a different site, but may have to go through the monthly diffs in this link and pull out anything under the /fosscad id.

    These are much bigger, since they’re the entirety of reddit (or top 20,000 subs, something like that), so download and parse is going to take a lot longer. Several hundred gigs to pull and parse. On review, no way I can do it all this morning but I can at least get some downloads cooking. Going to start pulling comments/subs for 2023, but won’t be able to check on any of this until roughly 6pm eastern.