Stack Overflow Changes Data Dump Process
Philippe (via Hacker News):
I’m going to start with an important statement: this is primarily only a change in location for where the data dump is accessed. Moving forward, we’ll be providing the data dump from a section of the site user profile on a Stack Exchange profile.
There are a number of reasons for this: first, this is an attempt to put commercial pressure on LLM manufacturers to join us and our existing partners in the “socially responsible AI“ usage that we’re advocating for - to get them to give back to the communities whose data they consume.
Second, we want to help make the process of accessing data dumps quicker and more efficient. While Archive.org has been a great partner to us, as you may know, both internally and externally, people have encountered challenges with uploading and downloading the dumps with any reasonable speed.
[…]
We are requiring that all partners in socially responsible AI comply with the CC BY-SA attribution requirements, attributing content to the community members who contributed it.
They will no longer be uploading the data dump to archive.org, reducing redundancy.
At best, this is extremely inconvenient; at worst, it guarantees no one will ever again have a consistent “dump”.
I’m going to guess: no one involved in making this decision has ever downloaded and worked with the full data dump. It’s already slow and fairly inconvenient; the one bright spot is that a decent torrent client lets you start it and do other stuff while waiting. Best-case, you devote a fast enough pipe to this that the hundreds of extra clicks necessary are rewarded with shorter turnaround… But somehow, I doubt it.
You have been engaging on this topic disingenuously for a year.
It was your intention to turn off the dumps a year ago, and now you're trying to make them as inconvenient as possible.
You are making it very easy to pull access to our own content that brings you profit. Even if we trusted the company now, this would make it not just possible, but trivial, for some future nefarious company leadership to backstab the community. And guess what: we already have the nefarious company leadership in the present.
Just over a year ago when I was still staff at the company, I was personally in the unenviable position of having been instructed by the Stack Overflow CEO to disable the Data Dump, and to not re-enable it because he wanted to end the dump. That decision ultimately snowballed until Stack Overflow made commitments to continue the data dump quarterly. Data Superstar Aaron ultimately made some improvements and there was a shift made to the delivery schedule, to make it align better with quarterly boundaries. This is all excellent news for those of us who use the data dumps, and/or are proponents for equal data, and/or are defenders of the open data commitments made by and for the community.
Now, just one quarter after the company’s most recent commitment to a schedule, it’s shifting, again. For no reason. Apparently undoing the most recent schedule-shift by bumping (at least) a month.
How do you plan to enforce “I agree that I will use this file for non-commercial use. I will not use it for any other purpose, and I will not transfer it to others without permission from Stack Overflow.” when the CC BY-SA license explicitly forbids adding downstream restrictions?
Previously:
