An evaluation of our new sourcing policy

← Back to all essays | Author's essays An evaluation of our new sourcing policy

by CriticalResist
Published: 2 (last update: 2024-06-16)
5-15 minutes

2

Read more

In the first days of February 2024, the ProleWiki "trusted" editorship (the editors who have been embedded long enough in the project that they receive the right to vote on internal matters) voted to implement a new sourcing policy at the behest of the administration. In less than a day, the new sorely-needed new policy went from a rough draft to a working document, to being implemented (implementation being the moment it goes live with all the underlying processes set up).

Now that this policy has been in place for some time, it is important to reflect back on it and compare it to what existed before. Feel free to skip sections that don't seem interesting to you, this document was written in such a way that it can be read in any order.

Background of the new policy

It is more important, at the start of a project, to focus on actually doing things rather than think about doing things. Hence ProleWiki was started on the basis that we would work out all the things that needed working out in due time -- the important part was to first write and publish content on it.

Our sourcing policy was thus not fleshed out for a time, and basically amounted to please source as much as possible, if you feel like it.

By sourcing, we mean adding external references to claims written on the encyclopedia so that readers can fact-check for themselves.

This general agile approach to setting up the project was successful, and it's not my intention here to downplay it. We have been growing consistently every year since 2020 but, reaching our fourth year in existence, several shortcomings appeared as we grew in terms of the editorship as well as content articles, and not just in sourcing.

In early 2024, we discovered that an editor whom we had agreed to let into the project despite our different tendencies had admitted to writing a completely unsourced article either as a joke or a troll to us. This prompted us to rework our sourcing policy which had never been looked at until then.

Implementing the new sourcing policy

In less than a day, we went from realizing the problem (we need to start getting people to source their contributions) to implementing it on the wiki. This was a big achievement, and it's important to realize these as well.

Truthfully, we had been in need of such a policy for quite a while, simply due to the growth ProleWiki experienced throughout 2023. While we do have an account request system with vetting questions to become an editor, we don't know truly know new editors until they have interacted with our community for a while.

Ultimately, it was decided that all edits, all additions on wiki pages, had to be sourced somehow. The only claims that would not need sources were "common sense" claims, such as names or dates, which we consider to be information so prevalent on the Internet, anyone can fact check for themselves.

But this was only step 2 of implementing the new policy, i.e. going from an idea to actually making it an actionable reality. When working on a project the size of ProleWiki, there are many more steps between the start and finish.

Once the idea was fleshed out in a way that could be implemented and be made actionable (owing to natural limits such as time, effort required, technical possibilities or capabilities, maintenance, and workload on the editorship), we also had to decide on several other questions which we quickly breezed through for the first run:

First, unsourced edits would be subject to instant removal. This was the "nuclear" option but would ensure that editors quickly integrate the new policy (and I say this as an editor that did not source often!). It also was not a killswitch per se; all edits to a page are saved in the page history, and can easily be accessed and reused.

Secondly, we had to make the entire editorship aware of the new policy, including yet-to-be editors. You see, it's certainly simple enough to send an email to every editor or ping them on our Discord server. However, future editors, who don't yet have an account, also need to be made aware of our policies. We had set up an onboarding page that gets sent out to every new account in 2023, and as such announce new changes there.

Thirdly, we had to set up the technical aspects of the new policy. MediaWiki, the software that powers ProleWiki, has a patrolling feature which we decided to repurpose for this. This needed to be activated and user permissions created so that we could create a patrolling group. The group also had to be staffed, making two different tasks.

Finally, the editorship had to vote on and validate this proposal.

All that to say: a whole host of things need to happen between the idea and implementation, and someone needs to make them happen. This is why I congratulate the editorship on being able to roll out this much-needed policy in less than a day, especially as it involved several people (as you might be aware, the more people involved in a project, the slower it moves).

Of course, the first version of a feature is never the last one. We believe in constant improvement, and whether that improvement is tiny or large, it remains an improvement. We have, since February, refined our sourcing policy here and there based on editor feedback. These changes aimed to reduce confusion on wording (especially on what has to or doesn't have to be sourced), and explain the "penalties".

We've actually only had to undo an edit once or twice. Most of the time, our patrollers notified the editor directly who sourced their edits very quickly.

Evaluating the impact of the new policy

When evaluating policies, we have to look at two very different segments: the editorship, and the readership. The two segments are formed of very different demographics, and have to be looked out in isolation from each other.

The challenge when enacting any new feature, policy, method, etc. on the wiki is working in a way that maximizes benefits for both groups at the same time, and the new sourcing policy was not spared.

On the editorship

The biggest expected challenge in regards to the editorship was to have this policy accepted and willingly followed by the editors. This was one reason we went for the killswitch option in regards to not respecting the policy -- undoing the edit, letting the editor

Remember that the old sourcing policy did not exist formally, and was basically add sources if you feel like it.

To some surprise, the transition from the old to new policy was completely frictionless. Like I said before, only a few edits had to be undone. Overall, we did not count many instances of the "existing" editorship having to be reminded to source their edits. We do sometimes have to remind newer editors to add references, but these are usually fixed quickly and are generally the result of forgetfulness in large edits.

There was some friction regarding the term "common sense". We allow common sense claims to remain unsourced, which was proposed at first just to keep some streamlined process to editing, and so that we wouldn't clog a page with dozens of useless references. This was only ever meant to apply to easily verifiable facts or, in other words, facts people can reasonably take for granted, such as somebody's full name or date of birth. This was sometimes applied by editors to mean what common sense usually means, i.e. "things everyone already knows". This is a problem of improperly defining terms, which creates confusion as everyone will have their own interpretation of it. We are slowly reaching a consensus on what common sense means and will soon redefine it to clear up any misunderstandings.

The editorship is otherwise generally happy with the new policy, especially as it reduces disagreements. We sometimes had editors disagreeing on whether a claim should appear on an article, as one believed A and another believed its opposite, B. An emergent situation with the new policy that we did not anticipate is that this reduces such discussions. If claim A is sourced, then it is not our claim specifically, but the claim of whoever we are quoting. Thus, editors can also add claim B and source it as well.

This also simplifies the process of correcting disagreements on the content of our pages. We have, at times, come across edits made by now-inactive editors who have become unresponsive or unreachable. Because these claims are unsourced, it is impossible for us to fact-check them, and usually leads us to removing them if we can't confirm them independently. This can happen on any claim, even something that seems as straightforward as pulling a quote from a book! With the new sourcing policy, we are able to confirm a claim much more easily. In a couple of cases, we found that the content written on the page was not necessarily reflected in the source; not that it was wrong necessarily, but that it was written in such a way that the page and the source differed slightly. Instead of having to remove the passage entirely since we can't find a source, we are able to instead correct the paragraph to bring it back in line with the source.

On the readership

We have yet to collect feedback from the readership due to the difficulty we face in forming our questions, as well as the fact that at the moment, the new sourcing policy only targets new edits made after the policy went in effect -- older unsourced edits are written down in a file to be looked over eventually. We would have to find readers who knew ProleWiki before and after the new sourcing policy, which is difficult when you can't target them specifically and can only send out a survey to anyone who sees it.

However, from earlier surveys, we know that many readers visit ProleWiki to find sources, either as books to read or topics to learn more on. As such, we can only assume more references can only be helpful to this large segment.

This assumption is further defended by the fact that ultimately, nobody would reasonably be upset by more references and sources. They are not obligated to look at them, and it ultimately makes us more trustworthy as our claims are not the product of whoever edited the page at this time (which the readers have no reason to be familiar with), but of someone else which they can check for themselves.

Overall evaluation

Overall, the new sourcing policy is bringing much more benefits than drawbacks. The editorship quickly integrated it in their workflow, and we have not received any negative responses from the readership -- on the contrary.

It is a net benefit for ProleWiki and while it was a very deep modification of how we worked up until then, it was very well-received and was very much needed at this stage. Ultimately, this new policy has made us more trustworthy and has brought our editorship closer together

Challenges to work out with the new policy

We are still faced with challenges from this new policy, and write them all down to be looked at eventually, usually in batches.