Technology

The Network Neighborhood icon, appearing in Windows 95 and NT4, allowed people to share files between computers connected on a home network. At universities and LAN parties, people used to make copies of the videos they watched, long before the growth of large video sharing websites. YouTube did so well during the video hosting wars that it was purchased by Google. Being a massive monolith of a video sharing website, it’s also rife with censorship. I archive videos that I find interesting, which are also at high risk of disappearing. I wrote a program called Youboot to scan my collection of downloaded videos, and tell me which ones are no longer available and the reason they were removed. The best offense, in the war against thought-crime, is to watch the videos the powerful want to burn.

For years, I used the tools youtube-dl and it’s currently maintained fork, yt-dlp, as a Swiss Army Knife for downloading videos from websites. I’ve called its Python API directly from tools I’ve written to archive content from Locals and The Blaze. Years ago I found an excellent wrapper script for archiving videos. Over the years I’ve modified it, and the current form is a fish shell function that saves metadata and subtitles (both user created and auto-generated) into a standard path that includes the YouTube channel name, video title and YouTube ID.

#!/bin/fish

function yt
yt-dlp --add-metadata -f bestvideo+bestaudio --write-sub --write-auto-subs --sub-format srt/best --sub-lang en --embed-subs -o "/media/storage/streaming/youtube/%(uploader)s/%(title)s-%(id)s.%(ext)s" $argv[1]
end

I used to download everything before watching it, as the YouTube website just sucks from a user interface perspective. It’s only gone from bad to horrible. Today I use FreeTube for watching videos (and sometimes NewPipe on my phone, although I rarely watch videos on mobile). But I’ll still call this shell function on videos I want to archive, or suspect might disappear. Before I used this function, I’d just call youtube-dl or yt-dlp on the video URL. By default, these tools don’t store metadata, so not all the videos I’ve archived contain the channel name.

Using Youboot

Youboot is a Python application and will require the poetry build tool. You can setup Youboot using the following:

git clone https://gitlab.com/djsumdog/youboot
cd youboot
poetry install

The following command will scan a directory for videos that appear to have a YouTube video ID:

# Scan a directory for YouTube Videos
poetry run python -m youboot scan-directory /path/to/scan

# Assume the parent directory is the name of the channel
poetry run python -m youboot scan-directory --channel-name-in-path /media/storage/streaming/youtube

The regular expression¹ for finding YouTube video IDs will likely scrape up some videos that are not downloaded from YouTube. They can be removed from future scans and reports by using the exclude-video command.

poetry run python -m youboot exclude-video 20_17-15-30
[23:39:22] INFO     Setting 20_17-15-30 as excluded

The pull-metadata command will attempt to retrieve YouTube metadata for each video. It will error out if a rate limit is encountered. When this happens, simply wait a while and try again.

[youtube] Extracting URL: https://youtube.com/watch?v=4ErvqnEFb-E
[youtube] 4ErvqnEFb-E: Downloading webpage
[youtube] 4ErvqnEFb-E: Downloading ios player API JSON
[youtube] 4ErvqnEFb-E: Downloading web creator player API JSON
ERROR: [youtube] 4ErvqnEFb-E: Sign in to confirm you’re not a bot. This helps protect our community. Learn more

The command list-errors will output the unique list of errors returned when searching for YouTube metadata.

❯ poetry run python -m youboot list-errors

list-removed-videos will print a table of removed in the console. The command export_csv will create a comma separated value file that can be opened with spreadsheet programs. totals will render a table of aggregated counts and percentages for each removal type. export_html_table can be used to generate HTML output, which I’ve used to create the table below.

Censored Videos

I scanned my storage drives, archives and backups to try to find all the YouTube videos I’ve downloaded. Out of the 1143 videos I have, 207 are no longer available for various reasons, and 21 are unlisted. 18% of the videos I’ve archived are no longer available on YouTube.

Some of these videos are not missing for nefarious reasons. For example, someone might decide to share a video to paid supporters first. Then, they delete the original and upload a public release that removes mistakes, or adds suggestions, based on comments by supporters. Some videos might violate another entity’s copyright. A creator might discover a large factual mistake in a video, and remove it to prevent embarrassment or the spreading of a lie. But looking through the titles, it seems like an uncomfortable number of these videos may have been taken down for political reasons.

YouTube is not just Video

In the years before YouTube, videos were shared between people without regard to how they were presented or their content. One of YouTube’s original slogans was “Broadcast yourself².” People often utilized YouTube’s features for text, link and video overlays. Those classic features have been removed, and older videos often reference onscreen links that no longer exist. However, people don’t often even see old videos, as the website favors suggesting recently published content. For a time, YouTube even removed the ability to sort by oldest videos first³. Older YouTube videos were limited to 10 or 15 minute clips, so people cut up their longer videos and used playlists for longer documentaries.

Standards and Advertising

Television networks have Standards and Practices for their shows, ensuring everything they broadcast meets certain editorial requirements⁴⁵⁶⁷. As Google sought to monetize YouTube, they implemented similar standards to encourage content favorable to advertisers. Blender, an open source 3D modeling tool, had a series of video tutorials on a channel they refused to monetize. This likely contributed to a suspension for their tutorial channel, leading the Blender team to start mirroring their videos to a PeerTube server they hosted themselves⁸.

Today we are all told to “like and subscribe,” “see the link in the description” and other instructions that help “feed the algorithm.” The old mantra of “Broadcast yourself,” died a long time ago, as did Google’s unofficial motto of “don’t be evil⁹.” In its early days, YouTube seems liked a much larger version of public access television. However, as it has grown, the need for advertising revenue to keep the massive video site afloat has brought about Community Guidelines. A new ridiculous vocabulary has been created by people attempting to side-step monetization rules. We now have terms like “he unalived himself” for suicide, “corn” when discussing pornography, and “pew pew” when talking about firearms.

This gradual creep has changed YouTube from a fun platform for sharing videos, into something that feels similar to the large broadcast networks that came before it. This has allowed YouTube to soak up the benefits of hundreds of thousands of young creators piloting content, while also removing the content of anyone who becomes remotely popular, who don’t align with Google’s political viewpoints.

YouTube’s API Wars

Both Reddit and X/Twitter have engaged in predatory API pricing to effectively kill all 3rd party apps and clients for their services. Nitter is a privacy frontend and proxy to Twitter that didn’t use the official API. Yet, due to changes in how the Twitter guest API worked, nearly all Nitter instances no longer work¹⁰.

YouTube has never had an official public API, but that hasn’t stopped Google from going after 3rd party developers. YouTube had the source code for youtube-dl taken down via a DMCA request to GitHub, although it was later reinstated¹¹. yt-dlp has become the successor fork of the project after youtube-dl was litigated against by the German government¹². YouTube has also gone after Invidious, a self-hosted open source frontend proxy to YouTube¹³. Although there are dozens of Invidious instances run by independent volunteers around the world, YouTube has been more aggressive at rating limiting these servers and making them unusable.

Archive The Videos They Want to Burn

I cannot emphasize the importance of archiving content you feel is worthwhile to save. People should always have multiple local backups of the data they trust to the cloud. It’s easy for data you trust to another company’s computers to just disappear. With storage as cheap as it is today, there’s no excuse not to make regular backups of social media accounts as well. YouTube is not an archive of material. Some videos may be preserved for decades, but others can vanish in an instant. With YouTube being the largest video website on the planet, this type of censorship has far-reaching consequences when it comes to guiding stories, belief systems and narratives presented to people around the entire world.