Adding various mirror links

[DandelionSprout]

In light of recent GitHub account problems for CHEF-KOCH and NanoMeow, as well as how I've learned that raw.githubusercontent.com links didn't work in some places (Iran was implied to be an example of that), and in an attempt to reduce the amount of issue reports and r/ublockorigin threads that complain about connection issues, I've partaken in discussions in NanoAdblocker/NanoCore#220, where one of the discussed topics was to keep backups and mirrors of lists, and where I've now taken the initiative to host some mirrors of some of the regional lists. However, as I appear to lack pull request privileges in the uBlock repo, I suppose I'll link here to my suggested edits that are in gorhill/uBlock@master...DandelionSprout:patch-3.

This issue (Pull?) aims to test out four different approaches to mirrors and alternate links: GitCDN (A raw file fetcher that @jspenguin2017 and farroukhi have adopted), Repo.or.cz (A small-time Czech git host that I've used with fair success), NotABug.org (A git host that is somewhat more secure than Repo.or.cz as far as edit credentials go, and which have been used by the LVT list), and others' GitHub mirroring of non-GitHub lists (in case the sources' domains run into problems with server admins, domain bills, or certificates).

If some or all of these approaches don't run into any problems, then these kinds of backup links could be rolled out big time, and could even grow to include jsDelivr and UNPKG, which Jspenguin2017 has taken a liking to.

[jspenguin2017]

I think we should first agree on which mirror to use. repo.or.cz should be fine for backing up repositories (and commit histories), but not so much as content server. If ever a fail over happen, it will bring down their servers pretty quickly.

[DandelionSprout]

You have a good point, but that's in part why I thought of this pull as a test of sorts. I could've attempted to make both repo.or.cz and notabug.com mirrors for every one of the 14 regional lists that both were GitHub-only and which had permissive licences, but I thought it'd be better to see how some examples of them would fare in everyday use before I'd do it on a large scale.

[DandelionSprout]

I've also prioritised the links in a set order, based on source originality, update frequencies, server uptimes, and server capacities.

Self-hosted sites > GitHub > GitCDN > NotABug > Repo.or.cz

[jspenguin2017]

I would vote for Filter Source > GitCDN > jsDelivr > UNPKG > GitHub > GitLab. I can set up a repository that mirrors all filters defined in assets.json and dual push to GitHub and GitLab.

[DandelionSprout]

If I'm not getting things completely wrong here, then GitCDN is just some kind of file fetcher from GitHub repos, as opposed to a true file host.

I'm unable to figure out how to host files on jsDelivr and UNPKG, but I see no reason why we can't interpolate these two suggestion trails into something á la Self-hosted sites > GitHub > GitCDN > GitLab > jsDelivr > NotABug > UNPKG > Repo.or.cz.

[DandelionSprout]

Ah, now I saw your addition of Filter Source >. Now I understand your stance properly. I could be in for Filter Source > GitCDN > jsDelivr > UNPKG > GitHub > GitLab > NotABug > Repo.or.cz.

[DandelionSprout]

I also suppose that it'd be easy-peasy to e.g. take your GitHub-GitLab supermirror once you've made it, and then mirror it on NotABug. But that's a scenario that I think would be better to decide upon once that supermirror has been made.

[jspenguin2017]

I might have to publish the repository to NPM before jsDelivr and UNPKG will work.

Having many mirrors is good, but we don't want assets.json to become too bloated though. I think 3~4 mirrors are enough, we can always add more if needed. We'll see what gorhill thinks about this.

[MartinKolarik]

I might have to publish the repository to NPM before jsDelivr and UNPKG will work.

jsDelivr works with GitHub too and can handle any amount of traffic. Let me know if you decide to use it and need any help.

[jspenguin2017]

@MartinKolarik I can't seem to get it to serve tip of master tree. It says:

Couldn't find the requested release version latest.

[MartinKolarik]

That's not supported yet but might be soon. Right now you have to make GitHub releases (or just git tags) for the "latest version" links to work, or use a commit sha in the URL (which I guess wouldn't work in this case because clients wouldn't know what's the latest commit).

[MartinKolarik]

@jspenguin2017 serving from master is possible now: https://cdn.jsdelivr.net/gh/uBlockOrigin/uAssets/

[jspenguin2017]

@MartinKolarik Oh, that's great! How long is the file cached for?

[MartinKolarik]

On the CDN it's 12 hours, on clients 7 days but that can be bypassed by adding a query string. There's also an API for purging the CDN cache if you need more frequent updates.

[DandelionSprout]

Just a quick headsup, even if this issue report may otherwise have fizzled out into nothing:

I hereby declare that NotABug is not the best of places to host a repo mirror. https://notabug.org/DandelionSprout/FilterMirrorRepo2 have been showing a 500 error for the past two months, after NotABug's mirror capabilities as a whole just randomly broke one day and doesn't seem to have been properly fixed since.

[jspenguin2017]

@DandelionSprout Can this be related? Is NotABug using an up to date fork of Gogs? gogs/gogs#5469

[DandelionSprout]

I have a hard time telling whether a remote command execution issue could cause 500 errors, but what I can confirm, is that NotABug is in fact not using an up-to-date version of Gogs (Compare the file update dates in https://notabug.org/hp/gogs to https://github.com/gogs/gogs), and that the 500's began showing up within mere weeks of gogs/gogs#5469 being reported.

[jspenguin2017]

Remote code execution (RCE) can result in complete server takeover, which can basically cause every imaginable issues.

[h1z1]

ipfs could be an option for this. Cloudflare as a bonus runs a gateway.

[gorhill]

Closing because no clear, specific actionable items. If there is any current, existing issue with a specific filter list, please open an issue for that one case. I will not add mirror URL for a list unless there is an existing issue for that list.