User:KiranBOT/AMP
Appearance
- This task is designed for removing en:Accelerated Mobile Pages link/URLs from articles.
- Stopping the bot: In case bot makes problematic edits, the recommended method is to add
{{bots|deny=KiranBOT}}at the top of the page where problematic edits are taking place. In case the issue is on multiple pages, this particular task can be stopped by anyone (doesn't need to be admin) by editing en:User:KiranBOT/shutoff/AMP. But turning off the bot should be last option as this task runs on multiple Wikipedias.
Technical details
[edit]- Currently the v2.2.7r runs continuously from toolforge, but it only updates the AMP URL that are in the database (more details in #Change-log).
- deployment notes
- the database
- To minimise repeated scraping and avoid revisiting the same AMP URLs, the bot relies on a database that stores AMP URLs along with their corresponding canonical URLs.
- During normal operation, whenever the bot detects an AMP URL on a wiki page, it first queries the database:
- If the URL is already present, the bot replaces the AMP URL with its stored canonical version, with no external scraping required.
- If the URL is not present, it is logged into a file for later processing.
- Almost every-other day, a separate program is run on my local machine that attempts to fetch the canonical versions of these new URLs. Successful results are added to the database, while non-functional or problematic URLs are flagged for review or removal.
- The system automatically avoids creating duplicate entries for the same URL, and looped/circular entries; ensuring the database remains clean and efficient.
Change-log
[edit]- nomenclature: I maintain the version naming for myself, to include the version in the edit summary so I can easily track the edits, errors, and updates I applied.
- • The naming partially follows semantic versioning, and partially follows the version naming system of en:FreeBSD (which uses branched history, and terms/versions like "CURRENT", "STABLE", and "RELEASE" link).
- • s versions are "stable", with some minor changes. They might behave unexpectedly if an unforeseen situation arises for which the bot is unprepared. r versions are "release versions". They are fine-tuned and thoroughly tested updates of "stable".
- • rarely, the version may be followed by a combination of "lm", "rs", and "rc". These indicate "local machine", "reverse search", and "recent changes" respectively. If there is no suffix, it means the bot is running from toolforge, going through recent changes.
- Around January 15, 2025 a database was integrated for storing scraped URLs, and the ability to repair the URLs was disabled.
- v2.1.1r February 7: The bot/task was approved on Spanish Wikipedia (eswiki).
- v2.1.3r February 8: The database was moved to my computer/local machine from toolforge. The bot now runs entirely from my local machine instead of toolforge.
- This was done because of some technical issues, and to avoid server overload.
- Also, to avoid en:WP:toolforge's IP address being denied/blacklisted by domains due to repeated requests to various URLs of the same domain.
- v2.1.7s 11 February: In case the canonical URL leads to a subscription page, parked domain, or domain selling page, those URLs are skipped.
- v2.1.7r 15 February:
- v2.1.8s 17 February: Added functionality for maintaining edit summaries in multiple languages.
- v2.2.1s 18 February: branched version of v2.1.7r — goes through recent changes; the database is moved back to toolforge, including the rest of the program.
- v2.1.9s 20 February: bug fix — db query for URLs with and without trailing
/; added a list to skip problematic or false positive root domains (egbandcamp.com) - v2.2.2s 20 February: same as of v2.1.9s
- v2.1.9r 28 February: does not fetch canonical URLs — relies only on the database. Runs intermittently from my computer.
- v2.2.2r 28 February: goes through recent changes, runs continuously from toolforge. Does not fetch canonical URLs — relies only on the database.
- v2.2.3s 27 March: bug fix regarding the `archive-url` parameter; buggy edit before update: en:special:diff/1282243830 (goes through recent changes, continuous run from toolforge)
- v2.2.4s 27 March: same as v2.2.3s (searches for AMP URLs in pages, runs sparsely from my computer)
- v2.2.4r 25 April: release version of v2.2.3s
- v2.2.5r 7 May: approved as global bot; added 21 small wikis
- v2.2.7s 13 May: added another mechanism to identify archive URLs; added functionality to upgrade http to https
- v2.2.7r 13 June 2025: release version of v2.2.7s; active on 100+ Wikipedias. On June 15, the bot was deployed on all active Wikipedias.
- v2.2.9s
- v3.1.1s 13 August 2026: better handling of edit conflicts.
Planned changes
[edit]- on Russian Wikipedia, add a delay of a couple of days between recent changes and bot edits to prevent vandalism from being whitelisted/overlooked. —usernamekiran (talk) 23:17, 8 September 2025 (UTC)
Approvals
[edit]- January 19: The task was approved on English Wikipedia (enwiki)
- February 7: Spanish Wikipedia (eswiki) (discussion)
- February 19: Polish Wikipedia (plwiki) (discussion)
- March 3: German Wikipedia (dewiki) (discussion)
- March 3: Italian Wikipedia (itwiki) (discussion)
- March 3: French Wikipedia (frwiki) (discussion)
- April 18: Portuguese Wikipedia (ptwiki) (discussion)
- May 7: approved as global bot (v2.2.5r) (discussion)