Delving Bitcoin Scraper Ignores Updates to Existing Posts

[kouloumos]

The current Delving Bitcoin scraper skips indexing when a document with the same ID already exists. This means updates to existing posts are not accounted for.

When adding Delving Bitcoin to scraperv2, we should address this issue and consider the following approaches:

Approach 1: Leverage Metadata Stored in ES Index

James' discourse-archive uses an on-disk .metadata.json file to store the last_sync_date, allowing it to avoid syncing from scratch each time. Currently, we are not using this functionality. Each GitHub Action workflow syncs everything from the beginning, which is manageable since Delving Bitcoin's history is relatively short.

In scraperv2, metadata from scrape jobs is stored in our Elasticsearch (ES) index. We can take advantage of this existing functionality to manage last_sync_date and only sync new or updated posts. However, this may require modifications to James' initial code to integrate this metadata mechanism.

Approach 2: Use a Custom Delving Bitcoin Archive

Another option is to forego using discourse-archive on every scraping run. Instead, we could maintain our own instance of the delving-bitcoin-archive and read the necessary data directly from a GitHub repository. This would be similar to how we handle data in the GitHubMetadataScraper.

Related

• The current code in delvingbitcoin_2_elasticsearch/achieve.py is extracted from jamesob/discourse-archive.
• https://delvingbitcoin.org/t/public-archive-for-delving-bitcoin/87/5
• https://meta.discourse.org/t/fetch-all-posts-from-a-topic-using-the-api/260886
• https://h-rd.org/archiving-a-discourse-forum/

[urvishp80]

@kouloumos I investigated the given issue and here is the proposed solution,

• commit .metadata.json on every run, this will allow us to download only the latest (recently changed) files based on the last_sync_date in it, avoiding the unnecessary downloads of the past unchanged files.
• introduce updated_at field in the ES docs, and compare it with the updated_at field in the downloaded JSON (raw) files to sync these posts in ES index.

[kouloumos]

@urvishp80 I don't like this approach, we already have scraperv2 which is tracking metadata using ES, so I prefer to add it there. I don't want that automated commit in this repo that your solution includes.

This is not a priority, don't work on this, when we do we are going to integrate delving bitcoin to scraperv2.