Schedule Regular Scraping of Bitcoin Improvement Proposals (BIPs, BLIPs, BOLTs)

[kouloumos]

Currently, we have scraping code for Bitcoin Improvement Proposals (BIPs, BLIPs, and BOLTs) from GitHub repositories, but it isn't set to run on a schedule. As a result, these sources are not being updated regularly and may contain outdated information.

We need to implement a cron job to ensure that these improvement proposals are scraped on a more frequent basis, so that our data stays current.

Tasks:

1. Create a cron job for regular scraping of BIPs, BLIPs, and BOLTs.
2. Define an appropriate frequency for scraping (e.g., daily or weekly).
3. Ensure the scraped data is stored properly in the Elasticsearch index.

[kouloumos]

Those repositories are covered in #81, activation of scraperv2 is pending

[staru09]

Is this open to work?

[kouloumos]

Is this open to work?

Have you read my comment above?

[staru09]

(Update)
Yes I did.
I haven't completely gone through the mentioned PR yet but I think all the tasks mentioned above are already been covered not sure of the cronjob part for scraping frequency. Is that so?

Or is it planned for next version of scraper?

[kouloumos]

(Update) Yes I did. I haven't completely gone through the mentioned PR yet but I think all the tasks mentioned above are already been covered not sure of the cronjob part for scraping frequency. Is that so?

Or is it planned for next version of scraper?

Scraperv2 already supports what's mentioned here. We just need to enable it. So there is nothing to do here. Send a message in the discord server and we can discuss further if you are looking for something to do.