Pretty sweet cloudflare has an off-the-shelf product for website crawling which integrates with their workers. Its a pupepeteer fork, so has the exact same API.
Pretty sweet cloudflare has an off-the-shelf product for website crawling which integrates with their workers. Its a pupepeteer fork, so has the exact same API.