Pretty sweet cloudflare has an off-the-shelf product for website crawling which integrates with their workers. Its a pupepeteer fork, so has the exact same API.

https://developers.cloudflare.com/browser-rendering