I'm using scrapinghub extensively for https://pagewatch.dev , is this project something you can use as a self-hosted replacement? Its not very clear how it works, what does the Realtime mean?
I believe they are contrasting it with how something like https://github.com/scrapy/scrapyd#readme or the ubiquitous "run scrapy under crontab" setups normally function: you kick off the spider, then you go get coffee and wait for the ItemPipeline to deliver the results somewhere else
With scrapyrt, you submit a POST or GET, and that URL blocks until the one requested thing completes or times out, enabling (only conceptually, mind you) an archive.is replacement using an XHR
Looks to just put an API in place. I think they are using creative license with the term "realtime" (since I presume all scraping actions are queued and async)
I don't get that impression from the description of the response from `POST`: https://scrapyrt.readthedocs.io/en/0.11.0/api.html#success-r... since it does not return a job-id that requires polling, it appears to block until your scrape request is completed or it times out
It's likely an implementation detail about whether the scrapyrt component is the one which blocks, but otherwise uses queuing and asynchronous invocations when interacting with Scrapy, or whether -- as the custom CrawlManager implies (https://scrapyrt.readthedocs.io/en/0.11.0/api.html#crawl-man...) -- scrapyrt actually takes over and makes the entire CrawlManager -> Scheduler -> Spider call stack synchronous and thus able to respond to a POST within https://scrapyrt.readthedocs.io/en/0.11.0/api.html#timeout-l... seconds with the actual Items
