Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

For fun and as a learning experience, I built a simple CL scrapping web site. I set it up on a small aws instance about 6 months ago and after playing around with and making modifications/updates to it over the course of a month or two, I largely forgot about it. After seeing this thread, it dawned on me that I had not received an e-mail update on any of my search alerts for a while. I ssh'ed into the server and got a 403 with 'wget craigslist.com', so my guess is that they are doing some sort of blocking. Looking at the logs, the block probably started about a month ago.


Modifying the user agent to mimic a normal browser might solve the problem.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: