In my spare time I am creating a web scraping environment. Why is it environment not a script or anything like that? Because those pretty much already exist.
Problems I did face:
– picking a right version of Firefox that would work smoothly with Selenium;
– setting Firefox preferences to speed up scraping;
– screenshots in Selenium by default are saved in PNG;
– making it work in several browsers in parallel – I did not want to use tabs;
– block content loaded by websites in https that could not be handled by DNS or Proxy.