Use Node JS and browser developer tools as intelligent webscraper for secure websites.

blog_web-scraper

Some of you probably already had the idea to combine (maybe even realtime) data from different websites on one page that holds all the relevant data without the crap you don’t need. There are some techniques to fetch and distribute static webpages, but when it comes to realtime data on a https location you can forget about it! And in fact it is a very good thing that you cannot simply embed an iframe with another website and get data out of it.

But unfortunately there are situations that require magic to make it happen. Continue reading “Use Node JS and browser developer tools as intelligent webscraper for secure websites.”