Page 1 of 1

WebScrape-v1.30b: Error when scraping "https://" url

Posted: Fri Jul 20, 2018 9:36 pm
by csmryad

I'm using Awasu App 3.2 with WebScrape 1.3b. It work fine except when I'm try to scrape "https" url:

Traceback (most recent call last):
File "", line 249, in ?
File "", line 183, in main
File "", line 139, in getWebpageInfo
File "urllib.pyo", line 77, in urlopen
File "urllib.pyo", line 185, in open
File "urllib.pyo", line 317, in open_http
File "urllib.pyo", line 330, in http_error
File "urllib.pyo", line 607, in http_error_301
File "urllib.pyo", line 588, in http_error_302
File "urllib.pyo", line 603, in redirect_internal
File "urllib.pyo", line 185, in open
File "urllib.pyo", line 386, in open_https
File "httplib.pyo", line 795, in endheaders
File "httplib.pyo", line 676, in _send_output
File "httplib.pyo", line 643, in send
File "httplib.pyo", line 1070, in connect
File "socket.pyo", line 74, in ssl
IOError: [Errno socket error] (2, 'The operation did not complete (read)')

Any idea about that ? Thank you

Re: WebScrape-v1.30b: Error when scraping "https://" url

Posted: Sun Jul 22, 2018 2:52 pm
by support
This plugin was written quite a few years ago by one of our old users, before SSL become as widely spread as it is now, so I imagine the plugin doesn't work properly with SSL. Unfortunately, he didn't leave us with the source code, so we can't fix it to support HTTPS sites. If you have some technical chops, you should be able to set up a proxy that accepts non-SSL requests, and returns the SSL page, then point the plugin at the proxy.