I'm trying to figure out how to make this script.
OK, I had a bit of a play around with this and it's not as simple as I first thought.
It seems that while WebScrape offers the DownloadUrl parameter, it appears to be ignoring it and only downloads whatever is specified in it's own INI file
So the trick of writing a replacement plugin may not tell you what you need to know since it's the plugin that's doing the download, not Awasu.
If you still want to try, this is what to do:
(*) We need a plugin that looks the same as WebScrape so take a copy of WebScrape.plugin and call it, say, foobar.plugin (in the same directory as WebScrape).
(*) Create a file called foobar.py (in the same directory) that looks like this:
Code: Select all
# get the path to the HTML file Awasu downloaded for us
configFilename = sys.argv
htmlFile = win32api.GetProfileVal( "DownloadUrl Response" , "DownloadUrlFile" , "" , configFilename )
# dump the HTML file
fp = open( htmlFile , "r" )
buf = fp.read()
fp = open( "foobar.log" , "w" )
fp.write( buf )
# invoke the real WebScrape
os.system( ".\\WebScrape.exe "" + configFilename + """ )
This dumps the HTML file Awasu downloaded and then invokes the real WebScrape.
(*) Exit Awasu, find your channel's .CHANNEL file and edit the ScriptFilename parameter to point to foobar.py instead of WebScrape.exe
(*) Restart Awasu and update the channel. You should find a file called foobar.log that contains a copy of the downloaded HTML file.
As I said, WebScrape is doing the download, not Awasu and IIRC, it doesn't honor any of IE's cookies or other settings. So maybe the best way to simulate what's happening is to clear all your IE cookies and cached files, then download the HTML page and play around with that in WebScrapeSettings.
You could also run Ethereal or some other HTTP monitor and watch what WebScrape is downloading...