Awasu Forums

abwilson

Sorry -- I didn't realize by "Webcrawler" you were referring to WebScrape, so I initially skipped your posting. The following is pretty close; however, WebScrape expects some form of quotation mark [" or '] to enclose href/img references (which the biotechnology.net page doesn't use),...

abwilson

It's really not practical because the plugin will time out waiting for a large number of channels.

Perhaps Taka will have another solution, but why not just put all 100 channels in their own channel folder and let Awasu do its thing?

abwilson

Okay, I found and fixed the problem, which was related to the https:// URI scheme; it doesn't handle authentication, but lets the "https://" through okay now. While I was at it I also included the latest feedparser module, which means the plugin now supports Atom 1.0 feeds too. Taka can ma...

abwilson

Sigh... Do you have another example? When I enter the URI I get:

Not Found
The requested URL /WebObjects/Core.woa/Feed/itunes.stanford.edu.1770152.01770196. was not found on this server.

Thanks

Allan

abwilson

When I click on the link (in Firefox), I get some message about iTunes -- not RSS as I expect. What should I be seeing?

Thanks

Allan

abwilson

Going back to a previous version after my build procedures had changed proved more difficult than expected. I am sending you an updated build; please try it.

Thanks

abwilson

Yes, though I've had to redo it a bit, since my build procedures also changed a while ago. Please send me (abwilson at awasu.com) your e-mail address and I'll send you a .zip file with the "sequential" version of MonitorURLs to try.

Thanks

abwilson

Sigh... It is likely my code. Despite what the Release Notes probably still say, the latest version of MonitorURLs does not use a standard Python library to retrieve Web pages, but rather does it using a new library I wrote to allow parallel (asynchronous) reads -- meaning that the plugin can fetch ...

abwilson

The plugin is written in Python and uses the standard Python library routines, which do support any (IE defined) proxy.

What is the/a site you are having trouble with?

abwilson

The DownloadUrl parameter is defined in the .plugin file and therefore appears in Awasu's dialog when you define the channel. What are the various settings in your channel properties?

abwilson

That should mean it tried to retrieve the page but gave up after not getting a reply from the site after waiting a certain length of time (30 seconds, I believe).

abwilson

Are you using the DownloadUrl facility provided by Awasu or are you having WebScrape fetch the page to be scraped? If WebScrape, I would expect it to fail with a pretty obvious error if it can't find a page.

abwilson

Why are you not using the RSS feed URLs? Try the following and see if it does what you want (the feed URLs below were obtained by clicking the RSS buttons on your HTTP URL pages): [ChannelParameters] title=Consolidated Feeds link=www.indeed.com description=Consolidated Indeed Feeds maxitemsperfeed=1...

abwilson

As I just responded to your other post, WebScrape won't do the job either -- it's also confused by the https URL.

Sorry, but I don't know what to recommend at this stage.

abwilson

Looking into WebScrape (as opposed to MonitorURLs) to give you the information you want, I've struck out again: The same https problem keeps WebScrape from properly retrieving the page. So, neither WebScrape nor MonitorURLs will do what you want.

Sorry I couldn't help with this one.