Page 1 of 1

How use batch processing to detect RSS in links

Posted: Thu Feb 08, 2018 4:01 pm
by awasu.user
Awasu has nice feature - detection RSS link on website. I use it a lof of time. On some exotic pages is not posible find RSS link in easy way. Awasu is solution, but I have troubles. I get spreadsheet with above 1000 news website. I have to found RSS on each.

Question:
1. How batch processing new url to use Awasu inbuild function to detect RSS link? (Perfect solution is open file and inbuild process Awasu one by one and show info about found new RSS or 404).

Re: How use batch processing to detect RSS in links

Posted: Thu Feb 08, 2018 7:36 pm
by support
There isn't really a way to automate this in Awasu, so you'll have to do it yourself. Fortunately, Python to the rescue - the process of detecting feed URL's is called "auto-discovery", and a Google search turns up a few possibilities e.g. this.

Re: How use batch processing to detect RSS in links

Posted: Thu Feb 08, 2018 8:04 pm
by awasu.user
I think about it, but when I get RSS link how next process them fx.

Code: Select all

page_first.com/feed/
pages_second.com/rss/
...
pages_n.com/feed/
to add them as source to Awasu? Is a posibility load rss feed URLs (more than one at time) to run New Channel wizard for each?

Re: How use batch processing to detect RSS in links

Posted: Thu Feb 08, 2018 8:20 pm
by support
Use the $/channels/create API call for each one.

If you really want the New Channel Wizard to come up, call $/channels/create instead.

Re: How use batch processing to detect RSS in links

Posted: Fri Feb 09, 2018 10:07 am
by awasu.user
If I send post data for each channel in format:

Code: Select all

<channel type="standard">
    <feedUrl> RSS_url </feedUrl>
</channel>
when I make API call is it start new channel wizard with RSS_url? Awasu be will wait until finish wizard started before? I don't want crash Awasu when I start adding above 1000 channels from script. I would set folder for channel manually. Not all feeds will be in the same folders

Re: How use batch processing to detect RSS in links

Posted: Fri Feb 09, 2018 10:20 am
by support
If you call $/channels/create, the Channel Wizard won't show at all, the channel will just be created (with default settings).

If you want to set folders, you can specify them in the XML you send when creating the channel. Call $/channels/list to see what the XML looks like, then send the same thing to Awasu when you create new channels.

Alternatively, don't set any folders, and just configure them manually after you've created them all.

Re: How use batch processing to detect RSS in links

Posted: Sat Feb 10, 2018 8:24 am
by kevotheclone
Sadly some companies have Atom/RSS feeds listed on a web page and don't use the "auto-discovery" mechanism, this New York Times page is one example.
You'd need to "web scrape" the page to extract the Atom/RSS feed URLs using a Python module like Beautiful Soup.

Another way to create many channels at once is to import an OPML file.
You could write some code using the above mentioned Python module feedfinder2 or Beautiful Soup and create an OPML file for each page scraped. Then import the OPML file(s) into Awasu.
Awasu even has an API call $/channels/import to start the OPML import dialog. ::-):

You can find some OPML files to import in the left and center columns of this page: http://feedshare.net/popular/, the right column is a list of Atom/RSS feeds not OPML.

Re: How use batch processing to detect RSS in links

Posted: Sat Feb 10, 2018 9:15 am
by support
kevotheclone wrote:
Sat Feb 10, 2018 8:24 am
Another way to create many channels at once is to import an OPML file.
Actually, that's a pretty clever idea :clap: Awasu impresses even me, sometimes... :-)

Import from OPML also has an option to create the imported channels in a specific folder, so as long as you want to put the new channels in the same folder, this would be a quick and easy way to do it.