Page 1 of 2

Channel Generated from search result: NO NewFeedItem=1 III

Posted: Tue Sep 16, 2008 7:19 pm
by Otonicara
... again session problem. I have tried login/logout and refresh but the problem is still there..

Well I wish it would be that simple as it seems to you. For both logs I have used official Awasu Hook (no modifications) with the name LogChannelActivity.hook (ChannelHooks / LogChannel Activity). The same hook in direct feeded (Dowloaded from Internet)(C1) Channel produce NewFeedItem and on Channels made from search reults (C2) doesn't! On Screen New News is displied on both channels..

Now comming to my development. For the data transfer I use additional hook made with your help. This hook transfer news to my Python program and consequently to Access Data base for C1 but not for C2. In C2 my Python program is not called as there is no NewFeedItem.. Realy strange and not consistent ..

New test:
I have used Fresh Awasu installation. I create only two channels C1 and C2. Than I wait that Awasu start refreshing (aprox. 5 minutes). I have also duble checked that the same Awasu original hook is used!

C1 Source: ... nglish.xml

C2 Key word "of"

Result is bellow.

16sep08 20:28:15 | EVENT: NewFeedItem

16sep08 20:28:15 | EVENT: NewFeedItem

16sep08 20:28:15 | EVENT: NewFeedItem

16sep08 20:28:15 | EVENT: NewFeedItem

16sep08 20:28:16 | EVENT: NewFeedItem

16sep08 20:28:17 | EVENT: NewFeedItem

16sep08 20:28:17 | EVENT: NewFeedItem

16sep08 20:28:17 | EVENT: NewFeedItem

16sep08 20:28:17 | EVENT: NewFeedItem

16sep08 20:28:18 | EVENT: NewFeedItem

16sep08 20:28:18 | EVENT: NewFeedItem

16sep08 20:28:18 | EVENT: NewFeedItem

16sep08 20:28:18 | EVENT: NewFeedItem

16sep08 20:28:19 | EVENT: NewFeedItem

16sep08 20:28:19 | EVENT: NewFeedItem

16sep08 20:28:19 | EVENT: NewFeedItem

16sep08 20:28:19 | EVENT: NewFeedItem

16sep08 20:28:20 | EVENT: NewFeedItem

16sep08 20:28:20 | EVENT: NewFeedItem

16sep08 20:26:08 | EVENT: EndChannelUpdate

16sep08 20:26:08 | EVENT: BeginChannelUpdate

16sep08 20:26:09 | EVENT: EndChannelUpdate

16sep08 20:31:10 | EVENT: BeginChannelUpdate

16sep08 20:32:02 | EVENT: EndChannelUpdate

16sep08 20:38:20 | EVENT: BeginChannelUpdate

16sep08 20:38:41 | EVENT: EndChannelUpdate

On 20:28:20 New News was delivered to C1

On 20:31:10 new news met C2 key word creteria (on screen Received: 20:31
Source: Marketwire)

HOOK used in the test above:

DisplayName=Log Channel Activity
Notes=This channel hook logs channel activity. Specify a file to log everything to and which events you are interested in.\n\nThe log file's extension must be set correctly and have an associated application.

' ---------------------------------------------------------------------


' ---------------------------------------------------------------------

DisplayName=&Open log file

DisplayName=&Clear log file

' ---------------------------------------------------------------------

Description=Include date stamps in log messages.

Description=Include time stamps in log messages.

' ---------------------------------------------------------------------

Description=Channel activity log file.

Description=Dump config files passed in by Awasu.

Re: Channel Generated from search result: NO NewFeedItem=1 I

Posted: Wed Sep 17, 2008 2:39 am
by support
Email the following stuff to me and I'll take a look at it:
(*) Your BUILD.INFO file (from the local data directory)
(*) Your CONFIG.INI file
(*) *All* your .CHANNEL files
(*) *All* the .HOOK files
(*) *All* the hook script files

Still not working ..

Posted: Wed Apr 29, 2009 11:54 am
by Otonicara
Hellow ..

We have been resolving this issue approximately a year ago. Later I put the issue on hold as I ran out of time. As this is the main reason way I bought Awasu Professional, and now again I would need this functionality for many purposes, I would really like the problem to be resolved..

In order that you can check it first hand I zipped entire Awasu directory and uploaded it to the bellow link. Please download and check what is wrong ..


Access DataBase:

Basic info:
Awasu Version: 2.4 Professional
Licence: Professional Valid Until End of April 2010

The Idea is that Python filter News Titles using keyword mechanism (Channel generated from News Search Results). Once Proper News is found, new is displayed in dedicated Channel and at the same time hook takes care that news data is also transferred to MC Access table.

Problem description
(check ZIP attachment for details):
MarketWire feed is hooked by official Awasu LogChannelActivity hook
BusinessWire feed is hooked by my Python program that exports data in MS Acccess
Both hooks works fine.. As you can see in ZIP attachment logfile includes News data and News data are also imported into Access table.

Than ..
Check TEST: of OR and (Channels generated from News Search Results). Official Awasu LogChannelActivity hook generates only start and end statements without any News Data. Consequently Python program has nothing to export to Access ..

Re: Still not working ..

Posted: Wed Apr 29, 2009 8:54 pm
by kevotheclone
I don't know if this is the problem or not but in your example of your Search Channel (C2) you are searching for the word "of". Awasu has a "stop word list" that lists many common words that are ignored by Awasu's content indexer (very common among search technology).

Maybe the stop word list is also used by Search Channels, but I'm not sure (Taka, your "fate" is calling :lol: ).

If it is, then you could probably remove "of" from the stop word list, or search for another common word that is not in the stop word list. "TSX" looks like it usually occurs in a feed item description that contains a stock ticker.

I know you were probably just searching for "of" because it is a common word and you wanted to test matching any word, but it may be conflicting with the stop word list.

Awasu's stop word list is typically located here:
{Awasu Install Directory}\search\StopWords.txt

Good luck! ::-):

Re: Still not working ..

Posted: Thu Apr 30, 2009 11:30 am
by support
kevotheclone wrote:Maybe the stop word list is also used by Search Channels

Search channels use the same search engine so yes, it will also be affected by stop words.

kevotheclone wrote:but I'm not sure (Taka, your "fate" is calling :lol: )

I'm clearly on its speed-dial :wall:

Re: Still not working ..

Posted: Thu Apr 30, 2009 11:32 am
by support
Otonicara wrote:In order that you can check it first hand I zipped entire Awasu directory and uploaded it to the bellow link.

Unfortunately, you've sent me the Awasu installation directory which only contains the software, which I already have :roll:

I need to see the local and roaming data directories.

Local Files ..

Posted: Thu Apr 30, 2009 12:57 pm
by Otonicara
Files you can find on:

I also tried with other keywords but still not working. I do hope you will be able to resolve the problem ..

Cheers, Ales

Re: Local Files ..

Posted: Sun May 03, 2009 12:22 pm
by support
The files you provided don't match what you described in your original post but since that was so long ago, it's hardly surprising.

I set up your installation and this is what happened:

(*) You seem to have only one channel configured with channel hooks, a search channel looking for "crt OR approvale OR approve OR approves OR approved". It is configured with our LogChannelActivity sample hook, and the one you wrote to save items to an Access database.

(*) In one of the ZIP files you provided, the log file for the LogChannelActivity hook indicates that the <tt>NewFeedItem</tt> and <tt>ConsolidatedNewFeedItems</tt> events are being raised correctly.

(*) I updated all your channels and a bunch of new items came in. However, <u>none of them matched the search query you are looking for in your main search channel</u>. Therefore, your channel hook didn't get called, therefore nothing got put into the database.

(*) As a test, I modified your channel hook script to log every time it got called. I also turned on logging for your hook in Awasu (by ticking <i>"Enable logging"</i> in the <i>Log</i> page of the channel hook's Properties dialog). I then modified the search channel's query <u>so that it would match one of the newly-received feed items</u>, then updated it. It found the new item and called your channel hook correctly. This was confirmed both by the logging Awasu does when it calls channel hooks, and the change I made to your script.

Awasu is working correctly. As I said in earlier posts, <u>your channel hook will only get called when new items arrive on the channel that it is attached to</u>. Your hook is registered for <tt>NewFeedItem</tt> events only, so if new feed items are not received, your hook won't get called. This means new feed items on your search channel (i.e. the one the channel hook is attached to), not any other channel. If new feed items arrive on another channel that you think should be added to the database, you need to make sure that your search channel was updated <u>after</u> they arrived (since the search engine can't possibly find items <u>before</u> they arrive). Open the <u>search channel</u> and new items will be flagged with the yellow icon. If you don't see any, then no new items have been received.

Getting News Data Structure witin a hook

Posted: Tue May 12, 2009 12:47 pm
by Otonicara
Thanks for taking a time. It will be really nice to finally resolve this mystery. Having control over the news is really useful for all sorts of reasons. That's why I was very happy year ago when I found Awasu on the web.

Of course hook is not triggered if there is no new feed item. In my case there are every day at least a few new items that match search criteria ant that can be seen inside search channel in Awasu (with the icon you mentioned). You gave me a hint wit additional log for hooked script. It looks like that my code has an error while is hooked on search channel and Bussines Wire direct source. On Market Wire direct sourec works normal. I would like to check data structure hook is passing to my code. Last year I was using your LogAcahnellActivity hook to get:

ItemTitle=CORRECTION FROM SOURCE: PlayBOX in Negotiations to Acquire New Visions Mobile%0A*0
ItemMetadata=<value>CORRECTION FROM SOURCE: PlayBOX in Negotiations to Acquire New Visions Mobile</value><group><value></value></group><group><value>OTC Bulletin Board:PYBX</value><value></value></group><value>1216986300</value>
ItemContent1_Content=LONDON, UNITED KINGDOM--(Marketwire - July 25, 2008)

This year I am getting only
[NewFeedItem] lines but no news contents as shown above...

What needs to be done to get HOOK news data as shown above? Without this I am lost as Awasu organize data in its own way - different as it is on main news source.

Cheers, Ales

Re: Getting News Data Structure witin a hook

Posted: Wed May 13, 2009 11:55 am
by support
Posting stuff that you got from a feed last year is not really going to be useful. You need to take a look at the feed as it stands now, and compare it with what Awasu is giving your channel hook.

You said that something is missing when you attach your hook to another channel but didn't say exactly what. However, I'm guessing that it's the <tt>ItemContent1_Content</tt> value that's not there. The most likely explanation is that the feed item doesn't actually have any item content :| You need to take a look at the raw feed XML and check what it has. Some feeds only publish a summary, some feeds don't have anything at all.

Re: Getting News Data Structure witin a hook

Posted: Fri May 15, 2009 1:17 am
by kevotheclone
Onticara, here's a debugging tip from my upcoming eBook: "True Confessions of an Awasu Junkie"

Python debugging
First off, if you are not using PythonWin as your Python editor give it a try and learn to use its debugging features so that you can set breakpoints, step through your code, watch variables and use the interactive window. If you have a Python editor that has these debugging features then get familiar with them, because I'm going to tell you how I debug my Channel Hooks.

Get your own Awasu "config" file
The Channel Hook, as nice as it is, just displays the config files that Awasu generates for it; they are not going to be exactly the same as the config files that your Channel Hook receives from Awasu. So first you need to temporarily modify your Channel Hook to make a copy of the exact config file that Awasu is generating for it.

I wrote this simple function to dump the config file to a (hard coded) text file that I specify. This is very similar to what the Channel Hook does (thanks Taka).

Code: Select all

def writeConfigFile(configFileName, outputFileName):
    # dump the config file
    fp = open(configFileName, "r")
    gLogFile = open(outputFileName, "w")

Add this function to your Channel Hook, call it early in your Python script and then immediately exit. Go to Awasu and refresh your Channel to get your Channel Hook called.

Code: Select all

configFilename = sys.argv[1]
writeConfigFile(configFileName, "C:\\MyConfigFile.txt"):

Once you've confirmed that your Channel Hook has run and wrote out the Awasu config file, "comment out" the two lines you added to your code so it looks like this:

Code: Select all

configFilename = sys.argv[1]
# writeConfigFile(configFileName, "C:\\MyConfigFile.txt"):
# sys.exit(0)

Now your Python script is virtually back to the way it originally was and you've got a config file that you can debug with. Forget about Awasu for awhile; shut it down if you must, we won't need it to debug our Channel Hook. You may also want to temporarily set the "read only" attribute on your config file so you don't accidentally overwrite it.

Debugging your Channel Hook
Using the PythonWin editor, load your Channel Hook script. If you think you know where the problem is/are go to the line(s) and add breakpoint(s) by pressing the F9 key (it toggles breakpoints on and off). Press Ctrl+R to display the "Run" dialog box. In the "Arguments" text box enter the path to the Awasu config file that your Channel Hook previously saved; there's no need to double the backslash character (\), however if the path contains spaces you should enclose the whole path with quotation marks. Select either "Run in the debugger" if you think you know where the problem is and you set your breakpoints correctly; or select "Step-through in the debugger" if you need to step through every line of code. I'm sure you know the difference between "Step", "Step Over", and "Step Out"; there are some places where "Step" will take you into some Python library code (you'll learn where this occurs in time, and you can "Step Over" them), just click the "Step Out" button to get back to your code. Add watches on your critical variables or use the "Interactive Window" to write out their values (just type the variables name and press the Enter key). Tip: Once you've typed a variable's name in the Interactive Window during a debugging session, you can auto-complete it the next time you need it by typing the first few character of it and the pressing Ctrl+Spacebar. If you have several variables with the same prefix (xTitle, xURL, xDescription), you can even just type the "x" and then press Ctrl-Spacebar multiple times to cycle through all of the variables that begin with that prefix.

This is my breaking point
Running my Channel Hook code in the debugger really helped me pinpoint my errors quickly, without having to keep refreshing my Channel and checking the error log that Awasu maintains for our Channel Hooks. I think it can help you too, best of luck!

Re: Getting News Data Structure witin a hook

Posted: Fri May 15, 2009 10:24 am
by support
kevotheclone wrote:Onticara, here's a debugging tip from my upcoming eBook: "True Confessions of an Awasu Junkie"

Cool! :cool: Let me know when we can pre-order :-) Although I daresay I'll probably be writing the foreward :roll:

It's been a while since I last look at them but I've been somewhat less than enthused with the Python IDE's and still just use a text editor and some judiciously placed print statements. However, I had a play with Eclipse for the first time the other day, with the Python plugin, and it looks like it could be quite nice.

We should maybe put this stuff up on the wiki :idea:

Thanks ... Sedning Email using Python ..

Posted: Tue May 19, 2009 10:17 am
by Otonicara
The post is very useful. I can finally see what my code is getting and do interactive testing at the same time... Great! THANKS!

Here is one more mystery to resolve. It is not directly Awasu related however is Python/News related. I would like to send email using python code to alert me once extremely important news is arrived. Bellow is the code. The only problem is that Outlook security request manual confirmation. Bellow is explained how security can be tricked in VB. I would appreciate advise how the same can be done in Python.. I have asked question on Python Tutors mailing list however this time I didn't receive any good advice (mostly I receive there very good tips). Python is really powerful programming language. Thanks to Awasu I start using it a year ago ..

On PC I have installed Outlook 2003.

Here is the code:
from win32com.client import Dispatch
session = Dispatch("MAPI.session")

session.Logon('OUTLOOK') # MAPI profile name
msg = session.Outbox.Messages.Add('Hellow', 'Test Message 400')
msg.Recipients.Add('ales', '')

On the web I found solution for VB.NET. There must be similar solution also for Python. ... tomate.php

Dim SecurityManager As New AddinExpress.Outlook.SecurityManager

I tried running VB code inside Python however with no success as this is not VBScript.

import win32com.client as win
vbscript = win.Dispatch("ScriptControl")
vbscript.language = "vbscript"
Sub Main\n\
OlSecurityManager.ConnectTo OutlookApp\n\
OlSecurityManager.DisableOOMWarnings = True\n\
End Sub\n')

Looking forward to your advice ...

Re: Thanks ... Sedning Email using Python ..

Posted: Tue May 19, 2009 12:46 pm
by support
I don't know much about MAPI (kevotheclone might be able to help here) but I suspect it might be a bit involved doing it from Python.

But there's no rule that says Awasu plugins and hooks have to be written in Python. You've got some VB code that does the job, so just write your hook in VB. All you need is a bit of wrapper code around it that gets all the information Awasu passes in.

Re: Thanks... Sending Email using Python...

Posted: Tue May 19, 2009 11:53 pm
by kevotheclone
Onticara, the code you provided that sets the "DisableOOMWarnings" property can only be used if you've have purchased add-in-express's "Security Manager", that costs $149.

There are other ways to send email; so unless you really need some special Outlook functionality (see below*), save your money. Windows has the Collaborative Data Object (CDO) library that you can use from VB/Python or any other languages capable of creating COM objects. Here's a good page with VB/CDO email examples:

Python also has built in email capabilities, so we don't even need Outlook or CDO.

Code: Select all

import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText
from email.Utils import COMMASPACE

def sendEmail(fromAddr, toAddr, subject, text, server, userID, password):
    assert type(fromAddr)==list
    assert type(toAddr)==list

    s = smtplib.SMTP(server) 
    s.login(userID, password)

    msg = MIMEMultipart()
    msg['from'] = COMMASPACE.join(fromAddr)
    msg['to'] = COMMASPACE.join(toAddr)
    msg['subject'] = subject
    s.sendmail(fromAddr, toAddr, msg.as_string())

sendEmail([''], \
          [''], \
          'my subject text', \
          'my body text', \
          'smptserver', \
          'userID', \

Note that the "from" and "to" addresses are actually Python List objects so that you can specify multiple email addresses. I'm not sure why the "from" email address should be a List object, but I found this code and have tested it and used it, so it should work for you too.

Depending upon your email server the login code may be slighty different. Also if your email server is using a non-standard TCP/IP port then you'll need to change this line:

Code: Select all

s = smtplib.SMTP(server)

to something like this:

Code: Select all

s = smtplib.SMTP(server, portNumber)

You're getting close, don't give up!

*Special Outlook functionality that you probably don't need for your Channel Hook
Do you need to have a copy of the email saved in your "Sent Mail" folder?
Do you need to send Meeting Requests?
Do you need to send Task Requests?
If not you probably don't need to use Outlook to send emails.