Categories
Programming

DataConnect.be blog is up.

I’ve set up a new blog connected to my programming efforts. It’s on DataConnect.be, the website I set up to store any tools/programs I making.

It’s a bit empty right now, and several links are still “dead”, like for example the demo, but I’m going to be adding more blog posts there that are program-related. I’ll note here any new posts that appear there.

On the progress front :

  • I’ve made some progress on following keywords, storing them, and collecting twitter user information. Everything is running stable on my laptop, in text version.
  • JQuery and JQuery-UI still rock, I’ve now found out how modal dialog boxes can be called up in a ajaxy way, which is really cool ! Coupled with cherrypy, I can initiate a dialog box that calls a page that does the db update and shows the results… really nice and much more user intuitive than being thrown on another page that tells you it’s done and you need to click “return” to see what changed.
  • in the following weeks I’ll be setting up a database, some keywords and a cron job on my webfaction host – I’ll need to see how they behave and if there are problems or tweaking to be done
  • still to do: setting up the graphics to show what I’m collecting, after all, text is not sexy.
  • after that, it’ll be writing the demo pages so that at least I can show what I’m doing
  • after that, the scary part comes – asking for feedback… 🙂
  • further than that I can’t or won’t look at the moment

I also discovered quite a few programs that analyse twitter, and for some time I got into a funk about one especially called hootsuit. It is really very very good ! It’s also very free, and it’s a bit worrisome to my idea of trying to gain money from my twitter analysis program if there are such good and great programs that are free. Obviously, a lot of care went into that program, and it’s very smooth and intuitive to use. Higly recommended to get a good overview of your tweets and following up on who said what. There are other programs out there (Social Oomp is one) that I admire (and which does ask money for the pro version).

That said, my focus lies on the keywords and not on the individual tweets. What I want to implement is different from what they are doing, so I still see added value for what I’m doing.

Categories
Programming

Ouch! Rate limiting on twitter request.

I just ran into the rate limiting of twitter. It hurts. 150 twitter requests an hour, at least for the search you can do more…

Have to think how to give twitter some rest inbetween my scurrying in their databases for user info… I’m gonna sleep on it.

Categories
Programming

Using CherryPy for webform authentication

If you are using CherryPy, I can recommend the webform-based authentication that Arnar Birgisson wrote for ease of use and extensability.

After trying out the included authentication models with CherryPy (I’m using 3.1.2, the last stable version at the moment of writing), I was disappointed in the results. Then I stumbled over a recommandation from someone on Nabble, a web-based programmar’s discussion forum, which pointed to the following wiki page on the CherryPy site:

http://tools.cherrypy.org/wiki/AuthenticationAndAccessRestrictions

The complete program code plus examples are on the page and are well explained.

You can have a skeleton login system (using a hardcoded dictionary) up and running in literally half an hour !

  • Just copy/paste the code on the page and save it as auth.py in your cherrypy script dir.
  • Add the hardcoded dictionary containing username and passwords to it (or script the db access, see the example included)
  • Put ‘require()’ everywhere on your cherrypy pages that need to have login protection – additionally, you can also have roles so that only admins can access certain pages.

Early last week I replaced that hardcoded dictionary and built the db lookup query for the login. Once that was working, I added a ‘my profile’ page to the application I’m working on.  Then I thought it would be nice for the admin to have a ‘create user’ form in the admin section to add users. Done that as well, using the jquery-ui to create tabs and seperate content in the admin section.

All in all, a nice week of nice work.

I’m starting to think this might make it’s way to my hosting server one of the coming weeks…  although I need to do some more work on showing the user only his keywords and not all the keywords, as well as doing something with the keywords to use them better.

Oh and one more thing: this works better under SSL than in the clear http: sky !

Categories
Programming

Simple Python threading.Thread example using Queue

I managed to write a really simple example of using threads in Python that I hope will give more insight on how to adapt my other programming stuff. And re-use this later on, in case I need to revisit this again it would be handy not to scour the internet again to assemble the bits and pieces of threading with Python.

The example below uses 3 threads, and processes 10 pairs of numbers (tuples) that I put in a list.

# Our list of work todo
inputlist_ori = [ (5,5),(10,4),(78,5),(87,2),(65,4),(10,10),(65,2),(88,95),(44,55),(33,3) ]

Those numbers are divided over those 3 threads by the Queue system.

The Queue system itself is limited to 5 slots, although this could easily be changed to more or less. You will notice in the console print that the message “Waiting for threads to finish.” appears after the fifth result, indicating that the queues are being used and the main program has continued on.

After putting everything in the queue system, the program waits for the threads to finish using the .join() function.

All spawned threads keep on being active, running forever, accepting jobs – that is, until the queue is empty, at which point they shut down.

I based most of my simple example on the examples in the Python threading tutorial (.pdf) work of Norman Matloff and Francis Hsu that I referenced before in a previous blog post. However, while their examples undoubtedly do more and are more extensive, they are also more complex. This example is deliberately made as simple as possible so to understand the basic principles of threading and the queue system.

Things I stumbled over:

  • Duh! You spawn the threads before you fill up the queues with stuff todo…
  • When printing out things to the console or python shell, things got jumbled because different threads took over from each other – to solve that I used the threading.Lock().acquire() and threading.Lock().release() to make sure that a thread could finish printing. Not sure if I understand completely all the possibilities this offers.
  • Still a bit stumped on getting more info, name, etc on the thread that is running at the moment – haven’t figured that out yet how to do that.

Feel free to comment and ask questions – if you can improve this program, please let me know !

# threading test
# Alex Boschmans
# www.boschmans.net
# January 2010

#
# IMPORT SECTION
#
import threading, Queue

#
# Variables setup
#
THREAD_LIMIT = 3                # This is how many threads we want
jobs = Queue.Queue(5)           # This sets up the queue object to use 5 slots
singlelock = threading.Lock()   # This is a lock so threads don't print trough each other (and other reasons)

# Our list of work todo
inputlist_ori = [ (5,5),(10,4),(78,5),(87,2),(65,4),(10,10),(65,2),(88,95),(44,55),(33,3) ]

#
# This is called from the main function
# It spawns the threads, fills up the queue with work items that the threads will use
# And then waits for the threads to finish
# This could use some more try:except code...
#
def draadje(inputlist):
    print "Inputlist received..."
    print inputlist

    # Spawn the threads
    print "Spawning the {0} threads.".format(THREAD_LIMIT)
    for x in xrange(THREAD_LIMIT):
        print "Thread {0} started.".format(x)
        # This is the thread class that we instantiate.
        workerbee().start()

    # Put stuff in queue
    print "Putting stuff in queue"
    for i in inputlist:
        # Block if queue is full, and wait 5 seconds. After 5s raise Queue Full error.
        try:
            jobs.put(i, block=True, timeout=5)
        except:
            singlelock.acquire()
            print "The queue is full !"
            singlelock.release()

    # Wait for the threads to finish
    singlelock.acquire()        # Acquire the lock so we can print
    print "Waiting for threads to finish."
    singlelock.release()        # Release the lock
    jobs.join()                 # This command waits for all threads to finish.

#
# Main thread class - based on threading.Thread
# This class is cloned/used as a thread template to spawn those threads.
# The class has a run function that gets a job out of the jobs queue
# And lets the queue object know when it has finished.
#
class workerbee(threading.Thread):
    def run(self):
        # run forever
        while 1:
            # Try and get a job out of the queue
            try:
                job = jobs.get(True,1)
                singlelock.acquire()        # Acquire the lock
                print "Multiplication of {0} with {1} gives {2}".format(job[0],job[1],(job[0]*job[1]))
                singlelock.release()        # Release the lock
                # Let the queue know the job is finished.
                jobs.task_done()
            except:
                break           # No more jobs in the queue

#
# Executes if the program is started normally, not if imported
#
if __name__ == '__main__':
    # Call the mainfunction that sets up threading.
    draadje(inputlist_ori)

Sigh. I just finished adding spaces to show where a def ends, and the damn code highlighter removed it again. Grrrrrr. If you want a copy of the code, let me know and I’ll update this post with a zipped copy of it.

Update: just discovered the “Syntaxhighlighter evolved” plugin and updated the code – indentation now works !!!

Categories
Programming

Not using regular expressions (re or regex) to find a #hashtag (python).

First, a quick reminder for myself: there’s an extremely good guide to regex on Andrew M. Kuchling’s pages.

Secondly, you don’t really *need* regex to parse for hashtags in a tweet – it’s a bit of overkill. The following code will do as well, and was written in 1 minute after searching 15 minutes in regex how to make certain to include hyphens ( – ) and other non-characters if they are put into the hashtag.

The regular expression that I find works quite well for all hashtags that don’t have a hyphen in it:

>>> hashtag = "This is a #hashtag #test-link #a should#not#work"
>>> x = re.compile(r'\B#\w+')
>>> x.findall(hashtag)
['#hashtag', '#test', '#a']

So the above code correctly finds all words beginning with a hashtage, and not the ones that contain a hashtag inside the word. Note that the hyphen and the word after it is not included.

This is the short code I wrote that does all I want:

>>> hashtag = "This is a #hashtag #test-link #a should#not#work"
>>> for word in hashtag.split():
	if word[0] == "#":
		print word		
#hashtag
#test-link
#a

In section 6 of the above-mentioned guide, Andrew states that in some cases string methods (like split) are faster than using regex. For simplicity, I’m going to use the latter code.

Update: Grrr – discovered that the tweets I am processing are in html so have href tags around them – which means ofcourse that there are no blanks for me to split words in. After another unsuccessful session with regex and just to continue I’ve used the BeautifulSoup html parsing library to get around that by stripping out all tags and then splitting the sentence up again. Probably not as efficient as immediately using regex, I’ll have to revisit this in the future.

Categories
Programming

Using threads in Python

I’ve been trying to setup threading in Python, so that in the back-end of my service system that I’m developing I can query more than one source at the same time. So instead of querying one server and waiting for feedback, I can launch 10 threads and thus query 10 servers and process each server’s feedback via it’s own thread.

So a very vague, generalising definition of a thread is an independent ‘process’ that performs a job that you give it. You can control how many threads that you launch. Each thread is a copy of the original thread that you describe (in essence a python def function that has been wrapped in a thread class).

Right now, my understanding of threads is a bit confused. So far it seems that threading has several different manners of implementing them:

  • using a number of threads that you launch, use, and forget about them (they go away)
  • improving on that by putting those threads in a thread pool, and when a thread finishes, re-using it for the next job (so you have  5 threads but 10 jobs to do, those five threads take five jobs, and the first thread that finishes takes on the sixth job, the second thread to finish the seventh job, and so on)
  • the final step seems to be (I haven’t got that far in my implementation) to set up worker bees that are managed by one thread (a better description is promised, as soon as I have understood it!)

Since I’ve been scouring the net for information over threads, here is a list of resources that discuss, give examples, and explain threads – it’s useful for me to refer to, it might be useful for you as well :

  • DaveN has an extensive post, with examples, building up gradually. It’s only at the end that you read that the code shown has never been run, which is a bit of a letdown. Still worth a good read though !!
  • A very thorough 25-page pdf documents that starts from the beginning is available on the site of UC Davis, University of California. It goes into all the nitty gritty details.
  • An example that uses workers in threads is found on the blog of Danial Taherzadeh.
  • Another one that discusses using multiple queues chained together can be found on IBM’s developerworks site.
  • And the blog post from Halotis that started my looking into threads…

Right now I’m using threads in a thread pool, but I’m not doing something right – I noticed that while I have 10 jobs to do, only the first five get done, and the others ‘disappear’.

I guess the only way to get it working is to continue reading the information above until it makes sense. Sometimes I wonder if I’m not slightly masochistic, looking for challenges like that… ai me poor pounding head ! 🙂

Categories
Programming

Feedparser.py and it’s uses…

I recently discovered feedparser.py, a library written by Mark Pilgrim that is amazing if you want to use python to consume rss feeds. It ‘normalizes’ the different versions of rss/atom out there into one request that you can use consistently. Doesn’t matter if it’s atom 0.1 or 0.3

A few links that are interesting together with feedparser.py as they show it’s usage:

I’m constantly amazed about the quality python code that is out there and you can just find via a simple google query. It certainly makes me think that choosing Python over, say Perl, was a good decision.

As for using feedparser.py to put relevant tweets on your website, note that you can also use javascript to achieve the same thing; go here for some twitter.com goodies and an explanation on how to set this up.

Categories
Programming

Cleaning up user input variables on the web (Python)

Only recently I’ve discovered the power of ‘re’ the python regular expression library. Instead of writing long functions that process text character by character to add or remove stuff, you use re, write and expression in regex that achieves what you want and basta! in a few lines things get done.

For example the following function will remove any html tags (preventing Cross Site Scripting) and escape the rest of whatever the user types in:

# Remove html tags and escape the input
def scrapeclean(text):
----# This matches open and closing tags and what's between them
----x = re.compile(r'<&#91;^<&#93;*?/?>')
----# Replace to nothing using sub and escape what's leftover and return the result all in one line!
----return cgi.escape(x.sub('',text))

Remove the dashes when you copy the code – they were added to show the necessary indentation. And for full disclosure : I took the compile statement from the following site (I’m not a regex expert).

So you can call this function from somewhere in your python code and the result will be ‘scraped clean’ of all tags beginning with < and ending with > plus any ampersands other other special characters get to be ‘escaped’.

YMMV – this is very likely not a complete protection against all the things a hacker can input in your website, but it’s certainly a start.

Categories
Blog News Programming

AIR Badge plugins for WordPress take away all the hard work!

This is soooo cool.

Creating an AIR badge for an Adobe AIR application is a bit of a hassle, but the following 2 WordPress plugins really do take all the work out of it. Any post where you want to add an Air badge, you’re done in 3 minutes.

You’ll need either one of the following plugins:

– The Original plugin, made by Peter Elst, which is easy to use
– The Updated plugin with built-in click tracking (requires a bit more work)

I decided to use the original plugin, and you need to do just 3 things to use it.

  1. install the plugin (in your admin menu, just go to the plugin section and use the “search plugins” button to search for “air badge”
  2. upload your .air file to the server (if you want to use the wordpress “Add Media” uploader, you might also need to install an additional plugin called “PJW mime-config” and add .air to the list.
  3. create your badge by writing the following magic words between the words ‘airbadge’ : application name, full URL to .air file, application version, image.jpg

You’re done ! Admire your work (and that of the guy who made it possible, Peter Elst)!

Categories
Programming

Python Package Manager

Python Package Manager Logo
Python Package Manager Logo

The Python Package Manager is here, a visual tool for the python developer to find and install all the necessary packages.

It shows you what is already installed on your system, with the option to deinstall the packages, and by typing into the search box you can find additional packages and install them, all graphically. It’s supposed to be cross-platform, but the homepage of the developer only provides a windows download option.

I just hope this gets used and keeps being supported, as it is a lot handier than using the command line ! I do think you still need easy_install and wxpython/wxwidgets though…