Convert Microsoft LDAP ObjectGuid from base64 to Python UUID

I needed to convert an objectGuid from Microsoft AD which was provided to me in base64. What I did not know is the Microsoft uses little-endian UUIDs instead of big-endian like the rest of world uses. This caused issues trying to convert the binary to UUID. Luckily, Python's UUID has bytes_le (little-endian) just for this use case.

Blogging this for the future.

Python Sort List of Dictionaries with Accented Characters

I needed to sort a list of dictionaries by a label key. It happens that the labels are in FR and normal sorting causes issues with ordering when accented characters are present. I happened to have PyUCA installed from PyPi.

Gunicorn dyno death spiral on Heroku -- Part II

After a lot of investigation, we've figured out there is an issue with NewRelic, Postgres and Gunicorn. I summarized the issue here:

https://discussion.heroku.com/t/gunicorn-dyno-death-spiral/136/13

After discussing this with Graham Dumpleton over Twitter there is an issue with libpq. Below is a summary of a rather long Twitter discussion. Anything in quotes is from Graham but could have been paraphrased or reworded slightly to make sense here. Didn't want anyone to think that using Graham's words as my own...

The real issue is caused by "the use of the NewRelic agent in a background thread which does SSL HTTP calls -- surfaced issues with libpq SSL connections to database." This can be replicated by the use of a script someone wrote when the bug was reported to Postgres in October, 2014 (see link below). It's not the fault of the NR agent -- just that it uses a background thread and triggers the same behavior. That's why you can reproduce the issue without the NR agent.

So the "NR agent will create a background thread, but if you had other threads for other reasons which did SSL connections, [it] still occurs. If process is completely single threaded without multiple request handler threads, nor background threads doing SSL, [then it] is okay."

http://www.postgresql.org/message-id/CAHUL3dpWYFnUgdgo95OHYDQ4kugdnBKPTjq0mNbTuBhCMG4xvQ@mail.gmail.com

So in a perfect storm, libpq deadlocks which causes large issues for Gunicorn. The reason why is how Gunicorn is designed and that a "main thread is used to handle requests and if that deadlocks then signals aren't handled or if it uses a pipe of death, it will never return to accept on connection where it gets message to shutdown."

By default, "Django creates a new database connection per request which exacerbates the problem." So the problem can mostly be alleviated by using some sort of database connection pooling -- either what is built-in into Django 1.6+ or something like django-postgresql-pool or PGBouncer however it still can cause issues for Gunicorn as the db pool only reduces the likelihood of the problem. Also, because of the way the main thread works -- that is why I personally saw that the Gunicorn timeout directive have no affect on the problem because the worker was still waiting for Postgres and therefore was still alive despite the fact that Heroku killed the request on the client side.

The only real work around until libpq is fixed is to use something other than Gunicorn like Uwsgi and deal with the Harakiri requests OR don't mix DB calls when calls to HTTPS resources. In our case, we are using Amazon S3 so some requests need to make a bucket request to get information and sometimes cause this deadlock issue.

I know many other people in my local user group that have written off Gunicorn on Heroku thinking the issue was Gunicorn and/or New Relic. However, the problem is in Postgres and it is exacerbated when the NR agent is used.

While we (GreatBizTools) has a workaround in place, it would be great if Heroku (and maybe you can get New Relic) to poke at the Postgres folks to fix this deadlock into that was reported to them in October, 2014. I can't image the number of folks that have been caught out in the rain in the past year or so by this. There are several blog posts that (now wrongly) point fingers at Heroku for not supporting Gunicorn correctly.

We can't be the only people to use Heroku on Python with New Relic and Postgres? It must be a really popular combination.

---

Edit: Heroku is now aware of the issue and is working with respective parties in order to rectify this low level issue. -- January 16th, 2015

Gunicorn dyno death spiral on Heroku

FYI -- Gunicorn dyno death spiral on Heroku -- Part II is now available

-----

We recently released our app XXXX on Heroku using Gunicorn however we quickly found in even the most modest of production load (as little as 10 users) that some dynos would stop responding and start throwing continuous H12 errors for hours.

We experience three separate events (from January 5-6) where one or more dynos would stop serving requests with Gunicorn and throw H12 errors for every request and the load metrics would spike from .2-.5 to 1.5 or higher on that particular dyno. The only remedy was to manually run heroku ps:restart web.X after reading logs and kill the appropriate dyno.

We experienced the same issue as outlined on this thread on the Heroku forums:

https://discussion.heroku.com/t/gunicorn-dyno-death-spiral/136

We were able to track it down to "bad clients" using the application -- they were always Verizon Wireless or Sprint Mobility aircards on laptop computers. We have a single client using this application so it was easy to confirm with them that the reverse IP was indeed Verizon Wireless or Sprint.

Our guess is that a client would not close a connect or respond with ACK messages for the streamed response and therefore exceed the 30 second limit. When Heroku performed an H12 on it, it left the worker on Gunicorn to continue working -- left tied up in an unrecoverable state. This would repeatedly happen (we were only running 3 workers per dyno) until all workers on a single dyno stopped responding. At this point, the the routing mesh would continue routing requests to this rogue dyno but the dyno would just return H12s until it was manually restarted.

We have confirmed it is NOT our application code. The application runs just fine when Gunicorn is swapped out with Uwsgi (we also tested Waitress with success as well). Currently, we are running Uwsgi on the XXXX application since the evening of January 6th. We have not experience any more events where dynos would death spiral out of control after switching to Uwsgi permanently. We still occasionally see a bad client and request -- however we are using the Harakiri option in Uwsgi and the rogue worker is killed and respawned after 25 seconds.

The question we have is why Heroku continues to recommend using Gunicorn when other people like ourselves have experienced terrible results with this particular application server.

Show Django-Debug-Toolbar when development IP addresses are dynamic (Vagrant, Landrush, etc.)

In certain circumstances, your IP address in development changes and therefore it's hard to have all the IP address in the list of INTERNAL_IPS that Django Debug Toolbar uses.  In your development.py settings file (you separate them out right?), you can add this to shortcircuit the logic and allow ANY IP address (be careful):

def show_toolbar(request):
    return True

DEBUG_TOOLBAR_CONFIG = {
    'SHOW_TOOLBAR_CALLBACK': 'YourAppName.settings.development.show_toolbar',
}

Django-Stronghold with Django-Debug-Toolbar

Django Stronghold intercepts calls to Django Debug Toolbar panels which cause the panels to show your login page. This can easily be fixed to exclude the Debug Toolbar urls in your settings.py file:

STRONGHOLD_DEFAULTS = True
STRONGHOLD_PUBLIC_URLS = (
    r'^/__debug__/.+$',
)

The STRONGHOLD_DEFAULTS tells Stronghold to include regex patterns for Static and Media file urls so your static assets will work too.

Correct connection settings for ElasticSearch / Django-Haystack on Bonsai.io

I should have read the documentation about this better, so I'm blogging this because I'm sure other people have had the same issue. Bonsai.io requires the username and password to be sent as an http_auth header.

from urlparse import urlparse

es = urlparse(os.environ.get('BONSAI_URL'))
port = es.port or 80

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': es.scheme + '://' + es.hostname + ':' + str(port),
        'INDEX_NAME': 'your_index_name',
    },
}

if es.username:
    HAYSTACK_CONNECTIONS['default']['KWARGS'] = {"http_auth": es.username + ':' + es.password}

Installing Cython - x86_64-linux-gnu-gcc: error: unrecognized command line option ‘-fstack-protector-strong’

I went around and around trying to get Cython install on my Ubuntu 14.04 LTS box.  I kept getting:

x86_64-linux-gnu-gcc: error: unrecognized command line option ‘-fstack-protector-strong’

Turns out that the ‘-fstack-protector-strong’ option was not added to the GCC compiler until version 4.9.  Upgraded my GCC to the latest available fixed the issue.

Uninstall all Python packages via pip

This is for my future reference because it's really handy if you don't want to drop a virtualenv or doing some crazy work on a vagrant box.  Even ignores any packages you installed from git or vcs sources with the -e flag

pip freeze | grep -v "^-e" | xargs pip uninstall -y

Reaction to Two Scoops of Django 1.6 FAQ - No Digital Edition?!?!

Considering that we publicly offered a free electronic copy to those who requested it, seeing piracy of the electronic editions from the first day of sales has been disheartening. As indie authors, it hurts to see this happening.

It is awful to think of the piracy that occurred, however this just punishes the people that want an electronic format and are honest in the first place. If somebody wants to pirate a copy, they will do it whether an electronic edition is available or not.

I can totally understand the argument on multiple ebook formats being an issue due to formatting.  The solution is simple -- only offer a PDF that is a rendering of the physical book.

It is legal (* from what I can tell) to scan a book that was legally acquired for personal use. This falls under the fair usage laws (similar ripping MP3s of CDs you own).  I've scanned books at home into PDF and then shredded them (I have fairly limited space for tech books at home).  The results are only fair and are better if I slice off the binding.

It is more than likely that with this edition of Two Scoops of Django, I will buy via Amazon and ship directly to 1dollarscan.com which will scan ($1 per 100 pages) a book into a PDF and then after two weeks they shred the book (they do not return books).  Talk about a waste of natural resources -- paper manufacturing, printing, ship to a warehouse and ship to a scanning company, etc.

Maybe an alternative is to sell PDFs but password protect them which makes it easier to figure the origin if a pirated copy is discovered and watermark the header on the top each page with the name, email and phone number of the original purchaser.

Physical books won't stop the pirates.  Yes, it does make it harder for them (marginally), however this just punishes the majority of honest people that used the electronic format in a responsible and legal manner for the few (relatively) that decided to pirate a copy.  So the only thing that has changed is increased the cost of my book by about $4 (the cost to scan at 1dollarscan.com) and I can no longer search it (unless I pay extra for the OCR option).

Edit on Feb. 7th, 2014 due to feedback -- I want to clarify some points:

This post is a reaction to not publishing a digital edition and points to how I'm probably solving the "no digital edition" issue for myself.  I will still purchase a legal physical copy of the book. I still want Danny and Audrey to take my money because I want the content.  I just expressed my wishes to get an official digital edition. Using a scanning service is a lazy hack, however Fair Use hack (as long as I don't distribute it).  Also, I do not support pirating media and therefore this post isn't a "how to pirate" manual -- merely a legitimate fair use solution to having a digital version for personal use.

Other than this post... I have not participated in any public discussions (other than the single automated tweet when this post was publishing).  I've now been counted and blamed as one of the trolls that "caused" future editions of Two Scoops of Django to not happen. This is a rather unfair assessment.

The point is pirates will still pirate a copy of this book because they want to.  If anything, this is a testament to quality of the content that Danny and Audrey produce.  I must admit it is sad that the availability of pirated copies of Two Scoops is a compliment to the authors.  It does reinforce that quality content is in demand and there are people that un-willing or just plain too cheap to buy a legitimate copy.

The real losers here are the legitimate users that want to buy a legal copy of a book. The only thing done by not making a digital edition available is that making the first pirated copy is just slighly harder to make. I would guess about 45 minutes at a flatbed scanner is about it. Not having a digital edition won't stop piracy at all and therefore I don't except that as a legitimate reason to not do a digital edition.  On to a real reason... time!

Two Scoops 1.5 edition was released with three digital edition types (mobi, epub and pdf).  The PDF looks like the print copy and I never used the epub or mobi editions.  Considering the target devices for epub and mobi, I suspect it took Danny and Audrey a LOT of time to probably do it right (which they indicated in their FAQ).  This was an ambitious and lofty goal to release so many formats. Kudos to them for doing it however I'm sure it contributed to the reason to not release a digital edition for this version.  So it understandable that time is a factor for the authors (rightfully so). The solution is export a PDF version of the book and skip the tedious hand grooming of epub and mobi formats.

Micro Python: Python for Microcontrollers

Micro Python: Python for Microcontrollers

I backed this Kickstarter for £28 (includes £4 for shipping to the USA).  Sounds like a bunch of fun over Arduino.

Micro Python is a lean and fast implementation of the Python programming language that is optimized to run on a microcontroller.  The Micro Python board is a small electronic circuit board that runs the Micro Python language.  The aim of this Kickstarter campaign is to make Micro Python open source software so you can use it in your own projects, and also to fund a small manufacturing run of Micro Python boards so that you can own one for yourself!

Python Tip: Thanking People is Important

Thanking people for the time they spend on helping you is important.  Email is great, but impersonal.  Consider taking the time to write a hand written message in a real card.  It's the personal touch that counts -- showing that you really thought about it and appreciate their efforts.

Installing VPython (Visual Python) on Linux

VPython on Linux is a pain to install because they recommend using WINE or compiling from source.  I was able to compile from source but it took over an hour to compile a special patched version of wxPython and then compile that into VPython.  It was a less than "fun" experience and requires knowledge of compiling.  I discovered this during Kirby Urner's Leveraging Python tutorial at DjangoCon US 2013. So I set out to find a better way with fellow attendee German Larrain.

We're waiting on a pull request to the main GitHub repo that fixes our problems however we figured out a way to install VPython using PIP.  Its as easy as using PIP ability to install from a GIT repo.  In the meantime, you can use German's fork of VPython at GitHub:

sudo pip install git+git://github.com/glarrain/vpython-wx.git

Edit (Nov. 2013):

The pull request mentioned above has been merged into the main VPython git repo.  However the installation instructions have changed due to the addition of the Numpy package and on my system I needed to install a newer WxPython (which much have been installed on my old system and why it wasn't on the original instructions).

Here's how I installed WxPython 2.9.4 on Ubuntu 13.10 (saucy -- no deb was available yet).  I couldn't get 2.9.5 to build on my system - it kept failing in the make process at random places. You do need GTK2 dev package (as indicated as the first command) if you do not have that installed yet and the gstreamer base dev package (second command).  If you get errors about missing .h files, then your system is missing a -dev version of some package that is indicated.

sudo apt-get install g++ libgtk2.0-dev freeglut3-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev python-dev libgtkglextmm-x11-1.2-dev libboost-python-dev libboost-signals-dev libboost-python-dev libboost-thread-dev tk
wget http://downloads.sourceforge.net/project/wxpython/wxPython/2.9.4.0/wxPython-src-2.9.4.0.tar.bz2
tar -xvf wxPython-src-2.9.4.0.tar.bz2
cd wxPython-src-2.9.4.0/wxPython
python build-wxpython.py --build_dir=../bld

Note the output that say something like -- we will need this later:

To run the wxPython demo you may need to:
 - set your PYTHONPATH variable to /home/peter/wxPython-src-2.9.4.0/wxPython
 - set your LD_LIBRARY_PATH to /home/peter/wxPython-src-2.9.4.0/bld/lib

Note that Numpy will take some time compiling all the C code on your system. I have an i7 with 8 cores it took a few minutes -- may look like it hangs, but be patient. Also, it may take some time for PIP to download the tarball depending on your internet connection speed -- it may look like it hangs downloading, but again be patient.

sudo pip install numpy
sudo pip install https://github.com/BruceSherwood/vpython-wx/archive/master.zip

At this point, we need to create .sh executable to run vidle.  Create a file named vidle.sh with the following text (replace paths with the stuff we noted above) and ensure this file is executable (chmod 777 vidle.sh).

export PYTHONPATH="/home/peter/wxPython-src-2.9.4.0/wxPython"
export LD_LIBRARY_PATH="/home/peter/wxPython-src-2.9.4.0/bld/lib"
python /usr/local/lib/python2.7/dist-packages/VPython-6.03-py2.7-linux-x86_64.egg/vidle/idle.py

Code Python anywhere online via your browser - PythonAnywhere

Code Python anywhere online via your browser - PythonAnywhere

Totally awesome with Python, Bash, MySQL consoles... GitHub integration, etc.

Recover Lost Admin Password in Django

./manage.py changepassword <username>

Run this in your project directory.

Excited for PyCon 2013 - Live Blogging It!

PyCon 2013 is the first conference since 2007 that I will be attending and not speaking at the conference itself.  So I actually feel like I can focus on being an attendee instead of a speaker.  So this is going to be a different experience for me and I'm already planning what I want to do at the conference.

I've decided I'm going to live blog the sessions that I attend at PyCon using Google Drive.  This means you'll be able to even watch me take my notes as the session goes on.  So I'll be blogging the live Google Doc links in a future blog post.  I'm so excited!  Watch this space...

pyMNtos - February Meeting Agenda - David Goodger on The Zen of Python Revisited and 3 Lightning Talks

Our next meeting on Thursday 2/21 (pizza at 7p / meeting at 7:30p -- Please RSVP) includes three Lightning Talks:

• MyHDL -- Chris Felton
• Docopt -- Andrew Carter
• Building Web UIs with Twitter Bootstrap -- Peter Farrell

David Goodger will be presenting a longer talk on The Zen of Python Revisited -- A fresh look at the meaning of Python's Zen. Does it mean what you think it means?

David is best known in the Python world for Docutils / reStructuredText and for chairing the 2008 & 2009 PyCons in Chicago.

We might broadcast this as a Google+ Hangout for those interested in attending remotely or out of state.  Let me know if you are interested as I don't want to setup all the mics and camera if nobody is going to come.

PyCharm - Initial Setup - Don't use OpenJDK

I installed PyCharm today and for the life of me couldn't get it to run right.  Don't use OpenJDK.  There is a check in the pycharm.sh to check for OpenJDK and warn...but I didn't see it because I didn't launch it from terminal.  I launched it from Nautilus.