Convert Microsoft LDAP ObjectGuid from base64 to Python UUID

I needed to convert an objectGuid from Microsoft AD which was provided to me in base64. What I did not know is the Microsoft uses little-endian UUIDs instead of big-endian like the rest of world uses. This caused issues trying to convert the binary to UUID. Luckily, Python's UUID has bytes_le (little-endian) just for this use case.

Blogging this for the future.

Python FizzBuzz

I watched Tom Scott's One Simple Interview Question on YouTube earlier and decided take a stab at a flexible implementation of FizzBuzz.

Moto X4 on Google FI - Microphone Muffled / Bad - Possible Fix

I've been having people complain they cannot hear me or that I sound extremely muffled when I making phone calls on my Moto X4. I confirmed it was definitely the phone because I tested it with a wired headphones/microphone without issues.

Many forums suggest returning the device as a RMA. This really isn't an option for my nearly 2 year old phone which is out of warranty. One interesting comment I read was related to the position of the microphone holes -- one of which is very near the fingerprint reader. My phone is always in a case, so the rear microphone hole seemed clear but the front microphone hole seemed slightly dirty.

The phone is rated IP68 which allows for submersion in water for 1 hour up to 1 meter. Since my next option was to buy a new phone, I decided to put the IP68 rating to the test.

Of course, what I did next -- only do at your own risk. I am not responsible for anything you decide to do to your own phone.

With 100% rubbing alcohol, I cleaned out the microphone holes with a cotton swab and let dry. Using a recording app, the sounds was marginally better.   More drops of alcohol but this time I used an electric toothbrush to vibrate all the junk which was more than I expected (ick).

Voila, things sounded 100% better and I no longer need a new phone!

Moto X4 on Google FI - Enable RCS Messages

On my Moto X4 (Google FI), I couldn't enable RCS messages in the Messages application as it was stuck "verifying" my phone number. After a bunch of support forums, this is the procedure that ultimately worked for me.

1. Turn on Airplane mode.
2. Open the Settings -> Apps & notifications.
1. If you don't see all your apps, first tap See all apps or App info.
2. At the top right, tap More More and then Show system.
3. Find the Carrier Services app.
1. Tap Force stop and confirm.
2. Tap Storage and then Clear data.
4. Go back to the "App info" screen and find the Messages app.
1. Tap Force stop and confirm
2.  Tap Storage and then Clear data.
5. Turn off Airplane mode.
6. Open Messages and checked the settings, it should said "Connected - Chat features are ready for use".

I'm blogging this so I don't forget the process if this ever comes up in the future.

Python Sort List of Dictionaries with Accented Characters

I needed to sort a list of dictionaries by a label key. It happens that the labels are in FR and normal sorting causes issues with ordering when accented characters are present. I happened to have PyUCA installed from PyPi.

Custom 404 Page for Django CMS

For a side project, I needed a 404 page that was editable by users in Django CMS. Suffice it say, it took a while to figure out how to do it without having the Django cache the CMS page response and provide the right 404 http status code. Other techniques wrongly serve the page as a 200 OK which is wrong for search engines.

1. First create a 404 page in Django CMS and publish it.
2. Set you handler404 in your urls.py (change the path to your view file accordingly):
   handler404 = 'shared.views.page_not_found'
3. The view:
   
   

   1:  from cms.views import details  
   2:  from django.http import HttpResponse  
   3:  def page_not_found(request, exception):  
   4:    response = details(request, '404')  # 404 is the slug you named your page in Django CMS
   5:    return HttpResponse(content=response.rendered_content, content_type='text/html; charset=utf-8', status=404)  
   

Gunicorn dyno death spiral on Heroku -- Part II

After a lot of investigation, we've figured out there is an issue with NewRelic, Postgres and Gunicorn. I summarized the issue here:

https://discussion.heroku.com/t/gunicorn-dyno-death-spiral/136/13

After discussing this with Graham Dumpleton over Twitter there is an issue with libpq. Below is a summary of a rather long Twitter discussion. Anything in quotes is from Graham but could have been paraphrased or reworded slightly to make sense here. Didn't want anyone to think that using Graham's words as my own...

The real issue is caused by "the use of the NewRelic agent in a background thread which does SSL HTTP calls -- surfaced issues with libpq SSL connections to database." This can be replicated by the use of a script someone wrote when the bug was reported to Postgres in October, 2014 (see link below). It's not the fault of the NR agent -- just that it uses a background thread and triggers the same behavior. That's why you can reproduce the issue without the NR agent.

So the "NR agent will create a background thread, but if you had other threads for other reasons which did SSL connections, [it] still occurs. If process is completely single threaded without multiple request handler threads, nor background threads doing SSL, [then it] is okay."

http://www.postgresql.org/message-id/CAHUL3dpWYFnUgdgo95OHYDQ4kugdnBKPTjq0mNbTuBhCMG4xvQ@mail.gmail.com

So in a perfect storm, libpq deadlocks which causes large issues for Gunicorn. The reason why is how Gunicorn is designed and that a "main thread is used to handle requests and if that deadlocks then signals aren't handled or if it uses a pipe of death, it will never return to accept on connection where it gets message to shutdown."

By default, "Django creates a new database connection per request which exacerbates the problem." So the problem can mostly be alleviated by using some sort of database connection pooling -- either what is built-in into Django 1.6+ or something like django-postgresql-pool or PGBouncer however it still can cause issues for Gunicorn as the db pool only reduces the likelihood of the problem. Also, because of the way the main thread works -- that is why I personally saw that the Gunicorn timeout directive have no affect on the problem because the worker was still waiting for Postgres and therefore was still alive despite the fact that Heroku killed the request on the client side.

The only real work around until libpq is fixed is to use something other than Gunicorn like Uwsgi and deal with the Harakiri requests OR don't mix DB calls when calls to HTTPS resources. In our case, we are using Amazon S3 so some requests need to make a bucket request to get information and sometimes cause this deadlock issue.

I know many other people in my local user group that have written off Gunicorn on Heroku thinking the issue was Gunicorn and/or New Relic. However, the problem is in Postgres and it is exacerbated when the NR agent is used.

While we (GreatBizTools) has a workaround in place, it would be great if Heroku (and maybe you can get New Relic) to poke at the Postgres folks to fix this deadlock into that was reported to them in October, 2014. I can't image the number of folks that have been caught out in the rain in the past year or so by this. There are several blog posts that (now wrongly) point fingers at Heroku for not supporting Gunicorn correctly.

We can't be the only people to use Heroku on Python with New Relic and Postgres? It must be a really popular combination.

---

Edit: Heroku is now aware of the issue and is working with respective parties in order to rectify this low level issue. -- January 16th, 2015

Gunicorn dyno death spiral on Heroku

FYI -- Gunicorn dyno death spiral on Heroku -- Part II is now available

-----

We recently released our app XXXX on Heroku using Gunicorn however we quickly found in even the most modest of production load (as little as 10 users) that some dynos would stop responding and start throwing continuous H12 errors for hours.

We experience three separate events (from January 5-6) where one or more dynos would stop serving requests with Gunicorn and throw H12 errors for every request and the load metrics would spike from .2-.5 to 1.5 or higher on that particular dyno. The only remedy was to manually run heroku ps:restart web.X after reading logs and kill the appropriate dyno.

We experienced the same issue as outlined on this thread on the Heroku forums:

https://discussion.heroku.com/t/gunicorn-dyno-death-spiral/136

We were able to track it down to "bad clients" using the application -- they were always Verizon Wireless or Sprint Mobility aircards on laptop computers. We have a single client using this application so it was easy to confirm with them that the reverse IP was indeed Verizon Wireless or Sprint.

Our guess is that a client would not close a connect or respond with ACK messages for the streamed response and therefore exceed the 30 second limit. When Heroku performed an H12 on it, it left the worker on Gunicorn to continue working -- left tied up in an unrecoverable state. This would repeatedly happen (we were only running 3 workers per dyno) until all workers on a single dyno stopped responding. At this point, the the routing mesh would continue routing requests to this rogue dyno but the dyno would just return H12s until it was manually restarted.

We have confirmed it is NOT our application code. The application runs just fine when Gunicorn is swapped out with Uwsgi (we also tested Waitress with success as well). Currently, we are running Uwsgi on the XXXX application since the evening of January 6th. We have not experience any more events where dynos would death spiral out of control after switching to Uwsgi permanently. We still occasionally see a bad client and request -- however we are using the Harakiri option in Uwsgi and the rogue worker is killed and respawned after 25 seconds.

The question we have is why Heroku continues to recommend using Gunicorn when other people like ourselves have experienced terrible results with this particular application server.

Python / Django / Selenium: Set viewport size (window size)

The default viewport size for PhantomJs is like a phone size width. I found plenty of examples of setting the viewport size of the window for Java, C# and Ruby but not much for Python.  It's ridiculously simple.  Below is an example basically for my future reminder, but here for your enjoyment. This sets the viewport 1280px wide by 720px high.

CustomLiveServerTestCase(LiveServerTestCase):
    def setUp(self):
        self.wd = webdriver.PhantomJS()
        self.wd.set_window_size(1280, 720)
        super(CustomLiveServerTestCase, self).setUp()

    def tearDown(self):
        super(CustomLiveServerTestCase, self).tearDown()
        self.wd.close()

Python / Django / Selenium: "Webdriver Exception: Unable to startphantomjs with ghostdriver"

You might be using an outdated version of Selenium or possibly running it on Windows. Try the latest version of Selenium for Python first.

However, if that doesn't solve your issues you might have a failing test and not closing the webdriver down in your teardown method in your test if you have a custom test case:

CustomLiveServerTestCase(LiveServerTestCase):
    def setUp(self):
        self.wd = webdriver.PhantomJS()
        super(CustomLiveServerTestCase, self).setUp()

    def tearDown(self):
        super(CustomLiveServerTestCase, self).tearDown()
        self.wd.close()

Show Django-Debug-Toolbar when development IP addresses are dynamic (Vagrant, Landrush, etc.)

In certain circumstances, your IP address in development changes and therefore it's hard to have all the IP address in the list of INTERNAL_IPS that Django Debug Toolbar uses.  In your development.py settings file (you separate them out right?), you can add this to shortcircuit the logic and allow ANY IP address (be careful):

def show_toolbar(request):
    return True

DEBUG_TOOLBAR_CONFIG = {
    'SHOW_TOOLBAR_CALLBACK': 'YourAppName.settings.development.show_toolbar',
}

Django-Stronghold with Django-Debug-Toolbar

Django Stronghold intercepts calls to Django Debug Toolbar panels which cause the panels to show your login page. This can easily be fixed to exclude the Debug Toolbar urls in your settings.py file:

STRONGHOLD_DEFAULTS = True
STRONGHOLD_PUBLIC_URLS = (
    r'^/__debug__/.+$',
)

The STRONGHOLD_DEFAULTS tells Stronghold to include regex patterns for Static and Media file urls so your static assets will work too.

Correct connection settings for ElasticSearch / Django-Haystack on Bonsai.io

I should have read the documentation about this better, so I'm blogging this because I'm sure other people have had the same issue. Bonsai.io requires the username and password to be sent as an http_auth header.

from urlparse import urlparse

es = urlparse(os.environ.get('BONSAI_URL'))
port = es.port or 80

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
        'URL': es.scheme + '://' + es.hostname + ':' + str(port),
        'INDEX_NAME': 'your_index_name',
    },
}

if es.username:
    HAYSTACK_CONNECTIONS['default']['KWARGS'] = {"http_auth": es.username + ':' + es.password}

Dell Sputnik Tip: Mouse / Cursor is broken or frozen after resuming from suspend or sleep

For some reason my third generation Dell Sputnik (aka Dell XPS13 with Ubuntu pre-installed) sometimes has a frozen or broken mouse / cursor after I suspend or sleep my laptop.  You can easily get it working again (this is not a permanent fix) by switching to a different TTY and then back to the TTY that holds the x-server GUI.

• ctrl+alt+F6 to switch to TTY6 (this will give you a terminal)
• Do nothing in the terminal
• ctrl-alt+F7 to switch back to the TTY7 which is the Ubuntu x-server
• Voila... your mouse should be working again.

If anyone has a permanent fix, please let me know.

Installing Cython - x86_64-linux-gnu-gcc: error: unrecognized command line option ‘-fstack-protector-strong’

I went around and around trying to get Cython install on my Ubuntu 14.04 LTS box.  I kept getting:

x86_64-linux-gnu-gcc: error: unrecognized command line option ‘-fstack-protector-strong’

Turns out that the ‘-fstack-protector-strong’ option was not added to the GCC compiler until version 4.9.  Upgraded my GCC to the latest available fixed the issue.

The City of Munich -- Switching Back to Windows from Linux?

 Munich’s much vaunted migration to Linux could be about to unravel, sending the city back into the fee-driven bosom of Microsoft.

http://www.omgubuntu.co.uk/2014/08/munich-city-linux-switching-back-windows

The City of Munich, Germany has been a Linux user for over 10 years but it appears that city employees are clamoring to switch back to Windows.

The issue I see is that they built their own distribution - LiMux. It was originally based on Debian and then switched to Ubuntu. The last stable release of LiMux was in 2011 and is based on Ubuntu 10.04 LTS which was released in April 2010. That version of Ubuntu is seriously old in the tooth now -- even Microsoft releases new version of Windoze faster.

Seems to me that the City of Munich suffers from "Not invented here" syndrome and would be better off if they just switch to an vendor with commercial support like Ubuntu or Red Hat instead of trying to build and maintain their own distro.

Uninstall all Python packages via pip

This is for my future reference because it's really handy if you don't want to drop a virtualenv or doing some crazy work on a vagrant box.  Even ignores any packages you installed from git or vcs sources with the -e flag

pip freeze | grep -v "^-e" | xargs pip uninstall -y

Logitech Unifying Receiver Pairing on Linux

This post is mostly for my future reference. Really simple to pair a new mouse with an existing receiver on Linux.

ltunify is a program resulting from the gathered knowledge on the Logitech HID++ protocol. It allows you to pair additional devices like keyboards and mice to your Unifying receiver, unpair existing devices and list information about connected devices. This section will show you how to install the ltunify program.

Besides a C compiler and a way to fetch sources (wget+tar or git), you will need Linux 3.2 or newer with the hid-logitech-dj module. On Debian and Ubuntu distributions, the required packages can be installed using:

sudo apt-get install git gcc

Fetch the sources and install ltunify to $HOME/bin/ using the next commands:

git clone https://git.lekensteyn.nl/ltunify.git
cd ltunify
make install-home

The following steps will assume that $HOME/bin is available in your path. If not, run:

export PATH="$HOME/bin:$PATH"

If everything went well, you should be able to run ltunify --help to show the available options:

Usage: ltunify [options] cmd [cmd options]
Logitech Unifying tool version dev
Copyright (C) 2013 Peter Wu <lekensteyn@gmail.com>

Generic options:
  -d, --device path Bypass detection, specify custom hidraw device.
  -D                Print debugging information
  -h, --help        Show this help message

Commands:
  list            - show all paired devices
  pair [timeout]  - Try to pair within "timeout" seconds (1 to 255,
                    default 0 which is an alias for 30s)
  unpair idx      - Unpair device
  info idx        - Show more detailed information for a device
  receiver-info   - Show information about the receiver
In the above lines, "idx" refers to the device number shown in the
 first column of the list command (between 1 and 6). Alternatively, you
 can use the following names (case-insensitive):
 Keyboard Mouse Numpad Presenter Trackball Touchpad

The below session shows you how to use a device index to unpair a mouse.

$ sudo ltunify list
Devices count: 1
Connected devices:
idx=1   Mouse   M525

$ sudo ltunify unpair 1
Device 0x01 Mouse successfully unpaired

$ sudo ltunify list
Devices count: 0
Connected devices:

$ sudo ltunify pair
Please turn your wireless device off and on to start pairing.
Found new device, id=0x01 Mouse

$ sudo ltunify list
Devices count: 1
Connected devices:
idx=1   Mouse   M525

It is also possible to select a device by device type (case-insensitive). When multiple devices of the same type are available, the first one will be selected.

$ sudo ltunify unpair mouse
Device 0x01 Mouse successfully unpaired

PyCon 2015 will...

PyCon 2015 will be held in Montreal again next year from April 8th - 16th, 2015. We hope to see you there!

Tutorials: April 8th - 9th
Conference: April 10th - 12th
Sprints: April 13th - 16th

Blogging this because it's hard to find the dates for PyCon 2015

Car2Go Minneapolis Promo Code - Expires 3/16/14

Didn't get a free lifetime membership to Car2Go Minneapolis last Fall? This is the last "cheap" promo for Car2Go and lifetime memberships. My wife and I used Car2Go in both Portland and Seattle during our recent vacation. We never had to rent a car!

The PFAR1030 promo code gives you a discounted $10 car2go registration fee (instead of the normal $35) AND 30 free minutes of car2go drive time. My promo code expires on 3/16.