re.nooble

displaying many to many fields in django with crispy-forms (or multiselect)

Mon, 13 May 2013 22:56:00 +0100

Das Problem:

You have a field with 246 items to chose from … and you’re asking your users to scroll through them, hold control down, scroll to the next one, ups, forgot the control, start again … grrr. Not pretty.

Do not fret. crispy-forms and chosen.js have come to the rescue. Here’s how it works and how simple it is:

1. Check out a chosen example to see how multi select works.

2. in your crispy form inside your form class add a line like:

Field('countries_of_operation', css_class='chosen', ),

This should render a form that looks something like:

<select multiple="multiple" class="selectmultiple chosen" 
id="id_countries_of_operation" name="countries_of_operation">

<option value="247">Antigua and Barbuda</option>

<option value="248">Algeria</option>

<option value="249">Azerbaijan</option>

etc ...

3. Add a little CSS and JS at the bottom of your page:

<script src="//ajax.googleapis.com/ajax/libs/jqueryui/1.10.2/jquery-ui.min.js">
</script>

<script type="text/javascript" 
src="{{ STATIC_URL }}js/chosen/chosen.jquery.min.js">
</script>

<link rel="stylesheet" href="{{ STATIC_URL }}js/chosen/chosen.css" 
type="text/css" media="screen" />

<script type="text/javascript">  $(".chosen").chosen();

</script>

4. Try it out!

From Paul Hudnut’s Bottom of the Pyramid Entrepreneur

Wed, 01 May 2013 20:24:00 +0100

From Paul Hudnut’s Bottom of the Pyramid Entrepreneur

I don't need to test that. What can possibly go wrong?

Tue, 23 Apr 2013 17:29:58 +0100

devopsreactions:

Image by IceMan1988

Reduce your PostgreSQL/PostGIS query time by 99%

Mon, 15 Apr 2013 14:39:00 +0100

In our search engine for renewables, we are running a bunch of PostGIS queries, e.g. to determine the wind speed at a user’s location. In our alpha search engine, we noticed that the PostGIS requests caused the longest query time of all 25 queries.

While experimenting with PostGIS, we discovered a nice time saver: In the alpha version, we translated the CSV data of measurement points (lat, lng, wind speed) into a shapefile with geometry points representing every measurement in the UK. The result looked like below (Southern England with a zoom on the Isle of Wight):

We extracted the data_point for a given location with the raw SQL statement in Django/Python:

We used the raw SQL statement in Django, since it provided the fastest query response. But it was still almost 300ms. Not enough!

So we looked for an alternative solution based on polygon intersect. Instead of looking for the closest data point to the geographic coordinate of the user’s location, we are now searching for the polygon which contains the user’s coordinate. Thanks to the WKT technology, intersect lookups run amazingly fast - in our case in 3ms.

If you have a shapefile with the point geometries (you can use QGIS to convert your CVS measurement data into shapefiles), you can use the QGIS geoprocessing function Voronoi Polygon (Vector | Geometry Tools | Voronoi Polygons) to convert the points into polygons for each measurement point/value. The result is a fancy British spider web:

PostGIS queries based on the polygon intersects can be done as follows:

data_point = WindData.objects.get(geom__intersects = location)

Since we do not have to sort the locations of the data points to find the closest data point anymore, the intersect query runs much faster. The WKT technology allows a hash comparison to determine if a point is part of a polygon or not. This amazing development provides the great speed increase.

The performance comparison in the Django shell says it all:

Nice reduction of almost 99% in the query time.

setting up our wifi antenna before a hackaton in the re.nooble...

Sun, 14 Apr 2013 15:41:00 +0100

setting up our wifi antenna before a hackaton in the re.nooble mountain accelerator :)

Handy Git tips to stop you getting fired

Tue, 09 Apr 2013 17:18:58 +0100

Handy Git tips to stop you getting fired:

Very nice summary of handy and life-saving git tipps!

patch from file using git diff

Thu, 28 Mar 2013 15:29:00 +0000

Say you want to send a friend a patch, or perhaps you often apply a change back and forth (say like preparing a file for production, and getting it back to testing). In our case base.html uses less.js to render the CSS during development, but in production it calls instead a minified CSS. So to convert base.html between the production and dev versions we created a patch. This is how you do it:

1. Modify the file

git diff dir/thefile.html --no-prefix > patch_thefile_from_dev2prod

The next time you need to change the file from dev to prod you simply apply the patch:

patch -p0 < patch_the_file_from_dev2prod

git workflow -- emergency fix or how to stash

Sat, 23 Mar 2013 13:20:00 +0000

You’re plowing away in your feature branch in the zone, you’ve messed with 10 files and it still doesn’t work. However you think that in four hours your creation will spring to life.

All of a sudden, you get 20 emails with 500 error reports from production. aaagh! You need to fix that, Quickly!!

$ git checkout master

Not so fast! If you do that, the 10 files you touched will move with you to master. But you’re not quite ready to commit anything either. It is time to stash away your changes.

$ git add [some-files]
$ git stash

Now you can

$ git checkout master

{ FIX THAT BUG, quick!}

Do your testing, fab update_testing, more testing, fab update_prod, and get back to where you were:

$ git checkout feature_branch
$ git stash apply

and you can pick up where you left it.

Note that the part where it says “bugfix” could include another branch, and includes testing, deployment, email, etc, etc.

Some references on multiple stashes, selective stashing, listing stashes or clearing them:

git workflow to develop a small ( 1 -3 day) feature as a single...

Mon, 04 Mar 2013 14:35:00 +0000

git workflow to develop a small ( 1 -3 day) feature as a single user (while other users, like Javi in the example commit to master. see: Git Workflow part 3 for more details. note the abbreviations:

gpu = git pull

gp = git push

co = git checkout

ga = git add

gcm = git commit -m

git workflow -- part 3

Sun, 03 Mar 2013 13:56:00 +0000

Ok, so here’s the third part of our series on our git-workflow (see part-1, part-2). We are 3 or 4 people working with git on a django project. So, what happens when one (and just one) of us needs to develope a small feature independently? This could be an extra form, an additional login option, a new field in the model.py or something that one person can tackle in 1 to 3 days.

Update the local database:

with a fab command that will copy the latest prod database to your local machine (see part-2 for details),

$ fab db_cp_prod2dev

then start a new branch for your feature

$ gpu

$ co -b dev_footer_menu_update

*remember I’m using the alliases introduced in the last post (gpu = git pull, co = git checkout, and co -b is like git branch branchname + git checkout branchname all in one). This takes me straight into the new branch as can be seen with:

$ git branch

* dev_footer_menu_update

  dev_some_other_feature

  master

Great. so you create a wonderful new footer, update the stuff, stage your changes often (with git add -u) and commit at the end of the day.

NOTE: If you had some files in master which you didn’t commit (some unfinished quick bugfix) you will need to go back to master and add/commit those before you continue on the branch. Otherwise, doing rebase, merge, etc can give you problems. So for example if you’ve worked on footer.html only in your branch, but your get:

$ git status

# On branch dev_footer_menu_update

# Changes not staged for commit:

#   (use "git add ..." to update what will be committed)

#   (use "git checkout -- ..." to discard changes in working directory)

#
#       modified:   fabfile.py

#       modified:   mysite/assets/bootstrap/somestyle.less

#       modified:   mysite/templates/footer/footer.html

You should go back to master and finish the somestyle.less and fabfile.py first (obviously there are ways around this with git, but it will make it easier to follow these rules). If it takes longer, it should have been in a branch to insure its independence. Ok, so we finish the work on footer.html and:

$ git status

# On branch dev_footer_menu_update

# Changes not staged for commit:

#   (use "git add ..." to update what will be committed)

#   (use "git checkout -- ..." to discard changes in working directory)

#       modified:   mysite/templates/footer/footer.html

$ ga -u # adds the files we modified, here footer.html

$ gcm "added the suppliers in the footer of the home template"

You go to bed, wake up in your polyphasic sleep, back to bed … and back to the computer. Oh! You forgot to style the footer. Back to work. Since we work accross 3 timezones, someone probably messed up your feature touching another file. To avert disaster you pull the changes that others have commited before :

$ co master   # means: git checkout master

$ gpu   # means: git pull

$ co dev_footer_menu_update

$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: added the suppliers in the footer of the home template

Rebase here will merge the stuff from master which other people have done into your branch. So you can work on a branch that has the latest upstream files. This may result in some conflicts, but it’s better to resolve those conflicts in your branch than later in master. Once you’ve done the rebase in your branch, merging your branch into master will be easy and painless (because you already have the latest master in your branch except for your additions).

So for example, after I commited a27aa52, I branched, worked and commited 18b831f. Once I rebase, I get the commits from my colleague Djangonaut Hannes into my branch: that is 6d412e8, 50c82fa, 43645e4.

by the way, that log is out “git log pretty”,

alias glp='git log --pretty=format:"%Cred%h%Creset - %Cgreen%an%Creset, %ar : %s"'

Ok, so we rebase to get other people’s updates (or merge if there are too many conflicts). Now we work some more on the styles, check we are on the right branch:

$ gb
* dev_footer_menu_update

dev_some_other_feature

master

TEST, add, commit, and merge:

$ ga -u

$ gcm "updated footer, now with classy styles!"
$ co master
$ git merge dev_footer_menu_update

And finally delete the branch:

$ git branch -d dev_footer_menu_update

So to recap these are the steps:

update the database
pull, finish and commit any work on master
create a branch, commit often
rebase our branch in case master is evolving without us
merge and delete the branch.

Now, that’s a lot of typing, so let’s review how we can do this with aliases and fab functions:

$ gpu

$ fab db_cp_prod2dev

$ co -b dev_footer_menu_update

{ WORK GOES HERE}

$ ga -u

$ gcm "some interesting commit"

$ fab update_branch

{ MORE WORK GOES HERE}

$ ga -u

$ gcm "some interesting commit"

$ fab end_branch

Where update_branch and end_branch do the following:

and

When all this is done, we can of course

fab update_testing

send an email, let the others know, have a look, run tests, and:

fab update_dev

when ready. For a graphical representation of the workflow check out the next post.

Note that sending an email (or hipchat, or equivalent) after testing is very important. Otherwise, it could happen that another developer pushes to the repository, updates testing and finds an error. But without communication, if you are unaware of that error you might update master incorporating your last version (and his last version which has errors). It would be possible to build a more fool-proof workflow to take care of this, but it’s a complexity tradeoff, and communicating won’t hurt either!

git workflow - part 2

Sun, 03 Mar 2013 01:42:00 +0000

Before creating a branch, doing a quick fix or messing with the models, we need to make sure our database is the same as the production one.

To do so we have a single bash (or fab) script which does:

This can all be summarized in a fab command which takes into account different OS (linux, mac), different absolute directories, and different naming of files (see fabfile.py in gist)

Ok, so we have one of those (Bfx), Urgent but simple bugfix.

If our database is out of date we can,

$ fab db_cp_prod2dev

in other words “database copy from production to development” (see gist above). And because it’s just a quick fix, we’ll stay on the master branch:

$ gb
  dev_some_feature
* master

Note that I will use a few git aliases to save my typing, in particular I have a ~/.bash_aliases with:

So, work, work, work, fix, fix, fix.

$ ga -u
$ gcm "quick fix, the margin in the CSS had gone cookoo, fixed .home-box class"
$ gp

Then send to testing

$ fab update_testing

which does,

and similarly send to production when ready

$ fab update_prod

Here’s a graphical overview of the whole thing:

And that’s it for bug-fixing alone. Next part, developing a small feature in a branch.

git workflow - part 1

Sun, 03 Mar 2013 01:05:00 +0000

This will be a series of exploratory articles on how to do django development in a team of 3 using git. Suggestions from the readers are more than welcome, but the solutions have to be cheap and fast as we’re a bootstraped startup.

Setting the stage:

www.example.com (production)

prod_db backed up automatically and regularly to dropbox and to a git repository with a cron job.
media files, served by a static nginx app and lives in a sub directory /media/ of the above nginx. Also compressed and backed up regularly to a dropbox account.
git repository of the django app, and dedicated virtual environment with requirements.txt tracked in the git repository.
static files (fonts, img, css, js) are in the django project under mysite/assets/ and are served by an nginx static app which has symbolic links to mysite/assets/.

www.top-secret-testing-url.com (testing)

testing_db is regularly copied from prod_db, but sometimes diverges for testing. It is also added regularly to the db_backup git repository with a cron job.
media files, served by a static nginx app and lives in a sub directory testing/media/ of the above nginx. This is so we don’t pollute the /media/ production directory with testing uploads.
git repository of the django app, and dedicated virtual environment with requirements.txt tracked in the git repository.
static files (fonts, img, css, js) are in the django project under my_testing_site/assets/ and are served by an nginx static app which has symbolic links from /testing/ to my_testing_site/assets/.

localhost (local)

dev_db is regularly wiped out and a new copy is loaded from prod_db
media files, served by django locally, can be loaded using the gziped backup file.
git repository of the django app, and dedicated virtual environment with requirements.txt tracked in the git repository.
static files (fonts, img, css, js) are in the django project under assets/ and are served by django.

The user cases:

We will explore a few cases that arise often:

(Bfx) Urgent but simple bugfix.

font end: wrong color, typo, simple css fix
minor typo leading to 404 or 500
Obvious isolated things that can be fixed in 5mn - 2h

(ft) Small feature branch (single dev).

adding a small extra functionality: ex login with fb when twitter and gmail login already work, or one extra field in a model.py.
can be done/tested by one person in 1-2 days
is a tick in a trello list

(FT) Feature branch.

adding a full functionality: ex. new model with templates and views, integrating a new django-package, solving deep performance issues, etc …
could require collaboration of 2-3 people in 1 or 2 branches.
Is a full trello card.
can take 2 - 10 days.

Along this exploration we need to remember how to keep our databases in sync with each other (including south migrations in testing and dev branches), how to sync our work when we work in different or common git branches at different speeds, how to backup everything, and how to create simple fabric scripts to simplify our life

Interesting Django Error Messages: 'function' object has no attribute 'as_view'

Sat, 02 Mar 2013 16:52:14 +0000

Django’s debug error messages are usually helpful and provide a good level of description. However, sometimes we encounter error messages, which “amaze” us. We would like to take the chance to summarise the results of our solutions here on this blog.

Today, we worked on a new online feature for re.nooble’s customers. While writing some classes in the particular `views.py` (yes, we implement class-based responses), we received a strange error message:

Error: ‘function’ object has no attribute ‘as_view’

That error usually is expressed if you try to override a class. That sometimes happens if you copy&paste code and forget to change e.g. the class name. But that was not the case today.

After a while of online error digging and code testing, we discovered that the `art` of decorating classes/methods changed with introducing class-based views.

However, this way of decorating methods can not be applied to classes.
If you apply `@login_required`to a class, you will receive the error message: ‘function’ object has no attribute ‘as_view’

So, how should you decorate classes in Django now?
For class-based views, you have two options of decorating your classes.

1) Decorating the URLconf
2) Decorating the class

Both options leads to the same result - restricting the access to a class only for logged users. The difference between the options is how the decorator is applied to the class instance.

Regarding the detailed implementation and the detailed descriptions, please have a look at the very helpful Django documentation for decorating classes.

Transferring Django Flatpages from the local Testing Server to Production

Sat, 02 Mar 2013 10:17:00 +0000

At the moment, we are creating a few flatpages for our re.nooble site. Obviously, they are created on a testing/development machine and need to be transferred to the production environment before the release of the latest code feature.

The few lines below will manage the transfer easily.

Create an API for your django models

Sun, 24 Feb 2013 23:11:20 +0000

http://www.nerdydork.com/basic-authentication-on-mod_wsgi.html

In our case we simply added

WSGIPassAuthorization On

to httpd.conf

font-awesome, twitter-bootstrap facepalm moment ...

Sun, 24 Feb 2013 23:08:06 +0000

Ok, so I have some django projects using the now ubiquitous twitter bootstrap, together with the fantastish Font-Awesome. No problem. And I like my css LESS flavoured, because … well, because who likes to write css rules over and over? So this results in a folder structure for the assets which looks something like this:

├── assets
│   ├── bootstrap
│   │   ├── docs
│   │   ├── img
│   │   ├── js
│   │   └── less
│   ├── css
│   ├── font
│   ├── img
│   └── js

To enable font-awesome you’ll have inside the less directory a file called font-awesome.less indicating the relative path to the font folder:

@FontAwesomePath:   "../../font";

because the font file lives two levels out. So during development I would call the less files from base.html simply as:

But when I ran the Makefile

lessc --compress ${BOOTSTRAP_LESS} > ${BOOTSTRAP_MIN}

This would keep the relative path

@FontAwesomePath:   "../../font";

however, the minified css file now lived only one level away from the font directory (as you can see above). So what was the obvious solution? Well … here’s my dumb way out: modify the Makefile

So to solve this problem, I modify the Makefile, use sed, test some regex, test regex some more, break, crash burn … and finally make it work. Was this a good solution? Well, two weeks later I realized there’s a much, much simpler solution: make a copy of the font/ directory in the bootstrap/ directory … like so:

├── assets
│   ├── bootstrap
│   │   ├── docs
│   │   ├── img
│   │   ├── font
│   │   ├── js
│   │   └── less
│   ├── css
│   ├── font
│   ├── img
│   └── js

so that:

@FontAwesomePath:   "../../font";

is always

@FontAwesomePath:   "../font";

DUUUHH!

Anyway. I just thought I would share this in case some unsuspecting n00b is looking for regex-es to solve this little issue

Complex regexp worked exactly as expected

Fri, 22 Feb 2013 11:12:13 +0000

devopsreactions:

Submitted by Leprosy

Perpetual To-Do List

Tue, 19 Feb 2013 15:44:36 +0000

facuberte:

Don’t ignore your dreams

Don’t work too much

Say what you think

Cultivate friendships

Be happy

By Paul Graham

http://paulgraham.com/todo.html

django, mysql and ubuntu

Tue, 19 Feb 2013 15:11:00 +0000

Here’s the setup from 0 to 127.0.0.1:8000 in 30 seconds:

I assume you already have python (below version 3.0) in your machine. You also have virtualenv (you can apt-get install python-virtualenv ) and pip.

After that it’s the usual: create some models, runserver and syncdb

femalesoftwareeng: When I finally get up after 12 hours of...

Sat, 16 Feb 2013 22:14:00 +0000

femalesoftwareeng:

When I finally get up after 12 hours of coding