Ok, so here’s the third part of our series on our git-workflow (see part-1, part-2). We are 3 or 4 people working with git on a django project. So, what happens when one (and just one) of us needs to develope a small feature independently? This could be an extra form, an additional login option, a new field in the model.py or something that one person can tackle in 1 to 3 days.

Update the local database:

with a fab command that will copy the latest prod database to your local machine (see part-2 for details),

$ fab db_cp_prod2dev

then start a new branch for your feature

$ gpu

$ co -b dev_footer_menu_update

*remember I’m using the alliases introduced in the last post (gpu = git pull, co = git checkout, and co -b is like git branch branchname + git checkout branchname all in one). This takes me straight into the new branch as can be seen with:

$ git branch 

* dev_footer_menu_update

  dev_some_other_feature

  master

Great. so you create a wonderful new footer, update the stuff, stage your changes often (with git add -u) and commit at the end of the day.

NOTE:  If you had some files in master which you didn’t commit (some unfinished quick bugfix) you will need to go back to master and add/commit those before you continue on the branch.  Otherwise, doing rebase, merge, etc can give you problems.  So for example if you’ve worked on footer.html only in your branch, but your get:

$ git status 

# On branch dev_footer_menu_update

# Changes not staged for commit:

#   (use "git add ..." to update what will be committed)

#   (use "git checkout -- ..." to discard changes in working directory)

#
#       modified:   fabfile.py

#       modified:   mysite/assets/bootstrap/somestyle.less

#       modified:   mysite/templates/footer/footer.html

You should go back to master and finish the somestyle.less and fabfile.py first (obviously there are ways around this with git, but it will make it easier to follow these rules).  If it takes longer, it should have been in a branch to insure its independence.  Ok, so we finish the work on footer.html and:

$ git status 

# On branch dev_footer_menu_update

# Changes not staged for commit:

#   (use "git add ..." to update what will be committed)

#   (use "git checkout -- ..." to discard changes in working directory)

#       modified:   mysite/templates/footer/footer.html

$ ga -u # adds the files we modified, here footer.html

$ gcm "added the suppliers in the footer of the home template"

You go to bed, wake up in your polyphasic sleep, back to bed … and back to the computer. Oh! You forgot to style the footer.  Back to work.  Since we work accross 3 timezones, someone probably messed up your feature touching another file. To avert disaster you pull the changes that others have commited before  :

$ co master   # means: git checkout master

$ gpu   # means: git pull

$ co dev_footer_menu_update

$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: added the suppliers in the footer of the home template

Rebase here will merge the stuff from master which other people have done into your branch.  So you can work on a branch that has the latest upstream files.  This may result in some conflicts, but it’s better to resolve those conflicts in your branch than later in master.  Once you’ve done the rebase in your branch, merging your branch into master will be easy and painless (because you already have the latest master in your branch except for your additions).  

So for example, after I commited a27aa52, I branched, worked and commited 18b831f. Once I rebase, I get the commits from my colleague Djangonaut Hannes into my branch: that is 6d412e8, 50c82fa, 43645e4.

by the way, that log is out “git log pretty”, 

alias glp='git log --pretty=format:"%Cred%h%Creset - %Cgreen%an%Creset, %ar : %s"'

Ok, so we rebase to get other people’s updates (or merge if there are too many conflicts). Now we work some more on the styles, check we are on the right branch:

$ gb
* dev_footer_menu_update

dev_some_other_feature

master 

TEST, add, commit, and merge:

$ ga -u  

$ gcm "updated footer, now with classy styles!"
$ co master
$ git merge dev_footer_menu_update

And finally delete the branch:

$ git branch -d dev_footer_menu_update

So to recap these are the steps:

• update the database
• pull, finish and commit any work on master
• create a branch, commit often
• rebase our branch in case master is evolving without us
• merge and delete the branch.

Now, that’s a lot of typing, so let’s review how we can do this with  aliases and fab functions:

$ gpu

$ fab db_cp_prod2dev

$ co -b dev_footer_menu_update

{ WORK GOES HERE}

$ ga -u

$ gcm "some interesting commit"

$ fab update_branch

{ MORE WORK GOES HERE}

$ ga -u

$ gcm "some interesting commit"

$ fab end_branch

Where update_branch and end_branch do the following:

When all this is done, we can of course

fab update_testing

send an email, let the others know, have a look, run tests, and:

fab update_dev

when ready. For a graphical representation of the workflow check out the next post.

Note that sending an email (or hipchat, or equivalent) after testing is very important. Otherwise, it could happen that another developer pushes to the repository, updates testing and finds an error. But without communication, if you are unaware of that error you might update master incorporating your last version (and his last version which has errors). It would be possible to build a more fool-proof workflow to take care of this, but it’s a complexity tradeoff, and communicating won’t hurt either!

This will be a series of exploratory articles on how to do django development in a team of 3 using git.  Suggestions from the readers are more than welcome, but the solutions have to be cheap and fast as we’re a bootstraped startup.

Setting the stage:

www.example.com (production)

• prod_db backed up automatically and regularly to dropbox and to a git repository with a cron job.
• media files, served by a static nginx app and lives in a sub directory /media/ of the above nginx.  Also compressed and backed up regularly to a dropbox account. 
• git repository of the django app, and dedicated virtual environment with requirements.txt tracked in the git repository.
• static files (fonts, img, css, js) are in the django project under mysite/assets/ and are served by an nginx static app which has symbolic links to mysite/assets/.

www.top-secret-testing-url.com (testing)

• testing_db  is regularly copied from prod_db, but sometimes diverges for testing. It is also added regularly to the db_backup git repository with a cron job.
• media files, served by a static nginx app and lives in a sub directory testing/media/ of the above nginx. This is so we don’t pollute the /media/ production directory with testing uploads.
• git repository of the django app, and dedicated virtual environment with requirements.txt tracked in the git repository.
• static files (fonts, img, css, js) are in the django project under my_testing_site/assets/ and are served by an nginx static app which has symbolic links from /testing/ to my_testing_site/assets/.

localhost (local)

• dev_db  is regularly wiped out and a new copy is loaded from prod_db
• media files, served by django locally, can be loaded using the gziped backup file.
• git repository of the django app, and dedicated virtual environment with requirements.txt tracked in the git repository.
• static files (fonts, img, css, js) are in the django project under assets/ and are served by django.

The user cases:

We will explore a few cases that arise often:

(Bfx) Urgent but simple bugfix.  

• font end: wrong color, typo, simple css fix
• minor typo leading to 404 or 500
• Obvious isolated things that can be fixed in 5mn - 2h

(ft) Small feature branch (single dev). 

• adding a small extra functionality: ex login with fb when twitter and gmail login already work, or one extra field in a model.py.
• can be done/tested by one person in 1-2 days
• is a tick in a trello list

(FT) Feature branch. 

• adding a full functionality: ex. new model with templates and views, integrating a new django-package, solving deep performance issues, etc …
• could require collaboration of 2-3 people in 1 or 2 branches.
• Is a full trello card.
• can take 2 - 10 days.

Along this exploration we need to remember how to keep our databases in sync with each other (including south migrations in testing and dev branches), how to sync our work when we work in different or common git branches at different speeds, how to backup everything, and how to create simple fabric scripts to simplify our life

Something on our host was driving me crazy for hours yesterday.  

after deploying to mysite.com/directory, when I followed a link (say about-us) it would take me to mysite.com/about-us instead of mysite.com/directory/about-us.

I thought it might be related to the Site settings and even tried the shell:

 >>> from django.contrib.sites.models import Site

 >>> Site.objects.get_current().domain 

but that returned mysite.com/directory.

Fortunately support at webfaction is pretty quick and smart and they pointed us to a variable in the settings file.  If you’re having that same problem just add:

#settings.py

FORCE_SCRIPT_NAME = "/directory"

(with no forward slash)

We use trello from Fog Creek Software to organize our coding, writing and business plans. We also use github for sharing code and development.

Both use markdown (trello in the description of a card, github in the README.md files).

Here’s a nice Cheat Sheet [PDF] I just found for the tags and tricks.

These are some wind turbines powering an Antarctic Research Station.  And that’s also about as South as you can get in your quest for renewable energy.  But today we’re writing about migrating your databases South!  Or the following scenario:

You create an übercool insightful Django model.  You put some data on it, make some templates, … looks good!  And then you realize it wasn’t as cool as you thought.  You forgot to add … say the minimum operating temperature for your wind turbines.  Bummer!  Enter South ( intelligent schema and data migration).

 manage.py schemamigration app_name --initial

if you have not done syncdb yet:

 manage.py migrate app_name

if you have already done syncdb:

 manage.py migrate app_name --fake

Then edit your models.py and add all the fields you forgot to add,

Then create the new migration file and finally migrate the database:

 manage.py schemamigration app_name --auto 

 manage.py migrate app_name 

see also [djangopro] for a good explanation.

Also to check the migrations you have done so far simply run:

 manage.py migrate --list

Perhaps you need to migrate from your local machine to a server, or ship some flatpages to your mate on the other side of the world.  Here’s how

$ python manage.py dumpdata flatpages --indent=2 > ma_backup.json

To recover it somewhere else simply do:

$ python manage.py loaddata ma_backup.json

Step 1 (install the stuff)

 $ sudo apt-get install postgresql-9.1 postgresql-server-dev-9.1 
 postgresql-9.1-postgis 
 $ sudo apt-get install binutils gdal-bin python-psycopg2 python-setuptools 

Step 2 (create templates) 

create a POSTGIS template (from which your databases will be derived)

$ POSTGIS_SQL_PATH=`pg_config --sharedir`/contrib/postgis-1.5 

# Creating the template spatial database. 

$ sudo -u postgres createdb -E UTF8 template_postgis

$ sudo emacs -nw /etc/postgresql/9.1/main/postgresql.conf 

 $ sudo service postgresql restart  

Add language support for PLPGSQL

 $ sudo -u postgres createlang -d template_postgis plpgsql 

Allows non-superusers the ability to create from this template

$ sudo -u postgres psql -d postgres -c 
 "UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis';" 

Load the PostGIS SQL routines

 $ sudo -u postgres psql -d template_postgis -f 
$POSTGIS_SQL_PATH/postgis.sql 
 $sudo -u postgres psql -d template_postgis -f 
$POSTGIS_SQL_PATH/spatial_ref_sys.sql 

(note that the dollar sign above after -f is to signify the variable $POSTGIS_SQL_PATH, it should all be a single line from “$sudo -u …. to … .sql”)

******** You may get an error here ***********
the Error is that /postgis.sql cannot be found. So I tried:

 $ sudo updatedb 
 $ locate "postgis.sql" # which gave me the answer: 
 /usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql
 /usr/share/postgresql/9.1/contrib/postgis-1.5/uninstall_postgis.sql

so I changed those two commands to commands to,

$ sudo -u postgres psql -d template_postgis -f 
/usr/share/postgresql/9.1/contrib/postgis-1.5/postgis.sql
 $ sudo -u postgres psql -d template_postgis -f 
/usr/share/postgresql/9.1/contrib/postgis-1.5/spatial_ref_sys.sql

Some user rights for users to be able to alter spatial tables

 $ sudo -u postgres psql -d template_postgis -c 
 "GRANT ALL ON geometry_columns TO PUBLIC;"
 $ sudo -u postgres psql -d template_postgis -c 
 "GRANT ALL ON geography_columns TO PUBLIC;"
 $ sudo -u postgres psql -d template_postgis -c 
 "GRANT ALL ON spatial_ref_sys TO PUBLIC;"

So now we are ready to create our own local database. Let’s call it test_db with user test_admin

 sudo -u postgres createuser --createdb test_admin
 shall the new role be a superuser? Y
 sudo -u postgres createdb -T template_postgis -O test_admin test_db
 sudo -u postgres psql test_db -c "ALTER USER test_admin with password 'random-obscure-password'"; 

I mostly followed the steps from ChicagoDjango

together with countless hours of debugging for those two simple errors! :-/ …

So I hope it’s of use to you all.