Category Archives: Web Development

Developing a Serious Game Application Part 2 – A Project Postmortem

Project Status and Future Development Goals

The project’s Github project page can be found at jtakayama.github.io/ics691-setupbooster/.

a25_landing_page

Among other issues, there were problems replacing the logo.

In many respects, this project did not go as planned. Several tasks ran overtime, several other long-term projects competed for my attention, and even when modifications to Makahiki did partially work, it was a challenge to figure out where exactly some components of each page were being loaded from.

a25_new_theme_blue

Above: The new theme uses blue and gray to match the logo, and to go for a more technological-looking background. However, theme changes do not seem to apply to Settings pages.


In terms of cosmetic changes, I switched the default theme to a customized blue-gray-white theme, created a custom set of default teams, users, and profiles to be used to initialize the database, and reworked the landing page and About page to describe SetupBooster instead of the Kukui Cup.
A gray page.

Above: The new “About” page.


So far, I have been unable to implement the article-creation and article-editing workflows. Non-functional widgets for creating articles, viewing a user’s average rating over all articles, and viewing user status information were created and are located on the user profile page.

The article creation widget.

The article widget, currently not functional.


The article creation widget currently writes an entry to the database, but there is no function implemented to retrieve the entry, and it does not have a corresponding editable, static page file.
rendering_glitch
The average_rating and player_status widgets are currently rendering in the same position due to a bug resulting from a conflict with a default Makahiki widget of priority 3 that has been disabled; the average_rating widget has priority 3 and the player_status widget has priority 4.

The primary future implementation goal is getting the article model to function correctly, since it is the central piece of the site. Once article pages can be rendered correctly for viewing and editing, the rest of the modules should be easy to integrate. Assuming each article can be identified by a unique ID and a revision number, creating view_article and edit_article widgets and passing them some variables to retrieve page content should be relatively easy.

Lesson 1: Gamification Benefits From a Clear End Goal

An issue I ran into when designing content for my software guides site “game” was that a site that exists to host user-generated content potentially indefinitely can be difficult to gamify. Competitions like the Kukui Cup or gamified personal challenges like World Without Oil ask players to make commitments and carry out actions within a specific time period. This allows the game designer to structure the experience by introducing levels, scaling difficulty, multiple rounds of prize giveaways, and constructing a narrative (if applicable). Other gamified concepts (fitness goal monitoring sites, social sites that measure TV viewing) can work for an indefinite amount of time because they can be driven by the user’s desire to achieve something or earn recognition for something of personal value. The gamified system can provide structure and incentives for the user, but in those situations, the user sets the timetable.

The new user groupings.

In SetupBooster, the teams don’t really serve as a means of organizing competitors against each other: they just distinguish between different types of users. In theory this could be done without teams completely (since Makahiki has superusers and non-superusers) but using the teams functionality could allow for additional “teams,” like temporarily banned users or more limited kinds of administrators.

In the case of the wiki / document-sharing site I planned for SetupBooster to eventually become, the issue is that the wiki model runs indefinitely, but individual users don’t really have a personal stake in the site unless they become popular or become administrators – both things that are determined by the community, not the software. A training game might help people get over the initial hurdles of editing a wiki. However, beyond this point, having users define the content of the wiki makes it difficult to create defined categories of prizes or activities.

In this situation, I think that only thing the wiki can reward are generalized measurements of activity: article creation, adding editors, making lots of comments, editing frequently, and similar statistics. Absent some kind of external prize, the wiki game eventually will become a long grind like parody games Progress Quest or Achievement Unlocked, with new badges and recognitions doled out every so many completed tasks. Arguably the content of the wiki could be the major selling point if the writers are good enough, but if that is the case, than the gamification really isn’t making much difference beyond lowering the barrier to entry.

Lesson 2: Don’t Work Alone If You Can Help It

As I stated in earlier posts, I had originally planned to work alone because I thought the final project would be more similar to my upcoming undergraduate project. (In retrospect, knowing what I know now about the process of modifying Makahiki, that would probably have been even more difficult.) All conflicts over scheduling and creative vision aside, the main reason to work in teams if you can – especially for students with heavy courseloads, long work hours, or other responsibilities – is that having more people working on the project increases the odds that at least one person will be able to work on something on any given night. It was too often the case that I was unable to put in much work on the project for days at a time. The tendency in such cases is to prioritize the things that are due earlier, which resulted in a Github commit pattern that bunched up noticeably just before major deadlines.

Teammates, if they care about deadlines, will pester each other to get the project done on schedule: this was the case with another team project I was working on for a cybersecurity class last week, and I believe it to be true of functional team projects in general. For more complex software projects, working alone and being responsible for the entire design of the system makes planning more difficult than for team projects, where each person can take a piece of the structure and work on it. On the other hand, breaking up a project can cause implementation problems if the different pieces don’t have the same gameplay style, access shared data in different and incompatible ways, or just end up not working because one piece of the puzzle is missing or late. Though I was working alone, I could work knowing that, for better or worse, all the pieces of the system were my responsibility to modify and I didn’t need to meet with someone to get part of the problem fixed.

Lesson 3: Documenting Your Code Is Valuable (even if you don’t have to)

Though Github’s Project Pages feature turned out to be easy to pick up, the process of documenting the vague flowcharts and lists I had created in earlier phases of the project was not. Being forced to describe the content of the game in detail caused me to realize some potential limitations of applying gamification to a wiki-like site, and to try to think more carefully about what kinds of integration I might be able to create between the tutorial game and the wiki interface. More importantly, it probably resulted in me having more of a product to show for my efforts than if I had just continued being stuck on resolving lower-level issues with the article model. Though in both cases I did not accomplish most of my initial implementation goals, having documentation to provide a blueprint for future progress increases the odds that I might be able to continue working on the project or come back to it later, rather than abandoning it because I ultimately had no idea where it would be going after the course was finished. Though there were several issues I was unable to resolve, having documented the plan for the final application refocused me on the larger goals of the project.

Advertisements

Developing a Serious Game Application – Part 1

Summary

Development of the serious game application has hit a roadblock due to problems in creating a valid Model representation of a text-only wiki article that are preventing the Makahiki database from being initialized. Once this problem is fixed, being able to perform standard Django database operations on the Articles should allow other standard pseudo-wiki-like features like article searches or revision management to be implemented to some degree.

False Starts: django-wiki

In my previous post on this project, I had been considering trying to incorporate modules from benjaoming’s django-wiki project into Makahiki. The problem with this approach is that it would essentially amount to taking an existing, full-fledged web application with its own authentication system and trying to link this system to Makahiki’s user database. At best, Makahiki would end up serving as a glorified login screen, with its widgets needing to pull from the external wiki database and integrate this with Makahiki’s database. While the django-wiki package appears to be a useful Django add-on, it did not seem as if it lent itself well to gamified integration with Makahiki in the limited development time I had.

The Article Model

The Article model, in its present, incomplete state, does not actually create an article that the user can edit. Rather, it stores metadata about the article – title, which editors can work on it, etc. – in the database. Since it seems impractical to store a hypothetical article (possibly with embedded images in a future iteration of the project) as a text field in a database, the goal is eventually to store a link to the actual article file as a field of the Article model. The database could then be queried to retrieve all revisions of the article (revision number is another field), and links to the article as generated by Django would point to the latest version. At present, Makahiki’s initialize_instance.py script cannot initialize the database because the error TypeError: __init__() got an unexpected keyword argument ‘required’ is preventing the Article model from being validated, so it is likely that an attempt to upload the application to Heroku in its present state would also fail. Oddly enough, the initialize_instance.py script worked once and produced this error only after I navigated to the Profile page; on the second use of initialize_instance.py, the script failed to complete its run and displayed the same error.

A TypeError.

The database was able to initialize once with the Article model, but this error appeared on the Profile page. The next use of initialize_instance.py failed to initialize the database completely, with the same error.

Implementing New Models

The practice of creating and being able to revise an article is very different from what Makahiki is currently designed to do: it is equipped to measure and reward user-generated actions and content, but naturally it is not possible for users to substantially alter the site. Now that I have a better understanding of the work that needs to be done to incorporate articles into the database, the greatest potential problem is the likelihood that most of the work over the remaining two weeks will need to go towards back-end, pseudo-wiki-implementing code, rather than into making Makahiki’s Smart Grid Game able to measure and reward training scenarios and user contributions within the wiki.

Future Objectives

In the short term, the goal is to fix the current version of the Article model such that the Article model will validate, and to then upload the application to Heroku. The next goal is to make the Article model or the article_mgr manager able to maintain references to the latest version of an article and provide a link to an editable version of the article, which will use the Markdown language that Makahiki already supports. A subsequent goal would be to modify one or more SmartGrid challenges to detect that a user has created a certain number of unique articles and/or revisions, and create a badge for it; alternately, I might attempt to incorporate a ratings module as well. Once the Article model is functional, building widgets to do basic database lookups for searches by title or author or loading a list of all revisions to an article should be relatively trivial. Articles are likely to be limited to text only for the foreseeable future, as support for image and file uploads is a lower priority given the current issues with the Article model and the feasibility of its gamification.

UPDATE: Heroku Application

My compilation and initialization failure was the result of extra commas in the models.py file. Once these were removed, the database could be initialized normally.

A draft version of the create_article widget.

The form does not have line breaks between fields, resulting in form help text seeming as if it is associated with the field below it instead of the field above it.


As mentioned, the changes at this point are minimal: the article creation form is displayed on the Profile page, and it redirects to an error page instead of a confirmation page or an editing page if the Submit button is clicked. Furthermore, it is not currently possible for any user, admin or non-admin, to delete the entry the Submit button creates in the database.

While trying to set up player0 as a dummy user, I was unable to continue the setup dialog beyond the point shown below. I can only conclude that my database model changes somehow broke other aspects of the setup process.

Now I cannot set up new users. Clicking on the right button to continue does nothing.

Now I cannot set up new users. Clicking on the right button to continue does nothing.

Developing for Makahiki – Part 1

Task 1: Hello World

This weeks’ class exercise in developing new widgets for Makahiki began as everything does in software engineering: with "Hello World.”" This tutorial breaks down the process of designing a new widget for Makahiki into a few basic steps:

  1. Initialize the new widget
  2. Edit its views.py file to determine what output it will return
  3. Edit its templates/index.html; this is the page section to display the values returned by its views.py file
  4. Add the widget to the list of widgets in Makahiki’s settings.py file
  5. In Makahiki’s challenge design configuration menus, add the new widget to a page
  6. Add a help topic for the widget
  7. Add the widget to fixtures/base_pages.json and fixtures/base_help.json to make it permanent (optional)
  8. Push changes to Github or Heroku as applicable.

I finished this without problems and got hello_world working locally.

The hello_world demo widget.

Above: The hello_world demo widget running locally.

The hello_world widget by itself, however, does not do much. The next set of tasks, after modifying score_mgr to support collecting data on user groups, involved adding more widgets to support data collection from player groups, which do not come with Makahiki by default.

Task 2: Add Group Widgets

Update score_mgr to Support Groups

My initial approach was one in which I attempted to add group support to score_mgr by creating new versions of its team data functions by replacing references to “team” with references to “group.” This was based on the assumption, based on models.py, that a Group could be accessed as a field of a Team (which is itself a field of a user_profile). This assumption was partially correct; the problem lay in the way I was processing the dictionary value returned from the widget’s views.py file, which was itself accessing profile__team__group__name to get the list of all groups.

Create a Group Scoreboard Widget

It was in attempting to create the group_scoreboard widget that I discovered that I was attempting to access the group value incorrectly. Initially, I thought that the error messages I experienced were due either to problems in the new functions I had added to start_mgr or the use of the admin account’s profile when the admin account was not linked to a group. Instead it was the result of another standard computer science problem, accessing a variable which did not exist.

A Django field error.

This problem turned out to be caused by a mistaken attempt to check for profile__team__group__isnull. “group” is not a property of profile__team.

As it turned out, the NoneType errors were caused by my views.py file. Originally, it returned the dictionary of group scores as follows:

return group_score

The code block Django uses to return named parameters is actually:

return {
    "group_score": group_score
}

which I had altered for some reason. Without this, views.py would always return null when Django processed index.html and attempted to access “group_score.”

There was another problem after this in which the widget displayed correctly but showed the text “last” and “points” instead of actual team names. The group_points_leaders function I created in score_mgr, modified from team_points_leader, has this block of code:

entries = ScoreboardEntry.objects.filter(
    round_name = round_name, profile__team__isnull=False).values(
    "profile__team__group__name").annotate(
    points=Sum("points"),
    last=Max("last_awarded_submission")).order_by("-points", "-last")

The entries variable is initialized to a dictionary and passed to views.py. I knew the problem was probably in my Django loop code in index.html, which tried to process the dictionary by looping through keys and values. The better way of accessing the dictionary in this case is to access individual, named parameters from each dictionary element:

{% for f in view_objects.group_scoreboard.group_score %}
   <!-- Some HTML formatting eliminated for clarity -->
   <tr>
   <td>{{ f.profile__team__group__name }}</td>
   <td>{{ f.points }}</td>
   </tr>
{% endfor %}

The group_scoreboard widget, showing the test group.

This section updated April 8, 2013.

Create a Group Resource Widget

I will update this section if I complete it.

Create a Group Prize Widget

I will update this section if I complete it.

Create Two Group Statistics Widgets

I will update this section if I complete it.

Suggestions for Makahiki Documentation Improvement

It is hard to say in this case if my problems are due to any issues in the Makahiki documentation, as they could just as easily result from my fixating on trying to salvage an unusable method of extending start_mgr to support groups. As far as I know, no one else had any problems with this step, so I am inclined to attribute my problems to configuration or coding errors on my end rather than problems with the documentation. Nevertheless, adding a type hierarchy diagram (manually created or automatically generated) to the documentation might be helpful. Though the widget system and user/group/team system is conceptually simple, it was not very clear to me which types had access to which other types.

This post will be updated if the score_mgr problems are resolved.

Makahiki Administration and Gamification, Version 3

Issues To Address

After reviewing the latest plan for my application, my professor pointed out a few areas of the design that might require further thought. The goal of my application was to provide a potentially wiki-like means for users to create guides, combined with a Stack Overflow-like means of awarding reputation and pointing out issues for future work.

The problem with this kind of system is that it is mostly disconnected from whatever documentation the developers produce – it may not keep up with software updates or newly discovered bugs, and at best it serves as a kind of patch on flaws in the developer documentation. As my professor pointed out, unless the authors are careful to mention specific details about their guide (what OS it applies to, what version of the software it’s meant to use, et cetera) it is difficult for a new user to know if a guide is applicable, and even more difficult to know if it will work. A reputation system may or may not be helpful here; it is only as good as the number of site users who bother to rate an article, and if they do not leave comments, it is not clear which parts of the article were useful or need to be fixed.

The challenge, then, is to design a system in which users are more closely connected to the production of developer documentation. It is probably not good to go “full wiki,” as the developers should still retain some control over the direction of development and the style and organization of documentation. Furthermore, the work of wiki administration and maintenance would probably require diverting personnel from actual development tasks, which would be a problem for smaller teams. This is more of an administrative issue, though, and in principle it could be taken care of by eventually nominating administrators from the user community to enforce a Wikipedia-like manual of style. Teaching this manual of style to new users is in and of itself a problem, since it might become a new barrier to entry.

The New Workflow

The revised workflow.

Above: The revised, wiki-based workflow.


The Makahiki system already contains features for recording when users have completed certain activities and for unlocking higher tiers of activities in response. The badge system is, from a game design standpoint, easy to think of as a reward system for wiki contributions, some awarded automatically and some by the decision of administrators. (For something similar, see Wikipedia’s Barnstars.) In this system, Makahiki’s Smart Grid Game would be re-purposed for a wiki training game, in which a user plays around in a sandbox and unlocks basic user privileges as he or she goes, earning badges as a certification of basic training. Just as Wikipedia regularly selects featured articles, the authors of the highest-rated guides in each category could periodically be invited to contribute to the documentation of new builds of a software system by the developers.

I think there is also a need for an “Issue” article type, for users who have found a problem and have documented it but do not know how to fix it. Instead of just a usual bug-tracking system, which users may be unaware of, and in which it can be difficult to figure out at first glance why an issue has a certain severity rating, articles on the issues themselves could become collection points for comments, and eventually become troubleshooting guides.

In this system, articles could be segregated by project or by general category. Hypothetically, developers (as administrators) could manage the creation of article categories and the registration of new projects with the site. Links to the current versions of official documentation could be provided on the project’s page, and users could contribute to multiple projects.

On the Surface: Minor UI Tweaks

User pages, version 2.

Above: A slight redesign of the user page.


I have decided to call the more generally targeted version of the site “SetupBooster,” for reasons which in large part have to do with the lack of unused domain names related to speed and launch metaphors. (Things like rocketstart.me and Kickstarter were reserved long ago.) It is probable that SetupBooster is already taken, but I will worry about less fundamental issues like site name and logo after the site itself is up and running. The slight redesign adds more mentions of user achievements and article ratings, and eliminates the newsfeed (which was essentially a duplicate of the newsfeed on the main page).

Since Issues is going to be an article type, the system for reporting problems with articles is being renamed to “bugs.” Both have similar meanings, but I am not sure which name would be more or less confusing for each. The article categorization system is probably just going to be an identifying string without any article format restrictions, as I can’t think of a specific format to write an issue report in that would differ substantially from an article – both would be required to include software version and operating system, and would feature a general description of some aspect of the software. This allows additional types of articles – suggestions for future development, or in the context of Makahiki, popular community-designed widgets with good setup documentation for creating them – to be added to the system more easily. A manual of style could be created for each type, but that is probably something best left to human administrators to enforce.

In One Month

In one month, I will attempt to integrate an open-source wiki application and a ratings module into Django and see if Makahiki’s Smart Grid Game can be configured to detect changes to versions of an article. Though the django-wikiapp and django-ratings look promising at first glance, I do not know what will be involved in integrating them with each other and the Makahiki badges system. At a minimum, it is my hope that, by the end of this semester, test users will be able to create a new user, run through a series of training games in their own article sandbox which reward them with badges, create and edit their own test articles, and rate their quality. Depending on how similar the wiki application is to Wikipedia, I may not need to implement a separate commenting system if the wiki application provides Talk pages. The short-term goal of this project is to create a ratings-driven Wikipedia-like system that runs on Django, uses the Smart Grid Game to train users, and possibly awards badges based on a running count of how often the user has done certain editing tasks.

Corrections to Heroku Posts

Corrections

Some corrections have been made to my previous posts as a result of errors pointed out by instructors and classmates. The articles listed below have been updated to reflect the corrections.

Corrections to Lessons Learned from Deploying Django on Heroku:

1. Step 2: Corrections to the section on pip freeze > requirements.txt:

NEW VERSION

(CORRECTION): Contrary to both previous versions of this post, you can run pip freeze at any time. This will change requirements.txt by adding the new dependencies that have been installed.

If you edit requirements.txt manually, run pip install -r requirements.txt to install any new dependencies:

(venv)mainuser@webdevbox:~/hellodjango$ pip install -r requirements.txt

PREVIOUS VERSION

(CORRECTION): An earlier version of this post stated that the pip freeze command had to be run while the server was running. Actually, as long as the pip install django psycopg2 dj-database-url command succeeded, the new dependencies will be added to the file when you run pip freeze. DO NOT RUN pip freeze WHILE THE SERVER IS RUNNING.

2. Step 2: Corrections to the section on DATABASES in settings.py:

NEW VERSION

Next, the tutorial states “Next, configure the application for the Heroku environment, including Heroku’s Postgres database. The dj-database-url module will parse the values of the DATABASE_URL environment variable and convert them to something Django can understand.” Add these lines to the end of helloheroku/settings.py as specified in the tutorial:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] =  dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

(CORRECTION) An earlier version of this post stated that values in settings.py’s DATABASES section needed to be changed. This was unnecessary. The DATABASE_URL environment variable must be set with Linux’s export command. Thanks to Zach Tomaszewski for pointing this out.

(venv)mainuser@webdevbox:~/hellodjango$ export DATABASE_URL=postgres://testadmin:testing@localhost/django_db

Here, testadmin is a postgresql user, testing is the password for that user, and django_db is the name of the local database. On the local machine, dj_database_url.config() reads the value of DATABASE_URL specified by export and returns its value, which becomes the value of DATABASES. On Heroku, dj_database_url.config() will read the value of DATABASES from Heroku’s environment variables instead.

PREVIOUS VERSION

Next, the tutorial states “Next, configure the application for the Heroku environment, including Heroku’s Postgres database. The dj-database-url module will parse the values of the DATABASE_URL environment variable and convert them to something Django can understand.” Before this will work, the DATABASES section should be changed to match the database you created earlier:

(venv)mainuser@webdevbox:~/hellodjango$ nano hellodjango/settings.py
# Some of the file omitted
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'django_testdb',
        'USER': 'testadmin',
        'PASSWORD': 'testing',
        'HOST': '',
        'PORT': '',
    }
}
# Now go to the very end of the file and add these lines, as in the tutorial:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] =  dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

3. Corrections to Step 3:

Note: This section was accidentally titled Part 3.

NEW VERSION

If, like me, you had at first forgotten to configure PostgreSQL and do an export, you will see errors like this if you try to run the server locally:

[…]

If this happens, set up the database as described in Step 1 and edit settings.py as described in Step 2, then try again.

PREVIOUS VERSION

If, like me, you had at first forgotten to configure postgreSQL, you will see errors like this:

[…]

If this happens, set up the database as described in Step 1 and edit settings.py to configure the DATABASES values as described in Step 2, then try again.

4. Corrections to Step 4:

NEW VERSION

  1. In settings.py, change DEBUG to False. You should always do this before deploying a Django application with gunicorn, because gunicorn does not respect the DEBUG flag. In general, always change DEBUG to False before deploying any production Django application.
    DEBUG = False
    
  2. Add the line gunicorn==0.16.1 to your requirements.txt
  3. Change your Procfile’s web: line to web: gunicorn hellodjango.wsgi
  4. Uncorrected sections omitted…

PREVIOUS VERSION

  1. Add the line gunicorn==0.16.1 to your requirements.txt
  2. Change your Procfile’s web: line to web: gunicorn hellodjango.wsgi
  3. Uncorrected sections omitted…

Corrections to Lessons Learned from Deploying Django on Heroku, Part 2:

Corrections to Step 2:

NEW VERSION

Now open helloheroku/settings.py in your text editor of choice and edit the following lines at the end of the file:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

The file for the example project used here should already have the dj_database_url lines. The SECURE_PROXY_SSL_HEADER setting is used by Django to confirm that a request served from behind a proxy is being served as HTTPS.

(CORRECTION) Ignore the DATABASES section; dj_database_url.config() will import the values for DATABASES based on what is specified in the export command.

OLD VERSION

Now open helloheroku/settings.py in your text editor of choice and edit the following lines:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle',
        'NAME': 'helloheroku_db',        # Or path to database file if using sqlite3.
        'USER': 'testadmin',             # Not used with sqlite3.
        'PASSWORD': 'testing',           # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
    }
}

# Now go to the end of the file:
# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

The file for the example project used here should already have the dj_database_url lines. The SECURE_PROXY_SSL_HEADER setting is used by Django to confirm that a request served from behind a proxy is being served as HTTPS.

Problems with Steps 3, 4, and 4A:

As it turns out, it’s supposed to be possible to get the site working with gunicorn as the server. As this would require substantial changes to Parts 3, 4, and 4A, I might just write a new article as an addition to this one, instead of trying to further revise the existing one.

In large part the problems with gunicorn and collectstatic stemmed from needing to run python manage.py collectstatic locally, and then to set DEBUG = False in settings.py.

One of our instructors, Yongwen Xu, gave this explanation:

6. “collectstatic” and “gunicorn” issues:
The static file handling is confusing both in Django and Heroku. One culprit is the flag “DEBUG” whose default value is set to True in settings.py. When the DEBUG flag is true, “manage.py runserver” will serve the static files automatically from the STATIC_ROOT and STATICFILES_DIRS settings, as defined in settings.py (“PROJECT_ROOT/site_media/static” and “PROJECT_ROOT/static”).
That is why students did not encounter this problem when using the “runserver” command locally and on Heroku.

The DEBUG setting is not honored by gunicorn, which is a server intended for production deployment (in which case the DEBUG flag should be set to false). So, when using gunicorn, the static files are NOT automatically served from STATIC_ROOT, thus, we don’t see any style in the app page, as in Jordan’s case.

One way we could improve the system is to add the following line to the urls.py, to explicitly tell the server to serve the static files from STATIC_ROOT:

urlpatterns += patterns('',
    (r'^' + settings.STATIC_URL[1:] + '(?P.*)$', 'django.views.static.serve',
         {'document_root': settings.STATIC_ROOT}), )

(Some text omitted…)

For local development, it’s best to always run “collectstatic”, so that the static files from both the app and the django admin are copied to the /site_media/static directory. Otherwise, even [if] the DEBUG flag is set to True, the style for the django admin interface will not be available because the admin static files are not in /site_media/static or /static dir.

On Heroku, the “collectstatic” is normally run by heroku automatically. see below on how to disable this:
https://devcenter.heroku.com/articles/django-assets

As a result of all this, for local development, “manage.py runserver” is easiest, while for production deployment, using “gunicorn” is recommended.

I will work on further revisions to the Part 2 post later, as time allows.

Lessons Learned from Deploying Django on Heroku, Part 2

NOTE: This article will set up a working Heroku application. However, there are issues with the way in which it handles collectstatic and gunicorn issues. For more information, see Corrections to Heroku Posts.

In a continuation of last week’s post, this post deploys a Github project on Heroku for a course assignment.

Heroku logo.

This blog is neither endorsed nor sponsored by Heroku.


SYSTEM DETAILS
Ubuntu 12.04.1 LTS (VirtualBox)

Prerequisites:

The prerequisites are the same as in my last article. Linux is recommended, since this guide uses its package manager. Additional steps may be required to install the equivalent software in Windows.

  • Heroku Toolbelt: $ sudo wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
  • Git: $ sudo apt-get install git
    As far as I know, Git should be installed as part of the Heroku toolbelt. If not, install it manually.
  • PostgreSQL: $ sudo apt-get install postgresql
  • Python-Dev: $ sudo apt-get install python-dev
  • Virtualenv: $ sudo apt-get install virtualenv
  • A Heroku account. All services provided by Heroku will be free so long as the 750-hour monthly limit is not exceeded. Examine Heroku pricing here.
  • Other required packages (psycopg2, dj-database-url) are installed during the tutorial.

This article covers the process of deploying a Github project with Twitter Bootstrap features to Heroku.

Step 1: Import The Github Project

Create a new folder, cd to it, and import the Github project:

mainuser@webdevbox:~/workspace/$ mkdir HelloHeroku && cd HelloHeroku
mainuser@webdevbox:~/workspace/HelloHeroku$ git init
Initialized empty Git repository in /home/mainuser/workspace/HelloHeroku/.git/
mainuser@webdevbox:~/workspace/HelloHeroku$ git remote add origin https://github.com/cammoore/responsive-heroku.git
mainuser@webdevbox:~/workspace/HelloHeroku$ git pull origin master
remote: Counting objects: 1975, done.
remote: Compressing objects: 100% (1899/1899), done.
remote: Total 1975 (delta 69), reused 1975 (delta 69)
Receiving objects: 100% (1975/1975), 22.76 MiB | 695 KiB/s, done.
Resolving deltas: 100% (69/69), done.
From https://github.com/cammoore/responsive-heroku
 * branch            master     -> FETCH_HEAD
mainuser@webdevbox:~/workspace/HelloHeroku$

Next, initialize and source the virtual environment. Use pip to install dependencies:

mainuser@webdevbox:~/workspace/HelloHeroku$ virtualenv venv --distribute
mainuser@webdevbox:~/workspace/HelloHeroku$ source venv/bin/activate
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ sudo pip install -r requirements.txt
// Output omitted

Step 2: Create a New Postgresql Database

If you did not configure the pg_hba.conf file as described in Step 1 of the tutorial in the previous article, do so now.

You will need to create a database for the new application:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ sudo su postgres
(venv)postgres@webdevbox:/home/mainuser/workspace/HelloHeroku$ psql
psql (9.1.8)
Type "help" for help.

postgres=# CREATE DATABASE helloheroku2_db OWNER testadmin;
CREATE DATABASE
postgres=# \q
postgres@webdevbox:/home/mainuser/workspace/HelloHeroku$ exit
exit
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$

Now open helloheroku/settings.py in your text editor of choice and edit the following lines at the end of the file:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

The file for the example project used here should already have the dj_database_url lines. The SECURE_PROXY_SSL_HEADER setting is used by Django to confirm that a request served from behind a proxy is being served as HTTPS.

(CORRECTION) Ignore the DATABASES section; dj_database_url.config() will import the values for DATABASES based on what is specified in the export command.

Now, export the database URL to your environment variables, and use set to check that it has been added. Note that you will need to re-export the environment variable every time you exit the terminal in which you started the virtual environment.

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ export DATABASE_URL=postgres://testadmin:testing@localhost/helloheroku_db
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ set | grep postgres
DATABASE_URL=postgres://testadmin:testing@localhost/helloheroku_db

Now add the changed file to Git:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ git add helloheroku/settings.py
// Output omitted
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ git commit -m "Configured settings.py to use postgresql database."
// output omitted

Step 3: Check or Configure Server Settings

The Github project used for this post is set to use Django’s own default web server by default, which runs with manage.py runserver:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ cat Procfile
web: python manage.py runserver 0.0.0.0:$PORT --noreload

NOTE: Though gunicorn was used in the last tutorial, it should not be used here.
Use pip to install the dependencies:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ pip install -r requirements.txt
// Output omitted

Step 4: Create Heroku Application

DISCLAIMER: If you plan to run the app locally, skip to Step 4A. For whatever reason, doing the python manage.py collectstatic manually in the local project before pushing it to Heroku may have caused my Heroku app to be unable to load the .less style files. I recommend that local and Heroku versions of this project be managed in separate directories to avoid this problem.

At this point, I was ready to create the Heroku application:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku create
Creating still-dawn-7589... done, stack is cedar
http://still-dawn-7589.herokuapp.com/ | git@heroku.com:still-dawn-7589.git
Git remote heroku added

Now, push the source code to Heroku:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ git push heroku master
// Output omitted

The Twitter Bootstrap static files are automatically detected by Heroku, which runs a python manage.py collectstatic on them:

// Output omitted
       Successfully installed Django argparse distribute dj-database-url psycopg2
       Cleaning up...
-----> Collecting static files
// Output omitted

It is important not to run python manage.py collectstatic locally before creating the Heroku project, or the .less files used by Twitter Bootstrap may not work correctly. (I am not sure that this was the cause of the problem. For a longer explanation see the “Conclusions” section.)

The page without CSS styles.

Above: If you run a “python manage.py collectstatic” command before pushing to Heroku, styles might not be applied to the page. (The hidden-journey-6676 app has been deleted.)


Thanks to Justin Lee for pointing this out.

Now, sync the Heroku database and set up a superuser for Django:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku run python manage.py syncdb
Running `python manage.py syncdb` attached to terminal... up, run.2022
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log
Creating table polls_poll
Creating table polls_choice

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'u15523'): helloherokuadmin
E-mail address: admin@example.com
Password: hhadmin
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$

Now, start the app and open it in your browser:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku ps:scale web=1
Scaling web processes... done, now running 1
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku open
Opening still-dawn-7589... done
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$

If you see this error, continue:

Error message.

Above: Errors related to unresponsive .less scripts can usually be ignored. The page will eventually load.


The final page will look like this:
The Twitter Bootstrap-powered page.

Above: The final page.

Step 4A: (Optional) Run The App Locally

To run the example app locally, follow this post up until the end of Step 3. Then synchronize the database:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ python manage.py syncdb
// Output omitted. It is generally the same as the output from 
// the "heroku run python manage.py syncdb" command in Step 3.

Now run collectstatic:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ python manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings.

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
// Output omitted...
Copying '/home/mainuser/workspace/HelloHeroku/static/js/prefixfree.min.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/Makefile'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/CONTRIBUTING.md'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/component.json'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/composer.json'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/package.json'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/README.md'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/LICENSE'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-popover.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-affix.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-transition.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-scrollspy.js'
// Output omitted...
258 static files copied.

This will copy all the .less, .js, and other Twitter Bootstrap files to the site_media directory in the HelloHeroku directory.

Afterwards, start the local Django server with this command.

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ python manage.py runserver

Next, go to http://127.0.0.1:8000/ to view your page.

Conclusions

Deploying Github projects on Heroku seems to work fairly well as long as the default Django server is used. I am not sure whether my deployment problems were caused by the collectstatic command or by using gunicorn as the server; however, even when the collectstatic command was handled by Heroku, using gunicorn as the server caused the same problem. This suggests that the problem stems from using gunicorn as the web server. Otherwise, Heroku seems to be well-integrated with Git and allows you to manage your projects much as you would manage GitHub or a regular remote repository.

This post was last revised on February 27, 2013. See Corrections to Heroku Posts for more details.

Lessons Learned From Deploying Django on Heroku

This article describes my experience with the Heroku/Django tutorial at https://devcenter.heroku.com/articles/django. It largely follows the original tutorial, but points out the areas where the current tutorial seemed to be unclear on what was required.

Heroku logo.

This blog is neither endorsed nor sponsored by Heroku.


SYSTEM DETAILS
Ubuntu 12.04.1 LTS (VirtualBox)

Prerequisites:

I recommend that anyone following the tutorial do so in Linux if possible. Versions of many of these packages exist for Windows, but the setup process will be more complicated.

*NOTE: This is not mentioned in the original tutorial, but an error occurs without it. See the rest of this post.

Heroku is a cloud application platform: the Heroku servers manage web applications which are isolated into Unix-like environments called dynos, which execute a single command on a packaged version of the application called a slug. The slug is compiled from a Git repository of the files that make up the web applications.

Step 1: Configure PostGreSQL

The tutorial says that you need to have PostgreSQL installed “to test locally.” They do not explain how to set it up properly. To set up a database in PostgreSQL, you need to first make sure that it accepts local connections. Open the file pg_hba.conf in the text editor (vi, nano, Emacs, etc.) of your choice.

mainuser@webdevbox:~/$ sudo nano /etc/postgresql/9.1/main/pg_hba.conf

Look for a line similar to the below image:

pg_hba.conf

Above: The pg_hba.conf file open in nano.


The last parameter in the line will say something like “peer” or “md5.” Change it to “trust”: we want it to allow all local connections. Now, restart postgresql:

mainuser@webdevbox:~/$ sudo service postgresql restart
 * Restarting PostgreSQL 9.1 database server                             [ OK ] 

You should now be able to switch to the postgres user and access a postgresql prompt (postgres=#):

mainuser@webdevbox:~/$ sudo su postgres
postgres@webdevbox:~/$ psql
postgres=#

If you like, you can set a service password here. By default, the postgres account has no password on UNIX / Linux. For a brief explanation, see the section on “Service Password” in this post by Dave Page.

postgres=# \password postgres
Enter new password:

Note that this is not the superuser password; without further configuration, you will not be able to use su / sudo / other switching commands as the postgres user. Changing/setting the superuser password for postgres is beyond the scope of this post.

Now create a user and create a table with that user as an administrator, then exit postgres and the postgres user:

postgres=# CREATE USER testadmin WITH CREATEDB PASSWORD 'testing';
CREATE ROLE
postgres=# CREATE DATABASE django_testdb OWNER testadmin;
CREATE DATABASE
postgres=# \q
postgres@webdevbox:~/$ exit
mainuser@webdevbox:~/$

Remember the new user, password, and database name; you will need them later.

Step 2: Set Up a Local Django App

As the tutorial says, run these commands to install Django and some other dependencies:

mainuser@webdevbox:~/$ mkdir hellodjango && cd hellodjango
mainuser@webdevbox:~/hellodjango$ virtualenv venv --distribute
New python executable in venv/bin/python
Installing distribute...............done.
Installing pip...............done.
mainuser@webdevboxL~/hellodjango$ source venv/bin/activate
(venv)mainuser@webdevbox:~/hellodjango$ pip install Django psycopg2 dj-database-url

I encountered a bug while psycopg2 was being compiled. (If you installed python-dev, you probably will not see this.)

// Some output omitted
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.6 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090108 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/usr/include/python2.7 -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.1/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-2.7/psycopg/psycopgmodule.o -Wdeclaration-after-statement

In file included from psycopg/psycopgmodule.c:27:0:

./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory

compilation terminated.

error: command 'gcc' failed with exit status 1

According to a psycopg2 FAQ, this error is caused by not having the python-dev package installed, so I installed it:

(venv)mainuser@webdevbox:~/hellodjango$ sudo apt-get install python-dev

The command should now work correctly. Continuing to follow the tutorial, you should run these commands:

(venv)mainuser@webdevbox:~/hellodjango$ pip install Django psycopg2 dj-database-url
// Output omitted
(venv)mainuser@webdevbox:~/hellodjango$ django-admin.py startproject hellodjango .
(venv)mainuser@webdevbox:~/hellodjango$ python manage.py runserver
0 errors found
Django version 1.4, using settings 'hellodjango.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
^C
(venv)mainuser@webdevbox:~/hellodjango$ pip freeze > requirements.txt
(venv)mainuser@webdevbox:~/hellodjango$ cat requirements.txt
Django==1.4
psycopg2==2.4.5
dj-database-url==0.2.0

(CORRECTION): Contrary to both previous versions of this post, you can run pip freeze at any time. This will change requirements.txt by adding the new dependencies that have been installed.

If you edit requirements.txt manually, run pip install -r requirements.txt to install any new dependencies:

(venv)mainuser@webdevbox:~/hellodjango$ pip install -r requirements.txt

The contents of your requirements.txt file may look different from the example in the tutorial. As long as the lines in the tutorial are part of the requirements.txt file, everything should be fine.

Next, the tutorial states “Next, configure the application for the Heroku environment, including Heroku’s Postgres database. The dj-database-url module will parse the values of the DATABASE_URL environment variable and convert them to something Django can understand.” Add these lines to the end of helloheroku/settings.py as specified in the tutorial:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] =  dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

(CORRECTION) An earlier version of this post stated that values in settings.py’s DATABASES section needed to be changed. This was unnecessary. The DATABASE_URL environment variable must be set with Linux’s export command. Thanks to Zach Tomaszewski for pointing this out.

(venv)mainuser@webdevbox:~/hellodjango$ export DATABASE_URL=postgres://testadmin:testing@localhost/django_db

Here, testadmin is a postgresql user, testing is the password for that user, and django_db is the name of the local database. On the local machine, dj_database_url.config() reads the value of DATABASE_URL specified by export and returns its value, which becomes the value of DATABASES. On Heroku, dj_database_url.config() will read the value of DATABASES from Heroku’s environment variables instead.

Next, the Procfile needs to be created. As far as I know, it goes in the top-level directory of the project (hellodjango, where manage.py is).

(venv)mainuser@webdevbox:~/hellodjango$ nano Procfile
web: python manage.py runserver 0.0.0.0:$PORT --noreload

As suggested by the tutorial, I used Github’s Python .gitignore file as the project .gitignore file. Though the line *.py[cod] probably covers *.pyc, I added lines for venv and *.pyc anyway, to be safe. Then I initialized the Git repository:

(venv)mainuser@webdevbox:~/hellodjango$ git init
Initialized empty Git repository in /home/mainuser/hellodjango/.git/
(venv)mainuser@webdevbox:~/hellodjango$ git add .
(venv)mainuser@webdevbox:~/hellodjango$ git commit -m "My Django app."
[master (root-commit) 6c2765c] My Django app.
 9 files changed, 262 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Procfile
 create mode 100644 hellodjango/settings.py
 create mode 100644 hellodjango/urls.py
 create mode 100644 hellodjango/wsgi.py
 create mode 100644 manage.py
 create mode 100644 requirements.txt
 create mode 120000 venv/include/python2.7
 create mode 120000 venv/local/include

Step 3: Deploying The App to Heroku

When you first installed the Heroku toolbelt, you should have been asked to generate SSH keys:

(venv)mainuser@webdevbox:~/$ heroku login
Enter your Heroku credentials.
Email: admin@example.com
Password: same-as-heroku-account
Could not find an existing public key.
Would you like to generate one? [Yn] Y
Generating new SSH public key.
Uploading ssh public key /Users/mainuser/.ssh/id_rsa.pub.

Otherwise, follow the developers’ example to generate new keys with ssh-keygen -t rsa. Once you have a public SSH key (check in /Users/<username>/.ssh), you should be able to create a Heroku app.

(venv)mainuser@webdevbox:~/$ heroku create
Creating still-badlands-2085... done, stack is cedar
http://still-badlands-2085.herokuapp.com/ | git@heroku.com:still-badlands-2085.git
Git remote heroku added

Now push your code to the Heroku repository:

(venv)mainuser@webdevbox:~/$ git push heroku master
Counting objects: 16, done.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (16/16), 4.34 KiB, done.
Total 16 (delta 0), reused 0 (delta 0)
-----> Python app detected
-----> No runtime.txt provided; assuming python-2.7.3.
-----> Preparing Python runtime (python-2.7.3)
-----> Installing Distribute (0.6.34)
-----> Installing Pip (1.2.1)
-----> Installing dependencies using Pip (1.2.1)
       Downloading/unpacking Django==1.4.4 (from -r requirements.txt (line 1))
         Running setup.py egg_info for package Django
           
       Downloading/unpacking argparse==1.2.1 (from -r requirements.txt (line 2))
         Running setup.py egg_info for package argparse
           
           no previously-included directories found matching 'doc/_build'
           no previously-included directories found matching 'env24'
           no previously-included directories found matching 'env25'
           no previously-included directories found matching 'env26'
           no previously-included directories found matching 'env27'
       Downloading/unpacking distribute==0.6.24 (from -r requirements.txt (line 3))
         Running setup.py egg_info for package distribute
           
           warning: no files found matching 'Makefile' under directory 'docs'
           warning: no files found matching 'indexsidebar.html' under directory 'docs'
       Downloading/unpacking dj-database-url==0.2.1 (from -r requirements.txt (line 4))
         Downloading dj-database-url-0.2.1.tar.gz
         Running setup.py egg_info for package dj-database-url
           
       Downloading/unpacking psycopg2==2.4.6 (from -r requirements.txt (line 5))
         Running setup.py egg_info for package psycopg2
           
           no previously-included directories found matching 'doc/src/_build'
       Installing collected packages: Django, argparse, distribute, dj-database-url, psycopg2
         Running setup.py install for Django
           changing mode of build/scripts-2.7/django-admin.py from 600 to 755
           
           changing mode of /app/.heroku/python/bin/django-admin.py to 755
         Running setup.py install for argparse
           
           no previously-included directories found matching 'doc/_build'
           no previously-included directories found matching 'env24'
           no previously-included directories found matching 'env25'
           no previously-included directories found matching 'env26'
           no previously-included directories found matching 'env27'
         Found existing installation: distribute 0.6.34
           Uninstalling distribute:
             Successfully uninstalled distribute
         Running setup.py install for distribute
           Before install bootstrap.
           Scanning installed packages
           Setuptools installation detected at /app/.heroku/python/lib/python2.7/site-packages
           Non-egg installation
           Removing elements out of the way...
           Already patched.
           /app/.heroku/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info already patched.
           
           warning: no files found matching 'Makefile' under directory 'docs'
           warning: no files found matching 'indexsidebar.html' under directory 'docs'
           Installing easy_install script to /app/.heroku/python/bin
           Installing easy_install-2.7 script to /app/.heroku/python/bin
           After install bootstrap.
           /app/.heroku/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info already exists
         Running setup.py install for dj-database-url
           
         Running setup.py install for psycopg2
           building 'psycopg2._psycopg' extension

// gcc output omitted           

           no previously-included directories found matching 'doc/src/_build'
       Successfully installed Django argparse distribute dj-database-url psycopg2
       Cleaning up...
-----> Collecting static files
       0 static files copied.

-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size: 29.2MB
-----> Launching... done, v6
       http://still-badlands-2085.herokuapp.com deployed to Heroku

To git@heroku.com:still-badlands-2085.git
 * [new branch]      master -> master
(venv)mainuser@webdevbox:~/hellodjango$ 

If you sign in to your Heroku account in a browser and navigate to My Apps, then to your-app-name-####, your app should be visible:

Heroku dashboard.

Above: The Heroku dashboard for app still-badlands-2085. If the “web” box is checked and the dyno count is 1, your app may be running already.


Now, you should be able to start the app with heroku ps:scale web=#, where # is the number of web dynos. Free accounts can have more than one app, but if more than one dyno is running for an extended period, you will exceed your 750 free hours (750 hours = 31.25 days) per month. Use heroku ps to view the app state, heroku open to view the app in your default browser, and heroku logs to view recent log information.

You can also start the app from your browser by checking the “web” checkbox, then clicking “Apply Changes.”

(venv)mainuser@webdevbox:~/hellodjango$ heroku ps:scale web=1
Scaling web processes... done, now running 1
(venv)mainuser@webdevbox:~/hellodjango$ heroku ps
=== web: `python manage.py runserver 0.0.0.0:$PORT --noreload`
web.1: up 2013/02/19 18:07:46 (~ 6m ago)

(venv)mainuser@webdevbox:~/hellodjango$ heroku open
Opening still-badlands-2085... done

The Django example app.

Above: The still-badlands-2085 app in Google Chrome.


Now you should be able to sync your Heroku database.

(venv)mainuser@webdevbox:~/hellodjango$ heroku run python manage.py syncdb
Running `python manage.py syncdb` attached to terminal... up, run.6600
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'u55828'): someadmin
E-mail address: admin@example.com
Password: somethingbetter
Password (again): somethingbetter
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
(venv)mainuser@webdevbox:~/hellodjango$

If, like me, you had at first forgotten to configure PostgreSQL and do an export, you will see errors like this if you try to run the server locally:

(venv)mainuser@webdevbox:~/hellodjango$ python manage.py syncdb
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
    cursor = connection.cursor()
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/db/backends/dummy/base.py", line 15, in complain
    raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.

If this happens, set up the database as described in Step 1 and edit settings.py as described in Step 2, then try again.

Step 4: Change The Web Server To Gunicorn

If you’ve been following the Heroku tutorial so far, there’s one last step: switching the webserver to gunicorn.

Gunicorn logo.

Gunicorn is a Python port of Ruby’s WSGI HTTP server, Unicorn.

  1. In settings.py, change DEBUG to False. You should always do this before deploying a Django application with gunicorn, because gunicorn does not respect the DEBUG flag. In general, always change DEBUG to False before deploying any production Django application.
    DEBUG = False
    
  2. Add the line gunicorn==0.16.1 to your requirements.txt
  3. Change your Procfile’s web: line to web: gunicorn hellodjango.wsgi
  4. Run pip to update your dependencies:
    (venv)mainuser@webdevbox:~/hellodjango$ pip install -r requirements.txt
    
  5. Run foreman start to run the application locally. Use Ctrl-C to quit the server when you’re done.
    (venv)mainuser@webdevbox:~/hellodjango$ foreman start
    18:36:03 web.1  | started with pid 3420
    18:36:04 web.1  | 2013-02-19 18:36:04 [3423] [INFO] Starting gunicorn 0.16.1
    18:36:04 web.1  | 2013-02-19 18:36:04 [3423] [INFO] Listening at: http://0.0.0.0:5000 (3423)
    18:36:04 web.1  | 2013-02-19 18:36:04 [3423] [INFO] Using worker: sync
    18:36:04 web.1  | 2013-02-19 18:36:04 [3426] [INFO] Booting worker with pid: 3426
    ^CSIGINT received
    18:41:06 web.1  | 2013-02-19 18:41:06 [3426] [INFO] Worker exiting (pid: 3426)
    18:41:06 system | sending SIGTERM to all processes
    18:41:06 web.1  | 2013-02-19 18:41:06 [3423] [INFO] Handling signal: int
    SIGTERM received
    18:41:06 web.1  | terminated by SIGTERM
    
  6. Push your changes to Heroku:
    (venv)mainuser@webdevbox:~/hellodjango$ git add Procfile requirements.txt hellodjango/settings.py
    (venv)mainuser@webdevbox:~/hellodjango$ git commit -m "Changed server to gunicorn."
    (venv)mainuser@webdevbox:~/hellodjango$ git push heroku master
    Counting objects: 7, done.
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (4/4), 364 bytes, done.
    Total 4 (delta 2), reused 0 (delta 0)
    -----> Python app detected
    -----> No runtime.txt provided; assuming python-2.7.3.
    -----> Using Python runtime (python-2.7.3)
    -----> Installing dependencies using Pip (1.2.1)
           Downloading/unpacking gunicorn==0.16.1 (from -r requirements.txt (line 7))
             Running setup.py egg_info for package gunicorn
               
           Installing collected packages: gunicorn
             Running setup.py install for gunicorn
               
               Installing gunicorn_paster script to /app/.heroku/python/bin
               Installing gunicorn script to /app/.heroku/python/bin
               Installing gunicorn_django script to /app/.heroku/python/bin
           Successfully installed gunicorn
           Cleaning up...
    -----> Collecting static files
           0 static files copied.
    
    -----> Discovering process types
           Procfile declares types -> web
    -----> Compiled slug size: 29.5MB
    -----> Launching... done, v7
           http://still-badlands-2085.herokuapp.com deployed to Heroku
    
    To git@heroku.com:still-badlands-2085.git
       6c2765c..697eb0f  master -> master
    mainuser@webdevbox:~/hellodjango$ 
    
  7. Run heroku logs. You should see some reference to gunicorn in the recent log entries.

Step 5: Stopping Your App

If you’re doing this for a course project like I did, or doing some testing later, you’ll want to leave the app running. When you’re done, shut off the process by scaling back the number of dynos to 0:

(venv)mainuser@webdevbox:~/hellodjango$ heroku ps:scale web=0
Scaling web processes... done, now running 0
(venv)mainuser@webdevbox:~/hellodjango$ 

Though I may have missed something, I could not find any advice on the official site on how to stop an app completely outside of the web interface (where you can uncheck the web) checkbox to stop the dyno). This tip comes from Stack Overflow.

Overall Thoughts

In my opinion, the biggest weakness of the tutorial is that it does not explain how to set up the Postgresql database or link to a tutorial which does. Maybe the developers assumed that anyone who is experienced enough in web development to be using Heroku would have used Postgresql before. Maybe they just expected people to Read The Manual. Otherwise, I found Heroku to be fairly easy to deploy and use.

This post was last revised on February 27, 2013. See Corrections to Heroku Posts for more details.