/var/

Ajax data with Fixed Data Table for React

Serafeim Papastefanos — Tue, 22 Dec 2015 15:20:00 +0200

Contents

Introduction
Our project
How FixedDataTable gets its data
AjaxCell: Non working version 1
AjaxCell: Non working version 2
AjaxCell: Non working version 3
AjaxCell: Final version
The Table container component
Some enchancements
Conslusion

Introduction

FixedDataTable is a nice React component from Facebook that is used to render tabular data. Its main characteristic is that it can handle many rows without sacrificing performance, however probably due to this fact I was not able to find any examples for loading data using Ajax - all examples I was able to find had the data already loaded (or created on the fly).

So, although FixedDataTable is able to handle many rows, I am against transfering all of them to the user whenever our page loads since at most one page of data will be shown on screen (30 rows or so ?) - any other actions (filtering, sorting, aggregate calculations etc) should be done on the server.

In the following I will present a simple, react-only example with a FixedDataTable that can be used with server-side, asynchronous, paginated data.

Our project

Let’s see an example of what we’ll build:

As a source of the data I’ve used the Star Wars API and specifically its People API by issuing requests to http://swapi.co/api/people/?format=json. This will return an array of people from the star wars universe in JSON format - the results are paginated with a page size of 10 (and we can switch to another page using the extra page= request parameter).

I will use es6 with the object spread operator (as described in a previous article) to write the code, using a single main.js as a source which will be transpiled to dist/bundle.js.

The placeholder HTML for our application is:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8" />
    <title>Hello Ajax Fixed Data Table!</title>
    <link rel="stylesheet" href='https://cdnjs.cloudflare.com/ajax/libs/fixed-data-table/0.6.0/fixed-data-table.min.css'>
  </head>
  <body>
    <div id="main"></div>
    <script type="text/javascript" src='dist/bundle.js' ></script>
  </body>
</html>

The versions that are used are react 14.3 and fixed-data-table 0.6.0. You can find this project in github @ https://github.com/spapas/react-tables — tag name fixed-data-table-ajax.

How FixedDataTable gets its data

The data of a FixedDataTable component is defined through a number of Column components and specifically, the cell attribute of that component which sould either return be a static string to be displayed in that column or a React component (usually a Cell) that will be displayed in that column or even a function that returns a string or a react component. The function (or component) passed to cell will have an object with a rowIndex attribute (among others) as a parameter, for example the following

<Column
  header={<Cell>Url</Cell>}
  cell={props => props.rowIndex}
  width={200}
/>

will just display the rowIndex for each cell.

So we can see that for each cell that FixedDataTable wants to display it will use its corresponding cell attribute. So, a general though if we wanted to support asynchronous, paginated data is to check the rowIndex and retrieve the correct page asynchronously. This would lead to a difficulty: What should the cell function return? We’ll try to resolve this using an AjaxCell function that should return a react componet to put in the cell:

AjaxCell: Non working version 1

A first sketch of an `AjaxCell component could be something like this:

// Careful - not working
const AjaxCell1 = ({rowIndex, col, ...props}) => {
    let page = 1;
    let idx = rowIndex;
    if(rowIndex>=pageSize) {
        page = Math.floor(rowIndex / pageSize) + 1;
        idx = rowIndex % pageSize;
    }
    fetch('http://swapi.co/api/people/?format=json&page='+page).then(function(response) {
        return response.json();
    }).then(function(j) {
        // Here we have the result ! But where will it go ?
    });
    return /// what ?
}

The col attribute will later be used to select the attribute we want to display in this cell. The page and idx will have the correct page number and idx inside that page (for example, if rowIndex is 33, page will be 4 and idx will be 3. The problem with the above is that the fetch function will be called asynchronously so when the AjaxCell returns it will not have the results (yet)! So we won’t be able to render anything :(

AjaxCell: Non working version 2

To be able to render something, we could put the results of fetching a page to a cache dictionary — and only fetch that page if it is not inside the cache - if it is inside the cache then we’ll just return the correct value:

// Careful - not working correctly
const cache = {}
const AjaxCell2 = ({rowIndex, col, forceUpdate, ...props}) => {
    let page = 1;
    let idx = rowIndex;
    if(rowIndex>=pageSize) {
        page = Math.floor(rowIndex / pageSize) + 1;
        idx = rowIndex % pageSize;
    }
    if (cache[page]) {
        return <Cell>{cache[page][idx][col]}</Cell>
    } else {
        console.log("Loading page " + page);
        fetch('http://swapi.co/api/people/?format=json&page='+page).then(function(response) {
            return response.json();
        }).then(function(j) {
            cache[page] = j['results'];
        });
    }
    return <Cell>-</Cell>;
}

The above will work, since it will return an empty (<Cell>-</Cell>) initially but when the fetch returns it will set the cache for that page and return the correct value (<Cell>{cache[page][idx][col]}</Cell>).

However, as can be understood, when the page first loads it will call AjaxCell2 for all visible cells — because fetch is asynchronous and takes time until it returns (and sets the cache for that page), so the fetch will be called for all cells!

AjaxCell: Non working version 3

fetch ing each page multiple times is of course not acceptable, so we’ll add a loading flag and fetch will be called only when this flag is false, like this:

// Not ready yet
const cache = {};
let loading = false;
const AjaxCell2 = ({rowIndex, col, ...props}) => {
    let page = 1;
    let idx = rowIndex;
    if(rowIndex>=pageSize) {
        page = Math.floor(rowIndex / pageSize) + 1;
        idx = rowIndex % pageSize;
    }
    if (cache[page]) {
        return <Cell>{cache[page][idx][col]}</Cell>
    } else if(!loading) {
        console.log("Loading page " + page);
        loading = true;

        fetch('http://swapi.co/api/people/?format=json&page='+page).then(function(response) {
            return response.json();
        }).then(function(j) {
            cache[page] = j['results'];
            loading = false;
        });
    }
    return <Cell>-</Cell>;
}

This works much better - the cells are rendered correctly and each page is loaded only once. However, if for example I tried to move to the end of the table quickly, I would see some cells that are always loading (they never get their correct value). This is because there is no way to know that the fetch function has actually completed in order to update with the latest (correct) value of that cell and will contain the stale placeholder (<Cell>-</Cell>) value.

AjaxCell: Final version

To clear the stale data we need to do an update to the table data when each fetch is finished — this should be done by a callback that will be passed to the AjaxCell, like this:

const cache = {};
let loading = false;
const AjaxCell = ({rowIndex, col, forceUpdate, ...props}) => {
    let page = 1;
    let idx = rowIndex;
    if(rowIndex>=pageSize) {
        page = Math.floor(rowIndex / pageSize) + 1;
        idx = rowIndex % pageSize;
    }
    if (cache[page]) {
        return <Cell>{cache[page][idx][col]}</Cell>
    } else if(!loading) {
        console.log("Loading page " + page);
        loading = true;

        fetch('http://swapi.co/api/people/?format=json&page='+page).then(function(response) {
            return response.json();
        }).then(function(j) {
            cache[page] = j['results'];
            loading = false;
            forceUpdate();
        });
    }
    return loadingCell;
}

So we pass a forceUpdate callback as a property which is called when a fetch is finished. This may result to some not needed updates to the table (since we would do a fetch + forceUpdate for non-displayed data) but we can now be positive that when the data is loaded the table will be updated to dispaly it.

The Table container component

Finally, the component that contains the table is the following:

class TableContainer extends React.Component {
    render() {
        return <Table
            rowHeight={30} rowsCount={87} width={600} height={200} headerHeight={30}>

            <Column
              header={<Cell>Name</Cell>}
              cell={ <AjaxCell col='name' forceUpdate={this.forceUpdate.bind(this)} /> }
              width={200}
            />
            <Column
              header={<Cell>Birth Year</Cell>}
              cell={ <AjaxCell col='birth_year' forceUpdate={this.forceUpdate.bind(this)} /> }
              width={200}
            />
            <Column
              header={<Cell>Url</Cell>}
              cell={ <AjaxCell col='url' forceUpdate={this.forceUpdate.bind(this)} /> }
              width={200}
            />

        </Table>
    }
}

I’ve made it a component in order to be able to bind the forceUpdate method of the component to and this and pass it to the forceUpdate parameter to the AjaxCell component. I’ve hard-coded the rowsCount value — instead we should have done an initial fetch to the first page of the API to get the total number of rows and only after that fetch had returned display the <Table> component (left as an exercise to the reader).

Some enchancements

Instead of displaying <Cell>-</Cell> (or </Cell>) when the page loads, I propose to define a cell with an embedded spinner, like

const loadingCell = <Cell>
    <img width="16" height="16" alt="star" src="data:image/gif;base64,R0lGODlhEAAQAPIAAP///wAAAMLCwkJCQgAAAGJiYoKCgpKSkiH/C05FVFNDQVBFMi4wAwEAAAAh/hpDcmVhdGVkIHdpdGggYWpheGxvYWQuaW5mbwAh+QQJCgAAACwAAAAAEAAQAAADMwi63P4wyklrE2MIOggZnAdOmGYJRbExwroUmcG2LmDEwnHQLVsYOd2mBzkYDAdKa+dIAAAh+QQJCgAAACwAAAAAEAAQAAADNAi63P5OjCEgG4QMu7DmikRxQlFUYDEZIGBMRVsaqHwctXXf7WEYB4Ag1xjihkMZsiUkKhIAIfkECQoAAAAsAAAAABAAEAAAAzYIujIjK8pByJDMlFYvBoVjHA70GU7xSUJhmKtwHPAKzLO9HMaoKwJZ7Rf8AYPDDzKpZBqfvwQAIfkECQoAAAAsAAAAABAAEAAAAzMIumIlK8oyhpHsnFZfhYumCYUhDAQxRIdhHBGqRoKw0R8DYlJd8z0fMDgsGo/IpHI5TAAAIfkECQoAAAAsAAAAABAAEAAAAzIIunInK0rnZBTwGPNMgQwmdsNgXGJUlIWEuR5oWUIpz8pAEAMe6TwfwyYsGo/IpFKSAAAh+QQJCgAAACwAAAAAEAAQAAADMwi6IMKQORfjdOe82p4wGccc4CEuQradylesojEMBgsUc2G7sDX3lQGBMLAJibufbSlKAAAh+QQJCgAAACwAAAAAEAAQAAADMgi63P7wCRHZnFVdmgHu2nFwlWCI3WGc3TSWhUFGxTAUkGCbtgENBMJAEJsxgMLWzpEAACH5BAkKAAAALAAAAAAQABAAAAMyCLrc/jDKSatlQtScKdceCAjDII7HcQ4EMTCpyrCuUBjCYRgHVtqlAiB1YhiCnlsRkAAAOwAAAAAAAAAAAA==" />
</Cell>

and return this instead.

Also, if your REST API returns too fast and you’d like to see what would happen if the server request took too long to return, you could change fetch like this

fetch('http://swapi.co/api/people/?format=json&page='+page).then(function(response) {
    return response.json();
}).then(function(j) {
    setTimeout( () => {
        cache[page] = j['results'];
        loading = false;
        forceUpdate();
    }, 1000);
});

to add a 1 second delay.

Conslusion

The above is a just a proof of concept of using FixedDataTable with asynchronously loaded server-side data. This of course could be used for small projects (I am already using it for an internal project) but I recommend using the flux architecture for more complex projects. What this more or less means is that a store component should be developed that will actually keep the data for each row, and a fetchCompleted action should be dispatched when the fetch is finished instead of calling forceUpdate directly.

PDFs in Django: The essential guide

Serafeim Papastefanos — Fri, 27 Nov 2015 10:20:00 +0200

Contents

Introduction
The players
Django integration
More advanced xhtml2pdf features
- Laying out
- Extra stuff
Conclusion

Introduction

I’ve noticed that although it is easy to create PDFs with Python, there’s no complete guide on how to integrate these tools with Django and resolve the problems that you’ll encounter when trying to actually create PDFs from your Django web application.

In this article I will present the solution I use for creating PDFs with Django, along with various tips on how to solve most of your common requirements. Specifically, here are some things that we’ll cover:

Learn how to create PDFs "by hand"
Create PDFs with Django using a normal Djang Template (similar to an HTML page)
Change the fonts of your PDFs
Use styling in your output
Create layouts
Embed images to your PDFs
Add page numbers
Merge (concatenate) PDFs

The players

We are going to use the following main tools:

ReportLab is an open source python library for creating PDFs. It uses a low-level API that allows "drawing" strings on specific coordinates on the PDF - for people familiar with creating PDFs in Java it is more or less iText for python.
xhtml2pdf (formerly named pisa) is an open source library that can convert HTML/CSS pages to PDF using ReportLab.
django-xhtml2pdf is a wrapper around xhtml2pdf that makes integration with Django easier.
PyPDF2 is an open source tool of that can split, merge and transform pages of PDF files.

I’ve created a django project https://github.com/spapas/django-pdf-guide with everything covered here. Please clone it, install its requirements and play with it to see how everything works !

Before integrating the above tools to a Django project, I’d like to describe them individually a bit more. Any files I mention below will be included in this project.

ReportLab

ReportLab offers a really low API for creating PDFs. It is something like having a canvas.drawString() method (for people familiar with drawing APIs) for your PDF page. Let’s take a look at an example, creating a PDF with a simple string:

from reportlab.pdfgen import canvas
import reportlab.rl_config


if __name__ == '__main__':
    reportlab.rl_config.warnOnMissingFontGlyphs = 0
    c = canvas.Canvas("./hello1.pdf",)
    c.drawString(100, 100, "Hello World")
    c.showPage()
    c.save()

Save the above in a file named testreportlab1.py. If you run python testreportlab1.py (in an environment that has reportlab of cours) you should see no errors and a pdf named hello1.pdf created. If you open it in your PDF reader you’ll see a blank page with "Hello World" written in its lower right corner.

If you try to add a unicode text, for example "Καλημέρα ελλάδα", you should see something like the following:

It seems that the default font that ReportLab uses does not have a good support for accented greek characters since they are missing (and probably for various other characters).

To resolve this, we could try changing the font to one that contains the missing symbols. You can find free fonts on the internet (for example the DejaVu font), or even grab one from your system fonts (in windows, check out c:\windows\fonts\). In any case, just copy the ttf file of your font inside the folder of your project and crate a file named testreportlab2.py with the following (I am using the DejaVuSans font):

# -*- coding: utf-8 -*-
import reportlab.rl_config
from reportlab.pdfbase import pdfmetrics
from reportlab.pdfbase.ttfonts import TTFont


if __name__ == '__main__':
    c = canvas.Canvas("./hello2.pdf",)
    reportlab.rl_config.warnOnMissingFontGlyphs = 0
    pdfmetrics.registerFont(TTFont('DejaVuSans', 'DejaVuSans.ttf'))

    c.setFont('DejaVuSans', 22)
    c.drawString(100, 100, u"Καλημέρα ελλάδα.")

    c.showPage()
    c.save()

The above was just a scratch on the surface of ReportLab, mainly to be confident that everything will work fine for non-english speaking people! To find out more, you should check the ReportLab open-source User Guide.

I also have to mention that the company behind ReportLab offers some great commercial solutions based on ReportLab for creating PDFs (similar to JasperReports) - check it out if you need support or advanced capabilities.

xhtml2pdf

The xhtml2pdf is a really great library that allows you to use html files as a template to a PDF. Of course, an html cannot always be converted to a PDF since, unfortunately, PDFs do have pages.

xhtml2pdf has a nice executable script that can be used to test its capabilities. After you install it (either globally or to a virtual environment) you should be able to find out the executable $PYTHON/scripts/xhtml2pdf (or xhtml2pdf.exe if you are in Windows) and a corresponding python script @ $PYTHON/scripts/xhtml2pdf-script.py.

Let’s try to use xhtml2pdf to explore some of its capabilities. Create a file named testxhtml2pdf.html with the following contents and run xhtml2pdf testxhtml2pdf.html:

<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
</head>
<body>
    <h1>Testing xhtml2pdf </h1>
    <ul>
        <li><b>Hello, world!</b></li>
        <li><i>Hello, italics</i></li>
        <li>Καλημέρα Ελλάδα!</li>
    </ul>
    <hr />
    <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus nulla erat, porttitor ut venenatis eget,
    tempor et purus. Nullam nec erat vel enim euismod auctor et at nisl. Integer posuere bibendum condimentum. Ut
    euismod velit ut porttitor condimentum. In ullamcorper nulla at lectus fermentum aliquam. Nunc elementum commodo
    dui, id pulvinar ex viverra id. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos
    himenaeos.</p>

    <p>Interdum et malesuada fames ac ante ipsum primis in faucibus. Sed aliquam vitae lectus sit amet accumsan. Morbi
    nibh urna, condimentum nec volutpat at, lobortis sit amet odio. Etiam quis neque interdum sapien cursus ornare. Cras
    commodo lacinia sapien nec porta. Suspendisse potenti. Nulla hendrerit dolor et rutrum consectetur.</p>
    <hr />
    <img  width="26" height="20" src="data:image/gif;base64,R0lGODlhEAAOALMAAOazToeHh0tLS/7LZv/0jvb29t/f3//Ub//ge8WSLf/
    rhf/3kdbW1mxsbP//mf///yH5BAAAAAAALAAAAAAQAA4AAARe8L1Ekyky67QZ1hLnjM5UUde0ECwLJoExKcppV0aCcGCmTIHEIUEqjgaORCMxIC6e0C
    cguWw6aFjsVMkkIr7g77ZKPJjPZqIyd7sJAgVGoEGv2xsBxqNgYPj/gAwXEQA7"  >
    <hr />
    <table>
        <tr>
            <th>header0</th><th>header1</th><th>header2</th><th>header3</th><th>header4</th><th>header5</th>
        </tr>
        <tr>
            <td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td>
        </tr>
        <tr>
            <td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td>
        </tr>
        <tr>
            <td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td>
        </tr>
        <tr>
            <td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td><td>Hello World!!!</td>
        </tr>
    </table>
</body>
</html>

Please notice the <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> in the above HTML — also it is saved as Unicode (Encoding - Covert to UTF-8 in Notepad++). The result (testxhtml2pdf.pdf) should have:

A nice header (h1)
Paragraphs
Horizontal lines
No support for greek characters (same problem as with reportlab)
Images (I am inlining it as a base 64 image)
A list
A table

Before moving on, I’d like to fix the problem with the greek characters. You should set the font to one supporting greek characters, just like you did with ReportLab before. This can be done with the help of the @font-face css directive. So, let’s create a file named testxhtml2pdf2.html with the following contents:

<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

    <style>
        @font-face {
            font-family: DejaVuSans;
            src: url("c:/progr/py/django-pdf-guide/django_pdf_guide/DejaVuSans.ttf");
        }

        body {
            font-family: DejaVuSans;
        }
    </style>
</head>
<body>
    <h1>Δοκιμή του xhtml2pdf </h1>
    <ul>
        <li>Καλημέρα Ελλάδα!</li>
    </ul>

</body>
</html>

Before running xhtml2pdf testxhtml2pdf2.html, please make sure to change the url of the font file above to the absolute path of that font in your local system . As a result, after running xhhtml2pdf you should see the unicode characters without problems.

I have to mention here that I wasn’t able to use the font from a relative path, that’s why I used the absolute one. In case something is not right, try running it with the -d option to output debugging information (something like xhtml2pdf -d testxhtml2pdf2.html). You must see a line like this one:

DEBUG [xhtml2pdf] C:\progr\py\django-pdf-guide\venv\lib\site-packages\xhtml2pdf\context.py line 857: Load font 'c:\\progr\\py\\django-pdf-guide\\django_pdf_guide\\DejaVuSans.ttf'

to make sure that the font is actually loaded!

PyPDF2

The PyPDF2 library can be used to extract pages from a PDF to a new one or combine pages from different PDFs to a a new one. A common requirement is to have the first and page of a report as static PDFs, create the contents of this report through your app as a PDF and combine all three PDFs (front page, content and back page) to the resulting PDF.

Let’s see a quick example of combining two PDFs:

import sys
from PyPDF2 import PdfFileMerger

if __name__ == '__main__':
    pdfs = sys.argv[1:]

    if not pdfs or len(pdfs) < 2:
        exit("Please enter at least two pdfs for merging!")

    merger = PdfFileMerger()

    for pdf in pdfs:
        merger.append(fileobj=open(pdf, "rb"))

    output = open("output.pdf", "wb")
    merger.write(output)

The above will try to open all input parameters (as files) and append them to a the output.pdf.

Django integration

To integrate the PDF creation process with django we’ll use a simple app with only one model about books. We are going to use the django-xhtml2pdf library — I recommend installing the latest version (from github using something like pip install -e git+https://github.com/chrisglass/django-xhtml2pdf.git#egg=django-xhtml2pdf ) since the pip package has not been updated in a long time!

Using a plain old view

The simplest case is to just create plain old view to display the PDF. We’ll use django-xhtml2pdf along with the followig django template:

<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
</head>
<body>
    <h1>Books</h1>
    <table>
        <tr>
            <th>ID</th><th>Title</th>
        </tr>
        {% for book in books %}
            <tr>
                <td>{{ book.id }}</td><td>{{ book.title }}</td>
            </tr>
        {% endfor %}
    </table>
</body>
</html>

Name it as books_plain_old_view.html and put it on books/templates directory. The view that returns the above template as PDF is the following:

from django.http import HttpResponse
from django_xhtml2pdf.utils import generate_pdf


def books_plain_old_view(request):
    resp = HttpResponse(content_type='application/pdf')
    context = {
        'books': Book.objects.all()
    }
    result = generate_pdf('books_plain_old_view.html', file_object=resp, context=context)
    return result

We just use the generate_pdf method of django-xhtml2pdf to help us generate the PDF, passing it our response object and a context dictionary (containing all books).

Instead of the simple HTTP response above, we could add a ‘Content Disposition’ HTTP header to our response (or use the django-xhtml2pdf method render_to_pdf_response instead of generate_pdf) to suggest a default filename for the file to be saved by adding the line

resp['Content-Disposition'] = 'attachment; filename="output.pdf"'

after the definition of resp.

This will have the extra effect, at least in Chrome and Firefox to show the "Save File" dialog when clicking on the link instead of retrieving the PDF and displaying it inside* the browser window.

Using a CBV

I don’t really recommend using plain old Django views - instead I propose to always use Class Based Views for their DRYness. The best approach is to create a mixin that would allow any kind of CBV (at least any kind of CBV that uses a template) to be rendered in PDF. Here’s how we could implement a PdfResponseMixin:

class PdfResponseMixin(object, ):
    def render_to_response(self, context, **response_kwargs):
        context=self.get_context_data()
        template=self.get_template_names()[0]
        resp = HttpResponse(content_type='application/pdf')
        result = generate_pdf(template, file_object=resp, context=context)
        return result

Now, we could use this mixin to create PDF outputting views from any other view! For example, here’s how we could create a book list in pdf:

class BookPdfListView(PdfResponseMixin, ListView):
    context_object_name = 'books'
    model = Book

To display it, you could use the same template as books_plain_old_view.html (so either add a template_name='books_plain_old_view.html' property to the class or copy books_plain_old_view.html to books/book_list.html).

Also, as another example, here’s a BookPdfDetailView that outputs PDF:

class BookPdfDetailView(PdfResponseMixin, DetailView):
    context_object_name = 'book'
    model = Book

and a corresponding template (name it books/book_detail.html):

<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
</head>
<body>
    <h1>Book Detail</h1>
    <b>ID</b>: {{ book.id }} <br />
    <b>Title</b>: {{ book.title }} <br />
</body>
</html>

To add the content-disposition header and a name for your PDF, you can use the following mixin:

class PdfResponseMixin(object, ):
    pdf_name = "output"

    def get_pdf_name(self):
        return self.pdf_name

    def render_to_response(self, context, **response_kwargs):
        context=self.get_context_data()
        template=self.get_template_names()[0]
        resp = HttpResponse(content_type='application/pdf')
        resp['Content-Disposition'] = 'attachment; filename="{0}.pdf"'.format(self.get_pdf_name())
        result = generate_pdf(template, file_object=resp, context=context)
        return result

You see that, in order to havea configurable output name for our PDF and be consistent with the other django CBVs, a pdf_name class attribute and a get_pdf_name instance method are added. When using the above mixin in your classes you can either provide a value for pdf_name (to use the same for all your instances), or override get_pdf_name to have a dynamic value!

How does django-xhtml2pdf loads resources

Before doing more advanced things, we need to understand how django-xhtml2pdf works and specifically how we can refer to things like css, images, fonts etc from our PDF templates. If you check the utils.py of django-xhtml2pdf you’ll see that it uses a function named fetch_resources for loading these resources. This function checks to see if the resource starts with /MEDIA_URL or /STATIC_URL and converts it to a local (filesystem) path. For example, if you refer to a font like /static/font1.ttf in your PDF template, xhtml2pdf will try to load the file STATIC_ROOT + /font1.ttf (and if it does not find the file you want to refer to there it will check all STATICFILES_DIRS entries).

Thus, you can just put your resources into your STATIC_ROOT directory and use the {% static %} template tag to create URL paths for them — django-xhtml2pdf will convert these to local paths and everything will work fine.

Please notice that you *need* to have configured “STATIC_ROOT“ for this to work — if STATIC_ROOT is empty (and, for example you use static directories in your apps) then the described substitution mechanism will not work. Also, notice that the /static directory inside your apps cannot be used for fetch resources like this (due to how fetch_resources is implemented it only checks if the static resource is contained inside the STATIC_ROOT or in one of the the STATICFILES_DIRS) so be careful to either put the static files you need to load from PDFs (fonts, styles and possibly images) to either the STATIC_ROOT or the one of the STATICFILES_DIRS.

Using a common style for your PDFs

If you need to create a lot of similar PDFs then you’ll probably want to use a bunch of common styles for them (same fonts, headers etc). This could be done using the {% static %} trick we saw on the previous section. However, if we include the styling css as a static file then we won’t be able to use the static-file-uri-to-local-path mechanism described above (since the {% static %} template tag won’t work in static files).

Thankfully, not everything is lost — Django comes to the rescue!!! We can create a single CSS file that would be used by all our PDF templates and include it in the templates using the {% include %} Django template tag! Django will think that this will be a normal template and paste its contents where we wanted and also execute the templates tags!

We’ll see an example of all this in the next section.

Changing the font (to a Unicode enabled one)

The time has finally arrived to change the font! It’s easy if you know exactly what to do. First of all configure your STATIC_ROOT and STATIC_URL setting, for example STATIC_ROOT = os.path.join(BASE_DIR,'static') and STATIC_URL = '/static/'.

Then, add a template-css file for your fonts in one of your templates directories. I am naming the file pdfstylefonts.css and I’ve put it to books/templates:

{% load static %}
@font-face {
    font-family: "Calibri";
    src: url({% static "fonts/calibri.ttf" %});
}
@font-face {
    font-family: "Calibri";
    src: url({% static "fonts/calibrib.ttf" %});
    font-weight: bold;
}
@font-face {
    font-family: "Calibri";
    src: url({% static "fonts/calibrii.ttf" %});
    font-style: italic, oblique;
}
@font-face {
    font-family: "Calibri";
    src: url({% static "fonts/calibriz.ttf" %});
    font-weight: bold;
    font-style: italic, oblique;
}

I am using Calibri family of fonts (copied from c:\windows\fonts) for this — I’ve also configured all styles (bold, italic, bold-italic) of this font family to use the correct ttf files. All the ttf files have been copied to the directory static/fonts/.

Now, add another css file that will be your global PDF styles. This should be put to the static directory and could be named pdfstyle.css:

h1 {
    color: blue;
}

*, html {
    font-family: "Calibri";
    font-size:11pt;
    color: red;
}

Next, here’s a template that lists all books (and contain some greek characters — the title of the books also contain greek characters) — I’ve named it book_list_ex.html:

{% load static %}
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <style>
        {% include "pdfstylefonts.css" %}
    </style>
    <link rel='stylesheet' href='{% static "pdfstyle.css" %}'/>
</head>
<body>
    <h1>Λίστα βιβλίων</h1>
    <img src='{% static "pony.png" %}' />
    <table>
        <tr>
            <th>ID</th><th>Title</th><th>Cover</th>
        </tr>
        {% for book in books %}
            <tr>
                <td>{{ book.id }}</td><td>{{ book.title }}</td><td><img src='{{ book.cover.url }}' /></td>
            </tr>
        {% endfor %}
    </table>
</body>
</html>

You’ll see that the pdfstylefonts.css is included as a Django template (so that {% static %} will work in that file) while pdfstyle.css is included using {% static %}. Als, notice that I’ve also added a static image (using the {% static %} tag) and a dynamic (media) file to show off how great the url-to-local-path mechanism works. Please notice that for the media files to work fine in your development environment you need to configure the MEDIA_URL and MEDIA_ROOT settigns (similar to STATIC_URL and STATIC_ROOT) and follow the serve files uploaded by a user during development tutorial on Django docs.

Finally, if you configure a PdfResponseMixin ListView like this:

class BookExPdfListView(PdfResponseMixin, ListView):
    context_object_name = 'books'
    model = Book
    template_name = 'books/book_list_ex.html'

you should see be able to see the correct (calibri) font (defined in pdfstylefonts.css), with unicode characters without problems including both the static and user uploaded images and with the styles defined in the pdf stylesheet (pdfstyle.css).

Configure Django for debugging PDF creation

If you experience any problems, you can configure xhtml2pdf to output DEBUG information. To do this, you may change your django logging configuration like this:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'xhtml2pdf': {
            'handlers': ['console'],
            'level': 'DEBUG',
        }
    }
}

This configuration will keep existing loggers ('disable_existing_loggers': False,) and will configure xhtml2pdf to log its output to the console, helping us find out why some things won’t be working.

Concatenating PDFs in Django

The final section of the PDF-Django-integration is to explain how we can concatenate PDFs in django using PyPDF2. There may be some other requirements like extracting pages from PDFs however the most common one as explained before is to just append the pages of one PDF after the other — after all using PyPDF2 is really easy after you get the hang of it.

To be more DRY, I will create a CoverPdfResponseMixin that will output a PDF with a cover. To be even more DRY, I will refactor PdfResponseMixin to put some common code in an extra method so that CoverPdfResponseMixin could inherit from it:

class PdfResponseMixin(object, ):
    def write_pdf(self, file_object, ):
        context = self.get_context_data()
        template = self.get_template_names()[0]
        generate_pdf(template, file_object=file_object, context=context)

    def render_to_response(self, context, **response_kwargs):
        resp = HttpResponse(content_type='application/pdf')
        self.write_pdf(resp)
        return resp


class CoverPdfResponseMixin(PdfResponseMixin, ):
    cover_pdf = None

    def render_to_response(self, context, **response_kwargs):
        merger = PdfFileMerger()
        merger.append(open(self.cover_pdf, "rb"))

        pdf_fo = StringIO.StringIO()
        self.write_pdf(pdf_fo)
        merger.append(pdf_fo)

        resp = HttpResponse(content_type='application/pdf')
        merger.write(resp)
        return resp

So, PdfResponseMixin now has a write_pdf method that gets a file-like object and outputs the PDF there. The new mixin, CoverPdfResponseMixin has a cover_pdf attribute that should be configured with the filesystem path of the cover file. The render_to_response method now will create a PdfFileMerger (which is empty initially) to which it appends the contents cover_pdf. After that, it creates a file-stream (using StreamIO) and uses write_pdf to create the PDF there and appends that file-stream to the merger. Finally, it writes the merger contents to the HttpResponse.

One thing that I’ve seen is that if you want to concatenate many PDFs with many pages sometimes you’ll get a strange an error when using PdfFileMerger. I was able to overcome this by reading and appending the pages of each PDF to-be-appended one by one using the PdfFileReader and PdfFileWriter objects. Here’s a small snippet of how this could be done:

pdfs = [] # List of pdfs to be concatenated
writer = PdfFileWriter()

for pdf in pdfs:
    reader = PdfFileReader(open(pdf, "rb"))
    for i in range(reader.getNumPages()):
        writer.addPage(reader.getPage(i))

resp = HttpResponse(content_type='application/pdf')
writer.write(resp)
return resp

More advanced xhtml2pdf features

In this section I will present some information on how to use various xhtml2pdf features to create the most common required printed document features, for example adding page numbers, adding headers and footers etc.

Laying out

To find out how you can create your pages you should read the defining page layouts section of the xhtml2pdf manual. There you’ll see that the basic components of laying out in xhtml2pdf is @page to create pages and @frames to create rectangular components inside these pages. So each page will have a number of frames inside it. These frames are seperated to static and dynamic. Staticsome should be used for things like headers and footers (so they’ll be the same across all pages) and dynamic will contain the main content of the report.

For pages you can use any page size you want, not just the specified ones. For example, in one of my projects I wanted to create a PDF print on a normal plastic card, so I’d used the following size: 8.56cm 5.398cm;. Also, page templates can be named and you can use different ones in the same PDF (so you could create a cover page with a different page template, use it first and then continue with the normal pages). To name a tempalte you just use @page template_name {} and to change the template use the combination of the following two xhtml2pdf tags:

<pdf:nexttemplate name="back_page" />
<pdf:nextpage />

Now, one thing I’ve noticed is that you are not able to use a named template for the first page of your PDF. So, what I’ve done is that I create an anonymous (default) page for the first page of the report. If I want to reuse it in another page, I copy it and name it accordingly. I will give an example shortly.

Now, for frames, I recommend using the -pdf-frame-border: 1; command for debugging where they are actually printed. Also, I recommend using a normal ruler and measuring completely where you want them to be. For example, for the following frame:

@frame photo_frame {
        -pdf-frame-border: 1;
        top:  2.4cm;
        left: 6.2cm;
        width: 1.9cm;
        height: 2.2cm;
}

I’d used a ruler to find out that I want it to start 2.4 cm from the top of my page (actually a credit card) and 6.2 cm from the left and have a width and height of 1.9 and 2.2 cm.

I recommend naming all frames to be able to distinguish them however I don’t think that their name plays any other role. However, for static frames you must define the id of the content they will contain using -pdf-frame-content in the css and a div with the corresponding id in the PDF template. For example, you could define a frame header like this

@frame header {
  -pdf-frame-content: headerContent;
  width: 8in;
  top: 0.5cm;
  margin-left: 0.5cm;
  margin-right: 0.5cm;
  height: 2cm;
}

and its content like this:

<div id='headerContent'>
  <h1 >
    Header !
  </h1>
</div>

Please notice that if for some reason the stuff you want to put in your static frames does not fit there the frame will be totally empty. This means that if you have size for three lines but you want to output five lines in a static frame then you’ll see no lines!

Now, for dynamic content you can expect the opposite behavior: You cannot select to which dynamic frame your content goes, instead the content just just flows to the first dynamic frame it fits! If it does not fit in any dynamic frames in the current page then a new page will be created.

I’d like to present a full example here on the already mentioned project of printing a plastic card for the books. I had two page layouts, one for the front page having a frame with the owner’s data and another frame with his photo and one for the back page having a barcode. This is a Django template that is used to print not only but a PDF with a group of these cards:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style>
    {% include "pdfstylefonts.css" %}
</style>
<style type='text/css'>
    @page {
        size: 8.56cm 5.398cm;
        margin: 0.0cm 0.0cm 0.0cm 0.0cm;
        padding: 0;

        @frame table_frame {
            /* -pdf-frame-border: 1; */
            top:  2.4cm;
            left: 0.4cm;
            width: 5.5cm;
            height: 3cm;
        }

        @frame photo_frame {
            /* -pdf-frame-border: 1; */
            top:  2.4cm;
            left: 6.2cm;
            width: 1.9cm;
            height: 2.2cm;
        }
    }

    @page front_page {
        size: 8.56cm 5.398cm;
        margin: 0.0cm 0.0cm 0.0cm 0.0cm;
        padding: 0;

        @frame table_frame {
            /* -pdf-frame-border: 1; */
            top:  2.4cm;
            left: 0.4cm;
            width: 5.5cm;
            height: 3cm;
        }

        @frame photo_frame {
            /* -pdf-frame-border: 1; */
            top:  2.4cm;
            left: 6.2cm;
            width: 1.9cm;
            height: 2.2cm;
        }
    }

    @page back_page {
        size: 8.56cm 5.398cm;
        margin: 0.0cm 0.0cm 0.0cm 0.0cm;
        padding: 0;

        @frame barcode_frame {
            /* -pdf-frame-border: 1; */
            top:  3.9cm;
            left: 1.8cm;
            width: 4.9cm;
            height: 1.1cm;
        }
    }

    *, html {
        font-family: "Calibri";
        font-size: 8pt;
        line-height: 80%;
    }

    .bigger {
        font-size:9pt;
        font-weight: bold;
    }
</style>
</head>
<body>
    {% for book in books %}
        <div>
            <! -- Here I print the card data. It fits exactly in the table_frame so ... ->
            {{ book.title }}<br />
            Data line 2<br />
            Data line 3<br />
            Data line 4<br />
            Data line 5<br />
            Data line 6<br />
            Data line 7<br />
            Data line 8<br />
            Data line 9<br />
            Data line 10<br />
        </div>
        <div>
            <! -- This photo here will be outputted to the photo_frame -->
            <img src='{{ book.cover.url }}' style='width:1.9cm ; height: 2.2cm ; ' />
        </div>

        <! -- Now the template is chaned to print the back of the card -->
        <pdf:nexttemplate name="back_page" />
        <pdf:nextpage />

        <div >
            <center>
                <! -- Print the barcode to the barcode_frame -->
                <pdf:barcode value="{{ book.id }}" type="code128" humanreadable="1" barwidth="0.43mm" barheight="0.6cm" align="middle" />
            </center>
        </div>
        <!-- Use the front_page template again for the next card -->
        <pdf:nexttemplate name="front_page" />
        <pdf:nextpage />
    {% endfor %}
</body>
</html>

Notice that I want to print exactly 10 lines in the table_frame (that’s why I use <br /> to go to next line) — if I had printed 3-4 lines (for example) then the photo would fit in the table_frame and would be printed there and not in the photo_frame! Also, another interesting thing is that if I had outputed 8-9 lines then the photo wouldn’t fit in that small space and would also be printed to the photo_frame.

Extra stuff

Some extra things I want to mention concerning xhtml2pdf:

<pdf:pagenumber> to output the current page number (please end the line after this tag)
<pdf:pagecount> to output the total page number (please end the line after this tag)

So if you want to print a footer with the pages, I recommend something like this:

<div id='footerContent'>
  Page <pdf:pagenumber>
  from <pdf:pagecount>
  total
</div>

Notice how the lines end after the tags.

<pdf:barcode>: Output a barcode in your PDF. Here are the possible barcode types: I2of5, Standard39, Extended39, Standard93, Extended93, MSI, Codabar, Code11, FIM, POSTNET, USPS_4State, Code128, EAN13, EAN8, QR.
You may add a page background image using the background-image property of @page.

For example, if you want the templates you create from your development/UAT systems to be different than the production ones you could do something like this:

@page {
    {% if DEBUG %}
        background-image: url({% static "debug.png" %});
    {% endif %}
    <!-- Other page properties -->
}

Conclusion

I hope that using the techniques described in this essential guide you’ll be able to create great looking PDF documents from your Django application and overcome any difficulties that may arise. If you feel that there’s something I’ve not covered properly (and is not covered by the documentation) please comment out and I’ll be happy to research it a bit and update the article with info.

I am using all the above in various production applications for a long time and everything is smooth so don’t afraid to create PDFs from Django!

Improve your client-side-javascript workflow more by using ES6

Serafeim Papastefanos — Mon, 16 Nov 2015 14:20:00 +0200

Contents

Introduction
NPM, —save, —save-dev and avoiding global deps
Using babel
Integrate babel with browserify
The object-spread operator
Conclusion

Update 22/12/2015: Add section for the object-spread operator.

Introduction

In a previous article we presented a simple workflow to improve your client-side (javascript) workflow by including a bunch of tools from the node-js world i.e browserify and its friends, watchify and uglify-js.

Using these tools we were able to properly manage our client side dependencies (no more script tags in our html templates) and modularize our code (so we could avoid monolithic javascript files that contained all our code). Another great improvement to our workflow would be to include ES6 to the mix!

ES6 (or EcmaScript 2015) is more or less "Javascript: TNG". It has many features that greatly improve the readability and writability of javascript code - for instance, thick arrows, better classes, better modules, iterators/generators, template strings, default function parameters and many others! More info on these features and how they could be used can be found on the es6features repository.

Unfortunately, these features are either not supported at all, or they are partially supported on current (November 2015) browsers. However all is not lost yet: Since we already use browserify to our client side workflow, we can easily enable it to read source files with ES6 syntax and transform them to normal javascript using a transformation through the babel tool.

In the following, we’ll talk a bit more about the node-js package manager (npm) features, talk a little about babel and finally modify our workflow so that it will be able to use ES6! Please notice that to properly follow this article you need to first read the previous one.

NPM, —save, —save-dev and avoiding global deps

In the previous article, I had recommended to install the needed tools (needed to create the output bundle browserify, watchify, uglify) globally using npm install -g package (of course the normal dependencies like moment.js would be installed locally). This has one advantage and two disadvantages: It puts these tools to the path so they can be called immediately (i.e browserify) but you will need root access to install a package globally and nothing is saved on package.json so you don’t know which packages must be installed in order to start developing the project!

This could be ok for the introductionary article, however for this one I will propose another alternative: Install all the tools locally using just npm install package --save. These tools will be put to the node_modules folder. They will not be put to the path, but if you want to execute a binary by yourself to debug the output, you can find it in node_modules/bin, for example, for browserify you can run node_modules/bin/browserify. Another intersting thing is that if you create executable scripts in your package.json you don’t actually need to include the full path but the binaries will be found.

Another thing I’d like to discuss here is the difference between --save and --save-dev options that can be passed to npm install. If you take a look at other guides you’ll see that people are using --save-dev for development dependencies (i.e testing) and --save for normal dependencies. The difference is that these dependencies are saved in different places in package.json and if you run npm install --production you’ll get only the normal dependencies (while, if you run npm install all dependencies will be installed). In these articles, I chose to just use --save everywhere, after all the only thing that would be needed for production would be the bundle.js output file.

Using babel

The babel library "is a JavaScript compiler". It gets input files in a variant of javascript (for example, ES6) and produces normal javascript files — something like what browserify transforms do. However, what babel does (and I think its the only tool that does this) is that it allows you to use ES6 features now by transpiling them to normal (ES5) javascript. Also, babel has various other transforms, including a react transform (so you can use this instead of the reactify browserify-transform)!

In any case, to be able to use ES6, we’ll need to install babel and its es6 presets (don’t forget that you need to have a package.json for the dependencies to be saved so either do an npm init or create a package.json file containing only {}):

npm install  babel babel-preset-es2015 --save

If we wanted to also use babel for react we’d need to install babel-preset-react.

To configure babel we can either add a babel section in our package.json or create a new file named .babelrc and put the configuration there.

I prefer the first one since we are already using the package.json. So add the following attribute to your package.json:

"babel": {
  "presets": [
    "es2015"
  ]
}

If you wanted to configure it through .babelrc then you’d just copy to it the contents of "babel".

To do some tests with babel, you can install its cli (it’s not included in the babel package) through npm install babel-cli. Now, you can run node_modules/.bin/babel. For example, create a file named testbabel.js with the following contents (thick arrow):

[1,2,3].forEach(x => console.log(x) );

when you pass it to babel you’ll see the following output:

>node_modules\.bin\babel testbabel.js
"use strict";

[1, 2, 3].forEach(function (x) {
  return console.log(x);
});

Integrate babel with browserify

To call babel from browserify we’re going to use the babelify browserify transform which actually uses babel to transpile the browserify input. After installing it with

npm install babelify --save

you need to tell browserify to use it. To do this, you’ll just pass a -t babelify parameter to browserify. So if you run it with the testbabel.js file as input you’ll see the following output:

>node_modules\.bin\browserify -t babelify testbabel.js
[...] browserify gibberish
"use strict";

[1, 2, 3].forEach(function (x) {
  return console.log(x);
});

[...] more browserify gibberish

yey — the code is transpiled to ES5!

To create a complete project, let’s add a normal requirement (moment.js):

npm install moment --save

and a file named src\main.js that uses it with ES6 syntax:

import moment from 'moment';

const arr = [1,2,3,4,5];
arr.forEach(x => setTimeout(() => console.log(`Now: ${moment().format("HH:mm:ss")}, Later: ${moment().add(x, "days").format("L")}...`), x*1000));

To create the output javascript file, we’ll use the browserify and watchify commands with the addition of the -t babelify switch. Here’s the complete package.json for this project:

{
  "dependencies": {
    "babel": "^6.1.18",
    "babel-preset-es2015": "^6.1.18",
    "babelify": "^7.2.0",
    "browserify": "^12.0.1",
    "moment": "^2.10.6",
    "uglify-js": "^2.6.0",
    "watchify": "^3.6.1"
  },
  "scripts": {
    "watch": "watchify src/main.js -o dist/bundle.js -v -t babelify",
    "build": "browserify src/main.js -t babelify | uglifyjs -mc warnings=false > dist/bundle.js"
  },
  "babel": {
    "presets": [
      "es2015"
    ]
  }
}

Running npm run build should create a dist/bundle.js file. If you include this in an html, you should see something like this in the console:

Now: 13:52:09, Later: 11/17/2015...
Now: 13:52:10, Later: 11/18/2015...

The object-spread operator

Many examples in the internet use the object spread operator which is not part of es6 so our proposed babel configuration does not support it! To be able to use this syntax, we’ll need to install the corresponding babel plugin by using npm install babel-plugin-transform-object-rest-spread --save and add it to our babel configuration in the plugins section, something like this:

"presets": [
  "es2015",
  "react"
],
"plugins": [
  "transform-object-rest-spread"
]

If everything is ok this should be transpiled without errors using node_modules\.bin\browserify testbabe.js -t babelify

let x = {a:1 , b:2 };
let y = {...x, c: 3};

Conclusion

Using the combination of babel and javascript we can easily write ES6 code in our modules! This, along with the modularization of our code and the management of client-side dependencies should make client side development a breeze!

Please notice that to keep the presented workflow simple and easy to replicate and configure, we have not used any external task runners (like gulp or grunt) — all configuration is kept in a single file (package.json) and the whole environment can be replicated just by doing a npm install. Of course, the capabilities of browserify are not unlimited, so if you wanted to do something more complicated (for instance, lint your code before passing it to browserify) you’d need to use the mentioned task runners (or webpack which is the current trend in javascript bundlers and actually replaces the task runners).

Django dynamic tables and filters for similar models

Serafeim Papastefanos — Mon, 05 Oct 2015 14:20:00 +0300

Contents

Introduction
The problem we’ll solve
Adding the dynamic table
Creating a dynamic form for filtering
Creating the dynamic CBV
Conclusion

Introduction

One of my favorite django apps is django-tables2: It allows you to easily create pagination and sorting enabled HTML tables to represent your model data using the usual djangonic technique (similar to how you create ModelForms). I use it to almost all my projects to represent the data, along with django-filter to create forms to filter my model data. I’ve written a nice SO answer with instructions on how to use django-filter along with django-tables2.

The problem we’ll solve

The main tool django-tables2 offers is a template tag called render_table that get an instance of a subclass of django_tables2.Table which contains a description of the table (columns, style etc) along with a queryset with the table’s data and output it to the page. A nice extra feature of render_table is that you could pass it just a simple django queryset and it will output it without the need to create the custom Table class. So you can do something like {% render_table User.objects.all() %} and get a quick output of your Users table.

In one of my projects I had three models that were used for keeping a different type of business log in the database (for auditing reasons) and was using the above feature to display these logs to the administrators, just by creating a very simple ListView (a different one for each Log type) and using the render_table to display the data. So I’d created three views like this

class AuditLogListView(ListView):
  model = AuditLog
  context_object_name = 'logs'

which all used a single template that contained a line {% render_table logs %} to display the table (the logs context variable contains the list of AuditLog s).

This was a nice DRY (but quick and dirty solution) that soon was not enough to fulfill the needs of the administrators since they neeeded to have default sorting, filtering, hide the primary key-id column etc. The obvious solution for that would be to just create three different Table subclasses that would more or less have the same options with only their model attributte different. I didn’t like this solution that much since it seemed non-DRY to me and I would instead prefer to create a generic Table (that wouldn’t have a model attributte) and would just output its data with the correct options — so instead of three Table classes I’d like to create just a single one with common options that would display its data (what render_table does).

Unfortunately, I couldn’t find a solution to this, since when I ommited the model attribute from the Table subclass nothing was outputed (there was no way to define fields to display without also defining the model). An obvious (and DRY) resolution would be to create a base Table subclass that would define the needed options and create three subclasses that would inherit from this class and override only the model attribute. This unfortunately was not possible becase inheritance does not work well with django-tables2!

Furthermore, there’s the extra hurdle of adding filtering to the above tables so that the admin’s would be able to quickly find a log message - if we wanted to use django-filter we’d need again to create three different subclasses (one for each log model type) of django_filter.FilterSet since django-filter requires you to define the model for which the filter will be created.

The best way to resolve such problems is to create your classes dynamically when you need ‘em. I’ve already described a way to dynamically create forms in django in a previous post. Below, I’ll describe a similar technique which can be used to create both dynamic tables and filters. Using this methodology, you’ll be able to add a new CBV that displays a table with pagination, order and filtering for your model by just inheriting from a base class!

Adding the dynamic table

To create the DRY CBV we’ll use the SingleTableView (django_tables2.SingleTableView) as a base and override its get_table_class method to dynamically create our table class using type. Here’s how it could be done using a mixin (notice that this mixin should be used in a SingleTableView to override its get_table_class method):

class AddTableMixin(object, ):
  table_pagination = {"per_page": 25}

  def get_table_class(self):
      def get_table_column(field):
          if isinstance(field, django.db.models.DateTimeField):
              return tables.DateColumn("d/m/Y H:i")
          else:
              return tables.Column()

      attrs = dict(
          (f.name, get_table_column(f)) for
          f in self.model._meta.fields if not f.name == 'id'
      )
      attrs['Meta'] = type('Meta', (), {'attrs':{"class":"table"}, "order_by": ("-created_on", ) } )
      klass = type('DTable', (tables.Table, ), attrs)

      return klass

Let’s try to explain the get_table_class method: First of all, we’ve defined a local get_table_column function that will return a django-tables2 column depending on the field of the model. For example, in our case we want to use a django_tables2.DateColumn with a specific format when a DateTimeField is encountered and for all other model fields just use the stock Column.

After that, we create a dictionary with all the attributes of the model. The self.model field will contain the model Class, so using its _meta.fields will return the defined fields. As we can see, we just use a generator expression to create a tuple with the name of the field and its type (using get_table_class) excluding the ‘id’ column. So, attrs will be a dictionary of field names and types.

The Meta class of this table is crated using type which creates a parentless class by defining all its attributes in a dictionary and set it as the Meta key in the previously defined attrs dict. Finally, we create the actual django_tables2.Table subclass by inheriting from it and passing the attrs dict. We’ll see an example of what get_table_class returns later.

Creating a dynamic form for filtering

Let’s create another mixin that could be used to create a dynamic django.Form subclass to the CBV:

class AddFormMixin(object, ):
  def define_form(self):
      def get_form_field(f):
          return forms.CharField(required=False)

      attrs = dict(
          (f, get_form_field(f)) for
          f in self.get_form_fields() )

      klass = type('DForm', (forms.Form, ), attrs)

      return klass

  def get_queryset(self):
      form_class = self.define_form()
      if self.request.GET:
          self.form = form_class(self.request.GET)
      else:
          self.form = form_class()

      qs = super(AddFormMixin, self).get_queryset()

      if self.form.data and self.form.is_valid():
          q_objects = Q()
          for f in self.get_form_fields():
              q_objects &= Q(**{f+'__icontains':self.form.cleaned_data.get(f, '')})

          qs = qs.filter(q_objects)

      return qs

  def get_context_data(self, **kwargs):
      ctx = super(AddFormMixin, self).get_context_data(**kwargs)
      ctx['form'] = self.form
      return ctx

The first method that will be called is the get_queryset method that will generate the dynamic form using define_form. This method has a get_form_fields local function (similar to get_table_fields) that can be used to override the types of the fields (or just fallback to a normal CharField) and then create the attrs dictionary and forms.Form subclass in a similar way as the Table subclass. Here, we don’t want to create a filter form from all fields of the model as we did on table, so instead we’ll use a get_form_fields method that returns the name of the fields that we want to use in the filtering form and needs to be defined in each CBV.

The get_queryset method is more interesting: First of all, since we are just filtering the queryset we’d need to submit the form with a GET (and not a POST). We see that if we have data to our request.GET dictionary we’ll instantiate the form using this data (or else we’ll just create an empty form). To do the actual filtering, we check if the form is valid and create a django.models.db.Q object that is used to combine (by AND) the conditions. Each of the individual Q objects that will be combined to create the complete one will be created using the line Q(**{f+'__icontains':self.form.cleaned_data.get(f, '')}) (where f will be the name of the field) which will create a dictionary of the form {'action__icontains': 'test text'} and then pass this as a keyword argument to the Q (using the ** mechanism) - this is the only way to pass dynamic kwargs to a function!

Finally, the queryset will be filtered using the combined Q object we just described.

Creating the dynamic CBV

Using the above mixins, we can easily create a dynamic CBV with a table and a filter form only by inheriting from the mixins and SingleTableView and defining the get_form_fields method:

class AuditLogListView(AddTableMixin, AddFormMixin, SingleTableView):
  model = AuditLog
  context_object_name = 'logs'

  def get_form_fields(self):
      return ('action','user__username', )

Let’s suppose that the AuditLog is defined as following:

class AuditLog(models.Model):
  created_on = models.DateTimeField( auto_now_add=True, editable=False,)
  user = models.ForeignKey(settings.AUTH_USER_MODEL, editable=False,)
  action = models.CharField(max_length=256, editable=False,)

Using the above AuditLogListView, a dynamic table and a dynamic form will be automaticall created whenever the view is visited. The Table class will be like this:

class DTable(tables.Table):
  created_on = tables.DateColumn("d/m/Y H:i")
  user = tables.Column()
  action = tables.Column()

  class Meta:
      attrs = {"class":"table"}
      order_by = ("-created_on", )

and the Form class like this:

class DForm(forms.Form):
  user__username = forms.CharField()
  action = forms.CharField()

An interesting thing to notice is that we can drill down to foreign keys (f.e. user__username) to create more interesting filters. Also we could add some more methods to be overriden by the implementing class beyond the get_form_fields. For example, instead of using self.model._meta.fields to generate the fields of the table, we could instead use a get_table_fields method that would be overriden in the implementing classes (and even drill down on foreign keys to display more data on the table using accessors).

Or, instead of just passing the the form field names we could also pass their type (instead of using get_form_fields) and their lookup method (instead of icontains) — similar to django-filter.

Please notice that instead of creating a django form instance for filtering, we could instead create a django-filter instance with a similar methodology. However, I preferred to just use a normal django form because it makes the whole process more clear and removes a level of abstraction (we just create a django.Form subclass while, if we used django-filter we’d need to create a django-filter subclass which would create a django.Form subclass)!

Conclusion

Using the above technique we can quickly create a table and filter for a number of Models that all share the same properties in the most DRY. This technique of course is useful only for quick CBVs that are more or less the same and require little customization. Another interesting thing is that instead of creating different SingleTableView s we could instead create a single CBV that will get the content type of the Model to be viewed as a parameter and retrieve the model (and queryset) from the content type!

A (little more) complex react and flux example

Serafeim Papastefanos — Tue, 08 Sep 2015 12:55:00 +0300

Contents

Introduction

In previous two parts of this series, we have seen how we could implement a not-so-simple single page application with full CRUD capabilities using react only (in the first part ) and then how to change it to use the flux architecture (in the second part).

In this, third, part, we will add some more capabilities to the previous app to prove how easy it is to create complex apps and continue the discussion on the Flux architecture. The source code of this project can be found at https://github.com/spapas/react-tutorial (tag name react-flux-complex).

Here’s a demo of the final application:

We can see that, when compared to the previous version this one has:

Sorting by table columns
Pagination
Message for loading
Client side url updating (with hashes)
Cascading drop downs (selecting category will filter out subcategories)
Integration with jquery-ui datepicker
Add a child object (author)
Add authors with a modal dialog (popup)
Delete authors (using the "-" button)
Colored ajax result messages (green when all is ok, red with error)
A statistics panel (number of books / authors)

In the paragraphs below, beyond the whole architecture, we’ll see a bunch of techniques such as:

Creating reusable flux components
Integrating react with jquery
Creating cascading dropdowns
Updating URLs with hashes
Adding pagination/sorting/querying to a table of results
Displaying pop ups

More about Flux components

Before delving into the source code of the above application, I’d like to make crystal what is the proposed way to call methods in the flux chain and which types of components can include other types of components. Let’s take a look at the following diagram:

The arrows display the dependencies of the Flux architecture - an arrow from X to Y means that a component of type Y could depend on an Component of type X (of example, a store could depend on an action or another store but not on a panel).

In more detail, we can see that in the top of the hierarchy, having no dependencies are the Dispatcher and the Constants. The Dispatcher is just a single component (which actually is a singleton - only one dispatcher exists in each single page react-flux application) that inherits from the Facebook’s dispatcher and is imported by the action components (since action methods defined in action components call the dispatch method of the dispatcher passing the correct parameters) and the stores which do the real actions when an action is dispatched, depending on the action type. The Constant components define constant values for the action types and are used by both the actions (to set the action types to the dispatch calls) and by the stores to be used on the switch statement when an action is dispatched. As an example, for BookActions we have the following

var BookConstants = require('../constants/BookConstants')
var AppDispatcher = require('../dispatcher/AppDispatcher').AppDispatcher;

var BookActions = {
    change_book: function(book) {
        AppDispatcher.dispatch({
            actionType: BookConstants.BOOK_CHANGE,
            book: book
        });
    },
// Other book actions [...]

and for BookStore

var $ = require('jquery');
var AppDispatcher = require('../dispatcher/AppDispatcher').AppDispatcher;
var BookConstants = require('../constants/BookConstants')

// [...]

BookStore.dispatchToken = AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case BookConstants.BOOK_EDIT:
            _editBook(action.book);
        break;
// [...] other switch branches

An interesting thing to see is that the actions depend only on the Dispatcher and on the Constants, while the stores also depend on actions (not only from the same action types as the store type, for example AuthorStore depends on both AuthorActions and MessageActions) and on other stores. This means that the actions should be a "clean" component: Just declare your action methods whose only purpose is to pass the action type along with the required parameters to the dispatcher.

On the other hand, the stores are depending on both actions and other stores. The action dependency is because sometimes when something is done in a store we need to notify another store to do something as a response to that. For example, in our case, when a book is updated we need to notify the message store to show the "Book updated ok" message. This preferrably should not be done by directly calling the corresponding method on the message store but instead by calling the corresponding action of MessageAction and passing the correct parameters (actually, the method that updates the message in MessageStore should be a private method that is called only through the action).

One thing to notice here is that you cannot call (and dispatch) an action in the same call stack (meaning it is directly called) as of another dispatch or you’ll get a Invariant Violation: Dispatch.dispatch(...): Cannot dispatch in the middle of a dispatch. error in your javascript console. Let’s see an example of what this means and how to avoid it because its a common error. Let’s say that we have the following code to a store:

AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case Constants.TEST_ERR:
            TestActions.action1();
        break;
        case Constants.TEST_OK1:
            setTimeout(function() {
                TestActions.action1();
            }, 0);

        break;
        case Constants.TEST_OK2:
            $.get('/get/', function() {
                TestActions.action1();
            });

        break;
    }
    return true;
});

Here, the TEST_ERR branch will throw an error because the TestAction.action1 is in the same call stack as this dispatch! On the other hand, both TEST_OK1 and TEST_OK2 will work: TEST_OK2 is the most usual, since most of the times we want to call an action as a result of an ajax call - however, sometimes we want to call an action without any ajax — in this case we use the setTimeout (with a timeout of 0 ms) in order to move the call to TestActions.action1() to a different call stack.

Now, as I mentioned before, there’s also a different-store dependency on stores. This dependency is needed because some stores may offer public methods for other stores to use (methods that don’t actually need to be dispatched through the dispatcher) and for the waitFor method , in case there are two different stores that respond to a single action and want to have one store executing its dispatch method before the other. (the waitFor method takes the dispatchToken, which is the result of Dispatcher.register of a different store as a parameter in order to wait for that action to finish).

Finally, the Panels depend on both actions (to initiate an action as a response to user input) and stores (to read the state of the each required store when it is changed). Of course, not all panels actually depend on stores, since as we already know the state of a React app should be kept as high in the hierarchy as possible, so a central component will get the state of a store and pass the required state attributes to its children through parameters. On the other hand, every component that responds to user input will have to import am action object and call the corresponding method — we should never pass callbacks from a parent to a child component as a parameter anymore unless we want to make the component reusable (more on this later).

Finally, as expected, because of their hierarchy the compoents depend on their child components (a BookTable depends on a BookTableRow etc).

Explaining the application components

There are a lot of components that are needed for the above application. A bird’s eye view of the components and their hierarchies can be seen on the following figure:

We will first explain a bit the components that could be easily reused by other applications and after that the components that are specifically implemented for our book/author single page application.

Reusable application components

We can see that beyond the specific components (BookPanel, BookTable etc) there’s a bunch of more general components (DropDown, DatePicker, PagingPanel, SearchPanel, MessagePanel) that have names like they could be used by other applications (for example every application that wants to implement a DropDown could use our component) and not only by the Book-Author application. Let’s take a quick look at these components and if they are actually reusable:

DropDown.react.js

var React = require('react');

var DropDown = React.createClass({
    render: function() {
        var options = [];
        options.push(<option key='-1' value='' >---</option>);
        if(this.props.options) {
            this.props.options.forEach(function(option) {
                options.push(<option key={option.id} value={option.id}>{option.name}</option>);
            });
        }

        return(
            <select ref='dropdown' value={this.props.value?this.props.value:''} onChange={this.onFormChange} >
                {options}
            </select>
        );
    },
    onFormChange: function() {
        var val = React.findDOMNode(this.refs.dropdown).value
        this.props.dropDownValueChanged(val);
    }
});

module.exports.DropDown = DropDown;

What we see here is that this component does not actually have a local state and will need three parameters:

the list of options (props.options)
the curently selected option (props.value)
a callback to be called when the dropdown is changed (props.dropDownValueChanged)

By passing the above three parameters this dropdown component can be reused whenever a dropdown is needed! As we can see here we need to pass a callback function to this dropdown and call the corresponding action directly. This is important to have a reusable component: The component that includes each dropdown decides which should be the action on the dropdown value change. Another option if we wanted to avoid callbacks would be to pass an identifier as a property for each dropdown and the dropdown would call a generic dropdown action and passing this identifier — however I think that actually passing the callback is easier and makes the code much more readable.

DatePicker.react.js

The datepicker component has more or less the same structure as with the dropdown: It needs one parameter for its current value (props.value) and another as the callback to the function when the date is changed (once again this is required to have a reusable component):

var React = require('react');

var DatePicker = React.createClass({
    render: function() {
        return(
            <input type='text' ref='date' value={this.props.value} onChange={this.handleChange} />
        );
    },
    componentDidMount: function() {
        $(React.findDOMNode(this)).datepicker({ dateFormat: 'yy-mm-dd' });
        $(React.findDOMNode(this)).on('change', this.handleChange);
    },
    componentWillUnmount: function() {

    },
    handleChange: function() {
        var date = React.findDOMNode(this.refs.date).value
        this.props.onChange(date);
    }
});

module.exports.DatePicker = DatePicker ;

Another thing we can see here is how can integrate jquery components with react: When the component is mounted, we get its DOM component using React.findDomNode(this) and convert it to a datepicker. We also set its change function to be the passed callback.

PagingPanel.react.js

The paging panel is not actually a reusable component because it requires BookActions (to handle next and previous page clicks) - so it can’t be used by Authors (if authors was a table that is). However, we can easily change it to be reusable if we passed callbacks for next and previous page so that the component including PagingPanel would call the correct action on each click. Having a reusable PagingPangel is not needed for our application since only books have a table.

var React = require('react');
var BookActions = require('../actions/BookActions').BookActions;

var PagingPanel = React.createClass({
    render: function() {
        return(
            <div className="row">
                {this.props.page==1?'':<button onClick={this.onPreviousPageClick}>&lt;</button>}
                &nbsp; Page {this.props.page} of {this.getTotalPages()} &nbsp;
                {this.props.page==this.getTotalPages()?'':<button onClick={this.onNextPageClick} >&gt;</button>}
            </div>
        );
    },
    onNextPageClick: function(e) {
        e.preventDefault();
        BookActions.change_page(this.props.page+1)
    },
    onPreviousPageClick: function(e) {
        e.preventDefault();
        BookActions.change_page(this.props.page-1)
    },
    getTotalPages: function() {
        return Math.ceil(this.props.total / this.props.page_size);
    }
})

module.exports.PagingPanel = PagingPanel;

The component is very simple, it needs three parameters:

the current page (props.page)
the page size (props.page_size)
the total pages number (props.total)

and just displayd the current page along with buttons to go to the next or previous page (if these buttons should be visible of course).

SearchPanel.react.js

The SearchPanel is another panel that could be reusable if we’d passed a callbeck instead of calling the BookActions.search action directly. The promise behavior has been explained in the previous posts and is needed to buffer the queries to the server when a user types his search query.

var React = require('react');
var BookActions = require('../actions/BookActions').BookActions;

var SearchPanel = React.createClass({
    getInitialState: function() {
        return {
            search: this.props.query,
        }
    },
    componentWillReceiveProps: function(nextProps) {
      this.setState({
            search: nextProps.query
      });
    },
    render: function() {
        return (
            <div className="row">
                <div className="one-fourth column">
                    Filter: &nbsp;
                    <input ref='search' name='search' type='text' value={this.state.search} onChange={this.onSearchChange} />
                    {this.state.search?<button onClick={this.onClearSearch} >x</button>:''}
                </div>
            </div>
        )
    },
    onSearchChange: function() {
        var query = React.findDOMNode(this.refs.search).value;
        if (this.promise) {
            clearInterval(this.promise)
        }
        this.setState({
            search: query
        });
        this.promise = setTimeout(function () {
            BookActions.search(query);
        }.bind(this), 400);
    },
    onClearSearch: function() {
        this.setState({
            search: ''
        });
        BookActions.search('');
    }
});

module.exports.SearchPanel = SearchPanel;

As we can see, this panel is a little different than the previous ones because it actually handles its own local state: When the component should get properties from its parent compoent, its state will be updated to the query attribute - so the current value of the search query gets updated through properties. However, when the value of the search input is changed, we see that the local state is changed immediately but the BookActions.search (or the corresponding callback) gets called only when the timeout has passed!

The above means that we can type whatever we want on the search input, but it at first it will be used only locally to immediately update the value of the input and, only after the timeout has fired the search action will be called. If we hadn’t used the local state it would be much more difficult to have this consistent behavior (we’d need to add two actions, one to handle the search query value change and another to handle the timeout firing — making everythimg much more complicated).

MessagePanel

The MessagePanel is really interesting because it is a reusable component that actually has its own action and store module! This component can be reused in different applications that need to display message but not on the same application (because a single state is kept for all messages). If we wanted to use a different MessagePanel for Books or Authors then we’d need to keep both in the state and also it to the action to differentiate between messages for author and for book. Instead, by keeping a single Messages state for both Books and Authors we have a much more simple version.

MessagePanel.react.js

The MessagePanel component has a local state which responds to changes on MessageStore. When the state of MessageStore is changed the MessagePanel will be re-rendered with the new message.

var React = require('react');
var MessageStore = require('../stores/MessageStore').MessageStore;

var MessagePanel = React.createClass({
    getInitialState: function() {
        return {

        };
    },
    render: function() {
        return(
            <div className="row">
                {this.state.message?<div className={this.state.message.color}>{this.state.message.text}</div>:""}
            </div>
        );
    },
    _onChange: function() {
        this.setState(MessageStore.getState());
    },
    componentWillUnmount: function() {
        MessageStore.removeChangeListener(this._onChange);
    },
    componentDidMount: function() {
        MessageStore.addChangeListener(this._onChange);
    }
})

module.exports.MessagePanel = MessagePanel;

MessageStore.js

The MessageStore has a (private) state containing a message that gets updated only when the ccorresponding action is dispached. The store has a single state for all messages - it doesn’t care if the messages are for books or authors.

var $ = require('jquery');
var EventEmitter = require('events').EventEmitter;
var AppDispatcher = require('../dispatcher/AppDispatcher').AppDispatcher;
var BookConstants = require('../constants/BookConstants')

var _state = {
    message: {}
};

var MessageStore = $.extend({}, EventEmitter.prototype, {
    getState: function() {
        return _state;
    },
    emitChange: function() {
        this.emit('change');
    },
    addChangeListener: function(callback) {
        this.on('change', callback);
    },
    removeChangeListener: function(callback) {
        this.removeListener('change', callback);
    }
});

MessageStore.dispatchToken = AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case BookConstants.MESSAGE_ADD:
            _state.message = action.message;
            MessageStore.emitChange();
        break;
    }
    return true;
});

module.exports.MessageStore = MessageStore;

MessageActions

Finally, there are two actions that are defined for the MessageStore: One for adding an ok message and one for adding an error message - both of which have the same message type (but pass a different color parameter).

var AppDispatcher = require('../dispatcher/AppDispatcher').AppDispatcher;
var BookConstants = require('../constants/BookConstants')

var MessageActions = {
    add_message_ok: function(msg) {
        AppDispatcher.dispatch({
            actionType: BookConstants.MESSAGE_ADD,
            message: {
                color: 'green',
                text: msg
            }
        });
    },
    add_message_error: function(msg) {
        AppDispatcher.dispatch({
            actionType: BookConstants.MESSAGE_ADD,
            message: {
                color: 'green',
                text: msg
            }
        });
    }
};

module.exports.MessageActions = MessageActions;

Non-reusable application components

I don’t want to discuss the source code for all the non-reusable components since some of them are more or less the same with the previous version and are easy to understand just by checking the source code (BookTableRow and ButtonPanel). However, I’ll discuss the other, more complex components starting from the inside of the react-onion:

BookTable.react.js

I want to display this component to discuss how sorting is implemented: Each column has a key which, when passed to django-rest-framework will sort the results based on that key (the __ does a join so by author__last_name we mean that we want to sort by the last_name field of the author of each book. Also, you can pass the key as it is to sort ascending or with a minus (-) in front (for example -author__last_name).

var React = require('react');
var BookTableRow = require('./BookTableRow.react').BookTableRow;
var BookActions = require('../actions/BookActions').BookActions;

var BookTable = React.createClass({
    render: function() {
        var rows = [];
        this.props.books.forEach(function(book) {
            rows.push(<BookTableRow key={book.id} book={book} />);
        });
        return (
            <table>
                <thead>
                    <tr>
                        <th><a href='#' onClick={this.onClick.bind(this, 'id')}>{this.showOrdering('id')} Id</a></th>
                        <th><a href='#' onClick={this.onClick.bind(this, 'title')}>{this.showOrdering('title')} Title</a></th>
                        <th><a href='#' onClick={this.onClick.bind(this, 'subcategory__name')}>{this.showOrdering('subcategory__name')} Category</a></th>
                        <th><a href='#' onClick={this.onClick.bind(this, 'publish_date')}>{this.showOrdering('publish_date')} Publish date</a></th>
                        <th><a href='#' onClick={this.onClick.bind(this, 'author__last_name')}>{this.showOrdering('author__last_name')} Author</a></th>
                        <th>Edit</th>
                    </tr>
                </thead>
                <tbody>{rows}</tbody>
            </table>
        );
    },
    onClick: function(v, e) {
        e.preventDefault();
        BookActions.sort_books(v);
    },
    showOrdering: function(v) {
        if (v==this.props.ordering) {
            return '+'
        } else if ('-'+v==this.props.ordering) {
            return '-'
        }
    }
});

module.exports.BookTable = BookTable ;

The only thing that needs explaining in this module is the line of the form

<th><a href='#' onClick={this.onClick.bind(this, 'id')}>{this.showOrdering('id')} Id</a></th>

that creates the title of each column and triggers ascending or descending sorting on this column by clicking on it. So, we can see that we’ve create an onClick function that actually expects a value - the key to that column. To allow passing that value, we use the bind method of the function object which will create new a function that has this key as its first parameter. If we didn’t want to use bind, we’d need to creatre 5 different function (onIdClick, onTitleClick etc)! The most common usage of bind is to actually bind a function to an object (that’s what the first parameter to this function does) so that calling this inside that function will refer to that object - here we leave the binding of the function to the same object and only do the parameter passing.

Also, the showOrdering checks if the current ordering is the same as that column’s key and displays either a + (for ascending) or - (for descending) in front of the column title.

AuthorDialog.react.js

This is a handmade pop-up dialog that gets displayed when a new author is added (the + button is clicked) using only css to center it on the screen when it is displayed. We can see that it is either visible on invisible based on the showDialog input property which actually is the only input this component requires. When it is visible and the ok or cancel button are pressed the corresponding action will be dispatched (which will actually close this popup by setting the showDialog to false):

var React = require('react');
var AuthorActions = require('../actions/AuthorActions').AuthorActions;

var AuthorDialog = React.createClass({

    render: function() {
        if (!this.props.showDialog) {
            return (
                <div />
            )
        } else {
            return(
                <div className='modal-dialog' id="dialog-form"  >
                    <label htmlFor="first_name">First name:</label> <input type='text' ref='first_name' name='first_name' /> <br />
                    <label htmlFor="last_name">Last name:</label> <input type='text' ref='last_name' name='last_name' /> <br />
                    <button onClick={this.onOk}>Ok</button>
                    <button onClick={this.onCancel} >Cancel</button>
                </div>

            );
        }
    },
    onCancel: function(e) {
        e.preventDefault();
        AuthorActions.hide_add_author();
    },
    onOk: function(e) {
        e.preventDefault();
        first_name = React.findDOMNode(this.refs.first_name).value;
        last_name = React.findDOMNode(this.refs.last_name).value;
        AuthorActions.add_author_ok({
            first_name: first_name,
            last_name: last_name
        });
    }
});

module.exports.AuthorDialog = AuthorDialog ;

AuthorPanel.react.js

The AuthorPanel displays the author select DropDown along with the + (add author) and - (delete author) buttons. It also contains the AuthorDialog which will be displayed or not depending on the value of the showDialog property.

var React = require('react');
var DropDown = require('./DropDown.react').DropDown;
var AuthorDialog = require('./AuthorDialog.react').AuthorDialog;
var AuthorActions = require('../actions/AuthorActions').AuthorActions;

var AuthorPanel = React.createClass({
    getInitialState: function() {
        return {};
    },
    render: function() {
        var authorExists = false ;
        if(this.props.authors) {
            var ids = this.props.authors.map(function(x) {
                return x.id*1;
            });

            if(ids.indexOf(1*this.props.author)>=0 ) {
                authorExists = true;
            }
        }

        return(
            <div className='one-half column'>
                <AuthorDialog showDialog={this.props.showDialog} />
                <label forHtml='date'>Author</label>
                <DropDown options={this.props.authors} dropDownValueChanged={this.props.onAuthorChanged} value={authorExists?this.props.author:''} />
                <button onClick={this.addAuthor} >+</button>
                {authorExists?<button onClick={this.deleteAuthor}>-</button>:""}
            </div>
        );
    },
    addAuthor: function(e) {
        e.preventDefault();
        console.log("ADD AUTHOR");
        AuthorActions.show_add_author();
    },
    deleteAuthor: function(e) {
        e.preventDefault();
        AuthorActions.delete_author(this.props.author);
        console.log("DELETE AUTHOR");
        console.log(this.props.author);
    },
});

module.exports.AuthorPanel = AuthorPanel;

As we can see, there are three properties that are passed to this component:

props.author: The currently selected author
props.authors: The list of all authors
props.onAuthorChanged: A callback that is called when the author is changed. Here, we could have used an action (just like for add/delete author) instead of a callback, however its not actually required. When the author is changed, it means that the currently edited book’s author is changed. So we could propagate the change to the parent (form) component that handles the book change along with the other changes (i.e title, publish date etc).

StatPanel.react.js

The StatPanel is an interesting, read-only component that displays the number of authors and books. This component requests updates from both the BookStore and AuthorStore - when their state is updated the component will be re-rendered with the number of books and authors:

var React = require('react');
var BookStore = require('../stores/BookStore').BookStore;
var AuthorStore = require('../stores/AuthorStore').AuthorStore;

var StatPanel = React.createClass({
    getInitialState: function() {
        return {};
    },
    render: function() {
        var book_len = '-';
        var author_len = '-';
        if(this.state.books) {
            book_len = this.state.books.length
        }
        if(this.state.authors) {
            author_len = this.state.authors.length
        }
        return(
            <div className="row">
                <div className="one-half column">
                    Books number: {book_len}
                </div>
                <div className="one-half column">
                    Authors number: {author_len}
                </div>
                <br />
            </div>
        );
    },
    _onBookChange: function() {
        this.setState({
            books:BookStore.getState().books
        });
    },
    _onAuthorChange: function() {
        this.setState({
            authors: AuthorStore.getState().authors
        });
    },
    componentWillUnmount: function() {
        AuthorStore.removeChangeListener(this._onAuthorChange);
        BookStore.removeChangeListener(this._onBookChange);
    },
    componentDidMount: function() {
        AuthorStore.addChangeListener(this._onAuthorChange);
        BookStore.addChangeListener(this._onBookChange);
    }
});

module.exports.StatPanel = StatPanel ;

We’ve added different change listeners in case we wanted to do some more computations for book or author change (instead of just getting their books / authors property). Of course the same behavior could be achieved with just a single change listener that would get both the books and authors.

BookForm.react.js

The BookForm is one of the most complex panels of this application (along with BookPanel) because it actually contains a bunch of other panels and has some callbacks for them to use.

We can see that, as explained before, when the current book form values are changed (through callbacks) the change_book action will be called.

var React = require('react');
var BookActions = require('../actions/BookActions').BookActions;
var DropDown = require('./DropDown.react.js').DropDown;
var StatPanel = require('./StatPanel.react.js').StatPanel;
var MessagePanel = require('./MessagePanel.react.js').MessagePanel;
var DatePicker = require('./DatePicker.react.js').DatePicker;
var ButtonPanel = require('./ButtonPanel.react.js').ButtonPanel;
var AuthorPanel = require('./AuthorPanel.react.js').AuthorPanel;
var CategoryStore = require('../stores/CategoryStore').CategoryStore;
var AuthorStore = require('../stores/AuthorStore').AuthorStore;
var loadCategories = require('../stores/CategoryStore').loadCategories;
var loadAuthors = require('../stores/AuthorStore').loadAuthors;

var BookForm = React.createClass({
    getInitialState: function() {
        return {};
    },
    render: function() {
        return(
            <form onSubmit={this.onSubmit}>
                <div className='row'>
                    <div className='one-half column'>
                        <label forHtml='title'>Title</label>
                        <input ref='title' name='title' type='text' value={this.props.book.title} onChange={this.onTitleChange} />
                    </div>
                    <div className='one-half column'>
                        <label forHtml='date'>Publish date</label>
                        <DatePicker ref='date' onChange={this.onDateChange} value={this.props.book.publish_date} />
                    </div>
                </div>
                <div className='row'>
                    <div className='one-half column'>
                        <label forHtml='category'>Category</label>
                        <DropDown options={this.state.categories} dropDownValueChanged={this.onCategoryChanged} value={this.props.book.category} />
                        <DropDown options={this.state.subcategories} dropDownValueChanged={this.onSubCategoryChanged} value={this.props.book.subcategory} />
                    </div>
                    <AuthorPanel authors={this.state.authors} author={this.props.book.author} onAuthorChanged={this.onAuthorChanged} showDialog={this.state.showDialog} />
                </div>

                <ButtonPanel book={this.props.book}  />
                <MessagePanel />
                <StatPanel  />
            </form>
        );
    },
    onSubmit: function(e) {
        e.preventDefault();
        BookActions.save(this.props.book)
    },
    onTitleChange: function() {
        this.props.book.title = React.findDOMNode(this.refs.title).value;
        BookActions.change_book(this.props.book);
    },
    onDateChange: function(date) {
        this.props.book.publish_date = date;
        BookActions.change_book(this.props.book);
    },
    onCategoryChanged: function(cat) {
        this.props.book.category = cat;
        this.props.book.subcategory = '';
        BookActions.change_book(this.props.book);
    },
    onSubCategoryChanged: function(cat) {
        this.props.book.subcategory = cat;
        BookActions.change_book(this.props.book);
    },
    onAuthorChanged: function(author) {
        this.props.book.author = author;
        BookActions.change_book(this.props.book);
    },
    _onChangeCategories: function() {
        this.setState(CategoryStore.getState());
    },
    _onChangeAuthors: function() {
        this.setState(AuthorStore.getState());
    },
    componentWillUnmount: function() {
        CategoryStore.removeChangeListener(this._onChangeCategories);
        AuthorStore.removeChangeListener(this._onChangeAuthors);
    },
    componentDidMount: function() {
        CategoryStore.addChangeListener(this._onChangeCategories);
        AuthorStore.addChangeListener(this._onChangeAuthors);
        loadCategories();
        loadAuthors();
    }
});

module.exports.BookForm = BookForm;

The above component listens for updates on both Category and Author store to update when the authors (when an author is added or deleted) and the categories are changed (for example to implement the cascading dropdown functionality), so the list of authors and the list of categories and subcategoreis are all stored in the local state. The book that is edited is just passed as a property - actually, this is the only property that this component needs to work.

BookPanel.react.js

Finally, BookPanel is the last component we’ll talk about. This is the central component of the application - however we’ll see that it is not very complex (since most user interaction is performed in other components). This component just listens on changes in the BookStore state and depending on the parameters either displays the "Loading" message or the table of books (depending on the state of ajax calls that load the books). The other parameters like the list of books, the ordering of the books etc are passed to the child components.

var React = require('react');
var BookStore = require('../stores/BookStore').BookStore;
var BookActions = require('../actions/BookActions').BookActions;
var SearchPanel = require('./SearchPanel.react').SearchPanel;
var BookTable = require('./BookTable.react').BookTable;
var PagingPanel = require('./PagingPanel.react').PagingPanel;
var BookForm = require('./BookForm.react').BookForm;

var reloadBooks = require('../stores/BookStore').reloadBooks;

var BookPanel = React.createClass({
    getInitialState: function() {
        return BookStore.getState();
    },
    render: function() {
        return(
            <div className="row">
                <div className="one-half column">
                    {
                        this.state.loading?
                        <div class='loading' >Loading...</div>:
                        <div>
                            <SearchPanel query={this.state.query} ></SearchPanel>
                            <BookTable books={this.state.books} ordering={this.state.ordering} />
                            <PagingPanel page_size='5' total={this.state.total} page={this.state.page} />
                        </div>
                    }
                </div>
                <div className="one-half column">
                    <BookForm
                        book={this.state.editingBook}
                    />
                </div>
                <br />
            </div>
        );
    },
    _onChange: function() {
        this.setState( BookStore.getState() );
    },
    componentWillUnmount: function() {
        BookStore.removeChangeListener(this._onChange);
    },
    componentDidMount: function() {
        BookStore.addChangeListener(this._onChange);
        reloadBooks();
    }
});

module.exports.BookPanel = BookPanel ;

The Actions

All actions are rather simple components without other dependencies as we’ve already discussed. They just define "actions" (which are simple functions) that create the correct parameter object type and pass it to the dispatcher. The only attribute that is required for this object is the actionType that should get a value from the constants. I won’t go into any more detail about the actions — please check the source code and all your questions will be resolved.

The Stores

First of all, all stores are defined through the following code that is already discussed in the previous part:

var MessageStore = $.extend({}, EventEmitter.prototype, {
    getState: function() {
        return _state;
    },
    emitChange: function() {
        this.emit('change');
    },
    addChangeListener: function(callback) {
        this.on('change', callback);
    },
    removeChangeListener: function(callback) {
        this.removeListener('change', callback);
    }
});

When the state of a store is changed its emitChange function will be called (I mean called manually from the code that actually changes the state and knows that it has actually been changed - nothing will be called automatically). When emitChange is called, all the components that listen for changes for this component (that have called addChangeListener of the store with a callback) will be notified (their callback will be called) and will use getState of the store to get its current state - after that, these components will set their own state to re-render and display the changes to the store.

Let’s now discuss the four stores defined — I will include only the parts of each file that are actually interesting, for everything else use the source Luke!

MessageStore.js

A very simple store that goes together with MessagePanel and MessageActions. It just keeps a state with the current message object and just changes this message when the MESSAGE_ADD message type is dispatched. After changing the message, the listeners (only one in this case) will be notified to update the displayed message:

var _state = {
    message: {}
};

MessageStore.dispatchToken = AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case BookConstants.MESSAGE_ADD:
            _state.message = action.message;
            MessageStore.emitChange();
        break;
    }
    return true;
});

AuthorStore.js

Here we see that the local state has an array of the authors and a showDialog flag that controls the state of the add author popup. For the AUTHOR_ADD and HIDE_ADD_AUTHOR cases of the dispatch we just change the state of this flag and emit the change; the BookForm listens for changes to the AuthorStore and will pass the showDialog to the AuthorPanel as a property which in turn will pass it to AuthorDialog and it will display the panel (or not) depending on the value of that flag. This flag will also take a false value when the add author ajax call returns.

The showDialog flag is not related to the actual data but is UI-related. This is something that we should keep in mind when creating stores: Stores don’t only contain the actual data (like models in an MVC application) but they should also contain UI (controller/view in an MVC architecture) related information since that is also part of the state!

We can see that the ajax calls just issue the corresponding HTTP method to the authors_url and when they return the add_message_ok or add_message_error methods of MessageActions will be called. These calls are in a different call stack so everything will work fine (please remember the discussion about dispatches in different call stacks before).

Finally, on the success of _load_authors the map array method is called to transform the returned data as we want it:

var $ = require('jquery');

var _state = {
    authors: [],
    showDialog: false
}

var _load_authors = function() {
    $.ajax({
        url: _props.authors_url,
        dataType: 'json',
        cache: false,
        success: function(data) {
            _state.authors = data.map(function(a){
                return {
                    id: a.id,
                    name: a.last_name+' '+a.first_name
                }
            });
            AuthorStore.emitChange();
        },
        error: function(xhr, status, err) {
            MessageActions.add_message_error(err.toString());
        }
    });
};

var _deleteAuthor = function(authorId) {
    $.ajax({
        url: _props.authors_url+authorId,
        method: 'DELETE',
        cache: false,
        success: function(data) {
            _load_authors();
            MessageActions.add_message_ok("Author delete ok");
            AuthorActions.delete_author_ok();
        },
        error: function(xhr, status, err) {
            MessageActions.add_message_error(err.toString());
        }
    });
};

var _addAuthor = function(author) {
    $.ajax({
        url: _props.authors_url,
        dataType: 'json',
        method: 'POST',
        data:author,
        cache: false,
        success: function(data) {
            MessageActions.add_message_ok("Author add  ok");
            _state.showDialog = false;
            _load_authors();
        },
        error: function(xhr, status, err) {
            MessageActions.add_message_error(err.toString());
        }
    });

};

AuthorStore.dispatchToken = AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case BookConstants.SHOW_ADD_AUTHOR:
            _state.showDialog = true;
            AuthorStore.emitChange();
        break;
        case BookConstants.HIDE_ADD_AUTHOR:
            _state.showDialog = false;
            AuthorStore.emitChange();
        break;
        case BookConstants.AUTHOR_ADD:
            _addAuthor(action.author);
        break;
        case BookConstants.AUTHOR_DELETE:
            _deleteAuthor(action.authorId);
        break;
    }
    return true;
});

CategoryStore.js

The CategoryStore has an interesting functionality concerning the load_subcategory function. This function is called whenever a book is changed (so its category form field may be changed and the subcategories may be reloaded based on this category) or is edited (so the category is that of the new book and once again the subcategories may need to because rerendered). It is important that we actually pass the current category to the book to the action. If for example we wanted to retrieve that from the state of the BookStore then we’d need to use the waitFor functionality of the dispatcher so that the category of the current book would be changed first then the load_category (that would read that value to read the subcategoreis) would be called after that.

Also, another thing to notice here is that there’s a simple subcat_cache that for each category contains the subcategories of that category so that we won’t do repeated ajax calls to reload the subcategories each time the category is changed.

var _state = {
    categories: [],
    subcategories: []
}

var _current_cat = ''
var _subcat_cache = []

var _load_categories = function() {
    $.ajax({
        url: _props.categories_url,
        dataType: 'json',
        cache: false,
        success: function(data) {
            _state.categories = data;
            CategoryStore.emitChange();
        },
        error: function(xhr, status, err) {
            console.error(this.props.url, status, err.toString());
        }
    });
};

var _load_subcategories = function(cat) {

    if(!cat) {
        _state.subcategories = [];
        CategoryStore.emitChange();
        return ;
    }
    if(_subcat_cache[cat]) {
        _state.subcategories = _subcat_cache[cat] ;
        CategoryStore.emitChange();
    }
    $.ajax({
        url: _props.subcategories_url+'?category='+cat,
        dataType: 'json',
        cache: false,
        success: function(data) {
            _state.subcategories = data;
            _subcat_cache[cat] = data;
            CategoryStore.emitChange();
        },
        error: function(xhr, status, err) {
            console.error(this.props.url, status, err.toString());
        }
    });
};

CategoryStore.dispatchToken = AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case BookConstants.BOOK_EDIT:
        case BookConstants.BOOK_CHANGE:
            _load_subcategories(action.book.category);
        break;
        case BookConstants.BOOK_EDIT_CANCEL:
            _state.subcategories = [];
            CategoryStore.emitChange();
        break;
    }
    return true;
});

BookStore.js

Here, beyond the book-related functionality we have also implemented the URL updating. The getUrlParameter that returns the value of a URL parameter has been taken from http://stackoverflow.com/questions/19491336/get-url-parameter-jquery. Depending on the url parameters, we set some initial properties of the local state and, on the other hand, when the search query, ordering or page are changed, the _update_href function is called to update the url parameters. This is not really related to the flux architecture beyond the initialization of state.

Another thing to notice is that the when the _search is executed whenever there’s a change in the list of books (query is updated, sorting is changed, page is changed or when an author is deleted since the books that have that author should now display an empty field). The setTimeout in the _search ajax return is to simulate a 400ms delay (in order for the "Loading" text to be visible).

function getUrlParameter(sParam) {
    var sPageURL = $(location).attr('hash');
    sPageURL = sPageURL.substr(1)
    var sURLVariables = sPageURL.split('&');
    for (var i = 0; i < sURLVariables.length; i++)  {
        var sParameterName = sURLVariables[i].split('=');
        if (sParameterName[0] == sParam)  {
            return sParameterName[1];
        }
    }
}

var _page_init = 1*getUrlParameter('page');
if(!_page_init) _page_init = 1 ;
var _ordering_init = getUrlParameter('ordering');
if(!_ordering_init) _ordering_init = '' ;
var _query_init = getUrlParameter('query');
if(!_query_init) _query_init = ''

var _state = {
    loading: false,
    books: [],
    message:{},
    page: _page_init,
    total: 0,
    editingBook: {},
    query: _query_init,
    ordering: _ordering_init
}


var _search = function() {
    _state.loading = true;
    BookStore.emitChange();

    $.ajax({
        url: _props.url+'?search='+_state.query+"&ordering="+_state.ordering+"&page="+_state.page,
        dataType: 'json',
        cache: false,
        success: function(data) {
            // Simulate a small delay in server response
            setTimeout(function() {
                _state.books = data.results;
                _state.total = data.count;
                _state.loading = false;
                BookStore.emitChange();
            }, 400);
        },
        error: function(xhr, status, err) {
            _state.loading = false;
            MessageActions.add_message_error(err.toString());
            BookStore.emitChange();
        }
    });
};

var _reloadBooks = function() {
    _search('');
};


var _clearEditingBook = function() {
    _state.editingBook = {};
};

var _editBook = function(book) {
    _state.editingBook = book;
    BookStore.emitChange();
};

var _cancelEditBook = function() {
    _clearEditingBook();
    BookStore.emitChange();
};

var _update_href = function() {
    var hash = 'page='+_state.page;
    hash += '&ordering='+_state.ordering;
    hash += '&query='+_state.query;
    $(location).attr('hash', hash);
}

BookStore.dispatchToken = AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case BookConstants.BOOK_EDIT:
            _editBook(action.book);
        break;
        case BookConstants.BOOK_EDIT_CANCEL:
            _cancelEditBook();
        break;
        case BookConstants.BOOK_SAVE:
            _saveBook(action.book);
        break;
        case BookConstants.BOOK_SEARCH:
            _state.query = action.query
            _state.page = 1;
            _update_href();
            _search();
        break;
        case BookConstants.BOOK_DELETE:
            _deleteBook(action.bookId);
        break;
        case BookConstants.BOOK_CHANGE:
            _state.editingBook = action.book;
            BookStore.emitChange();
        break;
        case BookConstants.BOOK_PAGE:
            _state.page = action.page;
            _update_href();
            _search();
        break;
        case BookConstants.AUTHOR_DELETE_OK:
            _search();
        break;
        case BookConstants.BOOK_SORT:
            _state.page = 1;
            if(_state.ordering == action.field) {
                _state.ordering = '-'+_state.ordering
            } else {
                _state.ordering = action.field;
            }
            _update_href();
            _search();
        break;
    }
    return true;
});

Conclusion

The application presented here has a number of techniques that will help you when you actually try to create a more complex react flux application. I hope that in the whole three part series I’ve thoroughly explained the flux architecture and how each part of (actions, stores, components) it works. Also, I tried to cover almost anything that somebody creating react/flux application will need to use — if you feel that something is not covered and could be integrated to the authors/book application I’d be happy to research and implement it!

django-rq redux: advanced techniques and tools

Serafeim Papastefanos — Tue, 01 Sep 2015 14:20:00 +0300

Contents

Introduction
Displaying your task progress
Displaying your queue statistics
Making sure that workers for your queue are actually running
Checking how many jobs are in the queue
Better exception handling
Multiple django-apps, single redis db
Low level debugging
Conclusion

Introduction

In the previous django-rq article we presented a quick introduction to asynchronous job queues and created a small (but complete) project that used rq and django-rq to implement asynchronous job queues in a django project.

In this article, we will present some more advanced techniques and tools for improving the capabilities of our asynchronous tasks and integrate them to the https://github.com/spapas/django-test-rq project (please checkout tag django-rq-redux git checkout django-rq-redux)

Displaying your task progress

Sometimes, especially for long-running tasks it is useful to let the user (task initiator) know what is the status of each task he’s started. For this, I recommend creating a task-description model that will hold the required information for this task with more or less the following fields (please also check LongTask model of django-test-rq):

class LongTask(models.Model):
  created_on = models.DateTimeField(auto_now_add=True)
  name = models.CharField(max_length=128, help_text='Enter a unique name for the task',)
  progress = models.PositiveIntegerField(default=0)
  result = models.CharField(max_length=128, blank=True, null=True)

Now, when the view that starts the task is POST ed, you’ll first create the LongTask model instance with a result of 'QUEUED' and a progress of 0 (and a name that identifies your task) and then you’ll start the real task asynchronously by passing the LongTask instance, something like this (also check LongTaskCreateView):

long_task = LongTask.objects.create(...)
long_runnig_task.delay(long_task)

In your asynchronous job, the first thing you’ll need to do is to set its result to ‘STARTED’ and save it so that the user will immediately see when he’s job is actually started. Also, if you can estimate its progress, you can update its progress value with the current value so that the user will know how close he is to finishing. Finally, when the job finished (or if it throws an expectable exception) you’ll update its status accordingly. Here’s an example of my long_running_task that just waits for the specifid amount of seconds:

@job('django-test-rq-low')
def long_runnig_task(task):
  job = get_current_job()
  task.job_id = job.get_id()

  task.result = 'STARTED'

  duration_in_second_persentages = task.duration*1.0 / 100
  for i in range(100):
      task.progress = i
      task.save()
      print task.progress
      time.sleep(duration_in_second_persentages)

  task.result = 'FINISHED'
  task.save()
  return task.result

To have proper feedback I propose to have your task-description model instance created by the view that starts the asynchronous task and not by the actual task! This is important since the worker may be full so the asynchronous task will need a lot of time until is actually started (or maybe there are no running workers - more on this later) and the user will not be able to see his task instance anywhere (unless of course you provide him access to the actual task queue but I don’t recommend this).

Displaying your queue statistics

django-rq has a really nice dashboard with a lot of queue statistics ( instructions here https://github.com/ui/django-rq#queue-statistics and also on django-test-rq project) which I recommend to always enable.

Also, there’s the individual use django-rq-dashboard project that could be installed to display some more statistics, however the only extra statistic that you can see throuh django-rq-dashboard is the status of your scheduled jobs so I don’t recommend installing it if you don’t use scheduling.

Making sure that workers for your queue are actually running

Using the django-rq dashboard you can make sure that all queues have at least one worker. However, sometimes workers fail, or maybe you’ve forgotten to start your workers or not configured your application correctly (this happens to me all the time for test/uat projects). So, for tasks that you want to display feedback to the user, you can easily add a check to make sure that there are active workers using the following code:

from rq import Worker
import django_rq

redis_conn = django_rq.get_connection('default')
if len([
    x for x in Worker.all(connection=redis_conn)
        if 'django-test-rq-low' in x.queue_names()
]) == 0:
    # Error -- no workers

With Worker.all() you get all workers for a connection and the queue_names() method returns the names that each worker serves. So we check that we have at least one worker for that queue.

This check can be added when the job is started and display a feedback error to the user (check example in django-test-rq).

For quick tasks (for example sending emails etc) you should not display anything to the user even if no workers are running (since the task will be queued and will be executed eventually when the workers are started) but instead send an email to the administrators so that they will start the workers.

Checking how many jobs are in the queue

To find out programatically how many jobs are actually in the queue (and display a message if the queue has too many jobs etc) you’ll need to use the Queue class, something like this:

from rq import Queue

redis_conn = django_rq.get_connection('default')
queue = Queue('django-test-rq-default', connection=redis_conn)
print queue.name
print len(queue.jobs)

Better exception handling

When a job fails, rq will put it in a failed jobs queue and finish with it. You (as administrator) won’t get any feedback and the user (unless he has access to that failed jobs queue) won’t be able to do anything aboutt this job.

In almost all cases you can’t rely only on this behavior but instead you have to install a custom exception handler. Using the custom exception handler you can do whatever you want for each failed job. For instance, you can create a new instance of a FailedTask model which will have information about the failure and the original task allow the user (or administrator) to restart the failed task after he’s fixed the error conditions.

Or, if you want to be informed when a job is failed, you can just send an email to ADMINS and fall back to the default behavior to enqueue the failed task the failed jobs queue (since job exception handlers can be chained).

A simple management command that starts a worker for a specific queue and installs a custom exception handler follows:

from django.conf import settings
from django.core.management.base import BaseCommand

import django_rq
from rq import Queue, Worker

def my_handler(job, *exc_info):
    print "FAILURE"
    print job
    print exc_info

class Command(BaseCommand):
    def handle(self, *args, **options):
        redis_conn = django_rq.get_connection('default')

        q = Queue(settings.DJANGO_TEST_RQ_LOW_QUEUE, connection=redis_conn)
        worker = Worker([q], exc_handler=my_handler, connection=redis_conn)
        worker.work()

This handler is for demonstration purposes since it just prints a message to the console (so please do not use it)!

Multiple django-apps, single redis db

One thing to keep in mind is that the only thing that seperates the queues are their name. If you have many django applications that define a "default" (or "low", "hight" etc) and they all use the same redis database to store their queue, the workers of each application won’t know which jobs belong to them and they’ll end up dequeuing the wrong job types. This will lead to an exception or, if you are really unlucky to a very nasty bug!

To avoid this, you can either use a different redis database (not database server) for each of your apps or add a prefix with the name of your app to your queue names:

Each redis database server can host a number of databases that are identified by a number (that’s what the /0 you see in redis://127.0.0.1:6379/0 means) and each one of them has a totally different keyspace. So, if you use /0 in an application and /1 in another application, you’ll have no problems. This solution has the disadvantage that you need to be really careful to use different database numbers for your projects and also the number of possible databases that redis can use is limited by a configuration file (so if you reach the maximum you’ll need to also increase that number)!

Instead of this, you can avoid using the ‘default’ queue, and use queues that contain your application name in their name, for example, for the sample project you could create something like ‘django-test-rq-default’, ‘django-test-rq-low’, ‘django-test-rq-high’ etc. You need to configure the extra queues by adding them to the RQ_QUEUES dictionary (check settings.py of django-test-rq) and then put the jobs to these queues using for example the job decorator (@job('django-test-rq-default')) and run your workers so that they will retrieve jobs from these queues (python manage.py rqworker django-test-rq-default) and not the default one (which may contain jobs of other applications).

If you use the default queue, and because you’ll need to use its name to many places, I recommend to add a (f.i) QUEUE_NAME = 'django-test-rq-default' setting and use this instead of just a string to be totally DRY.

Update 13/09/2015: Please notice that using a single redis database server (either with multiple numeric databases or in the same database using a keyword in keys to differentiate the apps) is not recommended as commenter Itamar Haber pointed out to me!

This is because for speed reasons redis uses a single thread to handle all requests (regardless if they are in the same or different numerical databases), so all resources may be used by a single, redis hungry, application and leave all others to starve!

Therefore, the recommended solution is to have a different redis server for each different application. This does not mean that you need to have different servers, just to run different instances of redis binding to different IP ports. Redis uses very little resourecs when it is idle (empty instance uses ~ 1 MB RAM) so you can run a lot of instances in a single server.

Long story short, my proposal is to have a redis.conf inside your application root tree (next to manage.py and requirements.txt) which has the redis options for each application. The options in redis.conf that need to be changed per application is the port that this redis instance will bind (this port also needs to be passed to django settings.py) and the pid filename if you daemonize redis — I recommend using a tool like supervisord instead so that you won’t need any daemonizing and pid files for each per-app-redis-instance!

Low level debugging

In this section I’ll present some commands that you can issue to your redis server using a simple telnet connection to get various info about your queues. You probably will never need to issue these commands to actually debug, but they will answer some of your (scientific) questions! In the following, > is things I type, # are comments, [...] is more output and everything else is the output I get:

> telnet 127.0.0.1 6379

# You won't see anything at first but you'll be connected and you can try typing things

> INFO

$1020
redis_version:2.4.10
redis_git_sha1:00000000
# [...]
db0:keys=83,expires=2
db1:keys=26,expires=1 # My redis server has two databases

# Now you'll see what you type!

> SELECT 1
+ OK # Now queries will be issued to database 1
> SELECT 0
+ OK # Now queries will be issued to database 0

KEYS rq* # List all rq related queues
*25
$43
rq:job:1d7afa32-3f90-4502-912f-d58eaa049fb1
$43
rq:queue:django-test-rq-low
$43
[...]

> SMEMBERS rq:workers # See workers
*1
$26
rq:worker:SERAFEIM-PC.6892

> LRANGE rq:queue:django-test-rq-low 0 100 # Check queued jobs
*2
$36
def896f4-84cb-4833-be6a-54d917f05271
$36
53cb1367-2fb5-46b3-99b2-7680397203b9

> HGETALL rq:job:def896f4-84cb-4833-be6a-54d917f05271 # Get info about this job
*16
$6
status
$6
queued
$11
description
$57
tasks.tasks.long_runnig_task(<LongTask: LongTask object>)
$10
created_at
$20
2015-09-01T09:04:38Z
$7
timeout
$3
180
$6
origin
$18
django-test-rq-low
$11
enqueued_at
$20
2015-09-01T09:04:38Z
$4
data
$409
[...] # data is the pickled parameters passed to the job !

> HGET rq:job:def896f4-84cb-4833-be6a-54d917f05271 status # Get only status
$6
queued

For more info on querying redis you can check the redis documentation and especially http://redis.io/topics/data-types and http://redis.io/commands.

Conclusion

Using some of the above techniques will help you in your asynchronous task adventures with rq. I’ll try to keep this article updated with any new techniques or tools I find in the future!

A comprehensive React and Flux tutorial part 2: Flux

Serafeim Papastefanos — Thu, 02 Jul 2015 14:20:00 +0300

Contents

Introduction
Flux components
The react-flux version
Explaining the data flow
A better code organization
Conclusion

Introduction

In the first part of this series we implemented a not-so-simple one page application with full CRUD capabilities. In this part, we will modify that application to make it use the Flux architecture. The full source code can be found at https://github.com/spapas/react-tutorial (tag name react-flux). In the In next part, we will create an even more complex application using react/flux!

I recommend reading Facebook’s Flux overview before reading this article — please read it even if you find some concepts difficult to grasp (I know I found it difficult the first time I read it), I will try to explain everything here. Also, because a rather large number of extra components will need to be created, we are going to split our javascript code to different files using browserify - you can learn how to use browserify here.

Flux components

To implement the Flux architecture, an application needs to have at least a store and a dispatcher.

The store is the central place of truth for the application and the dispacher is the central place of communications. The store should hold the state (and any models/DAOs) of the application and notify the react components when this state is changed. Also, the store will be notified by the dispatcher when an action happens (for example a button is clicked) so that it will change the state. As a flow, we can think of something like this:

a ui action on a component (click, change, etc) ->
 ^   dispatcher is notified ->
 |   store is notified (by the dispacher)->
 |   store state is changed ->
 └─  component is notified (by the store) and updated to reflect the change

One thing to keep in mind is that although each flux application will have only one dispatcher, it may have more stores, depending on the application’s architecture and separation of concerns. If there are more than store, all will be notified by the dispatcher and change their state (if needed of course). The ui will pass the action type and any optional parameters to the dispatcher and the dispatcher will notify all stores with these parameters.

An optional component in the Flux architecture is the Action. An action is a store related class that acts as an intermediate between the ui and the dispatcher. So, when a user clicks a button, an action will be called that will notify the dispatcher. As we will see we can just call the dispatcher directly from the components ui, but calling it through the Action makes the calls more consistent and creates an interface.

The react-flux version

Since we are using browserify, we will include a single file in our html file with a <script> tag and everything else will be included through the require function. We have the following packages as requirements for browserify:

"dependencies": {
  "flux": "^2.0.3",
  "jquery": "^2.1.4",
  "react": "^0.13.3",
  "reactify": "^1.1.1"
}

Also, in order to be able to use JSX with browserify, will use the reactify transform. To apply it to your project, change the scripts of your package.json to:

"scripts": {
  "watch": "watchify -v -d static/main.js -t reactify -o static/bundle.js",
  "build": "browserify static/main.js -t reactify  | uglifyjs -mc warnings=false > static/bundle.js"
},

main.js

The main.js file will just render the BookPanel component (the components.js file contains the source for all React components) and call the reloadBooks function from stores.js that will reload all books from the REST API:

var React = require('react');
var components = require('./components');
var stores = require('./stores');

React.render(<components.BookPanel url='/api/books/' />, document.getElementById('content'));

stores.reloadBooks();

constants.js

Before going into more complex modules, let’s present the constants.js which just exports some strings that will be passed to the dispatcher to differentiate between each ui action:

module.exports = {
    BOOK_EDIT: 'BOOK_EDIT',
    BOOK_EDIT_CANCEL: 'BOOK_EDIT_CANCEL',
    BOOK_SAVE: 'BOOK_SAVE',
    BOOK_SEARCH: 'BOOK_SEARCH',
    BOOK_DELETE: 'BOOK_DELETE',
};

As we can see, these constants are exported as a single object so when we do something like var BookConstants = require('./constants') we’ll the be able to refer to each constant through BookConstants.CONSTANT_NAME.

actions.js

The actions.js creates the dispatcher singleton and a BookActions object that defines the actions for books.

var BookConstants = require('./constants')
var Dispatcher = require('flux').Dispatcher;
var AppDispatcher = new Dispatcher();

var BookActions = {
    search: function(query) {
        AppDispatcher.dispatch({
            actionType: BookConstants.BOOK_SEARCH,
            query: query
        });
    },
    save: function(book) {
        AppDispatcher.dispatch({
            actionType: BookConstants.BOOK_SAVE,
            book: book
        });
    },
    edit: function(book) {
        AppDispatcher.dispatch({
            actionType: BookConstants.BOOK_EDIT,
            book: book
        });
    },
    edit_cancel: function() {
        AppDispatcher.dispatch({
            actionType: BookConstants.BOOK_EDIT_CANCEL
        });
    },
    delete: function(bookId) {
        AppDispatcher.dispatch({
            actionType: BookConstants.BOOK_DELETE,
            bookId: bookId
        });
    }
};

module.exports.BookActions = BookActions;
module.exports.AppDispatcher = AppDispatcher;

As we can see, the BookActions is just a collection of methods that will be called from the ui. Instead of calling BookActions.search() we could just call the dispatch method with the correct parameter object (actionType and optional parameter), both the BookActions object and the AppDispatcher singleton are exported.

The dispatcher is imported from the flux requirement: It offers a functionality to register callbacks for the various actions as we will see in the next module. This is a rather simple class that we could implement ourselves (each store passes a callback to the dispatcher that is called on the dispatch method, passing actionType and any other parameters). The dispatcher also offers a waitFor method that can be used to ensure that the dispatch callback for a store will be finished before another store’s dispatch callback ( when the second store uses the state of the first store — for example when implementing a series of related dropdowns ).

stores.js

The next module we will discuss is the stores.js that contains the BookStore object.

var $ = require('jquery');
var EventEmitter = require('events').EventEmitter;
var AppDispatcher = require('./actions').AppDispatcher;
var BookConstants = require('./constants')

var _state = {
    books: [],
    message:"",
    editingBook: null
}

var _props = {
    url: '/api/books/'
}

var _search = function(query) {
    $.ajax({
        url: _props.url+'?search='+query,
        dataType: 'json',
        cache: false,
        success: function(data) {
            _state.books = data;
            BookStore.emitChange();
        },
        error: function(xhr, status, err) {
            console.error(this.props.url, status, err.toString());
            _state.message = err.toString();
            BookStore.emitChange();
        }
    });
};

var _reloadBooks = function() {
    _search('');
};

var _deleteBook = function(bookId) {
    $.ajax({
        url: _props.url+bookId,
        method: 'DELETE',
        cache: false,
        success: function(data) {
            _state.message = "Successfully deleted book!"
            _clearEditingBook();
            _reloadBooks();
        },
        error: function(xhr, status, err) {
            console.error(this.props.url, status, err.toString());
            _state.message = err.toString();
            BookStore.emitChange();
        }
    });
};

var _saveBook = function(book) {
    if(book.id) {
        $.ajax({
            url: _props.url+book.id,
            dataType: 'json',
            method: 'PUT',
            data:book,
            cache: false,
            success: function(data) {
                _state.message = "Successfully updated book!"
                _clearEditingBook();
                _reloadBooks();
            },
            error: function(xhr, status, err) {
                _state.message = err.toString()
                BookStore.emitChange();
            }
        });
    } else {
        $.ajax({
            url: _props.url,
            dataType: 'json',
            method: 'POST',
            data:book,
            cache: false,
            success: function(data) {
                _state.message = "Successfully added book!"
                _clearEditingBook();
                _reloadBooks();
            },
            error: function(xhr, status, err) {
                _state.message = err.toString()
                BookStore.emitChange();
            }
        });
    }
};

var _clearEditingBook = function() {
    _state.editingBook = null;
};

var _editBook = function(book) {
    _state.editingBook = book;
    BookStore.emitChange();
};

var _cancelEditBook = function() {
    _clearEditingBook();
    BookStore.emitChange();
};

var BookStore = $.extend({}, EventEmitter.prototype, {
    getState: function() {
        return _state;
    },
    emitChange: function() {
        this.emit('change');
    },
    addChangeListener: function(callback) {
        this.on('change', callback);
    },
    removeChangeListener: function(callback) {
        this.removeListener('change', callback);
    }
});

AppDispatcher.register(function(action) {
    switch(action.actionType) {
        case BookConstants.BOOK_EDIT:
            _editBook(action.book);
        break;
        case BookConstants.BOOK_EDIT_CANCEL:
            _cancelEditBook();
        break;
        case BookConstants.BOOK_SAVE:
            _saveBook(action.book);
        break;
        case BookConstants.BOOK_SEARCH:
            _search(action.query);
        break;
        case BookConstants.BOOK_DELETE:
            _deleteBook(action.bookId);
        break;
    }
    return true;
});

module.exports.BookStore = BookStore;
module.exports.reloadBooks = _reloadBooks;

The stores.js module exports only the BookStore object and the reloadBooks method (that could also be called from inside the module since it’s just called when the application is loaded to load the books for the first time). All other objects/funtions are private to the module.

As we saw, the _state objects keep the global state of the application which are the list of books, the book that is edited right now and the result message for any update we are doing. The ajax methods are more or less the same as the ones in the react-only version of the application. However, please notice that when the ajax methods return and have to set the result, instead of setting the state of a React object they are just calling the emitChange method of the BookStore that will notify all react objects that "listen" to this store. This is possible because the ajax (DAO) methods are in the same module with the store - if we wanted instead to put them in different modules, we’d just need to add another action (e.g ReloadBooks) that would be called when the ajax method returns — this action would call the dispatcher which would in turn update the state of the store.

We can see that we are importing the AppDispatcher singleton and, depending on the action type we call the correct method that changes the state. So when a BookActions action is called it will call the corresponding AppDispatcher.register case branch which will call the corresponding state-changing function.

The BookStore extends the EventEmitter object (so we need to require the events module) in order to notify the React components when the state of the store is changed. Instead of using EventEmitter we could just implement the emit change logic ourselves by saving all the listener callbacks to an array and calling them all when there’s a state change (if we wanted to also add the ‘change’ parameter to group the listener callbacks we’d just make the complex more complex, something not needed for our case):

var BookStore = {
    listeners: [],
    getState: function() {
        return _state;
    },
    emitChange: function() {
        var i;
        for(i=0;i<this.listeners.length;i++) {
            this.listeners[i]();
        }
    },
    addChangeListener: function(callback) {
        this.listeners.push(callback);
    },
    removeChangeListener: function(callback) {
        this.listeners.splice(this.listeners.indexOf(callback), 1);
    }
};

components.js

Finally, the components.js module contains all the React components. These are more or less the same with the react-only version with three differences:

When something happens in the ui, the corresponding BookAction action is called with the needed parameter — no callbacks are passed between the components
The BookPanel component registers with the BookStore in order to be notified when the state changes and just gets its state from the store — these values are propagated to all other components through properties
The BookForm and SearcchPanel now hold their own temporary state instead of using the global state — notice that when a book is edited this book will be propagated to the BookForm through the book property, however BookForm needs to update its state through the componentWillReceiveProps method.

var React = require('react');
var BookStore = require('./stores').BookStore;
var BookActions = require('./actions').BookActions;

var BookTableRow = React.createClass({
    render: function() {
        return (
            <tr>
                <td>{this.props.book.id}</td>
                <td>{this.props.book.title}</td>
                <td>{this.props.book.category}</td>
                <td><a href='#' onClick={this.onClick}>Edit</a></td>
            </tr>
        );
    },
    onClick: function(e) {
        e.preventDefault();
        BookActions.edit(this.props.book);
    }
});

var BookTable = React.createClass({
    render: function() {
        var rows = [];
        this.props.books.forEach(function(book) {
            rows.push(<BookTableRow key={book.id} book={book} />);
        });
        return (
            <table>
                <thead>
                    <tr>
                        <th>Id</th>
                        <th>Title</th>
                        <th>Category</th>
                        <th>Edit</th>
                    </tr>
                </thead>
                <tbody>{rows}</tbody>
            </table>
        );
    }
});

var BookForm = React.createClass({
    getInitialState: function() {
        if (this.props.book) {
            return this.props.book;
        } else {
            return {};
        }
    },
    componentWillReceiveProps: function(props) {
        if (props.book) {
            this.setState(props.book);
        } else {
            this.replaceState({});
        }
    },
    render: function() {
        return(
            <form onSubmit={this.onSubmit}>
                <label forHtml='title'>Title</label><input ref='title' name='title' type='text' value={this.state.title} onChange={this.onFormChange} />
                <label forHtml='category'>Category</label>
                <select ref='category' name='category' value={this.state.category} onChange={this.onFormChange} >
                    <option value='CRIME' >Crime</option>
                    <option value='HISTORY'>History</option>
                    <option value='HORROR'>Horror</option>
                    <option value='SCIFI'>SciFi</option>
                </select>
                <br />
                <input type='submit' value={this.state.id?"Save (id = " +this.state.id+ ")":"Add"} />
                {this.state.id?<button onClick={this.onDeleteClick}>Delete</button>:""}
                {this.state.id?<button onClick={this.onCancelClick}>Cancel</button>:""}
                {this.props.message?<div>{this.props.message}</div>:""}
            </form>
        );
    },
    onFormChange: function() {
        this.setState({
            title: React.findDOMNode(this.refs.title).value,
            category: React.findDOMNode(this.refs.category).value
        })
    },
    onSubmit: function(e) {
        e.preventDefault();
        BookActions.save(this.state)
    },
    onCancelClick: function(e) {
        e.preventDefault();
        BookActions.edit_cancel()
    },
    onDeleteClick: function(e) {
        e.preventDefault();
        BookActions.delete(this.state.id)
    }
});

var SearchPanel = React.createClass({
    getInitialState: function() {
        return {
            search: '',
        }
    },
    render: function() {
        return (
            <div className="row">
                <div className="one-fourth column">
                    Filter: &nbsp;
                    <input ref='search' name='search' type='text' value={this.state.search} onChange={this.onSearchChange} />
                    {this.state.search?<button onClick={this.onClearSearch} >x</button>:''}
                </div>
            </div>
        )
    },
    onSearchChange: function() {
        var query = React.findDOMNode(this.refs.search).value;
        if (this.promise) {
            clearInterval(this.promise)
        }
        this.setState({
            search: query
        });
        this.promise = setTimeout(function () {
            BookActions.search(query);
        }.bind(this), 200);
    },
    onClearSearch: function() {
        this.setState({
            search: ''
        });
        BookActions.search('');
    }
});

var BookPanel = React.createClass({
    getInitialState: function() {
        return BookStore.getState();
    },
    render: function() {
        return(
            <div className="row">
                <div className="one-half column">
                    <SearchPanel></SearchPanel>
                    <BookTable books={this.state.books} />
                </div>
                <div className="one-half column">
                    <BookForm
                        book={this.state.editingBook}
                        message={this.state.message}
                    />
                </div>
                <br />
            </div>
        );
    },
    _onChange: function() {
        this.setState( BookStore.getState() );
    },
    componentWillUnmount: function() {
        BookStore.removeChangeListener(this._onChange);
    },
    componentDidMount: function() {
        BookStore.addChangeListener(this._onChange);
    }
});

module.exports.BookPanel = BookPanel ;

Only the BookPanel is exported — all other react components will be private to the module.

We can see that, beyond BookPanel, the code of all other components are more or less the same. However, not having to pass callbacks for state upddates is a huge win for readability and DRYness.

Explaining the data flow

I’ve added a bunch of console.log statements to see how the data/actions flow between all the components when the "Edit" book is clicked. So, when we click "Edit" we see the following messages to our console:

Inside BookTableRow.onClick
Inside BookActions.edit
Inside AppDispatcher.register
Inside AppDispatcher.register case BookConstants.BOOK_EDIT
Inside _editBook
Inside BookStore.emitChange
Inside BookPanel._onChange
Inside BookForm.componentWillReceiveProps
Inside BookForm.render

First of all the onClick method of BookTableRow will be called (which is the onClick property of the a href link) which will call BookActions.edit and pass it the book of that specific row. The edit method will create a new dispatcher object by setting the actionType and passing the book and pass it to AppDispatcher.register. register will go to the BookConstants.BOOK_EDIT case branch which will call the private _editBook function. _editBook will update the state of the store (by setting the _state.editingBook property and will call the BookStore.emitChange method which calls the dispatcher’s emit method, so all listening components will update. We only have one component that listens to this emit, BookPanel whose _onChange method is called. This method gets the application state from the BookStore and updates its own state. Now, the state will be propagated through properties - for example, for BookForm, first its componentWillReceiveProps method will be called (with the new properties) and finally its render method!

So the full data flow is something like this:

user action/callback etc ->
  component calls action ->
    dispatcher informes stores ->
      stores set their state ->
        state holding components are notified and update their state ->
          all other components are updated through properties

A better code organization

As you’ve seen, I’ve only created four javascript modules (components, stores, actions and constants) and put them all in the same folder. I did this for clarity and to keep everything together since our tutorial is a very small project. Facebook proposes a much better organization that what I did as can be seen in the TodoMVC tutorial: Instead of putting everything to a single folder, create a different foloder for each type of object: actions (for all your actions), components (for all your React components), constants and stores and put inside the objects each in a different javascript module, for example, the components folder should contain the following files:

BookTable.react.js
BookTableRow.react.js
BookForm.react.js
BookPanel.react.js
SearchPanel.react.js

Each one will export only the same-named React component and require only the components that it uses.

If you want to see the code of this tutorial organized like this go to the tag react-flux-better-organization.

Conclusion

In this two-part series we saw how we can create a full CRUD application with React.js and how can we enable it with the Facebook proposed Flux architecture. Comparing the react-only with the react-flux version we can see that we added a number of objects in the second version (dispatcher, store, actions, constants) whose usefulness may not be obvious from our example. However, our created application (and especially the better organized version) is war-ready and can easily fight any complexities that we throw to it! Unfortunately, if we really wanted to show the usefulness of the Flux architecture we’d need to create a really complex application that won’t be suitable for a tutorial.

However, we can already understand the obvious advantages of the React / Flux architecture:

Components can easily be re-used by changing their properties - DRY
Easy to grasp (but a little complex) data flow between components and stores
Separation of concerns - react components for the view, stores to hold the state/models, dispatcher to handle the data flow
Really easy to test - all components are simple objects and can be easily created fom tests
Works well for complex architectures - one dispatcher, multiple stores/action collections, react components only interact with actions and get their state from stores

I’ve tried to make the above as comprehensive as possible for the readers of these posts (and also resolve some of my own questions). I have to mention again that although React/Flux may seem complex at a first glance, when it is used in a complex architecture it will shine and make everything much easier. Everything is debuggable and we can always understand what’s really going on! This is in contrast with more complex frameworks that do various hidden stuff (two way data binding, magic in the REST etc) where, although it is easier to create a simple app, moving to something more complex (and especially debugging it) is a real nightmare!

A comprehensive React and Flux tutorial part 1: React

Serafeim Papastefanos — Fri, 05 Jun 2015 14:20:00 +0300

Contents

Introduction
Our project
A top-level view of the components
The react-only version
Local state
Adding form validation
Conclusion to the first part

Update 15/12/2015: Added some insights for form validation.

Introduction

React is a rather new library from Facebook for building dynamic components for web pages. It introduces a really simple, fresh approach to javascript user interface building. React allows you to define composable and self-contained HTML components through Javascript (or a special syntax that compiles to javascript named JSX) and that’s all — it doesn’t force you into any specific framework or architecture, you may use whatever you like. It’s like writing pure HTML but you have the advantage of using classes with all their advantages for your components (self-contained, composable, reusable, mixins etc).

Although React components can be used however you like in your client-side applications, Facebook proposes a specific architecture for the data flow of your events/data between the components called Flux. It’s important to keep in your mind that Flux is not a specific framework but a way to organize the flow of events and data in your applicition. Unfortunately, although Flux is a rather simple architecture it is a little difficult to understand without a proper example (at least it was difficult for me to understand by reading the Facebook documentation and some tutorials I found).

So, in this two-part tutorial we are going to build a (not-so-simple) single page CRUD application using React and Flux. Two versions of the same application will be built and explained: One with React (this part) only and one with React and Flux (part two). This will help us understand how Flux architecture fits to our project and why it greatly improves the experience with React.

Warning: This is not an introduction to react. Before reading this you need to be familiar with basic react usage, having read at least the following three pages from react documentation:

Our project

We are going to built a CRUD single-page application for editing views, let’s take a look at how it works:

Our application will be seperated to two panels: In the left one the user will be able to filter (search) for a book and in the right panel she’ll be able to add / edit / delete a book. Everything is supported by a django-rest-framework implemented REST API. You can find the complete source code at https://github.com/spapas/react-tutorial. I’ve added a couple of git tags to the source history in order to help us identify the differences between variou stages of the project (before and after integrating the Flux architecture).

Django-rest-framework is used in the server-side back-end to create a really simple REST API - I won’t provide any tutorial details on that (unless somebody wants it !), however you may either use the source code as is or create it from scratch using a different language/framework or even just a static json file (however you won’t see any changes to the data this way).

For styling (mainly to have a simple grid) we’ll use skeleton. For the ajax calls and some utils we’ll use jquery.

All client-side code will be contained in the file static/main.js. The placeholder HTML for our application is:

<html>
  <!-- styling etc ignored -->
<body>
  <h1>Hello, React!</h1>
  <div class="container"><div id="content"></div></div>
</body>
<script src="//fb.me/react-0.13.3.js"></script>
<script src="//fb.me/JSXTransformer-0.13.3.js"></script>
<script src="//code.jquery.com/jquery-2.1.3.min.js"></script>
<script type="text/jsx" src="{% static 'main.js' %}"></script>
</html>

We are using the version 0.13.3 of react and the same version of the JSXTransformer to translate JSX code to pure javascript.

A top-level view of the components

The following image shows how our components are composed:

So, the main component of our application is BookPanel which contains three components:

SearchPanel: To allow search (filtering) books based on their title/category
BookForm: To add/update/delete books
BookTable: To display all available books - each book is displayed in a BookTableRow component.

BookPanel is the only component having state — all other components will be initialized by property passing. The BookPanel element will be mounted to the #content element when the page loads.

The react-only version

The first version, using only react (and not flux) will use BookPanel as a central information HUB.

`SearchPanel`

SearchPanel renders an input element with a value defined by the search key of this component’same properties. When this is changed the onSearchChanged method of the component will be called, which in turn, retrieves the value of the input (using refs) and passes it to the properties onSearchChanged callback function. Finally, with the line {this.props.search?<button onClick={this.props.onClearSearch} >x</button>:null} we check if the search property contains any text and if yes, we display a clear filter button that will call properties onClearSearch method:

var SearchPanel = React.createClass({
  render: function() {
    return (
      <div className="row">
        <div className="one-fourth column">
          Filter: &nbsp;
          <input ref='search' type='text' value={this.props.search} onChange={this.onSearchChanged} />
          {this.props.search?<button onClick={this.props.onClearSearch} >x</button>:null}
        </div>
      </div>
    )
  },
  onSearchChanged: function() {
    var query = React.findDOMNode(this.refs.search).value;
    this.props.onSearchChanged(query);
  }
});

So, this component has three properties:

search which is the text to display in the input box
onSearchChanged (callback) which is called when the contents of the input box are changed
onClearSearch (callback) which is called when the button is pressed

Notice that this component doesn’t do anything - for all actions it uses the callbacks passed to it —this means that exactly the same component would easily be reused in a totally different application or could be duplicated if we wanted to have a different search component for the book title and category.

Another thing to notice is that the local onSearchChanged method is defined only to help us retrieve the value of the input and use it to call the onSearchChanged callback. Instead, we could just call the passed this.props.onSearchChanged — however to do this we’d need a way to find the value of the input. This could be done if we added a ref to the included SearchPanel from the parent component, so we’d be able to use something like React.findDOMNode(this.refs.searchPanel.refs.search).value to find out the value of the input (see that we use a ref to go to the searchPanel component and another ref to go to input component).

Both versions (getting the value directly from the child component or using the callback) could be used, however I believe that the callback version defines a more clear interface since the parent component shouldn’t need to know the implementation details of its children.

`BookTableRow`

BookTableRow will render a table row by creating a simple table row that will contain the title and category attributes of the passed book property and an edit link that will call the handleEditClickPanel property by passing the id of that book:

var BookTableRow = React.createClass({
  render: function() {
    return (
      <tr>
        <td>{this.props.book.title}</td>
        <td>{this.props.book.category}</td>
        <td><a href='#' onClick={this.onClick}>Edit</a></td>
      </tr>
    );
  },
  onClick: function(id) {
    this.props.handleEditClickPanel(this.props.book.id);
  }
});

This component is used by BookTable to render each one of the books.

`BookTable`

This component create the left-side table using an array of BookTableRow``s by passing it each one of the books of the ``books array property. The handleEditClickPanel property is retrieved from the parent of the component and passed as is to the row.

var BookTable = React.createClass({
  render: function() {
    var rows = [];
    this.props.books.forEach(function(book) {
      rows.push(<BookTableRow key={book.id} book={book} handleEditClickPanel={this.props.handleEditClickPanel}  />);
    }.bind(this));
    return (
      <table>
        <thead>
          <tr>
            <th>Title</th>
            <th>Category</th>
            <th>Edit</th>
          </tr>
        </thead>
        <tbody>{rows}</tbody>
      </table>
    );
  }
});

The key attribute is used by React to uniquely identify each component - we are using the id (primary key) of each book. Before continuing with the other components, I’d like to explain what is the purpose of that strange bind call!

Interlude: The `bind` function method

bind is a method of the function javascript object, since in javascript functions are objects! This method is useful for a number of things, however here we use it to set the this keyword of the anonymous function that is passed to foreach to the this keyword of the render method of BookTable, which will be the current BookTable instance.

To make things crystal: Since we are using an anonymous function to pass to forEach, this anonymous function won’t have a this keyword set to the current BookTable object so we will get an error that this.props is undefined. That’s why we use bind to set this to the current BookTable. If instead of using the anonymous function we had created a createBookTableRow method inside the BookTable object that returned the BookTableRow and passed this.createBookTableRow to forEach, we wouldn’t have to use bind (notice that we’d also need to make rows a class attribute and refer to it through this.rows for this to work).

`BookForm`

BookForm will create a form to either create a new book or update/delete an existing one. It has a book object property - when this object has an id (so it is saved to the database) the update/delete/cancel buttons will be shown, when it doesn’t have an id the add button will be shown.

var BookForm = React.createClass({
  render: function() {
    return(
      <form onSubmit={this.props.handleSubmitClick}>
        <label forHtml='title'>Title</label><input ref='title' name='title' type='text' value={this.props.book.title} onChange={this.onChange}/>
        <label forHtml='category'>Category</label>
        <select ref='category' name='category' value={this.props.book.category} onChange={this.onChange} >
          <option value='CRIME' >Crime</option>
          <option value='HISTORY'>History</option>
          <option value='HORROR'>Horror</option>
          <option value='SCIFI'>SciFi</option>
        </select>
        <br />
        <input type='submit' value={this.props.book.id?"Save (id = " +this.props.book.id+ ")":"Add"} />
        {this.props.book.id?<button onClick={this.props.handleDeleteClick}>Delete</button>:null}
        {this.props.book.id?<button onClick={this.props.handleCancelClick}>Cancel</button>:null}
        {this.props.message?<div>{this.props.message}</div>:null}
      </form>
    );
  },
  onChange: function() {
    var title = React.findDOMNode(this.refs.title).value;
    var category = React.findDOMNode(this.refs.category).value;
    this.props.handleChange(title, category);
  }
});

As we can see, this component uses many properties — most are passed callbacks functions for various actions:

book: This will either be a new book object (without and id) when adding one or an existing (from the database) book when updating one.
message: To display the result of the last operation (save/delete) — this is passed by the parent and probably it would be better if I had put it in a different component (and added styling etc).
handleSubmitClick: Will be called when the submit button is pressed to save the form (either by adding or updating).
handleCancelClick: Will be called when the cancel button is pressed — we decide that we want actually want to edit a book.
handleDeleteClick: Will be called when the delete button is pressed.
handleChange: Will be called whenever the title or the category of the currently edited book is changed through the local onChange method. The onChange will retrieve that values of title and category and pass them to handleChange to do the state update. As already discussed, we could retrieve the values immediately from the parent but this creates a better interface to our component.

`BookPanel`

The BookPanel component will contain all other components, will keep the global state and will also act as a central communications HUB between the components and the server. Because it is rather large class, I will explain it in three parts:

Component methods

In the first part of BookPanel, its react component methods will be presented:

var BookPanel = React.createClass({
  getInitialState: function() {
    return {
      books: [],
      editingBook: {
        title:"",
        category:"",
      },
      search:"",
      message:""
    };
  },
  render: function() {
    return(
      <div className="row">
        <div className="one-half column">
          <SearchPanel
            search={this.state.search}
            onSearchChanged={this.onSearchChanged}
            onClearSearch={this.onClearSearch}
          />
          <BookTable books={this.state.books} handleEditClickPanel={this.handleEditClickPanel} />
        </div>
        <div className="one-half column">
          <BookForm
            book={this.state.editingBook}
            message={this.state.message}
            handleChange={this.handleChange}
            handleSubmitClick={this.handleSubmitClick}
            handleCancelClick={this.handleCancelClick}
            handleDeleteClick={this.handleDeleteClick}
          />
        </div>
      </div>
    );
  },
  componentDidMount: function() {
    this.reloadBooks('');
  },
  // To be continued ...

getInitialState is called the first time the component is created or mounted (attached to an HTML component in the page and should return the initial values of the state - here we return an object with empty placeholders. componentDidMount will be called after the component is mounted and that’s the place we should do any initializationn — here we call the reloadBooks method (with an empty search string) to just retrieve all books. Finally, the render method creates a div that will contain all other components and initializes their properties with either state variables or object methods (these are the callbacks that were used in all other components).

Non-ajax object methods

// Continuing from above
onSearchChanged: function(query) {
  if (this.promise) {
    clearInterval(this.promise)
  }
  this.setState({
    search: query
  });
  this.promise = setTimeout(function () {
    this.reloadBooks(query);
  }.bind(this), 200);
},
onClearSearch: function() {
  this.setState({
    search: ''
  });
  this.reloadBooks('');
},
handleEditClickPanel: function(id) {
  var book = $.extend({}, this.state.books.filter(function(x) {
    return x.id == id;
  })[0] );

  this.setState({
    editingBook: book,
    message: ''
  });
},
handleChange: function(title, category) {
  this.setState({
    editingBook: {
      title: title,
      category: category,
      id: this.state.editingBook.id
    }
  });
},
handleCancelClick: function(e) {
  e.preventDefault();
  this.setState({
    editingBook: {}
  });
},
// to be continued ...

All the above function change the BookPanel state so that the properties of the child components will also be updated:

onSearchChanged is called when the search text is changed. The behavior here is interesting: Instead of immediately reloading the books, we create a timeout to be executed after 200 ms (also notice the usage of the bind function method to allow us call the reloadBooks method). If the user presses a key before these 200 ms, we cancel the previous timeout (using clearInterval) and create a new one. This technique greatly reduces ajax calls to the server when the user is just typing something in the search box — we could even increase the delay to reduce even more the ajax calls (but hurt the user experience a bit since the user will notice that his search results won’t be updated immediately).
onClearSearch is called when the clear filter button is pressed and removes the search text and reloads all books.
handleEditClickPanel is called when the edit link of a BookTableRow is clicked. The book with the passed id will be found (using filter) and then a clone of it will be created with ($.extend) and will be used to set the editingBook state attribute. If instead of the clone we passed the filtered book object we’d see that when the title or category in the BookForm were changed they’d also be changed in the BookTableRow!
handleChange just changes the state of the currently edited book based on the values passed (it does not modify the id of the book)
handleCancelClick when the cancel editing is pressed we clear the editingBook state attribute. Notice the e.preventDefault() method that needs to be there in order to prevent the form from submitting since the form submitting would result in an undesirable full page reload!

Ajax object methods

Finally, we need a buncch of object methods that use ajax calls to retrieve or update books:

  // Continuing from above
  reloadBooks: function(query) {
    $.ajax({
      url: this.props.url+'?search='+query,
      dataType: 'json',
      cache: false,
      success: function(data) {
        this.setState({
          books: data
        });
      }.bind(this),
      error: function(xhr, status, err) {
        console.error(this.props.url, status, err.toString());
        this.setState({
          message: err.toString(),
          search: query
        });
      }.bind(this)
    });
  },
  handleSubmitClick: function(e) {
    e.preventDefault();
    if(this.state.editingBook.id) {
      $.ajax({
        url: this.props.url+this.state.editingBook.id,
        dataType: 'json',
        method: 'PUT',
        data:this.state.editingBook,
        cache: false,
        success: function(data) {
          this.setState({
            message: "Successfully updated book!"
          });
          this.reloadBooks('');
        }.bind(this),
        error: function(xhr, status, err) {
          console.error(this.props.url, status, err.toString());
          this.setState({
            message: err.toString()
          });
        }.bind(this)
      });
    } else {
      $.ajax({
        url: this.props.url,
        dataType: 'json',
        method: 'POST',
        data:this.state.editingBook,
        cache: false,
        success: function(data) {
          this.setState({
            message: "Successfully added book!"
          });
          this.reloadBooks('');
        }.bind(this),
        error: function(xhr, status, err) {
          console.error(this.props.url, status, err.toString());
          this.setState({
            message: err.toString()
          });
        }.bind(this)
      });
    }
    this.setState({
      editingBook: {}
    });
  },
  handleDeleteClick: function(e) {
  e.preventDefault();
  $.ajax({
    url: this.props.url+this.state.editingBook.id,
    method: 'DELETE',
    cache: false,
    success: function(data) {
      this.setState({
          message: "Successfully deleted book!",
          editingBook: {}
      });
      this.reloadBooks('');
    }.bind(this),
    error: function(xhr, status, err) {
      console.error(this.props.url, status, err.toString());
      this.setState({
          message: err.toString()
      });
    }.bind(this)
    });
  },
});

reloadBooks will try to load the books using an ajax GET and pass its query parameter to filter the books (or get all books if query is an empty string). If the ajax call was successfull the state will be updated with the retrieved books and the search text (to clear the search text when we reload because of a save/edit/delete) while if there was an error the state will be updated with the error message. The books will be returned as a json array of book objects.
handleSubmitClick checks if the state’s editingBook has an id and will do either a POST to create a new book or a PUT to update the existing one. Depending on the result of the operation will either reload books and clear the editingBook or set the error message.
handleDeleteClick will do a DELETE to delete the state’s editingBook and clear it.

Notice that all success and error functions above were binded to this so that they could update the state of the current BookPanel object.

After each succesfull ajax call we do a reload to the books to keep the state between server and client side consistent. We could instead update the book array immediately before doing the ajax call - however this would increase complexity (what should happen if an error happens at the ajax call) without improving the UX that much. A better solution would be to add a loading css animation.

Local state

One thing to keep in mind is that there should only be one place of truth for the state and that the state should be as high in the component hierarch as possible. All changes should be propagated to thhe child components through properties. That’s the only way to be sure of the your data flow: When a change should happen (either because of a user action or an asynchronous call e.g ajax or timeout) just go to the state-holding component up in the hierarchy and change its state. Then the changes will be propagated in a top-down fashion from that component to its children.

The above paragraph does not mean that the state should be contained in just a single component! For example, let’s say that we had an AuthorPanel in the same web application and both BookPanel and AuthorPanel would be contained in a BookAuthorPanel. In this case, we should keep a different state object for BookPanel and AuthorPanel, we wouldn’t need to combine them into a single object contained in the BookAuthorPanel since they are unrelated!

One decision that we took in BookForm (and also to the SearchPanel before) is to not keep a local state for the book that is edited but move it to BookPanel. This means that whenever the value of the title or category is changed the parent component will be informed (through handleChange) so that it will change its state and the new values will be passed down to BookForm as properties so our changes will be reflected to the inputs. To make it more clear, when you press a letter on the title input:

the onChange method of BookForm will be called,
it will get the values of both title and category fields
and call handleChange with these values.
The handleChange method of BookPanel will update the editingBook state attribute,
so the book property of BookForm will be also updated
and the new value of the title will be displayed (since the components will be re-rendered due to the state update)

Conceptually, the above seems like a lot of work for just pressing a simple key! However, due to how react is implemented (virtual DOM) it won’t actually introduce any performance problems in our application. If nevertheless we wanted to have a local state of the currently edited book inside the BookForm then we’d need to use state.book and update the state using the componentWillReceiveProps method of BookForm: If we have a book to edit in the properties then copy it to the state or else just create an empty book. Also, the onChange method of the BookForm won’t need to notify the parent component that there is a state change (but only update the local state) and of course when the submit button is pressed the current book should be passed to the parent component (to either save it or delete it) since it won’t know the book that is currently edited.

So we could either have a local state object for the edited book in the BookForm or just save it as a property of the global state object in BookPanel — both solutions are correct. What wouldn’t be correct is if we had two copies of the same information in both the states of BookPanel and BookForm, for example the book id that is edited.

Adding form validation

Commenter Nitish Kumar made a nice comment about how we could do some validating to our form fields.

First of all, this adds to the complexity of the application so maybe it would be better to do it in the flux-architectured version. However, it’s not really difficult to also have form validation using react-only:

Here’s what we need:

A property (or properties) to pass the errors for each form field to the BookForm component
A way to display the form errors if they exist in BookForm
Execute an error handler when the form contents change and update the state if there’s an error

As an example, I’d like to check that the book title is not written in uppercase, so I will add another attribute to the book propery of BookForm called titleError and display it if it is defined, something like this:

{this.props.book.titleError?<div>{this.props.book.titleError}</div>:null}

Now, the handleChange method of BookPanel should do the actual check and change the sate, so I’m changing it like this:

handleChange: function(title, category) {
    var titleError = undefined;
    if(title.length > 1 && title === title.toUpperCase()) {
        titleError='Please don\'t use only capital letters for book titles';
    }
    this.setState({
        editingBook: {
            title: title,
            titleError: titleError,
            category: category,
            id: this.state.editingBook.id
        }
    });
},

So, the titleError attribute of editingBook will be defined only if there’s an error.

Finally, we must remember to not create/update the book if there’s an error so we add the following check to the handleSubmitClick method:

if(this.state.editingBook.titleError) {
    this.setState({
        message: "Please fix errors!"
    });
    return ;
}

The above could be easily generalized for checks on multiple fields. I am not really fond of doing the form validation in the BookPanel component (I’d prefer to do it on BookForm or even better on the corresponding field - if it was a component) however this would mean that I’d need to keep a local state with the error to the BookForm (please see the discussion of the previous paragraph).

I’ve added a new tag named react-only-validation to the https://github.com/spapas/react-tutorial/ repository that contains the changes to have the validation.

Conclusion to the first part

We just created a not-so-simple React single page application offering CRUD for an object. This application of course could be improved by adding a pagination component to the table, but I’m going to leave that as an excersie to the reader: To do that, I propose to create another component named TablePager that would have two properties: currentPage and a changePageTo callback. The currentPage would also be needed to added to the global state. When the changePageTo` is called by the ``TablePager it would update the global currentPage and will do a reloadBooks that will use the currentPage to fetch the correct page.

For application state keeping, we have selected to store all state attributes in BookPanel, for every state changing action we create a method that we pass in the child components so that the children can call it when the state needs to be updated.

As we will see in the next part, with Flux, Facebook proposes a design pattern for where to store the state and how to update it. We’ll see how to convert our React application to also use Flux and find out how Flux will help us when developing complex client-side applications!

Using browserify and watchify to improve your client-side-javascript workflow

Serafeim Papastefanos — Wed, 27 May 2015 14:20:00 +0300

Contents

The problem
The solution
Installing required tools
Starting your (node.js) project
Running browserify for the first time
Using external libraries
Introducing watchify
Creating your own modules
Uglifying your bundle
The client-side javascript workflow
Conlusion

The problem

Once upon a time, when people started using client-side code to their projects they (mainly due to the lack of decent client-side libraries but also because of the NIH syndrome) were just adding their own code in script nodes or .js files using document.getElementById to manipulate the DOM (good luck checking for all possible browser-edge cases) and window.XMLHttpRequest to try doing AJAX.

After these dark-times, the age of javascript-framework came: prototype, jquery, dojo. These were (and still are) some great libraries (even NIH-sick people used them to handle browser incompatibilities): You just downloaded the .js file with the framework, put it inside your project static files and added it to your page with a script tag and then filled your client-side code with funny $(‘#id’) symbols!

Coming to the modern age in client-side development, the number of decent libraries has greatly increased and instead of monolithic frameworks there are different libraries for different needs. So instead of just downloading a single file and adding the script node for that file to your HTML, you need to download the required javascript files, put them all in your static files directory and then micro-manage the script nodes for each of your pages depending on which libraries each page needs! So if you want to use (for example) moment.js to your client-side code you need to go to all HTML pages that use that specific client-side code and add a moment.js-script element!

As can be understood this leads to really ugly situations like people avoiding refactoring their code to use external libraries, using a single-global module with all their client side code, using CDN to avoid downloading the javascript libraries and of course never upgrade their javascript libraries!

The solution

browserify and watchify are two sister tools from the server-side-javascript (node.js and friends) world that greatly improve your javascript workflow: Using them, you no longer need to micro-manage your script tags but instead you just declare the libraries each of your client-side modules is using - or you can even create your own reusable modules! Also, installing (or updating) javascript libraries is as easy as running a single command!

How are they working? With browserify you create a single main.js for each of your HTML pages and in it you declare its requirements using require. You’ll then pass your main.js through browserify and it will create a single file (e.g bundle.js) that contains all the requirements (of course each requirement could have other requirements - they’ll be automatically also included in the resulting .js file). That’s the only file you need to put to the script tag of your HTML! Using watchify, you can watch your main.js for changes (the changes may also be in the files included from main.js) and automatically generate the resulting bundle.js so that you’ll just need to hit F5 to refresh and get the new version!

browserify not only concatenates your javascript libraries to a single bundle but can also transform your coffesscript, typescript, jsx etc files to javascrpt and then also add them to the bundle. This is possible through a concept called transforms — there are a lot of transforms that you can use.

Below, I will propose a really simple and generic workflow that should cover most of your javascript needs. I should mention that I mainly develop django apps and my development machine is running windows, however you can easily use exactly the same workflow from any kind of server-side technology (ruby, python, javascript, java, php or even static HTML pages!) or development machine (windows, linux, osx) - it’s exactly the same!

Installing required tools

As already mentioned, you need two node.js tools. Just install them globally using npm (installing node.js and npm is really easy - there’s even a package for windows):

npm install -g browserify watchify

The -g switch installs the packages globally so you can use the browserify and watchify commands from your command prompt - after that entering browserify or watchify from your command prompt should be working.

Starting your (node.js) project

Although you may already have a project structure, in order to use browserify you’ll need to create a node.js project (project from now on) that needs just two things:

a package.json that lists various options for your project
a node_modules directory that contains the packages that your project uses

To create the package.json you can either copy paste a simple one or run npm init inside a folder of your project. After npm init you’ll need to answer a bunch of questions and then a package.json will be created to the same folder. If you don’t want to answer these questions (most probably you only want to use node.js for browserify - instead you wouldn’t be reading this) then just put an empty json string {} in package.json.

I recommend adding package.json to the top-level folder of your version-controlled souce-code tree - please put this file in your version control - the node_modules directory will be be created to the same directory with package.json and should be ignored by your version control.

Running browserify for the first time

Not the time has come to create a main.js file. This could be put anywhere you like (based on your project structure) - I” suppose that main.js is inside the src/ folder of your project. Just put a console.log("Hello, world") to the main.js for now. To test that everything is working, run:

browserify src/main.js

You should see some minified-js gibberish to your console (something like (function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a=typeof ...) ) which means that everything works fine. Now, create a dist directory which would contain your bundle files and run

browserify src/main.js -o dist/bundle.js

the -o switch will put the the same minified-js gibberish output to the dist/bundle.js file instead of stdout. Finally, include a script element with that file to your HTML and you should see "Hello, world" to your javascript console when opening the HTML file!

Using external libraries

To use a library from your main.js you need to install it and get a reference to it through require. Let’s try to use moment.js: To install the library run

npm install moment --save

This will create a moment directory inside node_modules that will contain the moment.js library. It will also add a dependency to your package.json (that’s what the --save switch does), something like this:

"dependencies": {
  "moment": "^2.10.3"
}

Whenever you install more client-side libraries they’ll be saved there. When you want to re-install everything (for instance when you clone your project) you can just do a

npm install

and all dependencies of package.json will be installed in node_modules (that’s why node_modules should not be tracked).

After you’ve installed moment.js to your project change src/main.js to:

moment = require('moment')
console.log(moment() );

and rerun browserify src/main.js -o dist/bundle.js. When you reload your HTML you’ll see the that you are able to use moment - all this without changing your HTML!!!

As you can understand, in order to use a library with browserify, this library must support it by having an npm package. The nice thing is that most libraries already support it — let’s try for another example to use underscore.js and (for some reason) we need version underscore 1.7 :

npm install underscore@1.7--save

you’ll se that your package.json dependencies will also contain underscore.js 1.7:

{
  "dependencies": {
    "moment": "^2.10.3",
    "underscore": "^1.7.0"
  }
}

If you want to upgrade underscore to the latest version run a:

npm install underscore --upgrade --save

and you’ll see that your package.json will contan the latest version of underscore.js.

Finally, let’s change our src/man.js to use underscore:

moment = require('moment')
_ = require('underscore')

_([1,2,3]).map(function(x) {
  console.log(x+1);
});

After you create your bundle you should se 2 3 4 in your console!

Introducing watchify

Running browserify every time you change your js files to create the bundle.js feels like doing repetitive work - this is where wachify comes to the rescue; watchify is a tool that watches your source code and dependencies and when a change is detected it will recreate the bundle automagically!

To run it, you can use:

watchify src/main.js -o dist/bundle.js -v

and you’ll see something like: 155544 bytes written to dist/bundle.js (0.57 seconds) — try changing main.js and you’ll see that bundle.js will also be re-written!

Some things to keep in mind with watchify usage:

The -v flag outputs the verbose text (or else you won’t se any postive messages) - I like using it to be sure that everything is ok.
You need to use the -o flag with watchify — you can’t output to stdout(we’ll see that this will change our workflow for production a bit later)
watchify takes the same parameters with browserify — so if you do any transformations with browserify you can also do them with watchify

In the following, I’ll assume that you are running the watchify src/main.js -o dist/bundle.js -d so your bundles will always be re-created when changes are found.

Creating your own modules

Using browserify we can create our own modules and require them in other modules using the module.exports mechanism!

Creating a module is really simple: In a normal javascript file either assign directly to module.exports or include all local objects you want to be visible as an attribute to module.exports — everything else will be private to the module.

As an example, let’s create an src/modules folder and put a file module1.js inside it, containing the following:

var variable = 'variable'
var variable2 = 'variable2'
var funct = function(x) {
  return x+1;
}
var funct2 = function(x) {
  return x+1;
}

module.exports['variable'] = variable
module.exports['funct'] = funct

As you see, although we’ve defined a number of things in that module, only the variable and funct attributes of module.exports will be visible when the module is used. To use the module, change main.js like this:

module1 = require('./modules/module1')
console.log(module1.funct(9))

When you refresh your HTML you’ll see 10 in the console. So, require will return the module.exports objects of each module. It will either search in your project’s node_modules (when you use just the modfule name, for example moment, or locally (when you start a path with either ./ or ../ — in our case we required the module module1.js from the folder modules).

As a final example, we’ll create another module that is used by module1: Create a file named module2.js inside the modules folder and the following contents:

var funct = function(x) {
    return x+1;
}

module.exports = funct

After that, change module1.js to this:

module2 = require('./module2')

var variable = 'variable'
var funct = function(x) {
    return module2(x)+1;
}

module.exports['variable'] = variable
module.exports['funct'] = funct

So module1 will import the module2 module (from the same directory) and call it (since a function is assignedd to module.exports). When you refresh your HTML you should see 11!

Uglifying your bundle

If had taken a look at the file size of your bundle.js when you’d included moment.js or underscore.js you’d see that the file size has been greatly increased. Take a peek at bundle.js and you’ll see why: The contents of the module files will be concatenated as they are, without any changes! This may be nice for development / debugging, however for production we’d like our bundle.js to be minified — or uglyfied as it’s being said in the javascript world.

To help us with this, uglifying we’ll use uglify-js. First of all, please install it globally

npm install uglify-js -g

and you’ll be able to use the uglifyjs command to uglify your bundles! To use the uglifyjs command for your bundle.js try this

uglifyjs dist\bundle.js  > dist\bundle.min.js

and you’ll see the size of the bundle.min.js greatly reduced! To achieve even better minification (and code mangling as an added bonus) you could pass the -mc options to uglify:

uglifyjs dist\bundle.js -mc > dist\bundle.min.js

and you’ll see an even smaller bundle.min.js!

As a final step, we can combine the output of browserify and uglify to a single command using a pipe:

browserify src/main.js | uglifyjs -mc > dist/bundle.js

this will create the uglified bundle.js! Using the pipe to output to uglifyjs is not possible with watchify since watchify cannot output to stdout — however, as we’ll see in the next section this is not a real problem.

The client-side javascript workflow

The proposed client-side javascript workflow uses two commands, one for the development and one for creating the production bundle.

For the development, we’ll use watchify since we need to immediately re-create the bundle when a javascript source file is changed and we don’t want any uglification:

watchify src/main.js -o dist/bundle.js -v

For creating our production bundle, we’ll use browserify and uglify:

browserify src/main.js  | uglifyjs -mc warnings=false > dist/bundle.js

(i’ve added warnings=false to uglfiyjs to suppress warnings).

The above two commands can either be put to batch files or added to your existing workflow (for example as fabric commands if you use fabric). However, since we already have a javascrpt project (i.e a package.json) we can use that to run these commands. Just add a scripts section to your package.json like this:

{
  "dependencies": {
    "moment": "^2.10.3",
    "underscore": "^1.8.3"
  },
  "scripts": {
    "watch": "watchify src/main.js -o dist/bundle.js -v",
    "build": "browserify src/main.js  | uglifyjs -mc warnings=false > dist/bundle.js"
  }
}

and you’ll be able to run npm run watch to start watchifying for changes and npm run build to create your production bundle!

Conlusion

In the above we saw two (three if we include uglifyjs) javascript tools that will greatly improve our javascript workflow. Using these we can easily require (import) external javascript libraries to our project without any micromanagement of script tags in html files. We also can seperate our own client-side code to self-contained modules that will only export interfaces and not pollute the global namespace. The resulting production client-side javascript file will be output minimized and ready to be used by the users’ browsers.

All the above are possible with minimal changes to our code and development workflow:

create a package.json and install your dependencies
require the external libraries (instead of using them off the global namespace)
define your module’s interace through module.exports (instead of polluting the global namespace)
change your client javascript files to bundle.js
run npm run watch when developing and npm run build before deploying

Show 404 page on django when DEBUG=True

Serafeim Papastefanos — Wed, 29 Apr 2015 10:20:00 +0300

The default 404 error page on django can be easily overriden by adding a template named 404.html to the top level directory of your templates. However, on your development environment you’ll never be able to see this template because when DEBUG=True django will render the debug not found page to help you debug your url configuration.

If you want to display that page in your development environment you can always change the DEBUG setting to False, however there’s a better way: Add a url pattern for django’s default 404 view - just add the following to your urls.py:

import django.views.defaults

urlpatterns = patterns('',
    # Other url patterns ...
    url(r'^404/$', django.views.defaults.page_not_found, ),
)

You’ll then be able to see your 404 page by visiting the defined URL!

Calling the REST API of Pusher from python

Serafeim Papastefanos — Fri, 06 Feb 2015 12:20:00 +0200

Introduction

Pusher is one of the best real time frameworks right now. Using it you can add real time events in your projects without the need to configure and use HTTP servers that support real-time events in your environment. I used it recently in a project and it worked really good, having a very simple API and a nice interface for debugging your requests.

The only problem I’ve found was that the Pusher python API misses some features that the APIs for other languages have, specifically finding out the users on a presence channel.

Pusher supports real-time events through the use of "channels". Each pusher client will subscribe to a channel and receive messages that are sent to that channel. A special kind of channel are presence channels which keep a list of their subscribers. You can query the Pusher REST API (or f.e the Pusher Javascript API) to find out the names of the users in a presence channel - however this is not currently possible with the python API.

Unfortuanately, calling the Pusher REST API is not so easy, since it needs a complicated singining of each request, so I’ve written this post to help developers that need to call this API from python (to get the users of a presence channel or for any other method the REST API supports).

Signing the request

Quoting from the Pusher REST API, to sign a request we need a signature, which:

The signature is a HMAC SHA256 hex digest. This is generated by signing a string made up of the following components concatenated with newline characters \n:

The uppercase request method (e.g. POST)

The request path (e.g. /some/resource)

The query parameters sorted by key, with keys converted to lowercase, then joined as in the query string. Note that the string must not be url escaped (e.g. given the keys auth_key: foo, Name: Something else, you get auth_key=foo&name=Something else)

So, we need to create a string and then sign it using our Pusher api_key and secret. To help with this, we create a Token class which will be initialzed with out pusher key/secret and correctly sign a string:

class Token(object,):
    def __init__(self, key, secret):
        self.key = key
        self.secret = secret

    def sign(self, string):
        return hmac.new(self.secret, string, hashlib.sha256).hexdigest()

It uses the hmac and hashlib python modules.

Generating the complete query string

We can now create a function that will sign a request using an instance of the above token:

def create_signed_query_string(token, partial_path, request_params):
    params = {
        'auth_key': token.key,
        'auth_timestamp': int(time.time()),
        'auth_version': '1.0'
    }
    params.update(request_params)
    keys = sorted(params.keys() )
    params_list = []
    for k in keys:
        params_list.append( '{0}={1}'.format(k, params[k]) )

    query_string = '&'.join(params_list)

    sign_data = '\n'.join(['GET', partial_path, query_string])
    query_string += '&auth_signature=' + token.sign(sign_data);
    return query_string

create_signed_query_string receives an instance of a Token, the path that we want to request without the server part (for example /apps/33/users/my-channel) and a dictionary of request parameters. It then adds three extra fields to the request parameters (auth_key, auth_timestamp, auth_version) and creates a list of these parameters in the key=value form, where the keys are alphabetically sorted. After that it joins the above key=value parameters using & to create the query_string and then it creates the string to be signed (sign_data) by concatenating the HTTP methdo (GET) with the path and the query_string. Finally, it appends the signing result as an extra query parameter named (auth_signature).

Requesting the users of the presence channel

The create_signed_query_string can now be used to get the users of a presence channel like this:

def get_users(app_id, key, secret, channel):
    partial_path =  '/apps/{0}/channels/{1}/users'.format(app_id, channel)
    token = Token(key, secret)
    qs = create_signed_query_string(token, partial_path, {})
    full_path = 'http://api.pusherapp.com/{0}?{1}'.format(partial_path, qs)
    r = requests.get(full_path)
    return r.text

The get_users function will generate the path of the pusher REST API (using our pusher app_id and channel name) and initialize a signing Token using the pusher key and secret. It will then pass the previous to create_signed_query_string to generate the complete query_string and generate the full_path to which a simple HTTP GET request is issued. The result will be a JSON list of the users in the presence channel.

Complete example

A complete example of getting the presence users of a channel is the following:

import time
import hashlib
import hmac
import requests

app_id = 'pusher_app_id'
key = 'pusher_key'
secret = 'pusher_secret'
channel = 'pusher_presence_channel'


class Token(object,):
    def __init__(self, key, secret):
        self.key = key
        self.secret = secret

    def sign(self, string):
        return hmac.new(self.secret, string, hashlib.sha256).hexdigest()


def create_signed_query_string(token, partial_path, method, request_params):
    params = {
        'auth_key': token.key,
        'auth_timestamp': int(time.time()),
        'auth_version': '1.0'
    }
    params.update(request_params)
    keys = sorted(params.keys() )
    params_list = []
    for k in keys:
        params_list.append( '{0}={1}'.format(k, params[k]) )

    query_string = '&'.join(params_list)

    sign_data = '\n'.join([method, partial_path, query_string])
    query_string += '&auth_signature=' + token.sign(sign_data);
    return query_string


def get_users(channel):
    partial_path =  '/apps/{0}/channels/{1}/users'.format(app_id, channel)
    token = Token(key, secret)
    qs =  create_signed_query_string(token, partial_path, 'GET' {})
    full_path = 'http://api.pusherapp.com/{0}?{1}'.format(partial_path, qs)
    r = requests.get(full_path)
    return r.text

print get_users(channel)

Conclusion

With the above we are able to not only easily get the users of a Pusher presence channel in python but to also call any method we want from the Pusher REST API by implementing a function similar to get_users.

Asynchronous tasks in django with django-rq

Serafeim Papastefanos — Tue, 27 Jan 2015 14:20:00 +0200

Contents

Introduction
Job queues in python
django-test-rq
rqworker and rqscheduler
- For development
- For production
Conclusion

Update 01/09/15: I’ve written a new post about rq and django with some more advanced techniques !

Introduction

Job queuing (asynchronous tasks) is a common requirement for non-trivial django projects. Whenever an operation can take more than half a second it should be put to a job queue in order to be run asynchronously by a seperate worker. This is really important since the response to a user request needs to be immediate or else the users will experience laggy behavior and start complaining! Even for fairly quick tasks (like sending email through an SMTP server) you need to use an asynchronous task if you care about your users since the time required for such a task is not really limited.

Using job queues is involved not only for the developers of the application (who need to create the asynchronous tasks and give feedback to the users when the’ve finished since they can’t use the normal HTTP response) and but also for the administrators, since, in order to support job queues at least two more componets will be needed:

One job queue that will store the jobs to be executed next in a first in first queue. This could be the normal database of the project however it’s not recommended for performance reasons and most of thetimes it is a specific component called "Message Broker"
One (or more) workers that will monitor the job queue and when there is work to do they will dequeue and execute it

These can all run in the same server but if it gets saturated they can easily be seperated (even more work for administrators).

Beyond job queuing, another relative requirement for many projects is to schedule a task to be run in the future (similar to the at unix command) or at specific time intervals (similar to the cron unix command). For instance, if a user is registered today we may need to check after one or two days if he’s logged in and used our application - if he hasn’t then probably he’s having problems and we can call him to help him. Also, we could check every night to see if any users that have registered to our application don’t have activated their account through email activation and delete these accounts. Scheduled tasks should be also run by the workers mentioned above.

Job queues in python

The most known application for using job queues in python is celery which is a really great project that supports many brokers, integrates nicely with python/django (but can be used even with other languages) and has many more features (most of them are only useful on really big, enterprise projects). I’ve already used it in a previous application, however, because celery is really complex I found it rather difficult to configure it successfully and I never was perfectly sure that my asynchronous task would actually work or that I’d used the correct configuration for my needs!

Celery also has many dependencies in order to be able to talk with the different broker backends it supports, improve multithreading support etc. They may be required in enterprise apps but not for most Django web based projects.

So, for small-to-average projects I recommend using a different asynchronous task solution instead of celery, particularly (as you’ve already guessed from the title of this post) RQ. RQ is simpler than celery, it integrates great with django using the excellent django-rq package and doesn’t actually have any more dependencies beyond redis support which is used as a broker (however most modern django projects already use redis for their caching needs as an alternative to memcached).

It even supports supports job scheduling through the rq-scheduler package (celery also supports job scheduling through celery beat): Run a different process (scheduler) that polls the job scheduling queue for any jobs that need to be run because of scheduling and if yes put them to the normal job queue.

Although RQ and frieds are really easy to use (and have nice documentation) I wasn’t able to find a complete example of using it with django, so I’ve implemented one (found at https://github.com/spapas/django-test-rq — since I’ve updated this project a bit with new stuff, please checkout tag django-test-rq-simple git checkout django-test-rq-simple) mainly for my own testing purposes. To help others that want to also use RQ in their project but don’t know from where to start, I’ll present it in the following paragraphs, along with some comments on how to actually use RQ in your production environment.

django-test-rq

This is a simple django project that can be used to asynchronously run and schedule jobs and examine their behavior. The job to be scheduled just downloads a provided URL and counts its length. There is only one django application (tasks) that contains two views, one to display existing tasks and create new ones and one to display some info for the jobs.

models.py

Two models (Task and ScheduledTask) for saving individual tasks and scheduled tasks and one model (ScheduledTaskInstance) to save scheduled instances of each scheduled task.

from django.db import models
import requests
from rq import get_current_job


class Task(models.Model):
    # A model to save information about an asynchronous task
    created_on = models.DateTimeField(auto_now_add=True)
    name = models.CharField(max_length=128)
    job_id = models.CharField(max_length=128)
    result = models.CharField(max_length=128, blank=True, null=True)


class ScheduledTask(models.Model):
    # A model to save information about a scheduled task
    created_on = models.DateTimeField(auto_now_add=True)
    name = models.CharField(max_length=128)
    # A scheduled task has a common job id for all its occurences
    job_id = models.CharField(max_length=128)


class ScheduledTaskInstance(models.Model):
    # A model to save information about instances of a scheduled task
    scheduled_task = models.ForeignKey('ScheduledTask')
    created_on = models.DateTimeField(auto_now_add=True)
    result = models.CharField(max_length=128, blank=True, null=True)

forms.py

A very simple form to create a new task.

from django import forms

class TaskForm(forms.Form):
    """ A simple form to read a url from the user in order to find out its length
    and either run it asynchronously or schedule it schedule_times times,
    every schedule_interval seconds.
    """
    url = forms.CharField(label='URL', max_length=128, help_text='Enter a url (starting with http/https) to start a job that will download it and count its words' )
    schedule_times = forms.IntegerField(required=False, help_text='How many times to run this job. Leave empty or 0 to run it only once.')
    schedule_interval = forms.IntegerField(required=False, help_text='How much time (in seconds) between runs of the job. Leave empty to run it only once.')

    def clean(self):
        data = super(TaskForm, self).clean()
        schedule_times = data.get('schedule_times')
        schedule_interval = data.get('schedule_interval')

        if schedule_times and not schedule_interval or not schedule_times and schedule_interval:
            msg = 'Please fill both schedule_times and schedule_interval to schedule a job or leave them both empty'
            self.add_error('schedule_times', msg)
            self.add_error('schedule_interval', msg)

views.py

This is actually very simple if you’re familiar with Class Based Views. Two CBVs are defined, one for the Task form + Task display and another for the Job display.

from django.views.generic.edit import FormView
from django.views.generic import TemplateView
from forms import TaskForm
from tasks import get_url_words, scheduled_get_url_words
from models import Task,ScheduledTask
from rq.job import Job
import django_rq
import datetime

class TasksHomeFormView(FormView):
    """
    A class that displays a form to read a url to read its contents and if the job
    is to be scheduled or not and information about all the tasks and scheduled tasks.

    When the form is submitted, the task will be either scheduled based on the
    parameters of the form or will be just executed asynchronously immediately.
    """
    form_class = TaskForm
    template_name = 'tasks_home.html'
    success_url = '/'

    def form_valid(self, form):
        url = form.cleaned_data['url']
        schedule_times = form.cleaned_data.get('schedule_times')
        schedule_interval = form.cleaned_data.get('schedule_interval')

        if schedule_times and schedule_interval:
            # Schedule the job with the form parameters
            scheduler = django_rq.get_scheduler('default')
            job = scheduler.schedule(
                scheduled_time=datetime.datetime.now(),
                func=scheduled_get_url_words,
                args=[url],
                interval=schedule_interval,
                repeat=schedule_times,
            )
        else:
            # Just execute the job asynchronously
            get_url_words.delay(url)
        return super(TasksHomeFormView, self).form_valid(form)

    def get_context_data(self, **kwargs):
        ctx = super(TasksHomeFormView, self).get_context_data(**kwargs)
        ctx['tasks'] = Task.objects.all().order_by('-created_on')
        ctx['scheduled_tasks'] = ScheduledTask.objects.all().order_by('-created_on')
        return ctx


class JobTemplateView(TemplateView):
    """
    A simple template view that gets a job id as a kwarg parameter
    and tries to fetch that job from RQ. It will then print all attributes
    of that object using __dict__.
    """
    template_name = 'job.html'

    def get_context_data(self, **kwargs):
        ctx = super(JobTemplateView, self).get_context_data(**kwargs)
        redis_conn = django_rq.get_connection('default')
        try:
            job = Job.fetch(self.kwargs['job'], connection=redis_conn)
            job = job.__dict__
        except:
            job = None

        ctx['job'] = job
        return ctx

tasks.py

Here two jobs are defined: One to be used for simple asynchronous tasks and the other to be used for scheduled asynchronous tasks (since for asynchronous tasks we wanted to group their runs per job id).

The @job decorator will add the delay() method (used in views.py) to the function. It’s not really required for scheduled_get_url_words since it’s called through the scheduled.schedule.

When a task is finished, it can return a value (like we do in return task.result) which will be saved for a limited amount of time (500 seconds by default - could be even saved for ever) to redis. This may be useful in some cases, however, I think that for normal web applications it’s not that useful, and since here we use normal django models for each task, we can save it to that model’s instance instead.

import requests
from models import Task, ScheduledTask, ScheduledTaskInstance
from rq import get_current_job
from django_rq import job


@job
def get_url_words(url):
    # This creates a Task instance to save the job instance and job result
    job = get_current_job()

    task = Task.objects.create(
        job_id=job.get_id(),
        name=url
    )
    response = requests.get(url)
    task.result = len(response.text)
    task.save()
    return task.result


@job
def scheduled_get_url_words(url):
    """
    This creates a ScheduledTask instance for each group of
    scheduled task - each time this scheduled task is run
    a new instance of ScheduledTaskInstance will be created
    """
    job = get_current_job()

    task, created = ScheduledTask.objects.get_or_create(
        job_id=job.get_id(),
        name=url
    )
    response = requests.get(url)
    response_len = len(response.text)
    ScheduledTaskInstance.objects.create(
        scheduled_task=task,
        result = response_len,
    )
    return response_len

settings.py

import os
BASE_DIR = os.path.dirname(os.path.dirname(__file__))

SECRET_KEY = '123'
DEBUG = True
TEMPLATE_DEBUG = True
ALLOWED_HOSTS = []

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'django_extensions',
    'django_rq',

    'tasks',
)

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

ROOT_URLCONF = 'django_test_rq.urls'
WSGI_APPLICATION = 'django_test_rq.wsgi.application'

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True

STATIC_URL = '/static/'

# Use redis for caches
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/0",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    }
}

# Use the same redis as with caches for RQ
RQ_QUEUES = {
    'default': {
        'USE_REDIS_CACHE': 'default',
    },
}

SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_CACHE_ALIAS = "default"
RQ_SHOW_ADMIN_LINK = True

# Add a logger for rq_scheduler in order to display when jobs are queueud
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '%(asctime)s %(levelname)s %(message)s'
        },
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
    },

    'loggers': {
        'django.request': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'rq_scheduler': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

By default, rq_scheduler won’t log anything so we won’t be able to see any output when new instances of each scheduled task are queued for execution. That’s why we’ve overriden the LOGGING setting in order to actually log rq_scheduler output to the console.

Running the project

I recommend using Vagrant to start a stock ubuntu/trusty32 box. After that, install redis, virtualenv and virtualenvwrapper and create/activate a virtualenv named rq. You can go to the home directory of django-test-rq and install requirements through pip install requirements.txt and create the database tables with python manage.py migrate. Finally you may run the project with python manage.py runserver_plus.

rqworker and rqscheduler

Before scheduling any tasks we need to run two more processes:

rqworker: This is a worker that dequeues jobs from the queue and executes them. We could run more than one onstance of this job if we need it.
rqscheduler: This is a process that runs every one minute and checks if there are scheduled jobs that have to be executed. If yes, it will add them to the queue in order to be executed by a worker.

For development

If you want to run rqworker and rqscheduler for your development environment you can just do it with running python manage.py rqworker and python mange.py rqscheduler through screen/tmux. If everything is allright you should see tasks being added to the queue and scheduled (you may need to refresh the homepage before seeing everything since a task may be executed after the response is created).

Also, keep in mind that rqscheduler runs once every minute by default so you may need to wait up to minute to see a ScheduledTask instance. Also, this means that you can’t run more than one scheduled task instance per minute.

For production

Trying to create daemons through screen is not sufficient for a production envornment since we’d like to actually have logging, monitoring and of course automatically start rqworker and rqscheduler when the server boots.

For this, I recommend using the supervisord tool which can be used to monitor and control a number of processes. There are other similar tools, however I’ve found supervisord the easier to use.

In order to monitor/control a process through supervisord you need to add a [program:progrname] section in supervisord’s configuration and pass a number of parameters. The progname is the name of the monitoring process. Here’s how rqworker can be configured using supervisord:

[program:rqworker]
command=python manage.py rqworker
directory=/vagrant/progr/py/rq/django-test-rq
environment=PATH="/home/vagrant/.virtualenvs/rq/bin"
user=vagrant

The options used will chdir to directory and execute command as user. The environment option can be used to set envirotnment variables - here we set PATH in order to use a specific virtual environment. This will allow you to monitor rqworker through supervisord and log its output to a file in /var/log/supervisor (by default). A similar entry needs to be added for rqscheduler of course. If everything has been configured correctly, when you reload the supervisord settings you can run sudo /usr/bin/supervisorctl and should see something like

rqscheduler                      RUNNING    pid 1561, uptime 0:00:03
rqworker                         RUNNING    pid 1562, uptime 0:00:03

Also, tho log files should contain some debug info.

Conclusion

Although using job queues makes it more difficult for the developer and adds at least one (and probably more) points of failure to a project (the workers, the broker etc) their usage, even for very simple projects is unavoidable.

Unless a complex, enterprise solution like celery is really required for a project I recommend using the much simpler and easier to configure RQ for all your asynchronous and scheduled task needs. Using RQ (and the relative projects django-rq and rq-scheduler) we can easily add production ready queueued and scheduled jobs to any django project.

In this article we presented a small introduction to RQ and its friends and saw how to configure django to use it in a production ready environment using a small django project (https://github.com/spapas/django-test-rq) which was implemented as a companion to help readers quickly test the concepts presented here.

Django model auditing

Serafeim Papastefanos — Wed, 21 Jan 2015 14:20:00 +0200

Contents

Introduction
Adding simple auditing functionality ourselves
Example
- models.py
- forms.py
- views.py
- urls.py
- templates
Using django-simple-history
Using django-reversion
Conclusion

Introduction

An auditing trail is a common requirement in most non-trivial applications. Organizations need to know who did the change, when it was done and what was actually changed. In this post we will see three different solution in order to add this functionality in Django: doing it ourselves, using django-simple-history and using django-reversion.

Update 24/09/2015: Added a paragraph describing the django-reversion-compare which is a great addon for django-reversion that makes finding differences between versions a breeze!

Adding simple auditing functionality ourselves

A simple way to actually do auditing is to keep four extra fields in our models: created_by, created_on, modified_by and modified_on. The first two will be filled when the model instance is created while the latter two will be changed whenever the model instance is saved. So we only have who and whe. Sometimes, these are enough so let’s see how easy it is to implement it in django.

We’ll need an abstract model that could be used as a base class for models that need auditing:

from django.conf import settings
from django.db import models

class Auditable(models.Model):
    created_on = models.DateTimeField(auto_now_add = True)
    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, related_name='created_by')

    modified_on = models.DateTimeField(auto_now = True)
    modified_by = models.ForeignKey(settings.AUTH_USER_MODEL, related_name='modified_by')

    class Meta:
        abstract = True

Models inheriting from Auditable will contain their datetime of creation and modification which will be automatically filled using the very usefull auto_now_add_ (which will set the current datetime when the model instance is created) and auto_now_ (which will set the current datetime when the model instance is modified).

Such models will also have two foreign keys to User, one for the user that created the and one of the user that modified them. The problem with these two fields is that they cannot be filled automatically (like the datetimes) because the user that actually did create/change the objects must be provided!

Since I am really fond of CBVs I will present a simple mixin that can be used with CreateView and UpdateView and does exactly that:

class AuditableMixin(object,):
    def form_valid(self, form, ):
        if not form.instance.created_by:
            form.instance.created_by = self.request.user
        form.instance.modified_by = self.request.user
        return super(AuditableMixin, self).form_valid(form)

The above mixin overrides the form_valid method of CreateView and UpdateView: First it checks if the object is created (if it is created it won’t be saved in the database yet thus it won’t have an id) in order to set the created_by attribute to the current user. After that it will set the modified_by attribute of the object to the current user. Finally, it will call the next form_valid method to do whatever is required (save the model instance and redirect to success_url by default).

The views using AuditableMixin should allow only logged in users (or else an exception will be thrown). Also, don’t forget to exclude the created_by and modified_by fields from your model form (created_on and modified_on will automatically be excluded).

Example

Let’s see a simple example of creating a small django application using the previously defined abstract model and mixin:

models.py

from django.conf import settings
from django.core.urlresolvers import reverse
from django.db import models

from auditable.models import Auditable


class Book(Auditable):
    name = models.CharField(max_length=128)
    author = models.CharField(max_length=128)

    def get_absolute_url(self):
        return reverse("book_list")

In the above we suppose that the Auditable abstract model is imported from the auditable.models module and that a view named book_list that shows all books exists.

forms.py

from django.forms import ModelForm


class BookForm(ModelForm):
    class Meta:
        model = Book
        fields = ['name', 'author']

Show only name and author fields (and not the auditable fields) in the Book ModelForm.

views.py

from django.views.generic.edit import CreateView, UpdateView
from django.views.generic import ListView

from auditable.views import AuditableMixin

from models import Book
from forms import BookForm


class BookCreateView(AuditableMixin, CreateView):
    model = Book
    form_class = BookForm


class BookUpdateView(AuditableMixin, UpdateView):
    model = Book
    form_class = BookForm


class BookListView(ListView):
    model = Book

We import the AuditableMixin from auditable.views and make our Create and Update views inherit from this mixin also in addition to CreateView and UpdateView. Pay attention that our mixin is placed before CreateView in order to call form_valid in the proper order: When multiple inheritance is used like this python will check each class from left to right to find the proper method and call it. For example, in our BookCreateView, when the form_valid method is called, python will first check if BookCreateView has a form_valid method. Since it does not, it will check if AuditableMixin has a form_valid method and call it. Now, we are calling the super(...).form_valid() in the AuditableMixin form_valid, so the form_valid of CreateView will also be called.

A simple ListView is also added to just show the info on all books.

urls.py

from django.conf.urls import patterns, include, url

from views import BookCreateView, BookUpdateView, BookListView

urlpatterns = patterns('',
    url(r'^accounts/login/$', 'django.contrib.auth.views.login', ),
    url(r'^accounts/logout/$', 'django.contrib.auth.views.logout', ),

    url(r'^create/$', BookCreateView.as_view(), name='create_book'),
    url(r'^update/(?P<pk>\d+)/$', BookUpdateView.as_view(), name='update_book'),
    url(r'^$', BookListView.as_view(), name='book_list'),
)

Just add the previously defined Create/Update/List views along with a login/logout views.

templates

You’ll need four templates:

books/book_list.html: Show the list of books
books/book_form.html: Show the book editing form
registration/login.html: Login form
registration/logout.html: Logout message

Using django-simple-history

django-simple-history can be used to not only store the user and date of each modification but a different version for each modification. To do that, for every model that is registered to be used with django-simple-history, it wil create a second table in the database hosting all versions (historical records) of that model. As we can understand this is really powerfull since we can see exactly what was changed and also do normal SQL queries on that!

Installation

To use django-simple-history in a project, after we do a pip install django-simple-history, we just need to add it to INSTALLED_APPS and add the simple_history.middleware.HistoryRequestMiddleware to the MIDDLEWARE_CLASSES list.

Finally, to keep the historical records for a model, just add an instace of HistoricalRecords to this model.

Example

For example, our previously defined Book model will be modified like this:

class SHBook(models.Model):
    name = models.CharField(max_length=128)
    author = models.CharField(max_length=128)

    def get_absolute_url(self):
        return reverse("shbook_list")

    history = HistoricalRecords()

When we run python manage.py makemigrations and migrate this, we’ll see that beyond the table for SHBook, a table for HistoricalSHBook will be created:

Migrations for 'sample':
  0002_historicalshbook_shbook.py:
    - Create model HistoricalSHBook
    - Create model SHBook

Let’s see the schema of historicalshbook:

CREATE TABLE "sample_historicalshbook" (
    "id" integer NOT NULL,
    "name" varchar(128) NOT NULL,
    "author" varchar(128) NOT NULL,
    "history_id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    "history_date" datetime NOT NULL,
    "history_type" varchar(1) NOT NULL,
    "history_user_id" integer NULL REFERENCES "auth_user" ("id")
);

So we see that it has the same fields as with SHBook (id, name, author) with the addition of the primary key (history_id) of this historical record, the date and user that did the change (history_date, history_user_id) and the type of the record (created / update / delete).

So, just by adding a HistoricalRecords() attribute to our model definition we’ll get complete auditing for the instance of that model

Usage

To find out information about the historical records we’ll just use the HistoricalRecords() attribute of that model:

For example, running SHBook.history.filter(id=1) will return all historical records of the book with id = 1. For each one of them we have can use the following:

get the user that made the change through the history_user attribute
get the date of the change through the history_date attribute
get the type of the change through the history_type attribute (and the corresponding get_history_type_dispaly)
get a model instance as it was then through the history_object attribute (in order to save() it and revert to this version)

Using django-reversion

django-reversion offers more or less the same functionality of django-simple-history by following a different philosophy: Instead of creating an extra table holding the history records for each model, it insteads converts all the fields of each model to json and stores that JSON in the database in a text field.

This has the advantage that no extra tables are created to the database but the disadvantage that you can’t easily query your historical records. So you may choose one or the other depending on your actual requirements.

Installation

To use django-reversion in a project, after we do a pip install django-reversion, we just need to add it to INSTALLED_APPS and add the reversion.middleware.RevisionMiddleware to the MIDDLEWARE_CLASSES list.

In order to save the revisions of a model, you need to register this model to django-reversion. This can be done either through the django-admin, by inheriting the admin class of that model from reversion.VersionAdmin or, if you don’t want to use the admin by reversion.register decorator.

Example

To use django-reversion to keep track of changes to Book we can modify it like this:

@reversion.register
class RBook(models.Model):
    name = models.CharField(max_length=128)
    author = models.CharField(max_length=128)

    def get_absolute_url(self):
        return reverse("rbook_list")

django-reversion uses two tables in the database to keep track of revisions: revision and version. Let’s take a look at their schemata:

.schema reversion_revision
CREATE TABLE "reversion_revision" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    "manager_slug" varchar(200) NOT NULL,
    "date_created" datetime NOT NULL,
    "comment" text NOT NULL,
    "user_id" integer NULL REFERENCES "auth_user" ("id")
);

.schema reversion_version
CREATE TABLE "reversion_version" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    "object_id" text NOT NULL,
    "object_id_int" integer NULL,
    "format" varchar(255) NOT NULL,
    "serialized_data" text NOT NULL,
    "object_repr" text NOT NULL,
    "content_type_id" integer NOT NULL REFERENCES "django_content_type" ("id"),
    "revision_id" integer NOT NULL REFERENCES "reversion_revision" ("id")
);

As we can understand, the revision table holds information like who created this revison (user_id) and when (date_created) while the version stores a reference to the object that was modified (through a GenericForeignKey) and the actual data (in the serialized_data field). By default it uses JSON to serialize the data (the serialization format is in the format field). There’s an one-to-one relation between revision and version.

If we create an instance of RBook we’ll see the following in the database:

sqlite> select * from reversion_revision;
1|default|2015-01-21 10:31:25.233000||1

sqlite> select * from reversion_version;
1|1|1|json|[{"fields": {"name": "asdasdasd", "author": "asdasd"}, "model": "sample.rbook", "pk": 1}]|RBook object|12|1

date_created and user_id are stored on revision while format, serialized_data, content_type_id and object_id_int (the GenericForeignKey) are stored in version.

Usage

To find out information about an object you have to use the reversion.get_for_object(object) method. In order to be easily used in templates I recommend creating the following get_versions() method in each model that is registered with django-reversion

def get_versions(self):
    return reversion.get_for_object(self)

Now, each version has a revision attribute for the corresponding revision and can be used to do the following:

get the user that made the change through the revision.user attribute
get the date of the change through the revision.date_created attribute
get the values of the object fields as they were in this revision using the field_dict attribute
get a model instance as it was on that revision using the object_version.object attribute
revert to that previous version of that object using the revert() method

Comparing versions with django-reversion-compare

A great addon for django-version is django-reversion-compare which helps you find out differences between versions of your objects. When you use django-reversion-compare, you’ll be able to select two (different) versions of your object and you’ll be presented with a list of all the differences found in the fields of that object between the two versions. The diff algorithm is smart, so you’ll be able to easily recognise the changes.

To use django-reversion-compare, after installing it you should just inherit your admin views from reversion_compare.admin.CompareVersionAdmin (instead of reversion.VersionAdmin) and you’ll get the reversion-compare views instead of reversion views in the admin for the history of the object.

Also, in case you need to give access to normal, non-admin users to the history of an object (this is useful for auditing reasons), you can use the reversion_compare.views.HistoryCompareDetailView as a normal DetailView to create a non-admin history and compare diff view.

Conclusion

In the above we say that it is really easy to add basic (who and when) auditing capabilities to your models: You just need to inherit your models from the Auditable abstract class and inherit your Create and Update CBVs from AuditableMixin. If you want to know exactly what was changed then you have two solutions: django-simple-history to create an extra table for each of your models so you’ll be able to query your historical records (and easily extra aggregates, statistics etc) and django-reversion to save each version as a json object, so no extra tables will be created.

All three solutions for auditing have been implemented in a sample project at https://github.com/spapas/auditing-sample.

You can clone the project and, preferrably in a virtual environment, install requirements (pip install -r requirements.txt), do a migrate (python manage.py migrate — uses sqlite3 by default) and run the local development server (python manage.py ruinserver).

Change the primary color of bootstrap material design

Serafeim Papastefanos — Tue, 16 Dec 2014 16:20:00 +0200

Introduction

Bootstrap Material Design is a great theme that sits on top of Bootstrap and transforms it to Material Design! The great thing about Bootstrap Material Design is that you just need to include its css and js files after your Bootstrap files and …

boom! Your page is Material Design compatible!

A nice feature of Bootstrap Material Design is that you can change its default color to a new one (I don’t really like the current - greenish one). This is easy for people with less skills however I found it rather challenging when I tried it. That’s why I will present a step by step tutorial on changing the default primary color of the Bootstrap Material Design theme:

Step 1: Get the code

Use git to make a local clone of the project with git clone https://github.com/FezVrasta/bootstrap-material-design.git. This will create a directory named bootstrap-material-design. Or you can download the latest version of the code using (https://github.com/FezVrasta/bootstrap-material-design/archive/master.zip) and unzip it to the bootstrap-material-design directory.

Step 2: Install node.js and npm

You need to have node.js and npm installed in your system - this is something very easy so I won’t go into any details about this. After you have installed both node.js and npm you need to put them in your path so that you’ll be able to run npm -v without errors and receive something like 1.4.14.

Step 3: Install less

less is a CSS preprocessor in which Bootstrap Material Design has been written. To install it, just enter the command npm install -g less. After that you should have a command named lessc which, when run would output something like: lessc 2.1.1 (Less Compiler) [JavaScript].

Step 4: Create the customizations files

Go to the directory where you cloned (or unzipped) the Bootstrap Material Design code and create a file named custom.less (so, that file should be in the same folder as with bower.json, Gruntfile.js etc) with the following contents:

@import "less/material.less";

// Override @primary color with one took from _colors.less
@primary: @indigo;

(I wanted to use the indigo color as my primary one - you may of course use whichever color from the ones defined in less/_variables.less you like)

This file may contain other default values for variables - if I find anything useful I will add it to this post (also please reply with any recommendations).

Update 13/10/2015 After a request from commenter Jofferson Ramirez Tiquez, here’s a custom.less that overrides more colors from _variables.css (beyond the primary color, it changes the success color to teal and info and warning to the corresponding hex color values):

@import "less/material.less";

@primary: @indigo;
@success: @teal;
@info: #CFD8DC;
@warning:#455A64;

Step 5: Create your custom material css file

Finally, run the following command: lessc custom.less > material-custom.css. This will create a file named material-custom.css that contains your custom version of Bootstrap Material Design! If you want your material-custom.css to be compressed, add the -x option like this: lessc -x custom.less > material-custom.css.

You may now include material-custom.css instead of material.css (or the minified version of it) to your projects and you’ll have your own primary color!

Retrieving Gmail blocked attachments

Serafeim Papastefanos — Thu, 23 Oct 2014 14:20:00 +0300

Before services like Dropbox were widely available, some people (including me) were using their Gmail account as a primitive backup solution: Compress your directory and send it to your gmail. There. Backup complete.

However, nothing is so easy…

Recently, I wanted to retrieve one of these backups, a .rar containing the complete source code (since it was written in TeX) of my PhD thesis. The problem was that Gmail blocked the access to these attachments saying

Anti-virus warning - 1 attachment contains a virus or blocked file. Downloading this attachment is disabled.

probably because I had a number of .bat files inside that .rar archive to automate my work :(

Now what ?

After searching the internet and not founding any solutions, I tried the options that gmail gives for each email. One particular one cought my interest: Show original

Clicking this option opened a text file with the original, MIME encoded message. The interesting thing of course was

------=_NextPart_000_004F_01CA0AED.E63C2A30
Content-Type: application/octet-stream;
      name="phdstuff.rar"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
      filename="phdstuff.rar"

UmFyIRoHAM+QcwAADQAAAAAAAAB0f3TAgCwANAMAAFQEAAACRbXCx8lr9TodMwwAIAAAAG5ld2Zp
bmFsLnR4dA3dEQzM082BF7sB+D3q6QPUNEfwG7vHQgNkiQDTkGvfhOE4mNltIJJlBFMOCQPzPeKD
...

So the whole attachment was contained in that text file, encoded in base64! Now I just needed to extract it from the email and convert it back to binary.

Important: Before going the python way, please check the 2 June 2015 update at the end of the article for an easier solution!

This was very easy to do using Python - some people had already asked the same thing on SO. So here’s a simple program that gets an email in text/mime format as input and dumps all attachments:

import email
import sys

if __name__=='__main__':
    if len(sys.argv)<2:
        print "Please enter a file to extract attachments from"
        sys.exit(1)

    msg = email.message_from_file(open(sys.argv[1]))
    for pl in msg.get_payload():
        if pl.get_filename(): # if it is an attachment
            open(pl.get_filename(), 'wb').write(pl.get_payload(decode=True))

Save this to a file named get_attachments.py and, after saving the original message to a file named 0.txt run python get_attachments.py 0.txt and you’ll see the attachments of your email in the same folder!

Disclaimer: I have to warn you that since Gmail claims that an attachment is not safe it may be actually not safe. So you must be 100% sure that you know what you are doing before retrievening your email attachments like this.

Update: Stefan created an improved version of the attachment extractor which is also compatible with Python 3.4!

Update, 12 January 2015: Ivana (at the comments section) proposed a different solution that may work for some files: Use a mobile Gmail client (I tested it with Android) and "Save to Drive" your attachment. You’ll then be able to download it from the Google Drive! I am not sure if this works for all attachments, however it worked for the source of my PhD thesis! I’m writing it may not work for all attachments because when you download something from Google Drive it does a virus check so it may not allow you to download the attachment and then you’ll still need to do it manually using the method below (however in that case you must be even more cautious for the case the attachment actualyl contains a malicious file).

Update, 2 June 2015: Commenter Sumit Chauhan (and Yuri Marx) proposed to change the extension of the downloaded MIME text file (original message) to eml and open it with Outlook. I don’t have Outlook in my system, however I tried opening it with Thunderbird and it worked!!! So please try this solution before trying the pythonic way (especially if you’re not familiar with python).

Django non-HTML responses

Serafeim Papastefanos — Mon, 15 Sep 2014 14:20:00 +0300

Contents

Introduction
How are CBVs rendered
Rendering to non-HTML
A non-HTML mixin
A more complex example
Conclusion

Introduction

I have already written about the many advantages (DRY!) of using CBVs in a previous article. In this article I will present the correct (pythonic) way to allow a normal CBV to return non-HTML responses, like PDF, CSV, XSL etc.

How are CBVs rendered

Before proceeding, we need to understand how CBVs are rendered. By walking through the class hierarchy in the CBV inspector, we can see that all the normal Django CBVs (DetailView, CreateView, TemplateView etc) are using a mixin named TemplateResponseMixin which defines a method named render_to_response. This is the method that is used for rendering the output of the Views. Let’s take a look at how it is defined (I’ll remove the comments):

class TemplateResponseMixin(object):
  template_name = None
  response_class = TemplateResponse
  content_type = None

  def render_to_response(self, context, **response_kwargs):
      response_kwargs.setdefault('content_type', self.content_type)
      return self.response_class(
          request=self.request,
          template=self.get_template_names(),
          context=context,
          **response_kwargs
      )

  def get_template_names(self):
      if self.template_name is None:
          raise ImproperlyConfigured(
              "TemplateResponseMixin requires either a definition of "
              "'template_name' or an implementation of 'get_template_names()'")
      else:
          return [self.template_name]

This method will try to find out which html template should be used to render the View (using the get_template_names method and template_name attribute of the mixin) and then render this view by instantiating an instance of the TemplateResponse class (as defined in the response_class attribute) and passing the request, template, context and other response_args to it.

The TemplateResponse class which is instantiated in the render_to_response method inherits from a normal HttpResponse and is used to render the template passed to it as a parameter.

Rendering to non-HTML

From the previous discussion we can conclude that if your non-HTML response needs a template then you just need to create a subclass of TemplateResponse and assign it to the response_class attribute (and also change the content_type attribute). On the other hand, if your non-HTML respond does not need a template to be rendered then you have to override render_to_response completely (since the template parameter does not need to be passed now) and either define a subclass of HttpResponse or do the rendering in the render_to_response.

Since almost always you won’t need a template to create a non-HTML view and because I believe that the solution is DRY-enough by implementing the rendering code to the render_to_response method (without subclassing HttpResponse) I will implement a mixin that does exactly that.

Subclassing HttpResponse will not make our design more DRY because for every subclass of HttpResponse the render_to_response method would also need to be modified (by subclassing TemplateResponseMixin) to instantiate the subclass of ``HttpResponse with the correct parameters. For instance, the existing TemplateResponseMixin cannot be used if the subclass of HttpResponse does not take a template as a parameter (solutions like passing None to the template parameter are signals of bad design).

In any case, changing just the render_to_response method using a Mixin is in my opinion the best solution to the above problem. A Mixin is a simple class that can be used to extend other classes either by overriding functionality of the base class or by adding extra features. Django CBVs use various mixins to extend the base Views and add functionality.

A non-HTML mixin

So, a basic skeleton for our mixin would be something like this:

class NonHtmlResponseMixin(object):
    def render_to_response(self, context, **response_kwargs):
        response = HttpResponse(content_type='text/plain')
        response.write( "Hello, world" )
        return response

The previous mixin overrides the render_to_response method to just return the text "Hello, world". For instance we could define the following class:

class DummyTextResponseView(NonHtmlResponseMixin, TemplateView,):
  pass

which can be added as a route to urls.py (using the as_view method) and will always return the "Hello, world" text.

Here’s something more complicated: A Mixin that can be used along with a DetailView and will output the properties of the object as text:

class TextPropertiesResponseMixin(object):
  def render_to_response(self, context, **response_kwargs):
      response = HttpResponse(content_type='text/plain; charset=utf-8')
      o = self.get_object()
      o._meta.fields
      for f in o._meta.fields:
          response.write (u'{0}: {1}\n'.format(f.name,  unicode(o.__dict__.get(f.name)) ) )
      return response

and can be used like this

class TextPropertiesDetailView(TextPropertiesResponseMixin, FooDetailView,):
  pass

The above mixin will use the get_object() method of the DetailView to get the object and then output it as text. We can create similar mixins that will integrate with other types of CBVs, for instance to export a ListView as an CSV or generate an png from a DetailView of an Image file.

A more complex example

The previous examples all built upon an existing view (either a TemplateView, a DetailView or a ListView). However, an existing view that will fit our requirements won’t always be available. For instance, sometimes I want to export data from my database using a raw SQL query. Also I’d like to be able to easily export this data as csv or excel.

First of all, we need to define a view that will inherit from View and export the data as a CSV:

import unicodecsv
from django.db import connection
from django.views.generic import View

class CsvRawSqlExportView(View, ):
  sql = 'select 1+1'
  headers = ['res']
  params = []

  def get(self, request):
      def generate_data(cursor):
          for row in cursor.fetchall():
              yield row

      cursor = connection.cursor()
      cursor.execute(self.sql, self.params)
      generator = generate_data(cursor)
      return self.render_to_response(generator)

  def render_to_response(self, generator, **response_kwargs):
      response = HttpResponse(content_type='text/plain; charset=utf-8')
      response['Content-Disposition'] = 'attachment; filename=export.csv'
      w = unicodecsv.writer(response, encoding='utf-8')
      w.writerow(self.headers)
      for row in generator:
          w.writerow(row)

      return response

The above View has three attributes: * sql, which is a string with the raw sql that will be executed * headers, which is an array with the names of each header of the resulting data * params, which is an array with parameters that may need to be passed to the query

The get method executes the query and passes the result to render_to_response using a generator. The render_to_response method instantiates an HttpResponse object with the correct attributes and writes the CSV to the response object using unicodecsv.

We can now quickly create a route that will export data from the users table:

url(
    r'^raw_export_users/$',
    views.CsvRawSqlExportView.as_view(
        sql='select id, username from auth_user',
        headers=['id', 'username']
    ) ,
    name='raw_export_users'
),

If instead of CSV we wanted to export to XLS (using xlwt), we’d just need to create a Mixin:

class XlsRawSqlResponseMixin(object):
  def render_to_response(self, generator, **response_kwargs):
      response = HttpResponse(content_type='application/ms-excel')
      response['Content-Disposition'] = 'attachment; filename=export.xls'
      wb = xlwt.Workbook(encoding='utf-8')
      ws = wb.add_sheet("data")

      for j,c in enumerate(self.headers):
              ws.write(0,j,c)

      for i,row in enumerate(generator):
          for j,c in enumerate(row):
              ws.write(i+1,j,c)

      wb.save(response)
      return response

and create a View that inherits from CsvRawSqlExportView and uses the above mixin:

class XlsRawSqlExportView( XlsRawSqlResponseMixin, CsvRawSqlExportView ):
  pass

and route to that view to get the XLS:

url(
    r'^raw_export_users/$',
    views.XlsRawSqlExportView.as_view(
        sql='select id, username from auth_user',
        headers=['id', 'username']),
    name='raw_export_users'
),

Conclusion

Using the above techniques we can define CBVs that will output their content in various content types beyond HTML. This will help us write write clean and DRY code.

Implementing a simple, Heroku-hosted REST service using Flask and mongoDB

Serafeim Papastefanos — Mon, 30 Jun 2014 15:23:00 +0300

Contents

Introduction
Requirements
Implementing the REST service
Testing it locally
Deploying to Heroku

Introduction

In the following, I will describe how I used Flask, a very nice web microframework for python along with mongoDB, the most popular No-SQL database to implement a simple REST service that was hosted on Heroku. This REST service would get readings from a number of sensors from an Android device.

I chose Flask instead of Django mainly because the REST service that I needed to implement would be very simple and most of the Django bells and whistles (ORM, auth, admin, etc) wouldn’t be needed anyway. Also, Flask is much quicker to set-up than Django since almost everything (views, urls, etc) can be put inside one python module.

Concerning the choice of a NoSQL persistance solution (mongoDB), I wanted to have a table (or collection as it is called in the mongoDB world) of readings from the sensor. Each reading would just have a timestamp and various other arbitrary data depending on the type of the reading, so saving it as a JSON document in a NoSQL database is a good solution.

Finally, all the above will be deployed to Heroku which offers some great services for deploying python code in the cloud.

Requirements

I propose creating a file named requirements.txt that will host the required packages for your project, so you will be able to setup your projects after creating a virtual environment with virtualenv just by running pip install -r requirements.txt. Also, the requirements.txt is required for deploying python to Heroku.

So, for my case, the contents of requirements.txt are the following:

Flask==0.10.1
Flask-PyMongo==0.3.0
Flask-RESTful==0.2.12
Jinja2==2.7.3
MarkupSafe==0.23
Werkzeug==0.9.6
aniso8601==0.82
gunicorn==19.0.0
itsdangerous==0.24
pymongo==2.7.1
pytz==2014.4
six==1.7.2

Flask-PyMongo is a simple wrapper for Flask around pymongo which is the python mongoDB driver.
Flask-RESTful is a simple library for creating REST APIs - it needs aniso8601 and pytz.
Jinja2 is the template library for Flask (we won’t use it but it is required by Flask installation) - it needs MarkupSafe.
Werkzeug is a WSGI utility library - required by Flask
gunicorn is a WSGI HTTP server - needed for deployment to Heroku
itsdangerous is used to sign data for usage in untrusted environments
six is the python 2/3 compatibility layer

Implementing the REST service

Instead of using just one single file for our Flask web application, we will create a python module to contain it and a file named runserver.py that will start a local development server to test it:

So, in the same folder as the requirements.txt create a folder named flask_rest_service and in there put two files: __init__.py and resources.py.

The __init__.py initializes our Flask application, our mongoDB connection and our Flask-RESTful api:

import os
from flask import Flask
from flask.ext import restful
from flask.ext.pymongo import PyMongo
from flask import make_response
from bson.json_util import dumps

MONGO_URL = os.environ.get('MONGO_URL')
if not MONGO_URL:
    MONGO_URL = "mongodb://localhost:27017/rest";

app = Flask(__name__)

app.config['MONGO_URI'] = MONGO_URL
mongo = PyMongo(app)

def output_json(obj, code, headers=None):
    resp = make_response(dumps(obj), code)
    resp.headers.extend(headers or {})
    return resp

DEFAULT_REPRESENTATIONS = {'application/json': output_json}
api = restful.Api(app)
api.representations = DEFAULT_REPRESENTATIONS

import flask_rest_service.resources

So what happens here? After the imports, we check if we have a MONGO_URL environment variable. This is how we set options in Heroku. If such option does not exist in the environment then we are in our development environment so we set it to the localhost (we must have a running mongoDB installation in our dev environment).

In the next lines, we initialize our Flask application and our mongoDB connection (pymongo uses a MONGO_URI configuration option to know the database URI).

The output_json is used to dump the BSON encoded mongoDB objects to JSON and was borrowed from alienretro’s blog — we initialize our restful REST API with this function.

Finally, we import the resources.py module which actually defines our REST resources.

import json
from flask import request, abort
from flask.ext import restful
from flask.ext.restful import reqparse
from flask_rest_service import app, api, mongo
from bson.objectid import ObjectId

class ReadingList(restful.Resource):
    def __init__(self, *args, **kwargs):
        self.parser = reqparse.RequestParser()
        self.parser.add_argument('reading', type=str)
        super(ReadingList, self).__init__()

    def get(self):
        return  [x for x in mongo.db.readings.find()]

    def post(self):
        args = self.parser.parse_args()
        if not args['reading']:
            abort(400)

        jo = json.loads(args['reading'])
        reading_id =  mongo.db.readings.insert(jo)
        return mongo.db.readings.find_one({"_id": reading_id})


class Reading(restful.Resource):
    def get(self, reading_id):
        return mongo.db.readings.find_one_or_404({"_id": reading_id})

    def delete(self, reading_id):
        mongo.db.readings.find_one_or_404({"_id": reading_id})
        mongo.db.readings.remove({"_id": reading_id})
        return '', 204


class Root(restful.Resource):
    def get(self):
        return {
            'status': 'OK',
            'mongo': str(mongo.db),
        }

api.add_resource(Root, '/')
api.add_resource(ReadingList, '/readings/')
api.add_resource(Reading, '/readings/<ObjectId:reading_id>')

Here we define three Resource classes and add them to our previously defined api: Root, Reading and ReadingList.

Root just returns a dictionary with an OK status and some info on our mongodb connection.

Reading has gets an ObjectId (which is the mongodb primary key) as a parameter and depending on the HTTP operation, it returns the reading with that id when receiving an HTTP GET and deletes the reading with that id when receiving an HTTP DELETE.

ReadingList will return all readings when receiving an HTTP GET and will create a new reading when receiving an HTTP POST The post function uses the parser defined in __init__ which requires a reading parameter with the actual reading to be inserted.

Testing it locally

In order to run the development server, you will need to install and start mongodb locally which is beyond the scope of this post. After that create a file named runserver.py in the same folder as with the requirements.txt and the flask_rest_service folder. The contents of this file should be:

from flask_rest_service import app
app.run(debug=True)

When you run this file with python runserver.py you should be able top visit your rest service at http://localhost:5000 and get an "OK" status.

Deploying to Heroku

To deploy to Heroku, you must create a Procfile that contains the workers of your application. In our case, the Procfile should contain the following:

web: gunicorn flask_rest_service:app

Also, you should add a .gitignore file with the following:

*.pyc

Finally, to deploy your application to Heroku you can follow the instructions here: https://devcenter.heroku.com/articles/getting-started-with-python:

Initialize a git repository and commit everything:

git init
git add .
git commit -m

Create a new Heroku application (after logging in to heroku with heroku login) and set the MONGO_URL environment variable (of course you have to obtain ths MONGO_URL variable for your heroku envirotnment by adding a mongoDB database):

heroku create
heroku config:set MONGO_URL=mongodb://user:pass@mongoprovider.com:27409/rest

And finally push your master branch to the heroku remote repository:

git push heroku master

If everything went ok you should be able to start a worker for your application, check that the worker is running, and finally visit it:

heroku ps:scale web=1
heroku ps
heroku open

If everything was ok you should see an status-OK JSON !

Django generic FormViews for objects

Serafeim Papastefanos — Fri, 11 Apr 2014 10:23:00 +0300

Contents

Introduction
A quick introduction to the FormView
A quick introduction to the SingleObjectMixin
Being generic and DRY
Being more generic and DRY
Other options

Introduction

We recently needed to create a number of views for changing the status of an Application model instance for our organization. An Application model instance can be filled and then cancelled, submitted, acceptted etc - for each of these status changes a form should be presented to the user. When the user submits the form the status of the Application will be changed.

To implement the above requirement we created a generic FormView that acts on the specific model instance. This used two basic CBV components: The FormView for the form manipulation and the SingleObjectMixing for the object handling.

Django Class Based Views (CBVs) can be used to create reusable Views using normal class inheritance. Most people use the well-known CreateView, UpdateView, DetailView and ListView, however, as we will see below, the FormView will help us write DRY code.

I have to notice here that an invaluable tool to help you understanding CBVs is the CBV inspector which has a nice web interface for browsing the CBV hierarchies, attributes and methods.

A quick introduction to the `FormView`

A simple FormView can be defined like this (CBV FormView):

class MyFormView(FormView):
  form_class = forms.MyFormView
  template_name = 'my_template.html'

The above can be used in urls.py like this:

urlpatterns = patterns('',
  url(r'^my_formview/$', views.MyFormView.as_view() , name='my_formview' ),

This will present a form to the user when he visits the my_formview url — however this form won’t do anything. To allow the form to actually do something when it’s been submitted we need to override the form_valid method.

def form_valid(self, form):
    value = form.cleaned_data['value']
    messages.info(self.request, "MyForm submitted with value {0}!".format(value) )
    return HttpResponseRedirect( reverse('my_formview') )

As you can see the submitted form is passed in the method and can be used to receive its cleaned_data. The FormView has various other options for instance a form_invalid method, an initial attribute to set the initial values for the form etc.

A quick introduction to the `SingleObjectMixin`

A SingleObjectMixin adds a number of attributes & methods to a view that can be used for object manipulation. The most important ones is the model and queryset attributes and the get_queryset and get_object(). To use the SingleObjectMixin in your CBV just add it to the list of the classes to inherit from and define either the model or the queryset attribute. After that you may pass a pk parameter to your view and you will get an object context variable in the template with the selected object!

Being generic and DRY

We can more or less now understand how we should use FormView and SingleObjectMixin to generate our generic FormView for acting on objects: Our FormView should get the object using the SingleObjectMixin and change it when the form is submitted using the values from the form. A first implementation would be the following:

class GenericObjectFormView1(FormView, SingleObjectMixin):

    def form_valid(self, form):
        obj = self.get_object()
        obj.change_status(form)
        return HttpResponseRedirect( obj.get_absolute_url() )

So our GenericObjectFormView1 class inherits from FormView and SingleObjectMixin. The only thing that we have to assure is that the Model we want to act on needs to implement a change_status method which gets the form and changes the status of that object based on its value. For instance, two implementations can be the following:

class CancelObjectFormView(GenericObjectFormView1):
    template_name = 'cancel.html'
    form_class = forms.CancelForm
    model = models.Application

class SubmitObjectFormView(GenericObjectFormView1):
    template_name = 'submit.html'
    form_class = forms.SubmitForm
    model = models.Application

Being more generic and DRY

The previous implementation has two problems:

What happens if the status of the object should not be changed even if the form is valid?
We shouldn’t need to create a new template for every new GenericObjectFormView since all these templates will just output the object information, ask a question for the status change and output the form.

Let’s write a new version of our GenericObjectFormView that actually resolves these:

class GenericObjectFormView2(FormView, SingleObjectMixin):
    template_name = 'generic_formview.html'
    ok_message = ''
    not_ok_message = ''
    title = ''
    question =''

    def form_valid(self, form):
        obj = self.get_object()
        r = obj.change_status(form)
        if r:
            messages.info(self.request, self.yes_message)
        else:
            messages.info(self.request, self.not_ok_message)
        return HttpResponseRedirect( obj.get_absolute_url() )

    def get_context_data(self, **kwargs):
        context = super(GenericYesNoFormView, self).get_context_data(**kwargs)
        context['title'] = self.title
        context['question'] = self.question
        return context

The above adds an ok and not ok message which will be outputed if the status can or cannot be changed. To accomplish this, the change_status method should now return a boolean value to mark if the action was ok or not. Also, a generic template will now be used. This template has two placeholders: One for the title of the page (title attribute) and one for the question asked to the user (question attribute). Now we can use it like this:

class CancelObjectFormView(GenericObjectFormView2):
    form_class = forms.CancelForm
    model = models.Application
    ok_message = 'Cancel success!'
    not_ok_message = 'Not able to cancel!'
    title = 'Cancel an object'
    question = 'Do you want to cancel this object?'

class SubmitObjectFormView(GenericObjectFormView2):
    form_class = forms.SubmitForm
    model = models.Application
    ok_message = 'Submit  ok'
    not_ok_message = 'Cannot submit!'
    title = 'Submit an object'
    question ='Do you want to submit this object?'

Other options

We’ve just got a glimpse of how we can use CBVs to increase the DRYness of our Django applications. There are various extra things that we can add to our GenericObjectFormView2 as attributes which will be defined by inheriting classes. Some ideas is to check if the current user actually has access to modify the object (hint: override the get_object method of SingleObjectMixin) or render the form diffirently depending on the current user (hint: override the get_form_kwargs method of FormView).

A Wagtail tutorial

Serafeim Papastefanos — Thu, 13 Feb 2014 21:20:00 +0200

Wagtail is a new Open Source Django-based CMS. In this 20 minute tutorial we will see how you can create a blog from scratch using Wagtail. If you want to see some more examples of usage please take a look at the wagtaildemo GitHub project.

To follow this tutorial you will need to have Python 2.7 installed with a working version of pip and virtualenv.

Update: The result of this tutorial has been deployed to Heroku: http://gentle-refuge-2590.herokuapp.com/ - you may visit the admin site and login with root / 123 to play with Wagtail! Please don’t do anything naughty !!! Also, notice that because of how Heroku works you won’t be able to upload anything.

Update 08/09/2015: This tutorial has been written during the first days of Wagtail, when documentation and tutorials about it were sparce; right now it should be considered by all accounts obsolete and not be followed! Instead, you should read the official (and very well written) tutorial @ http://docs.wagtail.io/en/v1.0/getting_started/tutorial.html in the official Wagtail documentation!

Installing the wagtail dependencies

It is recomended to create a new virtual environment that will host the wagtail tutorial. After you have changed to the virtual environment you will need to installl the Wagtail requirements. Create a file named requirements.txt containing the following:

Django==1.6.2
South==1.0.0
django-compressor==1.4
django-modelcluster==0.3
-e git://github.com/torchbox/wagtail.git#egg=wagtail
django-taggit==0.11.2
django-libsass==0.2

and run pip install -r requirements.txt. If you use Microsoft Windows you will experience problems with Pillow and lxml. Please download the installation executables from https://pypi.python.org/pypi/Pillow/2.3.0 and https://pypi.python.org/pypi/lxml/3.3.1, install them using easy_install Pillow-2.3.0.x-py2.7.exe and easy_install lxml-3.3.1.x-py2.7.exe (from inside your virtual environment) and then install the other requirements. Also please use the latest version of Wagtail (hosted on github) because it has some changes from the pypi (so don’t do a pip install wagtail).

Warning: Unfortuanately, no official binaries for libsass (which is a django-libsass requirement) are (yet) available for Windows. I have compiled a version for win32 and python 2.7 (which I use) using the instructions found here. You can download this version as a wheel package. Notice that because libsass was compiled with VC Express 2013 you should also install the Visual C++ 2013 redistributable package from Microsoft.

Please use it at your own risk !!!

To install wheels you have to use a version of pip >= 1.4 (so do an easy_install -U pip from your virtual environment if you have a previous version) and then you can just do a normal pip install libsass-0.3.0-cp27-none-win32.whl.

Warning no2: More unfortuanately, there seem to be a number of issues with how libsass handles @import statements in Windows. Until this is fixed, Windows users are recommened to use the command line (Ruby) sass compiler. To use it, please install Ruby in your system and then install sass with gem install sass -v ">=3.3.0alpha" --pre. After that, in the COMPRESS_PRECOMPILERS setting of your settings.py (discussed in the next section), change the line ('text/x-scss', 'django_libsass.SassCompiler'), to ('text/x-scss', 'sass --scss {infile} {outfile}'),.

Creating and configuring your project

Wagtail has to “live” inside a normal Django project so you now may create a new Django project by issuing:

python <PATH_OF_YOUR_VIRTUAL_ENV>/scripts/django-admin.py startproject wagtailtutorial

If you use Unix you can just run django-admin.py etc however if you try to do the same in windows you will find out that windows tries to run the .py file with the python executable that is assigned through explorer (which of course is your main python installation) and not through your path! That’s why all our commands will be in the form python script.py to make sure that Windows picks the python executable from your path (which, if you’re inside the virtual enviroment will be the correct one).

Inside the wagtailtutorial folder you will see a file named manage.py and another folder named wagtailtutorial. Inside this wagtailtutorial folder you will find settings.py and urls.py which need to be changed.

Starting with urls.py, remove everything and change it like this:

from django.conf.urls import patterns, include, url
from django.conf.urls.static import static
from django.views.generic.base import RedirectView
from django.contrib import admin
from django.conf import settings
import os.path

from wagtail.wagtailcore import urls as wagtail_urls
from wagtail.wagtailadmin import urls as wagtailadmin_urls
from wagtail.wagtailimages import urls as wagtailimages_urls
from wagtail.wagtailembeds import urls as wagtailembeds_urls
from wagtail.wagtaildocs import admin_urls as wagtaildocs_admin_urls
from wagtail.wagtaildocs import urls as wagtaildocs_urls
from wagtail.wagtailsnippets import urls as wagtailsnippets_urls
from wagtail.wagtailsearch.urls import frontend as wagtailsearch_frontend_urls, admin as wagtailsearch_admin_urls
from wagtail.wagtailusers import urls as wagtailusers_urls
from wagtail.wagtailredirects import urls as wagtailredirects_urls

admin.autodiscover()


# Signal handlers
from wagtail.wagtailsearch import register_signal_handlers as wagtailsearch_register_signal_handlers
wagtailsearch_register_signal_handlers()


urlpatterns = patterns('',
    url(r'^django-admin/', include(admin.site.urls)),

    url(r'^admin/images/', include(wagtailimages_urls)),
    url(r'^admin/embeds/', include(wagtailembeds_urls)),
    url(r'^admin/documents/', include(wagtaildocs_admin_urls)),
    url(r'^admin/snippets/', include(wagtailsnippets_urls)),
    url(r'^admin/search/', include(wagtailsearch_admin_urls)),
    url(r'^admin/users/', include(wagtailusers_urls)),
    url(r'^admin/redirects/', include(wagtailredirects_urls)),
    url(r'^admin/', include(wagtailadmin_urls)),
    url(r'^search/', include(wagtailsearch_frontend_urls)),
    url(r'^documents/', include(wagtaildocs_urls)),

    # For anything not caught by a more specific rule above, hand over to
    # Wagtail's serving mechanism
    url(r'', include(wagtail_urls)),
)


if settings.DEBUG:
    from django.contrib.staticfiles.urls import staticfiles_urlpatterns

    urlpatterns += staticfiles_urlpatterns() # tell gunicorn where static files are in dev mode
    urlpatterns += static(settings.MEDIA_URL + 'images/', document_root=os.path.join(settings.MEDIA_ROOT, 'images'))

You can se that there is a signal handler when a searchable thing is added or changed to handle indexing for search, normal django admin is mapped under /django-admin since /admin is use for Wagtail (of course you may map Wagtail wherever you’d like), inclusion of various wagtail related urls and finally a Wagtail handling everything else. Finally there are some handlers for media and static files.

After that please change your settings.py like this:

# Django settings for wagtailtutorial project.

import os

PROJECT_ROOT = os.path.join(os.path.dirname(__file__), '..', '..')

DEBUG = True
TEMPLATE_DEBUG = DEBUG

ADMINS = ()
MANAGERS = ADMINS

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': PROJECT_ROOT+'/wagtailtutorial.db',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',  # Set to empty string for localhost.
        'PORT': '',  # Set to empty string for default.
    }
}

CONN_MAX_AGE = 600  # number of seconds database connections should persist for
ALLOWED_HOSTS = []
TIME_ZONE = 'Europe/London'
LANGUAGE_CODE = 'en-gb'
SITE_ID = 1
USE_I18N = True
USE_L10N = False
USE_TZ = True
MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media')
MEDIA_URL = '/media/'
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'static')
STATIC_URL = '/static/'
STATICFILES_DIRS = ()

STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'compressor.finders.CompressorFinder',
)

SECRET_KEY = 'wq21wtjo3@d_qfjvd-#td!%7gfy2updj2z+nev^k$iy%=m4_tr'

TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.Loader',
    'django.template.loaders.app_directories.Loader',
)

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'wagtail.wagtailcore.middleware.SiteMiddleware',
    'wagtail.wagtailredirects.middleware.RedirectMiddleware',
)

from django.conf import global_settings
TEMPLATE_CONTEXT_PROCESSORS = global_settings.TEMPLATE_CONTEXT_PROCESSORS + (
    'django.core.context_processors.request',
)

ROOT_URLCONF = 'wagtailtutorial.urls'
WSGI_APPLICATION = 'wagtailtutorial.wsgi.application'
TEMPLATE_DIRS = ()

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    # 'django.contrib.sites',  # Wagtail uses its own site management logic
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'south',
    'compressor',
    'taggit',
    'modelcluster',
    'django.contrib.admin',

    'wagtail.wagtailcore',
    'wagtail.wagtailadmin',
    'wagtail.wagtaildocs',
    'wagtail.wagtailsnippets',
    'wagtail.wagtailusers',
    'wagtail.wagtailimages',
    'wagtail.wagtailembeds',
    'wagtail.wagtailsearch',
    'wagtail.wagtailredirects',

    'tutorial',
)

EMAIL_SUBJECT_PREFIX = '[wagtailtutorial] '

INTERNAL_IPS = ('127.0.0.1', '10.0.2.2')

COMPRESS_PRECOMPILERS = (
    ('text/x-scss', 'django_libsass.SassCompiler'),
)

# Auth settings
LOGIN_URL = 'django.contrib.auth.views.login'
LOGIN_REDIRECT_URL = 'wagtailadmin_home'

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}


# WAGTAIL SETTINGS
WAGTAIL_SITE_NAME = 'wagtailtutorial'

# Override the search results template for wagtailsearch
WAGTAILSEARCH_RESULTS_TEMPLATE = 'tutorial/search_results.html'
WAGTAILSEARCH_RESULTS_TEMPLATE_AJAX = 'tutorial/includes/search_listing.html'

WAGTAILSEARCH_ES_INDEX = 'wagtailtutorial'

The most important thing to notice is that the INSTALLED_APPS contains the usual apps from django.*, south for database migrations, django-compressor to support compressing static files (and automatic translating from less to css), django-taggit to add support for tags, and django-modelcluster which adds support from clusters (groups) of models. It also contains the wagtail.* applications and the tutorial application which is where we will create our blog. Also there are two Wagtail related middleware (one to add a site attribute to each request and one to hand redirects), configuring django-compressor to use django_libsass to compile less files, and some other, not so important Wagtail settings.

So let’s create the missing tutorial application by issuing:

python manage.py startapp tutorial

Now we’ll have a tutorial folder waiting to define our blog structure inside the wagtailtutorial folder!

Checking to see if everything works

Before continuing with our blog creation let’s make sure that everything works, first of all by generating the database schema:

python manage.py syncdb

In the superuser question answer yes and add a superuser. Then you can run the migrations - however because there are some problems with the way SQLite3 runs migrations it is recommended to run the migrations in two steps:

python manage.py migrate 0001 --all
python manage.py migrate

Finally, you now may try a

python manage.py runserver

and visit http://127.0.0.1:8000. If everything worked fine you will get a

Welcome to your new Wagtail site!

page — congratulations !

The homepage is rather simple (for now!) but you may already navigate to http://127.0.0.1:8000/admin and from there, login to Wagtail admin with the superuser you created earlier. Now you may start experiencing Wagtail !

Exploring Wagtail admin

When you login to Wagtail admin you will see a menu at the left with the options:

Explorer is used to actually manage the content of your site
Search is used for searching through your content
Images is used to manage your images
Documents is used to manage your Documents
Snippets is used for side bars etc
Users is used for User management
Redirects is used to redirect to a specific page
Editors picks is used to promote search results

Of the previous, the one needing more explanation is Explorer: Clicking it you will see that a label named “Welcome to your Wagtail site!” will open. This is the root Page of your site. If you click at it you will go to the actions of this page. The actions you can do here is: - Add Child Page - Edit - View Live - Move - Delete - Unpublish

The Pages (and more generally the Content) of Wagtail are Django Models that are saved through a Tree hierarchy. However, if you click “Add Child Page” you won’t be able to add anything because you must create your own Page types (we will see how it is done in the next section).

When you click edit you will be able to edit the parts of the page (each part is a normal Field of the model). Also, you will see that the form is split into two tabs: Content and Promote. In the Content for instance you will see that the “Welcome to your Wagtail site!” has only a “Title” CharField. These are the fields that will be available to all Pages since “Welcome to your Wagtail site!” has a class of Page from which every other page should inherit. After you finish editing a page you may save it as a draft, publish, sent it for moderation (if you don’t have the rights to publish it) etc.

Creating our blog

Each of our Page types is a normal Django Model which inherits from Page. Let’s suppose that our posts should contain a title, a body and a created date. Add the following to tutorials/models.py:

from django.db import models

from wagtail.wagtailcore.models import Page
from wagtail.wagtailcore.fields import RichTextField
from wagtail.wagtailadmin.edit_handlers import FieldPanel    

class BlogPage(Page):
    body = RichTextField()
    date = models.DateField("Post date")
    search_name = "Blog Page"

    indexed_fields = ('body', )

BlogPage.content_panels = [
    FieldPanel('title', classname="full title"),
    FieldPanel('date'),
    FieldPanel('body', classname="full"),
]

and run python manage.py syncdb to create the tutorial_blogpage table.

Now, if you visit again the /admin and click on the “Add Child Page” action of “Welcome to your new Wagtail Site!” you will see the “Blog Page” page and after you click it you will be able to see a form with the Fields you defined and title(title, body, date). The title is a field inherited from Page, along with the fields in the Promote tab.

In our declaration of BlogPage we added three FieldPanels on its content_panes. A FieldPanel is a special edit handler for each Field. That is why when you try to edit the body you will see a rich text toolbar that enables you to not only format text but also embed images, documents and even oembed links. If you hadn’t included the FieldPanel('body', classname="full") then you wouldn’t see the rich text editor.

So now we can add as many posts as we like!

The time has come to take a look at our fine blog post: After we publish our page we click to the view live action and…

TemplateDoesNotExist at /hello-world/

tutorial/blog_page.html

A template seems to be missing — actually we have totally forgotten about the presentation of our blog — we’ll talk about this in the next section !

However, before going there we should also create an Index page type collecting our posts.

For this, we will change our models.py to this:

from django.db import models

from wagtail.wagtailcore.models import Page, Orderable
from wagtail.wagtailcore.fields import RichTextField
from wagtail.wagtailadmin.edit_handlers import FieldPanel  ,MultiFieldPanel,InlinePanel, PageChooserPanel
from modelcluster.fields import ParentalKey

class BlogPage(Page):
    body = RichTextField()
    date = models.DateField("Post date")
    indexed_fields = ('body', )
    search_name = "Blog Page"

BlogPage.content_panels = [
    FieldPanel('title', classname="full title"),
    FieldPanel('date'),
    FieldPanel('body', classname="full"),
]


class LinkFields(models.Model):
    link_page = models.ForeignKey(
        'wagtailcore.Page',
        null=True,
        blank=True,
        related_name='+'
    )

    panels = [
        PageChooserPanel('link_page'),
    ]

    class Meta:
        abstract = True

class RelatedLink(LinkFields):
    title = models.CharField(max_length=255, help_text="Link title")
    panels = [
        FieldPanel('title'),
        MultiFieldPanel(LinkFields.panels, "Link"),
    ]

    class Meta:
        abstract = True


class BlogIndexPageRelatedLink(Orderable, RelatedLink):
    page = ParentalKey('tutorial.BlogIndexPage', related_name='related_links')

class BlogIndexPage(Page):
    intro = models.CharField(max_length=256)
    indexed_fields = ('body', )
    search_name = "Blog Index Page"

BlogIndexPage.content_panels = [
    FieldPanel('title', classname="full title"),
    FieldPanel('intro', classname="full"),
    InlinePanel(BlogIndexPage, 'related_links', label="Related links"),
]

The above adds a way to put related_links to a BlogIndexPage — these related_links are the BlogPages that actually belong to the blog - so now, we can add a blog index page and add all our blog posts to it!

Now that we’ve created the BlogPage and BlogIndexPage pages it’s time to take a look at how we will actually display our blog…

Creating templates for Pages

A normal Django template can be used to display each page type. Wagtail either generates automatically (by seperating underscors with capital letters in camelcase for instance BlogIndexPage -> blog_index_page) or you can use the template class attribute. Let’s add a templates foler within the tutorial foler and add another folder named tutorial inside templates and ten add a file named blog_index_page.html to templates with the following content (you must have the following hierarchy wagtailtutorial/tutorial/templates/tutorial/blog_index_page.html):

<html>
<body>
    <h1>{{ self.title }}</h1>
        Intro: {{ self.intro }}
        <hr />
        {% for rl in self.related_links.all %}
            <p>{{ rl.title }}: <a href='{{ rl.link_page.url }}'>{{ rl.link_page }}</a></p>
        {% endfor %}
    </body>
</html>

So self is the context name of the BlogPageIndex instance that is used to render this page. Beyond that, it’s normal django.

Now you can view your Blog Index — however before clicking on a link to also view your posts add the template for your BlogPost (tutorial/blog_page.html):

{% load rich_text %}
<html>
<body>
    <h1>{{ self.title }}</h1>
        Date: {{ self.date }}
        {{ self.body | richtext }}
    </body>
</html>

The extra thing here is the richtext filter which renders a RichTextField correctly. Of course the above templates are just examples - check wagtaildemo for a much better template design.

Woo-hoo — now I can totally start blogging !!! \o/

…

But … when I go to 127.0.0.1:8000 it displays the ugly “Welcome to your new Wagtail site!”. I don’t want to see that any more !!!

No problemo - check the next section of the tutorial :)

Changing your home page

Wagtail uses the concept of sites to define groups of pages hosted in the same server. To make more clear what site is, you have to use the good old django admin which can be found at http://127.0.0.1:8000/django-admin/. Check the localhost[default] site entry: You will see that each site has a name, a root page (currently the “Welcome to your new Wagtail site!”) and an is_default check. So, change the root page of the localhost site to your BlogIndexPage and go to http://127.0.0.1:8000/ … Yes ! The blog is alive :)

Where to go from here

You are now ready to start adding more pages to your blog, adding functionality to it (don’t forget that everything is just Django models and templates so you can change it at will), or even creating a completely different kind of site by adding other Page types. For a more complete site with lots of examples please check wagtaildemo. There is no complete documentation yet however you can

Use the source Luke!

Django dynamic forms

Serafeim Papastefanos — Tue, 24 Dec 2013 14:20:00 +0200

Contents

Introduction
Describing a django form in JSON
Creating the form fields
Creating the actual form
Using the dynamic form

Introduction

To define a form in django, a developer has to create a class which extends django.forms.Form and has a number of attributes extending from django.forms.Field. This makes it very easy for the developer to create static forms, but creating dynamic forms whose fields can be changed depending on the data contributed by the users of the application is not so obvious.

Of course, some people may argue that they can do whatever they want just by spitting html input tags to their templates, however this totally violates DRY and any serious django developer would prefer to write MUMPS than creating html dynamically.

The implementation I will present here had been developed for an old project: In that project there was a number of services which could be edited dynamically by the moderators. For each service, the moderators would generate a questionnaire to get input from the users which would be defined using JSON. When the users needed to submit information for each service, a dynamic django form would be generated from this JSON and the answers would be saved to no-SQL database like MongoDB.

Describing a django form in JSON

A JSON described django form is just an array of field JSON objects. Each field object has three required attributes: name which is the keyword of the field, label which is how the label of the field and type which is the type of the input of that field. The supported types are text, textarea, integer, radio, select, checkbox. Now, depending on the type of the field, there could be also some more required attributes, for instace text has a max_length attribute and select has a choices attribute (which is an array of name/value objects). Also there are two optional attributes, required with a default value of False and help_text with a default value of ”.

As you can understand these map one by one to the corresponding attributes of the actual django form fields. An example containing a complete JSON described form is the following:

[
    {
        "name": "firstname",
        "label": "First Name",
        "type": "text",
        "max_length": 25,
        "required": 1
    },
    {
        "name": "lastname",
        "label": "Last Name",
        "type": "text",
        "max_length": 25,
        "required": 1
    },
    {
        "name": "smallcv",
        "label": "Small CV",
        "type": "textarea",
        "help_text": "Please insert a small CV"
    },
    {
        "name": "age",
        "label": "Age",
        "type": "integer",
        "max_value": 200,
        "min_value": 0
    },
    {
        "name": "marital_status",
        "label": "Marital Status",
        "type": "radio",
        "choices": [
            {"name": "Single", "value":"single"},
            {"name": "Married", "value":"married"},
            {"name": "Divorced", "value":"divorced"},
            {"name": "Widower", "value":"widower"}
        ]
    },
    {
        "name": "occupation",
        "label": "Occupation",
        "type": "select",
        "choices": [
            {"name": "Farmer", "value":"farmer"},
            {"name": "Engineer", "value":"engineer"},
            {"name": "Teacher", "value":"teacher"},
            {"name": "Office Clerk", "value":"office_clerk"},
            {"name": "Merchant", "value":"merchant"},
            {"name": "Unemployed", "value":"unemployed"},
            {"name": "Retired", "value":"retired"},
            {"name": "Other", "value":"other"}
        ]
    },
    {
        "name": "internet",
        "label": "Internet Access",
        "type": "checkbox"
    }
]

The above JSON string can be easily converted to an array of dictionaries with the following code:

import json
fields=json.loads(json_fields)

Creating the form fields

The most import part in the django dynamic form creation is to convert the above array of field-describing dictionaries to actual objects of type django.forms.Field.

To help with that I implemented a class named FieldHandler which gets an array of field dictionaries and after initialization will have an attribute named formfields which will be a dictionary with keys the names of each field an values the corresponding django.forms.Field objects. The implementation is as follows:

import django.forms

class FieldHandler():
    formfields = {}
    def __init__(self, fields):
        for field in fields:
            options = self.get_options(field)
            f = getattr(self, "create_field_for_"+field['type'] )(field, options)
            self.formfields[field['name']] = f

    def get_options(self, field):
        options = {}
        options['label'] = field['label']
        options['help_text'] = field.get("help_text", None)
        options['required'] = bool(field.get("required", 0) )
        return options

    def create_field_for_text(self, field, options):
        options['max_length'] = int(field.get("max_length", "20") )
        return django.forms.CharField(**options)

    def create_field_for_textarea(self, field, options):
        options['max_length'] = int(field.get("max_value", "9999") )
        return django.forms.CharField(widget=django.forms.Textarea, **options)

    def create_field_for_integer(self, field, options):
        options['max_value'] = int(field.get("max_value", "999999999") )
        options['min_value'] = int(field.get("min_value", "-999999999") )
        return django.forms.IntegerField(**options)

    def create_field_for_radio(self, field, options):
        options['choices'] = [ (c['value'], c['name'] ) for c in field['choices'] ]
        return django.forms.ChoiceField(widget=django.forms.RadioSelect,   **options)

    def create_field_for_select(self, field, options):
        options['choices']  = [ (c['value'], c['name'] ) for c in field['choices'] ]
        return django.forms.ChoiceField(  **options)

    def create_field_for_checkbox(self, field, options):
        return django.forms.BooleanField(widget=django.forms.CheckboxInput, **options)

As can be seen, in the __init__ method, the get_options method is called first which returns a dictionary with the common options (label, help_text, required). After that, depending on the type of each field the correct method will be generated with getattr(self, "create_field_for_"+field['type'] ) (so if type is text this will return a reference to the create_field_for_text method) and then called passing the field dictinary and the options returned from get_options. Each one of the create_field_for_xxx methods will extract the required (or optional) attributes for the specific field type, update options and initialize the correct Field passing the options as kwargs. Finally the formfields attribute will be updated with the name and Field object.

Creating the actual form

To create the actual dynamic django.forms.Form I used the function get_form which receives a string with the json description, parses it to a python array, creates the array of fields with the help of FieldHandler and then generates the Form class with type passing it django.forms.Form as a parent and the array of django.forms.Field from FieldHandler as attributes:

def get_form(jstr):
    fields=json.loads(jstr)
    fh = FieldHandler(fields)
    return type('DynaForm', (django.forms.Form,), fh.formfields )

Using the dynamic form

The result of get_form can be used as a normal form class. As an example:

import dynaform

def dform(request):
    json_form = get_json_form_from_somewhere()
    form_class = dynaform.get_form(json_form)
    data = {}
    if request.method == 'POST':
        form = form_class(request.POST)
        if form.is_valid():
            data = form.cleaned_data
    else:
        form = form_class()

    return render_to_response( "dform.html", {
        'form': form,  'data': data,
    }, RequestContext(request) )

So, we have to get our JSON form description from somewhere (for instance a field in a model) and then generate the form class with get_form. After that we follow the normal procedure of checking if the request.method is POST so we pass the POST data to the form and check if it is value or we just create an empty form. As a result we just pass the data that was read from the form to the view for presentation.

Django authority data

Serafeim Papastefanos — Tue, 05 Nov 2013 14:20:00 +0200

Contents

Introduction
Defining authorities
Adding authority data
Conclusion

Introduction

One common requirement in an organization is to separate users in authorities (meaning departments / units / branches etc) and each authority have its own data. So users belonging to the "Athens Branch" won’t be able to edit data submitted from users of the "Thessaloniki Branch".

This is a special case of the more general row-level-security in which each instance of a domain object will have an ACL. Row-level-security would need a many-to-many relation between object instances and authorities, something that would be overkill in our case.

Authority data is also a more general case of the user-data meaning that each user can have access to data that he inserts in the system. Implementing user-data is easy using the techniques we will present below.

We have to notice that the django permissions do not support our requirements since they define security for all instances of a model.

Defining authorities

In order to have custom authorities I propose first of all to add an Authority model that would define the authority. Even if your authorities only have a name I believe that adding the Authority model would be beneficial. Now, there are many ways to separate normal django users (django.contrib.auth.models.User) to authorities:

Using groups

Just define a django.contrib.auth.models.Group for each authority and add the users to the groups you want using the django-admin. Your Authority model would have an one-to-one relation with the django.contrib.auth.models.Group so you will be able to find out the other information of the authority (since django groups only have names).

Now you can just get the groups for the user and find out his authorities. This could lead to problems when users belong to django groups that are not related to authorities so you must filter these out (for instance by checking which groups actually have a corresponding Authority).

By storing the authority to the session

When the user logs in you can add an attribute to the session that would save the authority of the user. To do that, you should define a custom middleware that checks to see if there is an authority attribute to the session and if not it will do whatever it needs to find it and set it. An example is this:

class CustomAuthorityMiddleware:
  def process_request(self, request):
    if not request.session.get('authority'):
      authority = get_the_authority(request.user)
      request.session['authority']=authority

This way, whenever you want to find out the authority of the user you just check the session.

By using a Custom User Profile

Just create a django user profile and add to it a ForeignKey to your Authority model:

class Profile(models.Model):
  user = models.OneToOneField('django.auth.User')
  authority = models.ForeignKey('authorities.Authority', blank=True, null=True )

class Authority(models.Model):
  id = models.IntegerField(primary_key = True)
  name = models.CharField(max_length=64, )
  auth_type = models.CharField(max_length=16, )

You can get the authority of the user through request.user.profile.authority.

Getting the authority of the user has to be DRY

Whatever method you use to define the authorities of your users you have to remember that it is very important to define somewhere a function that will return the authority (or authorities) of a user. You need to define a function even in the simple case in which your function would just return request.user.profile.authority. This will greatly help you when you wish to add some logic to this, for instance "quickly disable users belonging to Authority X or temporary move users from Authority Y to authority Z".

Let us suppose that you have defined a get_user_authority function. Also, you need to define a has_access function that would decide if a users/request has access to a particular object. This also needs to be DRY.

Adding authority data

To define authority data you have to add a field to your model that would define its authority, for instance like this:

class AuthorityData(models.Model):
  authority = models.ForeignKey('authorities.Authority', editable=False,)

This field should not be editable (at least by your end users) because they shouldn’t be able to change the authority of the data they insert.

If you want to have user-data then just add a models.ForeignKey('django.auth.User', editable=False)

Now, your Create and Update Class Based Views have to pass the request to your forms and also your Detail and Update CBV should allow only getting objects that belong to the authority of the user:

class AuthorityDataCreateView(CreateView):
  model=models.AuthorityData

  def get_form_kwargs(self):
      kwargs = super(AuthorityDataCreateView, self).get_form_kwargs()
      kwargs.update({'request': self.request})
      return kwargs

class AuthorityDataDetailView(DetailView):
  def get_object(self, queryset=None):
      obj = super(AuthorityDataDetailView, self).get_object(queryset)
      if if not user_has_access(obj, self.request):
          raise Http404(u"Access Denied")
      return obj

class AuthorityDataUpdateView(UpdateView):
  model=models.AuthorityData

  def get_form_kwargs(self):
      kwargs = super(AuthorityDataUpdateView, self).get_form_kwargs()
      kwargs.update({'request': self.request})
      return kwargs

  def get_object(self, queryset=None):
      obj = super(AuthorityDataUpdateView, self).get_object(queryset)
      if if not user_has_access(obj, self.request):
          raise Http404(u"Access Denied")
      return obj

Your ModelForm can now use the request to get the Authority and set it (don’t forget that you should not use Meta.exclude but instead use Meta.include!):

class AuthorityDataModelForm(forms.ModelForm):
    class Meta:
      model = models.AuthorityData
      exclude = ('authority',)

    def __init__(self, *args, **kwargs):
      self.request = kwargs.pop('request', None)
      super(ActionModelForm, self).__init__(*args, **kwargs)


    def save(self, force_insert=False, force_update=False, commit=True):
      obj = super(AuthorityDataModelForm, self).save(commit=False)
      if obj:
          obj.authority = get_user_authority(self.request)
          obj.save()
      return obj

The previous work fine for Create/Detail/Update CBVs but not for ListsViews. List views querysets and in general all queries to the object have to be filtered through authority.

class AuthorityDataListView(ListView):
  def get_queryset(self):
    queryset = super(AuthorityDataModelForm, self).get_queryset()
    return queryset.filter(authority = get_user_authority(request))

Conclusion

Using the above techniques we can define authority (or just user) data. Your AuthorityData should have a ForeignKey to your Authority and you have configure your queries, ModelForms and CBVs to use that. If you have more than one models that belong to an authority and want to stay DRY then you’d need to define all the above as mixins.

Using custom authorities with spring-security LDAP authentication

Serafeim Papastefanos — Mon, 14 Oct 2013 08:55:00 +0300

Contents

Introduction
A basic spring security setup
Spring security LDAP with custom authorities
Conclusion

Introduction

One very useful component of the spring java framework is spring-security since it allows consistent usage of various security providers for authentication and authorization. Although I’ve found a great number of basic spring-security tutorials on the internet, I wasn’t able to find a complete solution for my own requirements:

Logging in with LDAP but configuring the authorities [*] of the logged in user with the help of a custom method and not through LDAP.

I think that the above is a common requirement in many organizations: There is a central LDAP repository in which the usernames and passwords of the users are stored, but the groups of the users are not stored there. Or maybe the groups that are actually stored in the LDAP cannot be transformed easily to application specific groups for each application.

You may find the working spring project that uses ldap and a custom groups populator here: https://github.com/spapas/SpringLdapCustomAuthorities/

A basic spring security setup

I’ve created a very basic setup for spring-security for a spring-mvc project. Please take a look here for a more thorough explanation of a simple spring-security project http://www.mkyong.com/spring-security/spring-security-hello-world-example/ and here http://www.codeproject.com/Articles/253901/Getting-Started-Spring-Security for a great explanation of the various spring-security classes.

In my setup there is a controller that defines two mappings, the "/" which is the homepage that has a link to the "/enter" and the "/enter" which is an internal page in which only authorized users have access. When the user clicks on "enter" he will be represented with a login form first. If the use logs in successfully, the enter.jsp will list the username and the authorities of the logged in user through the following spring-security tags:

<%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
[...]
Username: <sec:authentication property="principal.username" /><br />
Authorities: <sec:authentication property="principal.authorities"/><br />

The authentication provider is an in memory service in which the username, password and authorities of each user are defined in the XML. So this is a simple spring-security example that can be found in a number of places on the internet. The security rules, login form and the authentication provider are configured with the following security-config.xml:

<beans:beans xmlns="http://www.springframework.org/schema/security"
   xmlns:beans="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.springframework.org/schema/beans
                   http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
                   http://www.springframework.org/schema/security
                   http://www.springframework.org/schema/security/spring-security-3.1.xsd">

   <http pattern="/static/**" security="none" />

   <http use-expressions="true">
       <intercept-url pattern="/" access="permitAll" />
       <intercept-url pattern="/enter" access="hasRole('user')" />
       <intercept-url pattern="/**" access="denyAll" />
       <form-login default-target-url="/" />
       <logout  logout-success-url="/" />
   </http>

   <authentication-manager>
       <authentication-provider>
           <user-service>
               <user name="spapas" password="123" authorities="admin, user, nonldap" />
               <user name="serafeim" password="123" authorities="user" />
           </user-service>
       </authentication-provider>
   </authentication-manager>

</beans:beans>

When we run this application and go to the /enter, we will get the following output:

Username: spapas

Authorities: [admin, nonldap, user]

Spring security LDAP with custom authorities

The previous application can be modified to login through LDAP and get the authorities from a custom class. The main differences are in the pom.xml which adsd the spring-security-ldap dependency, the addition of a CustomLdapAuthoritiesPopulator.java which does the actual mapping of username to authority and various changes to the security-config.xml.

As you will see we had to define our security beans mainly using spring beans and not using the various elements from security namespace like <ldap-server> and <ldap-authentication-provder>. For a good tutorial on using these elements and ldap in spring security in general check these out: http://docs.spring.io/spring-security/site/docs/3.1.x/reference/ldap.html and http://krams915.blogspot.gr/2011/01/spring-security-mvc-using-ldap.html.

<beans:beans xmlns="http://www.springframework.org/schema/security"
   xmlns:beans="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.springframework.org/schema/beans
                   http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
                   http://www.springframework.org/schema/security
                   http://www.springframework.org/schema/security/spring-security-3.1.xsd">

   <http pattern="/static/**" security="none" />

   <http use-expressions="true" >
     <intercept-url pattern="/" access="permitAll" />
     <intercept-url pattern="/enter" access="hasRole('user')" />
     <intercept-url pattern="/**" access="denyAll" />
     <form-login default-target-url="/" />
     <logout  logout-success-url="/" />
   </http>

   <beans:bean id="contextSource"
         class="org.springframework.security.ldap.DefaultSpringSecurityContextSource">
     <beans:constructor-arg value="ldap://login.serafeim.gr:389/dc=serafeim,dc=gr"/>
     <beans:property name="anonymousReadOnly" value="true"/>
   </beans:bean>

   <beans:bean
         id="userSearch"
         class="org.springframework.security.ldap.search.FilterBasedLdapUserSearch">
     <beans:constructor-arg index="0" value=""/>
     <beans:constructor-arg index="1" value="(uid={0})"/>
     <beans:constructor-arg index="2" ref="contextSource" />
   </beans:bean>

   <beans:bean
         id="ldapAuthProvider"
         class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider">
     <beans:constructor-arg>
       <beans:bean class="org.springframework.security.ldap.authentication.BindAuthenticator">
         <beans:constructor-arg ref="contextSource"/>
         <beans:property name="userSearch" ref="userSearch" />
         <!--
         <beans:property name="userDnPatterns">
           <beans:list><beans:value>uid={0},ou=People</beans:value></beans:list>
         </beans:property>
         -->
       </beans:bean>
     </beans:constructor-arg>
     <beans:constructor-arg>
       <beans:bean class="gr.serafeim.springldapcustom.CustomLdapAuthoritiesPopulator" />
     </beans:constructor-arg>
   </beans:bean>

   <authentication-manager>
     <authentication-provider ref="ldapAuthProvider" />
   </authentication-manager>

</beans:beans>

So, in the above configuration we’ve defined three spring beans: contextSource, userSearch and ldapAuthProvider. The <authentication-manager> element uses the ldapAuthProvider as an authentication provider. Below we will explain these beans:

contextSource

The contextSource bean defines the actual LDAP server that we are going to connect to. It has the class o.s.s.ldap.DefaultSpringSecurityContextSource. This will need to be passed to other beans that would need to connect to the server for a number of operations. We pass to it the url of our LDAP server and set its anonymousReadOnly property to true. The anonymousReadOnly defines if we can anonymously connect to our LDAP server in order to perform the search operation below. If we cannot connect anonymously then we have to set its userDn and password properties.

A very interesting question is if the <ldap-server> element of the spring security namespace is related to the o.s.s.ldap.DefaultSpringSecurityContextSource like our contextSource. To find out, we need to check the o.s.s.config.SecurityNamespaceHandler class of the spring-security-config.jar. In there we see the loadParsers method which has the line: parsers.put(Elements.LDAP_SERVER, new LdapServerBeanDefinitionParser());. The constant o.s.s.config.Elements.LDAP_SERVER has the value of "ldap-server" as expected, so we need to see what does the class o.s.s.config.ldap.LdapServerBeanDefinitionParser do. This class has a parse() method that receives the xml that was used to instantiate the <ldap-server> element and, depending an on the actualy configuration, instantiates a bean of the class o.s.s.ldap.DefaultSpringSecurityContextSource with an id of o.s.s.securityContextSource that will be used by the other elements in the security namespace !

This actually solves another question I had concerning the following error:

No bean named ‘org.springframework.security.authenticationManager’ is defined: Did you forget to add a gobal <authentication-manager> element to your configuration (with child <authentication-provider> elements)? Alternatively you can use the authentication-manager-ref attribute on your <http> and <global-method-security> elements.

What happens is that when spring-security-configuration encounters an <authentication-manager> it will instantiate a bean named o.s.s.authenticationManager having the class o.s.s.authentication.ProviderManager and will create and pass to it a providers list with all the authentication providers that are defined inside the <authentication-manager> element with <authentication-provider> nodes. So, if you encounter the above error, the problem is that for some reason your <authentication-manager> is not configured correctly, so no o.s.s.authenticatioManager bean is created!

userSearch

The userSearch bean is needed if we don’t know exactly where our users are stored in the LDAP directory so we will use this bean as a search filter. If we do know our user tree then we won’t need this bean at all as will be explained later. It has the class o.s.s.ldap.search.FilterBasedLdapUserSearch and gets three constructor parameters: searchBase, searchFilter and contextSource. The searchBase is from where in the LDAP tree to start searching (empty in our case), the searchFilter defines where is the username (uid in our case) and the contextSource has been defined before.

ldapAuthProvider

This is the actual authentication-provider that the spring-security authentication-manager is going to use. It is an instance of class o.s.s.ldap.authentication.LdapAuthenticationProvider which has two main properties: An o.s.s.ldap.authentication.LdapAuthenticator implementation and an o.s.s.ldap.userdetails.LdapAuthoritiesPopulator implementation. The first interface defines an authenticate method and is used to actually authenticate the user with the LDAP server. The second interface defines a getGrantedAuthorities which returns the roles for the authenticated user. The LdapAuthoritiesPopulator parameter is actually optional (so we can use LDAP to authenticate only the users) and we can provide our own implementation to have custom authorities for our application. That is exactly what we’ve done here.

The two arguments used to initialize the ldapAuthProvoder are one instance of o.s.s.ldap.authentication.BindAuthenticator which is a simple authenticator that tries to bind with the given credentials to the LDAP server to check the credentials and one instance of a custom class named g.s.s.CustomLdapAuthoritiesPopulator which is the actual implementation of the custom roles provider. The BindAuthenticator gets the contextSource as a constructor parameter and its userSearch property is set with the userSearch bean defined previously. If we instead knew the actual place of the users, we could use the commented out userDnPatterns property which takes a list of possible places in the LDAP catalog which will be checked for the username.

CustomLdapAuthoritiesPopulator

The CustomLdapAuthoritiesPopulator just needs to implement the LdapAuthoritiesPopulator interface. Here’s our implmentation:

package gr.serafeim.springldapcustom;

import java.util.Collection;
import java.util.HashSet;
import org.springframework.ldap.core.DirContextOperations;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.authority.SimpleGrantedAuthority;
import org.springframework.security.ldap.userdetails.LdapAuthoritiesPopulator;
import org.springframework.stereotype.Component;

@Component
public class CustomLdapAuthoritiesPopulator implements LdapAuthoritiesPopulator {
       @Override
       public Collection<? extends GrantedAuthority> getGrantedAuthorities(
                       DirContextOperations userData, String username) {
               Collection<GrantedAuthority> gas = new HashSet<GrantedAuthority>();
               if(username.equals("spapas")) {
                       gas.add(new SimpleGrantedAuthority("admin"));
               }
               gas.add(new SimpleGrantedAuthority("user"));
               return gas;
       }
}

The getGrantedAuthorities just checks the username and add another role if it is a specific one. Of course here we would autowire our user roles repository and query the database to get the roles of the user, however I’m not going to do that for the case of simplicity.

Example

When we run this application and go to the /enter, after logging in with our LDAP credentials as spapas, we will get the following output:

Username: spapas

Authorities: [admin, user]

Conclusion

In the previous a complete example of configuring a custom authorities populator was represented. Using this configuration we can login through the LDAP server of our organization but use application specific roles for our logged-in users.

[*]	Which is how spring calls the groups/roles the user belongs to

git branches

Serafeim Papastefanos — Tue, 08 Oct 2013 13:20:00 +0300

Contents

Introduction
Creating a new git repository
Branching
Remote branches

Introduction

A branch is a very interesting git feature. With this you may have more than one branches in the same repository. The main usage of this feature would be to create different versions of your source code to parallel test development of different features. When the development of each of these features has been finished then the different versions would need to be combined (or merged) to a single version. Of course, merging is not always the result of branching - some branches may exist indefinitely or other may just be deleted without merging.

I will try to experiment with it and comment on the results.

Creating a new git repository

Let’s start by creating a new git repository:

D:\>mkdir testgit
D:\>cd testgit
D:\testgit>echo contents 11 > file1.txt
D:\testgit>echo contents 222 > file2.txt
D:\testgit>echo 3333 > file2.txt
D:\testgit>copy con file3.txt
line 1 of file 3

line 3 of file 3

test

line 7 of file 3
^Z
       1 files copied.

D:\testgit>git init
Initialized empty Git repository in D:/testgit/.git/
D:\testgit>git add .
D:\testgit>git commit -m Initial
[master (root-commit) 96ca9af] Initial
 3 files changed, 9 insertions(+)
 create mode 100644 file1.txt
 create mode 100644 file2.txt
 create mode 100644 file3.txt

To see the branch we are in we can use the git branch command. Also git status outputs the current branch:

D:\testgit>git status
# On branch master
nothing to commit, working directory clean
D:\testgit>git branch
* master

So, it seems that when we create a new repository, a "master" branch is created.

Branching

Lets create a new branch and change our working branch to it:

D:\testgit>git branch slave
D:\testgit>git branch
* master
  slave
D:\testgit>git checkout slave
Switched to branch 'slave'
D:\testgit>git branch
  master
* slave

We can see that now the slave branch is the current one. Let’s do some changes and add commit them to the slave branch:

D:\testgit>git branch
  master
* slave
D:\testgit>echo new file1 contents > file1.txt
D:\testgit>git commit -m "Slave modification"
[slave b6083ad] Slave modification
 1 file changed, 1 insertion(+), 1 deletion(-)
D:\testgit>git checkout master
Switched to branch 'master'
D:\testgit>more file1.txt
contents 11
D:\testgit>git checkout slave
Switched to branch 'slave'
D:\testgit>more file1.txt
new file1 contents

So the contents of file1.txt in the branch master is contents 11 while the contents of the same file in the branch slave is new file1 contents.

An interested behaviour is what happens with uncommit changes when changing branches. Let’s try deleting a file:

D:\testgit>del file2.txt
D:\testgit>git status
# On branch master
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       deleted:    file2.txt
#
no changes added to commit (use "git add" and/or "git commit -a")
D:\testgit>git checkout slave
D       file2.txt
Switched to branch 'slave'
D:\testgit>git status
# On branch slave
# Changes not staged for commit:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       deleted:    file2.txt
#
no changes added to commit (use "git add" and/or "git commit -a")
D:\testgit>git add -A
D:\testgit>git status
# On branch slave
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       deleted:    file2.txt
#
D:\testgit>git checkout master
D       file2.txt
Switched to branch 'master'
D:\testgit>git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       deleted:    file2.txt
#

So, our changes are not correlated with a branch until we commit them! Let’s commit them to the master repository and confirm that:

D:\testgit>git commit -m "Deleted file2.txt"
[master 6f8749d] Deleted file2.txt
 1 file changed, 1 deletion(-)
 delete mode 100644 file2.txt
D:\testgit>git status
# On branch master
nothing to commit, working directory clean
D:\testgit>dir file2.txt
[...]
File not found
D:\testgit>git checkout slave
Switched to branch 'slave'
D:\testgit>git status
# On branch slave
nothing to commit, working directory clean
D:\testgit>dir file2.txt
[...]
08/10/2013  05:59 pm                15 file2.txt

This is interesting… Let’s try modifying the file2.txt (which does not exist to the master branch):

D:\testgit>git branch
  master
* slave
D:\testgit>echo new file2 contents > file2.txt
D:\testgit>git add .
D:\testgit>git status
# On branch slave
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       modified:   file2.txt
#
D:\testgit>git checkout master
error: Your local changes to the following files would be overwritten by checkout:
       file2.txt
Please, commit your changes or stash them before you can switch branches.
Aborting

We won’t be able to change the current branch until we commit the conflicting change:

D:\testgit>git commit -m "Modified file2"
[slave b5af832] Modified file2
 1 file changed, 1 insertion(+), 1 deletion(-)
D:\testgit>git checkout master
Switched to branch 'master'

Remote branches

For each local repository you can define a number of remote repositories, or remotes as git calls them. When you clone a repository from github.com, your local repository will have one remote, named origin. We will try to add the same remote by hand. Let’s suppose that we have created a repository in github.com named testgit. After that we wil issue:

D:\testgit>git remote
D:\testgit>git remote add origin https://github.com/spapas/testgit.git
D:\testgit>git remote
origin

So no we have one remote named origin that is linked with https://github.com/spapas/testgit.git. Let’s try to push our master branch to the origin remote:

D:\testgit>git push origin master
Username for 'https://github.com': spapas
Password for 'https://spapas@github.com':
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (7/7), 531 bytes, done.
Total 7 (delta 1), reused 0 (delta 0)
To https://github.com/spapas/testgit.git
 * [new branch]      master -> master
D:\testgit>git branch -r
  master
* slave
  remote/origin/master

We see now that we have three branches. Two local (master slave) and one remote (origin/master). We will also add the slave remote (origin/slave):

D:\testgit>git branch -r
  origin/master
  origin/slave

Let’s do a change to our local repository and then push them to the remote:

D:\testgit>notepad file3.txt
D:\testgit>git add .
D:\testgit>git status
# On branch slave
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       modified:   file3.txt
#
D:\testgit>git commit -m "Changed file3.txt"
[slave ce3b7b9] Changed file3.txt
 1 file changed, 1 insertion(+), 1 deletion(-)
D:\testgit>git push origin slave
Username for 'https://github.com': spapas
Password for 'https://spapas@github.com':
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 299 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
To https://github.com/spapas/testgit.git
   b5af832..ce3b7b9  slave -> slave

Everything works as expected. The final thing to test is to try checking out a remote branch:

D:\testgit>git checkout master
Switched to branch 'master'
D:\testgit>echo new new file1 > file1.txt
D:\testgit>more file1.txt
 new new file1
D:\testgit>git checkout origin/master
M       file1.txt
Note: checking out 'origin/master'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 6f8749d... Deleted file2.txt
D:\testgit>git status
# Not currently on any branch.
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   file1.txt
#
no changes added to commit (use "git add" and/or "git commit -a")
D:\testgit>more file1.txt
new new file1

So, it seems that when we check out the remote branch, we won’t have any local branches, however the change we did to the file1.txt is transfered just like when switching from one local repository to another. We can then add the changes and commit:

D:\testgit>git add .
D:\testgit>git commit
[detached HEAD 506674c] foo
 1 file changed, 1 insertion(+), 1 deletion(-)
 D:\testgit>git status
# Not currently on any branch.
nothing to commit, working directory clean
D:\testgit>git branch
* (no branch)
  master
  slave

So we are working with an unnamed branch! We have to name it to be able to work without problems:

D:\testgit>git checkout -b named_branch
Switched to a new branch 'named_branch'
D:\testgit>git branch
  master
* named_branch
  slave

Finally we may push again the named_branch to our remote origin.

Using pelican to generate static sites on windows

Serafeim Papastefanos — Mon, 07 Oct 2013 10:20:00 +0300

Contents

Introduction
Installing pelican and generating a skeleton for your site
Modifying pelican tools for windows
Configuration of your skeleton site
- Settings
- Themes
- Plugins
Hosting in github pages
Publishing changes

Introduction

Pelican is a great static site generator. A static site generator is a tool that users a number of input files to generate a complete html site. No database or server side scripting is needed for the resulting site, that’s why many such sites are hosted on github pages (more on this later).

The input contains a number of html templates, css styles and the actual content of the site which most of the time is written in a lightweight markup language like reStructuredText or Markdown. The static site generator will generate the static pages by inserting the content in the appropriate places in the templates.

In the following sections we will describe the installation of pelican on Windows and the creation of the spapas.github.io site.

Installing pelican and generating a skeleton for your site

The official pelican quickstart can be found in http://docs.getpelican.com/en/latest/getting_started.html

To install pelican just enter:

>pip install pelican

After installing pelican, I propose creating a parent directory that will contain all your pelican sites, along with extra themes and plugins like this:

pelican/
  ├── pelican-themes
  ├── other-pelican-theme
  ├── spapas.github.io
  └── other-sites

After creating the pelican directory just go in it with a command line and run the pelican-quickstart command. It will ask you a number of questions, take a look at how I did answer these:

pelican>pelican-quickstart
Welcome to pelican-quickstart v3.2.2.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.


> Where do you want to create your new web site? [.] spapas.github.io
> What will be the title of this web site? Test github.io
> Who will be the author of this web site? Serafeim
> What will be the default language of this web site? [en]
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n)
> What is your URL prefix? (see above example; no trailing slash) http://spapas.github.io
> Do you want to enable article pagination? (Y/n)
> How many articles per page do you want? [10]
> Do you want to generate a Makefile to easily manage your website? (Y/n) n
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n) n
Done. Your new project is available at C:\progr\py\pelican\spapas.github.io

After that, you will have a pelican/spapas.github.io folder that will contain the following content:

spapas.github.io/
  ├── content
  ├── output
  ├── pelicanconf.py
  └── publishconf.py

The content folder will contain your content (rst or markdown), the output will contain the generated html after you run pelican for your site. The pelicanconf.py will have a number of options for the generation of the development version of your site while the publishconf.py will override some of the options of pelicanconf.py before generating the production version of your site that will actually be uploaded to github pages.

Modifying pelican tools for windows

Pelican uses a Makefile and a unix shell script to generate the static html files and start an http server for development. Because I prefer to use windows, I answered no to the questions of generating these when pelican-quickstarte asked me. Instead I have included the following files inside the spapas.github.io directory:

pelrun.bat, to generate the content for your debug site in the output directory:

pelican content --debug --autoreload  --output output --settings pelicanconf.py

pelserve.bat, to localy serve the generated debug site:

pushd output
python -m pelican.server
popd

pelpub.bat, to generate the production site in the output directory:

pelican content --output output --settings publishconf.py

Now, when you want to develop your site locally, enter:

spapas.github.io>start pelrun.bat
spapas.github.io>start pelserv.bat

If everything was ok until now, you can visit http://127.0.0.1:8000 and will get the following output:

Because of the -r option that is used in pelrun.bat whenever you do a change (for instance when you add an rst file in the content directory) it will be discovered and the output will be changed immediately!

Configuration of your skeleton site

Settings

There is a number of settings that you may configure in your site. The pelican settings reference can be found here: http://docs.getpelican.com/en/latest/settings.html. The pelicanconf.py and publishconf.py for this site can be found here:

https://github.com/spapas/spapas.github.io/blob/source/pelicanconf.py https://github.com/spapas/spapas.github.io/blob/source/publishconf.py

The most important difference is the RELATIVE_URLS directive which must be True to the debug and False to the production.

Themes

Beyond the default pelican theme, you can use configure pelican to use various other themes. To enable them, go to your pelican parent directory and clone the pelican-themes github repository:

pelican>git clone https://github.com/getpelican/pelican-themes.git

After that, you may select your theme from pelicanconf.py:

THEME = "../pelican-themes/theme-name"

In my configuration, I am using the Octopress Theme for Pelican, which you may get from here https://github.com/duilio/pelican-octopress-theme. Just clone it to your pelican directory and refer to it as above.

Plugins

Pelican has a number of plugins. To enable them, go to your pelican parent directory and clone the pelican-plugins github repository:

pelican>git clone https://github.com/getpelican/pelican-plugins.git

After that, you may add the following two lines to your pelicanconf.py:

PLUGIN_PATH = '../pelican-plugins'
PLUGINS = ['a-plugin']

Hosting in github pages

To host your static site in github pages you must first of all create a repository named username.github.io (for instance spapas.github.io) from github.

Then, generate your production output:

spapas.github.io>pelpub.bat

Finally, go to your output directory, create a git repository, add everything and push it to your repository:

spapas.github.io\output>git init
spapas.github.io\output>git add .
spapas.github.io\output>git commit -m Initial
spapas.github.io\output>git remote add origin https://github.com/spapas/spapas.github.io.git
spapas.github.io\output>git push origin master --force

The —force is to overwrite any previous versions - you don’t care about version control on your output (but you want it on your source).

You can now visit http://username.github.io and see your statically generated site !

Don’t forget to add your source to the version control! To do that, add a .gitignore file in your pelican/username.github.io direcory containing the following:

output

The above file will ignore the contents of the output directory from version control. After that, do the following:

spapas.github.io>git init
spapas.github.io>git add .
spapas.github.io>git commit -m Initial
spapas.github.io>git branch -m master source
spapas.github.io>git remote add origin https://github.com/spapas/spapas.github.io.git
spapas.github.io>git push origin source

The above will rename the master branch to source, will attach the origin remote to https://github.com/spapas/spapas.github.io.git and will push the source branch to it. Now you will have two branches in your username.github.io repository. One named origin/master that will be your actual site and will be displayed through http://username.github.io and one named origin/source that will contain the source of your site.

To learn more about branches and remotes you may check out the git branches article.

Publishing changes

Whenever you need to publish a new article or do changes to an existing one, you need to do the following:

Run pelpub.bat to create the new output
Add/commit and push changes from your pelican site(source) folder to the source remote
Add/commit and push changes from your output folder to the master remote

To help with this, here’s a ghdeploy.bat file that does all the above:

call pelpub.bat
git add -A
git commit -m "Deploying changes"
git push origin source
pushd output
git add -A
git commit -m "Deploying changes"
git push origin master
popd

If you’ve followed this far, by running pelpub.bat you’ll need to enter your github repository credentials (twice) and then everything (source and master) will be deployed! To make things even better, I propose to use ssh based authentication to your github account and add new remote names to your source and master by running the following:

git remote add origin2 git@github.com:spapas/spapas.github.io.git

to your pelican site and the output directories. After you change ghdeploy.bat to use origin2 instead of origin you’ll be able to deploy everything with just running it without entering any credentials!