Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

May 28 2017

Ionic 3 : Create a cross platform mobile application with Python Django and Django Rest Framework (DRF)

Ionic 3 with Django backend and DRF

In this two parts tutorial ,we are going to recreate our previous Ionic 3 product inventory manager application ,which used a local SQLite database ,to use a Python Django backend .

The application we'll be creating in this tutorial uses Ionic 3 as a frontend ,Django as a backend and Django Rest Framework to create Rest API which will be consumed by Ionic .

It's a simple product inventory management application which shows you many Django concepts such as

How to create a Django project from scratch .

How to create a Django app .

How to model and create database models .

How to migrate your database .

How to generate the Admin web interface to create ,update ,delete and view database records .

How to create a super user .

How to generate a browsable and documented Rest API with Django Rest Framework .

What is Django ?

Django is a Python based web framework which encourages rapid development which is used by many web developers create web applications .It has a clean and pragmatic design which can help ypu create complete prototypes in a hours .Django packages or apps for organization and reuse .You can either create your own apps or use community created apps to sove common web development problems without reinventing the wheel .

Django has a great and helpful community ,a very good documentation and a a lot of tutorials on the web which can help you pickup easily the framework .

What is Django Rest Framework (DRF) ?

Django Rest Framework or DRF is a Django package which allows you to build or generate a full browsable and documented Rest API from your models .Thanks to DRF you can easily get an API which allows you create , update ,delete and view database records from different meduims such as mobile apps or web apps .

What is Ionic ?

Ionic is a hybrid mobile framework to create cross platform mobile apps for Android ,iOS and Universal Windows
Platform (UWP) using web technologies such as HTML ,CSS and JavaScript .Ionic 3 (The latest version of Ionic when writing this tutorial ) is based on Cordova and Angular 4 .

Install Django and Create a project

Head over your terminal or command prompt then create a new virtual environment and activate it :

virtualenv env 
source env/bin/activate 

Install the latest version of Django :

pip install django 

Start a new Django project :

django-admin startproject ProductInventoryManager 

Create an app

cd ProductInventoryManager
python manage.py startapp core 

Go to your project settings.py and add your project to the list of installed apps

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'core'
]

Create your database and run a local development server :

python manage.py migrate 
python manage.py runserver 

A SQLite database will be created on your current directory .You can change the database to mySql or PostgresSQL anytime but for now lets use to a SQLite database .

A local server will be launched ,you can visit your web app at http://localhost:8200

Project requirements and database modeling

The next step is to create database models for our backend but lets first talk about our project requirements

Our product inventory system has products ,each product belongs to a family and has a location

Users need to be able to create products ,families ,locations and transactions

Users need to be able to edit products ,families ,locations and transactions

Users need to be able to delete products ,families ,locations and transactions

Users need to be able to list products ,families ,locations and transactions

Users can create products with an initial Quantity then update this Quantity using Transactions .

Now lets create our database models

Open core/models.py file and add these models :

A Product has a bunch of properties such as : Title ,Description ,Unit Price ,SKU (Stock Keeping Unit) ,Barcode (ISBN, UPC etc.) ,Quantity ,minQuantity ,Unit .

A Product belongs to a Family and has a location

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models


class Product(models.Model):

    sku = models.CharField(max_length=13,help_text="Enter Product Stock Keeping Unit")
    barcode = models.CharField(max_length=13,help_text="Enter Product Barcode (ISBN, UPC ...)")

    title = models.CharField(max_length=200, help_text="Enter Product Title")
    description = models.TextField(help_text="Enter Product Description")

    unitCost = models.FloatField(help_text="Enter Product Unit Cost")
    unit = models.CharField(max_length=10,help_text="Enter Product Unit ")

    quantity = models.FloatField(help_text="Enter Product Quantity")
    minQuantity = models.FloatField(help_text="Enter Product Min Quantity")

    family = models.ForeignKey('Family')
    location = models.ForeignKey('Location')

    def get_absolute_url(self):
        """
        Returns the url to access a particular instance of Product.
        """
        return reverse('product-detail-view', args=[str(self.id)])

    def __str__(self):

        return self.title

A Family has a : Reference,Title ,Description ,Unit ,minQuantity .

class Family(models.Model):

    reference = models.CharField(max_length=13, help_text="Enter Family Reference")
    title = models.CharField(max_length=200, help_text="Enter Family Title")
    description = models.TextField(help_text="Enter Family Description")

    unit = models.CharField(max_length=10,help_text="Enter Family Unit ")

    minQuantity = models.FloatField(help_text="Enter Family Min Quantity")

    def get_absolute_url(self):
        """
        Returns the url to access a particular instance of Family.
        """
        return reverse('family-detail-view', args=[str(self.id)])

    def __str__(self):

        return self.title

A Location has a : Reference ,Title , Description .

class Location(models.Model):

    reference = models.CharField(max_length=20, help_text="Enter Location Reference")
    title = models.CharField(max_length=200, help_text="Enter Location Title")
    description = models.TextField(help_text="Enter Location Description")

    def get_absolute_url(self):
        """
        Returns the url to access a particular instance of Location.
        """
        return reverse('family-detail-view', args=[str(self.id)])

    def __str__(self):

        return self.title

A Transaction has a : Date , Quantity , Unit Cost , Reason (New Stock - Usable Return - Unusable Return ) ,UPC (Universal Product Code ) ,Comment .

class Transaction(models.Model):

    sku = models.CharField(max_length=13,help_text="Enter Product Stock Keeping Unit")
    barcode = models.CharField(max_length=13,help_text="Enter Product Barcode (ISBN, UPC ...)")

    comment = models.TextField(help_text="Enter Product Stock Keeping Unit")

    unitCost = models.FloatField(help_text="Enter Product Unit Cost")

    quantity = models.FloatField(help_text="Enter Product Quantity")

    product = models.ForeignKey('Product')

    date = models.DateField(null=True, blank=True)

    REASONS = (
        ('ns', 'New Stock'),
        ('ur', 'Usable Return'),
        ('nr', 'Unusable Return'),
    )


    reason = models.CharField(max_length=2, choices=REASONS, blank=True, default='ns', help_text='Reason for transaction')

    def get_absolute_url(self):
        """
        Returns the url to access a particular instance of Product.
        """
        return reverse('transaction-detail-view', args=[str(self.id)])

    def __str__(self):

        return 'Transaction :  %d' % (self.id)

Registering models in admin interface

The Django admin can use your models to auto create a web interface which allows you to add ,delete ,update and view records .

To tell django about our newly created models .Open core/admin.py file the copy and paste

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.contrib import admin

from .models import Product ,Family ,Location ,Transaction  

admin.site.register(Product)
admin.site.register(Family)
admin.site.register(Location)
admin.site.register(Transaction)

Now to acess the admin web interface we need to create a super user so head over to your terminal an run

python manage.py createsuperuser 

Enter the credentials you want to use and hit Enter .

Next go to http://localhost:8200/admin then enter the credentials you just created .

You can now use the admin interface to create records for different models such as Products ,Families , Locations and Transactions .

Django 1.11 : Create super user for Admin back office

Django 1.11 create super user

After you successfully created your first project with Django 1.11 ,created an app and migrated your SQlite database .It's time to access the admin interface or back office generated automatically by Django which you can use to create ,delete ,edit and list your database tables data .

You can visit the admin interface from http://localhost:8200/admin

You'll be presented with a form to enter your username and password credentials .If you enter any values you'll get a message :

Please enter the correct username and password for a staff account. 
Note that both fields may be case-sensitive.

Since you have no staff user created yet but don't worry this can be fixed with one command .

Head over to your terminal ane run :

python manage.py createsuperuser

You'll be prompted for a username and a password (twice) ,enter them and hit Enter .

Next run your local server again :

python manage.py runserver 

Then visit your admin interface http://localhost:8200/admin and enter your credentials then hit Login button .

You should be successfully logged in .

May 27 2017

How to Configure Mailgun To Send Emails in a Django Project

In this tutorial you will learn how to setup a Django project to send emails using the Mailgun service.

Previously I’ve published in the blog a post about how to configure SendGrid to send emails. It’s a great service, but they don’t offer free plans anymore, nowadays it’s just a 30 days free trail. So, I thought about sharing the whole email setup with a better option to get started.

Mailgun is great and super easy to setup. The first 10,000 emails you send are always free. The only downside is that if you don’t provide a payment information (even though you are only going to use the first 10,000 free emails), there will be some limitations, such as requiring to configure “Authorized Recipients” for custom domains, which pretty much makes it useless, unless you know beforehand the email addresses you will be sending emails.

Anyway, let’s get started.

Initial Setup

Go to www.mailgun.com and create a free account. Sign in with your Mailgun account, click on Domains and then Add New Domain.

Add New Domain Button Screen Shot

I will setup the Mailgun service for a domain I own, “www.bottlenose.co”. For the setup, it’s advised to use the “mg” subdomain, so you will need to provide the Domain Name like this:

mg.bottlenose.co

From now on, always change bottlenose.co with your domain name.

Add New Domain Screen Shot

Click on Add Domain.

Domain Verification & DNS

To perform the next steps, you will need to access the DNS provider of your domain. Normally it’s managed by the service/website you registered your domain name. In my name, I registered the “www.bottlenose.co” domain using Namecheap.

The next steps should be more or less the name. Try to find something that says “manage”, “DNS records”, “Advanced DNS” or something similar.

DNS Records For Sending

In the Mailgun website you will see the following instructions:

DNS Records For Sending Screen Shot

Add the DNS records accordingly in your DNS provider:

Namecheap Advanced DNS TXT Records Screen Shot

Namecheap Advanced DNS MX Records Screen Shot

DNS Records For Tracking

In a similar way, add now a CNAME for tracking opens, clicks etc. You will see those instructions:

DNS Records For Tracking Screen Shot

Follow them accordingly:

Namecheap Advanced DNS CNAME Record Screen Shot

Remember, in the previous screenshot you are supposed to do in your DNS provider!

Wait For Your Domain To Verify

Now it’s a matter of patience. We gotta wait for the DNS to propagate. Sometimes it can take an eternity to propagate. But my experience with brand new domains is that it usually happens very quickly. Wait like 5 minutes and give it a shot.

Click on Continue to Domain Overview:

Continue to Domain Overview Button Screen Shot

You will now see something like this:

Domain Overview Screen Shot

Click on Check DNS Records Now and see if Mailgun can verify your domain (remember, this process can take up to 48 hours!).

If the verification was successful, you will see the screen below:

Active Domain Screen Shot

Configuring Django to Send Emails

To configure you Django Project, add the following parameters to your settings.py:

EMAIL_HOST = 'smtp.mailgun.org'
EMAIL_PORT = 587
EMAIL_HOST_USER = 'postmaster@mg.bottlenose.co'
EMAIL_HOST_PASSWORD = 'mys3cr3tp4ssw0rd'
EMAIL_USE_TLS = True

Note that we have some sensitive informations here, such as the EMAIL_HOST_PASSWORD. You should not put it directly to your settings.py file or commit it to a public repository. Instead use environment variables or use the Python library Python Decouple. I have also written a tutorial on how to use Python Decouple.

Here is a very simple snippet to send an email:

from django.core.mail import send_mail

send_mail('subject', 'body of the message', 'noreply@bottlenose.co', ['vitor@freitas.com'])

And here is how the email will look like, displaying properly your domain:

Email Sent

If you need to keep reading about the basic email functions, check my previous article about email: How to Send Email in a Django App.

May 25 2017

Django efficient implementation of Amazon s3 and Cloudfront CDN for faster loading.

Django by default to store the files in your local file system. To make your files load quickly and secure we need to go for any third party storage systems. AWS s3 is one of the storage service for the Internet. It is designed to make web-scale computing easier for developers. django has a package called django-storages which can be used to store the files in the amazon s3. and serve them from its cloud front service.

Implementing django-storages with amazon s3 as storage service in your django application:

To implment django-storages in your django application you need to follow the following steps.

1. Install django-storages: We can install django-storages with the following command.

pip install django-storages

2. Keep storages in your installed apps.

INSTALLED_APPS = (
    ...
    'storages',
    ...
)

3. Change your DEFAULT_FILE_STORAGE setting in your settings.py file: Django by default comes with a setting called DEFAULT_FILE_STORAGE which deals with file storage. By default its value is 'django.core.files.storage.FileSystemStorage'. By keeping this value to DEFAULT_FILE_STORAGE, django, by default tries to store the files on your local machine. To store your files in amazon s3 just change that value to 'storages.backends.s3boto3.S3Boto3Storage'

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

4. Configure your AWS ACCESS_KEY and SECRET_KEY with the following settings:

AWS_ACCESS_KEY_ID = "your aws access key"
AWS_SECRET_ACCESS_KEY = "your aws secret key"

5. Keep your bucket in the settings file: A bucket is a logical unit of storage in Amazon Web Services (AWS) object storage service, Simple Storage Solution S3. Buckets are used to store objects, which consist of data and metadata that describes the data. Create a bucket on AWS S3 web console and give that name in your settings file.

AWS_STORAGE_BUCKET_NAME = "your bucket name"

6. Keep file permissions with AWS_DEFAULT_ACL setting: There are some permissions for every file that is stored in your S3 bucket. There are 2 permissions in particular which will be used by default. They are private and public-read. To make a file accessable to only admin we have to make it private, If we make the file permission as public-read it can be accessable to public with read permissions.

AWS_DEFAULT_ACL = "public-read" # to make sure all your files gives read only access to the files

7. Keep your file specific meta information with AWS_HEADERS: This setting will be useful in keeping the file's meta information like Expires and Cache-Control which will play major role in the google page speed score. In fixing the Leverage Browser Caching issue we have to keep Cache-Control setting value and keep Expires value in fixing the keep expiration header issue in your google page speed validator.

AWS_HEADERS = {
    'Expires': 'Thu, 15 Apr 2010 20:00:00 GMT',
    'Cache-Control': 'max-age=86400',
}

By keeping all the above settings, we are ready to use the AWS S3 as a storage service for all the files in our project. But to make our files serve from cloud front we have keep other settings that relates to cloudfront in our application.

What is cloud front?

Amazon CloudFront is a web service that speeds up distribution of your static and dynamic web content, such as .html, .css, .php, and image files, to your users. CloudFront delivers your content through a worldwide network of data centers called edge locations. When a user requests content that you're serving with CloudFront, the user is routed to the edge location that provides the lowest latency (time delay), so that content is delivered with the best possible performance. 

To make sure our files served from cloudfront we have to keep the following settings in our application.

CLOUDFRONT_DOMAIN = "your cloudfornt domain"
CLOUDFRONT_ID = "your cloud front id"
AWS_S3_CUSTOM_DOMAIN = "same as your cloud front domain" # to make sure the url that the files are served from this domain

Thats all you have to do, now all the files that the user uploads or the static files will be stored in the S3 bucket that you gave and served from cloudfront. To make your static files load from cloudfront, first you need to do python manage.py collectstatic, using which all your static CSS and JS files will be stored in your S3 bucket under the folder of STATICFILES_DIR setting you gave.

May 23 2017

Interview with Andrew Godwin about deployment

This is an interview I took from Django core developer Andrew Godwin about deployment. Recorded in DjangoCon EU 2017, Florence, Italy.

https://djangodeployment.com/wp-content/uploads/2017/05/godwin.mp3

 

ANT: Hello everyone! This is Antonis from djangodeployment.com. It’s 5 April 2017 and I am at the European Django Conference in Florence. I am sitting with Andrew Godwin, who is a Django core developer mostly known as the creator of migrations and now Django Channels, and we are going to have a little chat about deployment. Hello Andrew!

AND: Hello! It’s lovely to talk to you!

ANT: Thanks! Some years ago you tried to create a deployment service which was called epio. So, could you tell me a little bit more about that?

AND: Of course! As a bit of background history, we’re talking quite a few years now when Heroku was just for Ruby at the time, and so I was sitting in DjangoCon US and me and a friend had the idea to, why not do Heroku for Python?, which of course was an idea common to several startups at that time. We took on the challenge of making a platform-as-a-service, which is what Heroku of course is. I think it was two or three years from start to end of that project. It was quite difficult, and eventually we sort of shut down gradually, the company itself moved to a project then called Orchard, which then later got sold on to Docker, Inc, so it still had a history through that time, but I left it around four years ago.

ANT: One question that I’ve been having about deployment, because I started a blog and a book about deployment and I see that people are really interested: So I asked my readers, why do you care when you can just press a button in Heroku and have your application deployed? I got a number of answers but I’d like to have your viewpoint.

AND: So that’s a very interesting and common question of course, and I have a number of different viewpoints. The first one for me is cost, ultimately. Heroku is very expensive for what they give you in terms of resources. Obviously the value out there is in terms of management and time saving and wages, but in general I would say that, as someone who’s experienced in deployment, I can do that. If that is in my range, I don’t need to spend too much for it. And the other part is customization. Heroku, I would say, is very good for a small or medium sized web site where you fit into the common pattern, but as you grow you inevitably become more specialized. You require different kinds of routing or network or storage and that’s the point where I think that something like Heroku restrains to make everything work, you can’t have one pattern that works for every big site. Now, would I recommend it? For a small site, absolutely. Would I run it? No. But that’s two very different things.

ANT: I think that services like Heroku, or, maybe, the Divio cloud, they treat deployment as a black box. That is essentially their purpose, to make deployment simpler for you, so you are not supposed to know anything about it. But do things eventually go wrong? And when things go wrong, is it useful to have knowledge of the mechanics of deployment in order to fix things? Or is it maturing now? Is it a matter of maturity?

AND: So it’s not just maturity, it’s also about how well you are equipped to deal with these problems. Having full knowledge of the system is not useful if you don’t have the resources to fix that system. You just know about your failure rather than not knowing about your failure. So I think it is in some cases better to know of a failure and be able to fix it, but doing that requires a lot of expertise that is hard to find, like good operations people are very very thin on the ground in almost every part of the world. So it really depends, is it worth for you to know how to fix the black box or can you take the failure? One good example that isn’t about deployment is a database. If MySQL or PostgreSQL fails, it’s technically open source, you can, in theory, go and fix it, but in all likelihood you don’t have the knowledge or the ability to fix the problem in PostgreSQL, and so, it’s not about if it’s technically close, it’s also if it’s by the nature of its complexity close to you. If I made you a system that is fully open, and gave you that system with the ssh keys and everything, but you didn’t know how it worked, it would still be effectively a black box, but now it’s a black box with no support contract. In a lot of cases it’s best to have a black box with a support contract than something you half-know yourself.

ANT: People compare Django deployment to PHP deployment, because PHP has a fame for easy deployment—you just throw your files in a directory and it just works. Is there any fundamental reason why Django has to be harder? And a related question is, is PHP deployment really simpler, or is it deceptive to think that you can just throw your files in a directory and that’s it?

AND: There’s two parts to that question. The first one that I will address is, is it really simpler? The answer is, for a small project yes. PHP has the unenviable ability to just take a directory, put a file in there, and you’re off. And for that reason I came into web development from PHP. I had my problems with it as a language but I still respect the ease of use of it. The problem is that simplicity quickly becomes irrelevant as you get bigger and bigger, and one of the downsides of PHP can often be, yes it is a directory, and so you have to have an htaccess file in the old days, there’s a few more security issues. In Python and Django it’s just having standards. Before the advent of WSGI—when I started doing Django WSGI was there but very new—it was much harder, you had to sort of find a custom web server, wrangle with it, and it wasn’t particularly right. So we have come a long way in having a standard where you can pick anyone of uWSGI, mod_wsgi, Gunicorn, a number of other servers, plug any Python application into them, and there you go. It is more difficult, but it’s only very marginally more difficult. And ultimately I think, what I see the difference is with PHP you can drag-and-drop files in an interface, with Django you need to have some command line access. I think for some people just the presence of a command line makes it seem more difficult—it may not be more difficult but it makes it seem more difficult. That’s ultimately the sticking point some people have.

ANT: One example I have been thinking about that, a very simple one, is that when you deploy you also need to setup the email settings so that it can email you in case of an error, and for people who have no experience in deployment this is also something that they have to learn. You have to do this also in PHP I guess.

AND: Of course, you do, the difference there being that PHP is not setup to email you errors by default. If you configure it you can make it show a 500 page; if you do it incorrectly, as if you leave DEBUG on Django, it will show you a full traceback and all these wonderful variables and things exposed in there. So it’s more the case that out of the box Django is looking for an email address to send errors to. In a way that might be outdated by itself, for example I use something like Sentry more often than emails, my mailbox is already full enough. It’s just the defaults, the way it arrives, like, you install Django, you open that settings file, and there’s like 40 different things that you’re like “Oh! Do I need this? Do I fill it in?” and one of those is the email address.

ANT: OK. This is everything, so thank you Andrew and thanks everyone for listening.

Did you like this post?

Interview
© 2017 Antonis Christofides and Andrew Godwin.
The interview is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License.

Photo by Bartek Pawlik
© 2017 DjangoCon Europe 2017
Original photo

The post Interview with Andrew Godwin about deployment appeared first on Django deployment.

May 22 2017

3 Reasons to Upgrade to the Latest Version of Django

When considering a website upgrade, many business stakeholders probably think about the frontend, i.e., how the website looks or the features users interact with. Perhaps less often considered is the importance of upgrading the backend; that is, the databases, applications, and servers powering all the behind-the-scenes activity. Infrastructure support and upgrades are necessary but often performed as a separate project from any improvements to design or user experience, rather than as part of a holistic update project.

With that in mind, it helps to have an understanding of why upgrading the backend should be considered a necessary part of any website upgrade project. We offer 3 reasons, focusing on our specialty of Django-based websites. Upgrading:

  • increases security,
  • reduces development and maintenance costs, and
  • ensures support for future growth.

Read on for details about each of these factors, or get in touch to speak with us about them.

Increase the security of your site

The Django framework is continually being improved and certain releases are designated as “Long Term Support” (LTS) versions. LTS versions receive security updates and bug fixes for a three-year period, as opposed to the usual 18 months. When your website uses an unsupported version of Django, newly uncovered bugs are not being fixed, patched, or supported by the Django and Open Source communities. No new security fixes are planned for retired versions, a situation that carries a number of risks.

These risks come in the form of vulnerabilities - weaknesses that leave your site open to attack. Attacks could potentially cause servers to go down, data to be leaked or stolen, or features to stop working. If a vulnerability is taken advantage of, it could lead to a loss of reputation and potentially a loss of revenue or legal ramifications. With high consumer expectations and increasing requirements from international data protection laws, this could prove disastrous for organizations or web applications without stringent upgrade plans in place.

If your site is using an older version of Django, a security patch may not be released for your version of Django. This means that a fix for the vulnerability would have to be authored and implemented by your development team, which, over time, is less cost effective than upgrading to the LTS version.

Upgrading to an LTS release offers significant benefits, including security updates as needed. Fixes for security issues and vulnerabilities are implemented quickly. There is no need to implement fixes yourself (or hire out expensive custom work). Taking proactive steps to upgrade reduces risk and can save you the trouble of expensive, reactive steps in the event of a cyberattack.

Reduce development and maintenance costs

In addition to improving security and ensuring support for future growth, upgrading also offers productivity benefits for development teams. Many extra lines of code may be required in order to continue to backport fixes for your website or app as issues occur or features are added. Adding all this code and continuing to use old versions of Django will eventually lead to technical debt, where the short-term fixes and outdated code end up creating extra work to patch and maintain a project in the long run.

Custom fixes and patches also introduce a large learning curve for new developers or contractors. The issue here is two-fold: Onboarding new developers is more time consuming than it needs to be, and if key personnel leave, you may lose knowledge which is integral to maintaining or updating the project.

Upgrading your version of Django reduces technical debt by eliminating old, no-longer-needed code. It also allows your development team to reduce the time and money spent on addressing security issues and bug fixes, freeing up time for them to work on website improvements or revenue-generating work.

Ensure support for future growth

Extensibility is the practice of keeping future growth in mind when working on a development project. We often hear from potential clients who built a website or web app in the early days of their business, when releasing features quickly took precedence over planning for future growth. Whether that growth is in the form of more data, more users, or more functionality, planning for it impacts current design and development decisions. When growth isn’t considered, scaling up the project and adding new features requires a disproportionate amount of work. If the original development was not intended to support the changes being made, custom workarounds must be introduced.

Where does this leave your web project? Technologically out of date, unnecessarily clunky, and less able to deliver a quality experience to site visitors.

Upgrading Django from an out-of-date version to a more recent LTS version not only provides access to software that is constantly receiving bug and security fixes; it also simplifies the upgrade process when a new version of Django is released with a feature needed by your project. If your project is two, three, even four releases behind, upgrading all at once could be cost-prohibitive. By regularly upgrading, you gain near-immediate access to new features in Django if and when needed. In other words, you can depend on a highly-engaged developer community actively working to add features rather than reinventing the wheel by developing them yourself.

Next steps

The wider open source development community is producing great tools and enhancements every day and the community continues to grow in size and support. Your project may find itself left behind if Django is left unsupported - or growing along with the community if you upgrade.

So where to get started? For clients considering an upgrade, we generally advise moving up to the most recent LTS release. While the latest version of Django offers the newest features, the LTS version represents a version of Django that will be more cost efficient to maintain given the community’s three-year commitment to releasing updates for it.

As Django specialists, Caktus developers have experience upgrading and maintaining Django-based websites. We have successfully completed upgrades for numerous clients and offer an upgrade protection plan that ensures your site will be maintained year to year as well as updated to the most recent LTS version of Django. Sound good? Get in touch to start the process of upgrading and securing your website, or take a look at some of our other services if you’ve got a larger project in mind.

How to Document API Requests using Django Rest Swagger

In Developing an application, we'll write our required API's. To Document these API's we use Django Rest Swagger.

Django Rest Swagger is used to Document your API for eg., Listing all Your project Rest API's in the browser with brief description abou the API.

It is Open Source so you can contribute to the project.

If your API's available to the users with description, so UI developers can understand and test your API's and use it accordingly.

First we need to install the swagger.

pip install django-rest-swagger

Add "rest_framework_swagger" to Django settings INSTALLED_APPS

INSTALLED_APPS = (

    ...

    'rest_framework_swagger',

)

Include the rest_framework_swagger URLs to a path of your choice

patterns = ('',

    ...

    url(r'^docs/$', schema_view, name="schema_view"),

)

In your API Views

from rest_framework.decorators import api_view, permission_classes, renderer_classes

from rest_framework.permissions import AllowAny

from rest_framework_swagger.renderers import OpenAPIRenderer, SwaggerUIRenderer

@api_view()

@permission_classes((AllowAny, ))

@renderer_classes([OpenAPIRenderer, SwaggerUIRenderer])

def schema_view(request):

    generator = schemas.SchemaGenerator(title='Rest Swagger')

    return Response(generator.get_schema(request=request))


If you are not authorized, This document List all the API which doesnt reguire any permission to get authorize.

If you want to authorize user you need to add these swagger settings in your settings.py file. By this you will be provide with Authorize Button in the header when user hits the swagger document URL, When you click on it you will be prompt with a pop-up to enter details of Api key authorization. By providing your API Key the user gets Authorize.

SWAGGER_SETTINGS = {

    'USE_SESSION_AUTH': False,

    'api_version': '0.1',

    'enabled_methods': [

        'get',

        'post',

    ],

    'SECURITY_DEFINITIONS': {

        "api_key": {

            "type": "apiKey",

            "name": "Authorization",

            "in": "header"

          },

    },

}

May 21 2017

Django and nginx file proxy - part two

You have our application up and running but there is a problem. You don't want the user to see that your media files are served from media url. How to fix that? This blog post will tell you one of the solutions. Let's go!

How to hide urls from the …

May 19 2017

PyGrunn: google machine learning APIs for python developers - keynote from Google Cloud

(One of my summaries of a talk at the 2017 PyGrunn conference).

Lee Boonstra and Dmitriy Novakovskiy gave the keynote.

Python at google. Python is widely used at google, it is one of its official languages. It is integral to many of the google projects, for instance youtube and 'app engine'. And lots of open source libraries. Every API has its own python client.

Google for python developers. What can you do as a python programmer on google? Google cloud platform. It consists of many parts and services that you can use.

  • You can embed machine learning services like tensorflow, speach API, the translation API, etc.
  • Serverless data processing and analytics. Pub/Sub, bigquery (map/reduce without needing to run your own hadoop cluster), etc.
  • Server stuff like kubernetes, container services.

Machine learning. There have been quite some presentation on this already. Look at it like this: how do you teach things to your kids? You show them! "That is a car", "that is a bike". After a while they will start learning the words for the concepts.

Machine learning is not the same as AI (artificial intelligence). AI is the process of building smarter computers. Machine learning is getting the computer to actually learn. Which is actually much easier.

Why is machine learning so popular now? Well:

  • The amount of data. There is much more data. So we finally have the data we need to do something with it.
  • Better models. The models have gotten much better.
  • More computing power. Parallellization and so. You now have the power to actually do it in reasonable time.

Why google? Tensorflow is very popular (see an earlier talk about tensorflow).

You can do your own thing and use tensorflow and the cloud machine learning engine. OR you can use one of the google-trained services like the vision API (object recognition, text recognision, facial sentiment, logo detection). Speech API/natural language API (syntax analysis, sentiment analysis, entity recognision). Translation API (realtime subtitles, language detection). Beta feature: the video intelligence API (it can detect the dogs in your video and tell you when in the video the dogs appeared...).

Code and demos. She gave a nice demo about what she could recognize with google stuff in an Arjan Robben image. It even detected the copyright statement text at the bottom of the photo and the text on his lanyard ("champions league final"). And it was 88% sure it was a soccer player. And 76% sure it might be a tennis player :-)

Using the API looked pretty easy. Nice detail: several textual items that came back from the API were then fed to the automatic translation API to convert them to Dutch.

Tensorflow demo. He used the MNIST dataset, a set of handwritten numbers often used for testing neural nets.

Dataflow is a unifield programming model for batchor stream data processing. You can use it for map/reduce-like operations and "embarrassingly parallel" workloads. It is open sourced as apache Beam (you can use it hosted on google's infrastructure).

The flow has four steps:

  • Cloud storage (storage of everything).
  • dataflow.
  • Bigquery (data storage).
  • Data studio (data visualization).

(The demo code can be found in the sheets that will be available, googling for it probably also helps).

https://abload.de/img/screenshot2017-05-08a1pzqd.png

Photo explanation: just a nice unrelated picture from the my work-in-progress german model railway

Dutch note: python+django programmeren in hartje Utrecht bij de oude gracht? Watersector, dus veel data en geo. Leuk! Nelen&Schuurmans is op zoek. Stuur mij maar een mailtje, want de vacaturetekst staat nog niet online :-)

PyGrunn: deep learning with tensorflow ("Trump tweet generator") - Ede Meijer

(One of my summaries of a talk at the 2017 PyGrunn conference).

He originally used a java solution for 'machine learning', but that didn't work very comfortably. He then switch to tensorflow, written in python.

Machine learning is learning from data without being explicitly programmed. You feed the computer lots of data and it learns from that. Some examples of the techniques used: linear regression, logistic regression, decision trees, artifical neural networks and much more.

Artificial neural networks are what tensorflow is about. A normal neural network has "input layer", a "hidden layer" and "an output layer". The nodes in the three layers are connected. The neural net tries to learn the "weights" of the connections.

Deep learning means you have neural networks with multiple hidden layers. Often it deals with features at different levels of abstractions. Images that have to be recognized can be cut in to several pieces of different sizes and fed to the net as those parts, but also as the full image. Training a model often works wit minimizing error by using the "gradual descent" method.

Tensor flow? What are tensors? Well, a 0D tensor is a scalar, 1D a vector, 2D a matrix, etc.

Then he started showing code examples. The ultimate test was trying to learn the network to talk like Trump. So it got fed a huge number of Trump twitter messages :-) It worked by defining a number of "layers" in order to predict the next character in a tweet. Very low-level, thus.

In the end the generated tweets started to look like Trump tweets. The code is here: https://github.com/EdeMeijer/trumpet

https://abload.de/img/screenshot2017-05-08ai9z30.png

Photo explanation: just a nice unrelated picture from the my work-in-progress german model railway

Dutch note: python+django programmeren in hartje Utrecht bij de oude gracht? Watersector, dus veel data en geo. Leuk! Nelen&Schuurmans is op zoek. Stuur mij maar een mailtje, want de vacaturetekst staat nog niet online :-)

PyGrunn: creating abstraction between consumer and datastore - Marco Vellinga

(One of my summaries of a talk at the 2017 PyGrunn conference).

Marco Vellinga started his talk with train-related analogies and photos, which I of course liked. They have a big monolythic application. Like a big train: move one part (a locomotive) and the rest also moves. In an application you don't want that: if you make a change in one part of the application, you don't want to have to change lots of other parts. Decoupled is better.

With abstraction, for instance a user object with username, password and billing information, you can change the backend (splitting user and billing info, for instance) without needing to change anything in the frontend.

They created an abstraction layer containing

  • core logic
  • validation
  • error handling
  • data contracts

There isn't really support for such a layer in Django, so you have to build it yourself. This was a lot of work and took a couple of tries to get it right.

They used Marshmallow for json serialization. For serializing queries (the equivalent of django's 'Q' objects) they build filterql.

For (API) versioning they build their own function/class decorator to mark functions as "version 1" or "version 2". They're calling it versionary. It is kind of spooky, but it is fully tested and documented.

The main question: is it worth building? Well, it paid off instantly when they put it into production because they immediately found invalid data that was already in the system. Because the abstraction layer put all the validation in one place, all the data that previously slipped through was now found out :-)

https://abload.de/img/screenshot2017-05-08aiql1f.png

Photo explanation: just a nice unrelated picture from the my work-in-progress german model railway

PyGrunn: django localization ('l10n') - Cees van Wieringen

(One of my summaries of a talk at the 2017 PyGrunn conference).

What is L10N (=LocalizatioN)? You have multiple terms like that.

  • Translation (t9n): just translating text.
  • Localization: the adaption of a product or content to a specific language or culture. This includes localization. But also date/time formats. Currency. Units of measurement. Number formatting.
  • i18n (internationalization) is preparing your product for localization. Using unicode. Django translation toolkit.

Translation in django. Enable it in your settings.py by adding I18N = True. And use _() around your strings.

Where does the underscore function come from? It is a gettext function:

from django.utils.translation import gettext as _

There are multiple gettext variants (gettext_lazy, for instance), so he thinks it best to use the full function name instead of the customary underscore.

Sometimes you have terms that can be translated in multiple ways. Django has a gettext function that allows adding a "domain". "Spring" can be a "season", but it can also be a "mechanical part".

For translating models, there are multiple django apps that handle it and that store the various translations in the database. But there is none that tries to grab the translation from the existing gettext *.po translation files. So he build one himself ("TransField").

Localization trick for lengths: use Distance from the contrib.gis package. You can store a length as "300m" and then ask for distance.km() and distance.yard(). They added similar classes for volumes and temperatures and areas and masses.

Then they build model fields like AreaField and MassField. For localization you can provide defaults ("I want my lengths in km and my weights in kg"). When outputting values, the correct localization and unit are applied automatically.

The code is at https://github.com/ceasaro/django-l10n-extensions

A recommended video (8 minutes) that shows all the problems you might encounter when translating: https://www.youtube.com/watch?v=0j74jcxSunY

https://abload.de/img/screenshot2017-05-08atja0y.png

Photo explanation: just a nice unrelated picture from the my work-in-progress german model railway

Dutch note: python+django programmeren in hartje Utrecht bij de oude gracht? Watersector, dus veel data en geo. Leuk! Nelen&Schuurmans is op zoek. Stuur mij maar een mailtje, want de vacaturetekst staat nog niet online :-)

May 17 2017

PyCon tutorials, day one

May 15 2017

Using Python and Google Docs to Build Books

Python F-Strings Are Fun!

When I started my latest fiction book, The Darkest Autumn, I wrote out the chapters as individual files. I did it in a text editor (Sublime) and saved the files to a git repo. The names of the files determined their order, chapters being named in this pattern:

the-darkest-autumn $ tree
.
├── 01_Beginnings.md
├── 02_Town_of_Ravenna.md
├── 03_Walls_of_Ravenna.md

As the book developed I thought about moving it to Scrivener. If you don't know, Scrivener is an excellent tool for writing. It encourages you to break up your work into chapters and scenes. The downside is that Scrivener is complex (I want to write, not figure out software) and Scrivener isn't designed for simultaneous collaboration. The latter issue is a very serious problem, as I like to have others review and comment on my writing as I go.

What I really wanted to do is combine the chapter breaking of Scrivener with the simplicity and collaboration of Google Docs. Preferably, I would put the book chapters into Google Docs as individual files and then send invites to my editor, wife, and my beta readers. By using Google Docs I could ensure anyone could access the work without having to create a new account and learn an unfamiliar system.

Unfortunately, at this time Google Docs has no way to combine multiple Google Docs contained in one directory into one large document for publication. To use Google Docs thhe way I want involves manually copy/pasting content from dozens of files into one master document any time you want to update a work. With even 5 to 10 documents this is time consuming and error prone (for me) to the point of being unusable. This is a problem because my fiction books have anywhere from 30 to 50 chapters.

Fortunately for me, I know how to code. By using the Python programming language, I can automate the process of combining the Google Docs into one master file which can be converted to epub, mobi (kindle), or PDF.

How I Combine Google Doc Files

First, I download all the files in the book's Google Docs directory.

Selecting Files With Google Docs

This generates and downloads a zip file called something like drive-download-20170505T230011Z-001.zip. I use unzip to open it:

unzip drive-download-20170505T230011Z-001.zip -d the-darkest-autumn

Inside the new the-darkest-autumn folder are a bunch of MS Word-formatted files named identically to what's stored on Google Docs:

$ tree the-darkest-autumn/
the-darkest-autumn
├── 01.\ Beginnings.docx
├── 02.\ Town\ of\ Ravenna.docx
├── 03.\ Walls\ of\ Ravenna.docx
├── 04.\ Gatehouse\ of\ Ravenna.docx
├── 05.\ Courage.docx
├── 06.\ To\ the\ Upper\ Valley.docx
├── _afterward.docx
├── _copyright.docx
├── _dedication.docx
└── _title.docx

Now it's time to bring in the code. By leveraging the python-docx library, I combine all the Word files into one large Word files using this Python (3.6 or higher) script:

"""bookify.py: Combines multiple Word docs in a folder.

"""

import os
import sys
from glob import glob

try:
    from docx import Document
    from docx.enum.text import WD_ALIGN_PARAGRAPH
    from docx.enum.text import WD_COLOR_INDEX
    from docx.shared import Inches, Pt
except ImportError:
    raise ImportError("You need to 'pip install python-docx'")

TEXT_FONT = "Trebuchet MS"


def add_matter(master_document, filename, chapter, after=False):
    """Builds """
    if not os.path.exists(filename):
        return master_document

    if after:
        master_document.add_page_break()

    # Build the heading
    heading = master_document.add_heading('', level=1)
    heading.alignment = WD_ALIGN_PARAGRAPH.CENTER
    runt = heading.add_run(chapter)
    runt.font.color.theme_color = WD_COLOR_INDEX.WHITE

    # Add the material
    document = Document(docx=filename)
    for index, paragraph in enumerate(document.paragraphs):
        new_paragraph = master_document.add_paragraph()
        new_paragraph.paragraph_format.alignment = paragraph.paragraph_format.alignment
        new_paragraph.style = paragraph.style
        # Loop through the runs of a paragraph
        # A run is a style element within a paragraph (i.e. bold)
        for j, run in enumerate(paragraph.runs):
            # Copy over the old style
            text = run.text
            # Add run to new paragraph
            new_run = new_paragraph.add_run(text=text)
            # Update styles for run
            new_run.bold = run.bold
            new_run.italic = run.italic
            new_run.font.size = run.font.size
            new_run.font.color.theme_color = WD_COLOR_INDEX.BLACK
    master_document.add_page_break()
    print(f'Adding {chapter}')
    return master_document


def add_chapter(master_document, filename, chapter):
    """Build chapters, i.e. where the story happens."""
    # Build the chapter
    document = Document(docx=filename)

    # Build the heading
    heading = master_document.add_heading('', level=1)
    heading.alignment = WD_ALIGN_PARAGRAPH.CENTER

    heading.add_run(chapter).font.color.theme_color = WD_COLOR_INDEX.BLACK
    heading.paragraph_format.space_after = Pt(12)

    for index, paragraph in enumerate(document.paragraphs):
        new_paragraph = master_document.add_paragraph()
        # Loop through the runs of a paragraph
        # A run is a style element within a paragraph (i.e. bold)
        for j, run in enumerate(paragraph.runs):

            text = run.text
            # If at start of paragraph and no tab, add one
            if j == 0 and not text.startswith('\t'):
                text = f"\t{text}"
            # Add run to new paragraph
            new_run = new_paragraph.add_run(text=text)
            # Update styles for run
            new_run.font.name = TEXT_FONT
            new_run.bold = run.bold
            new_run.italic = run.italic

        # Last minute format checking
        text = new_paragraph.text

    master_document.add_page_break()
    # Destroy the document object
    del document
    return master_document


def main(book):
    master_document = Document()

    master_document = add_matter(
      master_document,
      filename=f'{book}/_title.docx',
      chapter='Title Page'
    )
    master_document = add_matter(
        master_document,
        filename=f'{book}/_copyright.docx',
        chapter='Copyright Page'
    )
    master_document = add_matter(
        master_document,
        filename=f'{book}/_dedication.docx',
        chapter='Dedication'
    )

    for filename in glob(f"{book}/*"):
        if filename.startswith(f"{book}/_"):
            print(f'skipping {filename}')
            continue

        # Get the chapter name
        book, short = filename.split('/')
        chapter = short.replace('.docx', '')
        if chapter.startswith('0'):
            chapter = chapter[1:]
        print(f'Adding {chapter}')
        master_document = add_chapter(master_document, filename, chapter)

    master_document = add_matter(
        master_document,
        filename=f'{book}/_aboutauthor.docx',
        chapter='About the Author',
        after=True
    )
    master_document = add_matter(
        master_document,
        filename=f'{book}/_afterward.docx',
        chapter='Afterward',
        after=True
    )
    master_document.save(f'{book}.docx')
    print('DONE!!!')

if __name__ == '__main__':
    try:
        book = sys.argv[1]
    except IndexError:
        msg = 'You need to specify a book. A book is a directory of word files.'
        raise Exception(msg)

    main(book)

This is what it looks like when I run the code:

$ python bookify.py the-darkest-autumn/
Adding Title Page
Adding Copyright Page
Adding Dedication
Adding 1. Beginnings
Adding 2. Town of Ravenna
Adding 3. Walls of Ravenna
Adding 4. Gatehouse of Ravenna
Adding 5. Courage
Adding 6. To the Upper Valley
skipping the-darkest-autumn/_afterward.docx
skipping the-darkest-autumn/_copyright.docx
skipping the-darkest-autumn/_dedication.docx
skipping the-darkest-autumn/_title.docx
Adding Afterward
DONE!!!

And now I've got a Word document in the same directory called the-darkest-autumn.docx.

Converting Word to EPUB

While Kindle Direct Publishing (KDP) will accept .docx files, I like to convert it to .epub using Calibre:

$ ebook-convert the-darkest-autumn.docx the-darkest-autumn.epub \
--authors "Daniel Roy Greenfeld" \
--publisher "Two Scoops Press" \
--series Ambria \
--series-index 1 \
--output-profile kindle

And now I can check out my results by using Calibre's book viewer:

$ ebook-viewer the-darkest-autumn.epub

Add the Links!

As python-docx doesn't handle HTTP links at this time, I manually add them to the book using Calibre's epub editor. I add links to:

How Well Does It Work?

For me it works wonders for my productivity. By following a "chapters as files" pattern within Google Docs I get solid collaboration power plus some (but not all) of the features of Scrivener. I can quickly regenerate the book at any time without having to struggle with Scrivener or have to add tools like Vellum to the process.

I have a secondary script that fixes quoting and tab issues, written before I realized Calibre could have done that for me.

The book I started this project for, The Darkest Autumn, is available now on Amazon. Check it out and let me know what you think of what the script generates. Or if you want to support my writing (both fiction and non-fiction), buy it on Amazon and leave an honest review.

Thinking About the Future

Right now this snippet of code generates something that looks okay, but could be improved. I plan to enhance it with better fonts and chapter headers, the goal to generate something as nice as what Draft2Digital generates.

I've considered adding the OAuth components necessary in order to allow for automated downloading. The problem is I loathe working with OAuth. Therefore I'm sticking with the manial download process.

For about a week I thought about leveraging it and my Django skills to build it as a paid subscription service and rake in the passive income. Basically turn it into a startup. After some reflection I backed off because if Google added file combination as a feature, it would destroy the business overnight.

I've also decided not to package this up on Github/PyPI. While Cookiecutter makes it trivial for me to do this kind of thing, I'm not interested in maintaining yet another open source project. However, if someone does package it up and credits me for my work, I'm happy to link to it from this blog post.

Cover for The Darkest Autumn

May 14 2017

Django and nginx file proxy - part one

In this blog post series, I will show you how to use Nginx for hiding download urls. Django will serve us as a backend. Let's go!

In this series I will build simple web application - user upload a file via api and then she/he wants to download it. But …

May 12 2017

Decorator to add placeholders to field

Decorator to automagically add placeholders to form widgets. cls can be any class derived from django.forms.Form or django.forms.ModelForm. The field labels are used as value for the placeholder. This will affect all form instances of this class.

  • add_placeholders only to forms.TextInput and form.Textarea
  • add_placeholders_to_any_field ...

Fastest Redis configuration for Django

I have an app that does a lot of Redis queries. It all runs in AWS with ElastiCache Redis. Due to the nature of the app, it stores really large hash tables in Redis. The application then depends on querying Redis for these. The question is; What is the best configuration possible for the fastest service possible?

Note! Last month I wrote Fastest cache backend possible for Django which looked at comparing Redis against Memcache. Might be an interesting read too if you're not sold on Redis.

Options

All options are variations on the compressor, serializer and parser which are things you can override in django-redis. All have an effect on the performance. Even compression, for if the number of bytes between Redis and the application is smaller, then it should have better network throughput.

Without further ado, here are the variations:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": config('REDIS_LOCATION', 'redis://127.0.0.1:6379') + '/0',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        }
    },
    "json": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": config('REDIS_LOCATION', 'redis://127.0.0.1:6379') + '/1',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "SERIALIZER": "django_redis.serializers.json.JSONSerializer",
        }
    },
    "ujson": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": config('REDIS_LOCATION', 'redis://127.0.0.1:6379') + '/2',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "SERIALIZER": "fastestcache.ujson_serializer.UJSONSerializer",
        }
    },
    "msgpack": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": config('REDIS_LOCATION', 'redis://127.0.0.1:6379') + '/3',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "SERIALIZER": "django_redis.serializers.msgpack.MSGPackSerializer",
        }
    },
    "hires": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": config('REDIS_LOCATION', 'redis://127.0.0.1:6379') + '/4',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "PARSER_CLASS": "redis.connection.HiredisParser",
        }
    },
    "zlib": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": config('REDIS_LOCATION', 'redis://127.0.0.1:6379') + '/5',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "COMPRESSOR": "django_redis.compressors.zlib.ZlibCompressor",
        }
    },
    "lzma": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": config('REDIS_LOCATION', 'redis://127.0.0.1:6379') + '/6',
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "COMPRESSOR": "django_redis.compressors.lzma.LzmaCompressor"
        }
    },
}

As you can see, they each have a variation on the OPTIONS.PARSER_CLASS, OPTIONS.SERIALIZER or OPTIONS.COMPRESSOR.

The default configuration is to use redis-py and to pickle the Python objects to a bytestring. Pickling in Python is pretty fast but it has the disadvantage that it's Python specific so you can't have a Ruby application reading the same Redis database.

The Experiment

Note how I have one LOCATION per configuration. That's crucial for the sake of testing. That way one database is all JSON and another is all gzip etc.

What the benchmark does is that it measures how long it takes to READ a specific key (called benchmarking). Then, once it's done that it appends that time to the previous value (or [] if it was the first time). And lastly it writes that list back into the database. That way, towards the end you have 1 key whose value looks something like this: [0.013103008270263672, 0.003879070281982422, 0.009411096572875977, 0.0009970664978027344, 0.0002830028533935547, ..... MANY MORE ....].

Towards the end, each of these lists are pretty big. About 500 to 1,000 depending on the benchmark run.

In the experiment I used wrk to basically bombard the Django server on the URL /random (which makes a measurement with a random configuration). On the EC2 experiment node, it finalizes around 1,300 requests per second which is a decent number for an application that does a fair amount of writes.

The way I run the Django server is with uwsgi like this:

uwsgi --http :8000 --wsgi-file fastestcache/wsgi.py --master --processes 4 --threads 2

And the wrk command like this:

wrk -d30s  "http://127.0.0.1:8000/random"

(that, by default, runs 2 threads on 10 connections)

At the end of starting the benchmarking, I open http://localhost:8000/summary which spits out a table and some simple charts.

An Important Quirk

Time measurements over time
One thing I noticed when I started was that the final numbers' average was very different from the medians. That would indicate that there are spikes. The graph on the right shows the times put into that huge Python list for the default configuration for the first 200 measurements. Note that there are little spikes but generally quite flat over time once it gets past the beginning.

Sure enough, it turns out that in almost all configurations, the time it takes to make the query in the beginning is almost order of magnitude slower than the times once the benchmark has started running for a while.

So in the test code you'll see that it chops off the first 10 times. Perhaps it should be more than 10. After all, if you don't like the spikes you can simply look at the median as the best source of conclusive truth.

The Code

The benchmarking code is here. Please be aware that this is quite rough. I'm sure there are many things that can be improved, but I'm not sure I'm going to keep this around.

The Equipment

The ElastiCache Redis I used was a cache.m3.xlarge (13 GiB, High network performance) with 0 shards and 1 node and no multi-zone enabled.

The EC2 node was a m4.xlarge Ubuntu 16.04 64-bit (4 vCPUs and 16 GiB RAM with High network performance).

Both the Redis and the EC2 were run in us-west-1c (North Virginia).

The Results

Here are the results! Sorry if it looks terrible on mobile devices.

root@ip-172-31-2-61:~# wrk -d30s  "http://127.0.0.1:8000/random" && curl "http://127.0.0.1:8000/summary"
Running 30s test @ http://127.0.0.1:8000/random
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.19ms    6.32ms  60.14ms   80.12%
    Req/Sec   583.94    205.60     1.34k    76.50%
  34902 requests in 30.03s, 2.59MB read
Requests/sec:   1162.12
Transfer/sec:     88.23KB
                         TIMES        AVERAGE         MEDIAN         STDDEV
json                      2629        2.596ms        2.159ms        1.969ms
msgpack                   3889        1.531ms        0.830ms        1.855ms
lzma                      1799        2.001ms        1.261ms        2.067ms
default                   3849        1.529ms        0.894ms        1.716ms
zlib                      3211        1.622ms        0.898ms        1.881ms
ujson                     3715        1.668ms        0.979ms        1.894ms
hires                     3791        1.531ms        0.879ms        1.800ms

Best Averages (shorter better)
###############################################################################
██████████████████████████████████████████████████████████████   2.596  json
█████████████████████████████████████                            1.531  msgpack
████████████████████████████████████████████████                 2.001  lzma
█████████████████████████████████████                            1.529  default
███████████████████████████████████████                          1.622  zlib
████████████████████████████████████████                         1.668  ujson
█████████████████████████████████████                            1.531  hires
Best Medians (shorter better)
###############################################################################
███████████████████████████████████████████████████████████████  2.159  json
████████████████████████                                         0.830  msgpack
████████████████████████████████████                             1.261  lzma
██████████████████████████                                       0.894  default
██████████████████████████                                       0.898  zlib
████████████████████████████                                     0.979  ujson
█████████████████████████                                        0.879  hires


Size of Data Saved (shorter better)
###############################################################################
█████████████████████████████████████████████████████████████████  60K  json
██████████████████████████████████████                             35K  msgpack
████                                                                4K  lzma
█████████████████████████████████████                              35K  default
█████████                                                           9K  zlib
████████████████████████████████████████████████████               48K  ujson
█████████████████████████████████████                              34K  hires

Discussion Points

  • There is very little difference once you avoid the json serialized one.
  • msgpack is the fastest by a tiny margin. I prefer median over average because it's more important how it over a long period of time.
  • The default (which is pickle) is fast too.
  • lzma and zlib compress the strings very well. Worth thinking about the fact that zlib is a very universal tool and makes the app "Python agnostic".
  • You probably don't want to use the json serializer. It's fat and slow.
  • Using hires makes very little difference. That's a bummer.
  • Considering how useful zlib is (since you can fit so much much more data in your Redis) it's impressive that it's so fast too!
  • I quite like zlib. If you use that on the pickle serializer you're able to save ~3.5 times as much data.
  • Laugh all you want but until today I had never heard of lzma. So based on that odd personal fact, I'm pessmistic towards that as a compression choice.

Conclusion

This experiment has lead me to the conclusion that the best serializer is msgpack and the best compression is zlib. That is the best configuration for django-redis.

msgpack has implementation libraries for many other programming languages. Right now that doesn't matter for my application but if msgpack is both faster and more versatile (because it supports multiple languages) I conclude that to be the best serializer instead.

May 11 2017

May 10 2017

Django Version Viewer Announcement

Presenting a new open source plugin: The Django Version Viewer!

Imaginary Landscape is pleased to present a helpful new plugin for the Django Web Framework - The Django Version Viewer!

If you find yourself frequently switching between projects or perhaps looking at a codebase you haven’t seen in some time, it’s often useful to know the contents of the project’s virtual environment. Typically this information is only accessible either by examining your project’s requirements.txt (which may or may not accurately reflect what is installed in your environment) or by accessing your app’s server directly to examine the app’s virtual environment.

The Django Version Viewer is a handy tool that exposes this information in a handful of ways:

Through the Django Admin panel

The plugin conveniently exposes a link at the top of your admin index pane to expose a simple drop down listing all of your project dependencies.

Through a DjangoCMS Toolbar Menu Item

If Django CMS is installed, a new menu item will be added to the CMS Toolbar Page Menu that will allow opening the version viewer popup.

Through an API call

Alternatively, you can make a request to a URL at www.yourAppsDomain/django_version_viewer  to retrieve a list of your app’s installed package represented in json .

Learn more about the project at its GitHub repository.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl