Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

June 04 2018

Metaprogramming and Django - Using Decorators

While programming is about, in some way, doing code to transform data, metaprogramming can be seen as the task of doing code to change code. This category is often used to help programmers to enhance the readability and maintainability of the code, help with separation of concerns and respect one of the most important principles of software develop

Database concurrency in Django the right way

When developing applications which have real-time requirements or other specific needs for running asynchronous tasks outside the web application, it is common to adopt a task queue such as Celery. This allows, for example, for the server to handle a request, start an asynchronous task responsible of doing some heavyweight processing, and return an answer while the task is still running. Here, we are considering a similar scenario: a request is made, and the server has to do some processing on the request. Ideally, we want to separate the high time-demanding parts from the view processing flow, so we run those parts in a separate task. Now, let's suppose we have to do some database operations both in the view and the task when the request happens. If not done carefully, those operations can be a source for issues that can be hard to track.

Controlling access: a Django permission apps comparison

There are many ways to handle permissions in a project. This post highlights the main differences and approaches taken by some popular Django third party apps to handle permissions.

3 Django apps for sending great e-mails

3 Django apps for debugging and sending HTML styled e-mails

Uploading files from the frontend to Amazon S3

How to upload files from the frontend straight to S3 without sending to the server using django

10 Django apps you're not using but should be

Description of 10 essential apps every Django developer should be using in their projects

Classy Django REST Framework Release

Release of Classy Django REST Framework

A Basic SEO for Django

A Basic SEO for Django.

How to configure Sass and Bower with django-compressor - part 2 (deployment to Heroku and S3)

Quick guide on how to deploy Django with Sass, Bower and django-compressor to Heroku and S3

How to configure Sass and Bower with django-compressor - part 1 (local config)

Quick guide on how to configure Django with Sass, Bower and django-compressor

Django CORS

An introduction to CORS and how to enable it in Django.

June 03 2018

Reactify Django

# Reactify Django is coming. ...

June 01 2018

Django development with Docker — Testing, Continuous Integration and Docker Hub

May 31 2018

Ansible provision/deploy setup

I never got around to write down the ansible setup I figured out (together with others, of course, at my previous job) for deploying/provisioning django websites.

The whole setup (as a cookiecutter template) can be found in https://github.com/reinout/cookiecutter-djangosite-template . The relevant code is in the ansible/ directory. Note: this is a "cookiecutter template" from which you can generate a project so you'll see some {{ }} and {% %}: when you create the actual project these items will be filled in.

My goal was to keep the setup simple and safe. With "safe", I mean that you normally cannot accidentally do something wrong.

And it was intended for getting one website project onto the server(s). It is not a huge ansible setup to set up the whole infra in one fell swoop.

First the inventory files:

  • Yes, multiple: there are two of them. production_inventory and staging_inventory. The safe aspect is that we used to have a single inventory file with [production] and [staging] headings in it. If you wanted to update staging, you had to add --limit staging on the command line. If you forgot that...

    With two separate files, you never have this problem. Accidents are much less likely to occur.

  • The simple aspect is that variables are right there in the inventory. It are only a few variables after all: servername, hostname, checkout (master or tag).

    A "problem" with ansible is that you can place variables in lots of places (playbook, inventory, host variable file, group variable file and another two or three I don't remember right away). So where to look? I figured that the inventory was the right place for this kind of simple setup:

    [web]
    p-web-1234.internal.network
    
    [web:vars]
    site_name=myproject.example.org
    checkout_name="1.2"
    

Second, the ansible playbooks:

  • There are two. A provision.yml and a deploy.yml. Provisioning is for one-time setup (well, you probably want to change stuff sometimes, but you'll rarely run this one). Deploy is for the regular deploys of the actual code. The stuff you regularly do.

  • Provision should be run as a user that can do sudo su on the target machine. Provisioning installs packages and adds the nginx config file in /etc/. And it creates a user for running/deploying the django site (in my previous job, this user was historically called buildout).

  • The deploy playbook connects as the abovementioned deploy user, does a "git pull" (or whatever deploy mechanism you prefer), runs the database migration and so.

  • Now, how can the person who deploys connect as that deploy user? Well, the provision playbook creates the deploy user ("buildout"), disables the password and adds the public ssh keys of the deployers to the /home/buildout/.ssh/authorized_keys file:

    - name: Add user "buildout" and set an unusable password.
       user: name=buildout password='*' state=present shell="/bin/bash"
    
     - name: Add maintainers' ssh keys so they can log in as user buildout.
       authorized_key: user=buildout key=https://github.com/{{ item}}.keys
       with_items:
         - reinout
         - another_colleague
    

    It is simple because you only need a very limited number of people on any server with sudo rights. Or a very limited number of people with the password of a generic admin account. Re-provisioning is basically only needed if something changed in the nginx config file. In practice: hardly ever.

    It is simple because you don't need to give the deployers each a separate user account (or access to a password): their public ssh key is enough.

    It safe because it limits the (root-level) mistakes you can do during regular deploys. And the small amount of users you need on your system is also an advantage.

  • The ansible playbooks are short. Just a couple of tasks. Even though I'm normally all in favour of generic libraries and so: for a 65 line playbook I personally don't need the mental overhead of one or two generic ansible roles.

    The playbooks do basically the same thing I did years earlier with "Fabric" scripts. So: the basic structure has quite proven itself (for me).

There are all sorts of approaches you can take. Automatic deploys from your Jenkins or Gitlab, for instance. That'd be something I like to do once. For not-automatically-deployed projects, a simple setup such as I showed here has much to recommend itself.

GeoDjango and PostgreSQL

I recently had to set up GeoDjango on a site that used PostgreSQL as a database. As the online instructions for doing this were less than totally clear, I decided to write down how I did it.

May 30 2018

Django Local 404 Page

Learn how to display and customize a 404 page locally.

Best React Books 2018

List of current React/JavaScript books.

May 29 2018

Angular 6|5 Tutorial: Integrating Angular with Django

In the previous Angular 6 tutorial we've seen how to build a CRUD web application with a Django REST framework API back-end. In this tutorial we'll see how we can integrate the Angular 6 front-end with the Django back-end.

After creating both the back-end and front-end apps we need to integrate them i.e instead of taking the approach where both applications are completely separated we'll serve the front-end application using a Django view. In development we'll have both Django development server and Angular/Webpack dev server running but for production we'll only need a Django server.

To use this approach you need to tweak the Webpack settings of your front-end project, use the webpack-bundle-tracker (from npm) and use the django-webpack-loader package (from PyPI)

The webpack-bundle-tracker is a Webpack plugin that generates a stats file containing meta data information about the assets of your front-end application generated by Webpack.

We'll start by installing the webpack-bundle-tracker module then update the Webpack configuration file to make use of this plugin.

npm install webpack-bundle-tracker --save

Next you need to eject the Webpack configuration from the Angular 6 CLI using

ng eject

If the ejection is successful you'll find a webpack.config.js in the root of your folder.

Open webpack.config.js and import BundleTracker from webpack-bundle-tracker then locate the plugins entry and add the following code

var BundleTracker = require('webpack-bundle-tracker');

/*...*/
module.exports  = {
    /*...*/
    plugins:[
        /*...*/
        new BundleTracker({filename: '../webpack-stats.json'})
    ]
}


Next add the publicPath setting

"output": {
    "path":  path.join(process.cwd(), "dist"),
    "filename":  "[name].bundle.js",
    "chunkFilename":  "[id].chunk.js",
    "crossOriginLoading":  false,
    "publicPath":"http://127.0.0.1:4200/"//1
},

"devServer": {
    "historyApiFallback":  true,
    "publicPath":  "http://127.0.0.1:4200/",//2
}

If you serve your application you'll have a ../webpack-stats.json in the parent folder i.e the root of the Django project.

After ejecting your Webpack configuration from the Angular 6 CLI you won't be able to use ng serve instead you'll have to use npm run start to serve your application.

This is a screenshot of a project where you can see the webpack.config.js file in the front-end application and a generated webpack-stats.json in the root folder of the project

Next let's install The django-webpack-loader package which will take care of reading the webpack-stats.json and insert the assets in a Django template.

Head back to your terminal the run the following command:

pip install django-webpack-loader

In your settings.py file add webpack_loader to the list of installed apps:

INSTALLED_APPS = [
    #...
    'webpack_loader',
    'core'
]

Then add this configuration object:

WEBPACK_LOADER = {
    'DEFAULT': {
        'BUNDLE_DIR_NAME': '',

        'STATS_FILE': os.path.join(BASE_DIR, 'webpack-stats.json'),
    }
}

You can find more settings that you can use via this link.

Serving the Angular 6 Application

Now let's create the view to serve the Angular 6 application. Open core/views and add the following view function:

from django.shortcuts import render

def  home(request):
    return render(request, 'core/home.html')

Next you need to create the home.html template so create a templates/core folder inside the core application then add a home.html with the following content:


{% load render_bundle from webpack_loader %}

<html  lang="en">
<head>
<meta  charset="UTF-8">
<base  href="/">
<title>A Simple CRM with Django and Angular 6</title>
</head>
<body>

<app-root></app-root>

{% render_bundle 'inline' %}
{% render_bundle 'polyfills' %}
{% render_bundle 'styles' %}
{% render_bundle 'vendor' %}
{% render_bundle 'main' %}

</body>
</html>

Now you need to add the URL for the home page in urls.py:

from django.contrib import admin
from django.urls import path
from django.conf.urls import url
from core import views as coreviews

urlpatterns =[

url(r'^$',coreviews.home),

path('admin/', admin.site.urls)
]

That's it. You should now be able to see your Angular 6 page when visiting the Django web application

Fixing Hot Code Reload

If you change the source code of your front-end application you will not get updates hot code reloaded without manually refreshing the page if you are navigating your application from http://127.0.0.1:8000. That means HCR is not working properly so simply open webpack.config.js and add the following setting:

"devServer": {
    "historyApiFallback":  true,
    "publicPath":  "http://127.0.0.1:4200/",//2,
    "headers": {
        'Access-Control-Allow-Origin':  '\\*'//3
    }
}

That's because http://localhost:8000 sends requests to the Webpack dev server (http://localhost:4200) to get source code changes so we need to update headers to allow request from all origins.

Conclusion

Throughout this tutorial we have integrated both Angular 6 front-end and the Django REST API back-end.

May 28 2018

QuerySet Filters on Many-to-many Relations

May 25 2018

Djangocon: friday lightning talks

(One of my summaries of a talk at the 2018 European djangocon.)

The stenographers - Sheryll and Andrew

The stenographers are the ones that provide LIVE speech to text subtitles for the talks. Wow.

They use "shorthand machines" for steno. It is like a piano where you press multiple keys to form words. On wednesday the speakers talked 46000 words...

How fast do people talk? Anything between 180 and 250, though 300 also occurs. The handiest are speakers that speak in a regular tempo. And not too fast.

They ask presenters for notes and texts beforehand. That helps the software pick the right words.

Pytest-picked - Ana Paula Gomes

Say you have a codebase that has tests that take a long time. You've just changed a few files and don't want to run the full test before the commit. You could run git status and figure out the files to test.

But you can do it automatically with pytest-picked.

It is a small plugin, but she wants to improve it with better guessing and also with support for testing what changed on a branch.

How to build a treehouse - Harry Biddle

A talk about building actual treehouses!

One of the craziest is "Horace's cathedral", a huge one that was closed by the fire brigade for fire risks...

If you're going to do it, be nice to your tree. Look at "garnier bolt" for securing your tree house.

Give the tree room to grow.

Recommended book: "the man who climbs trees".

What about DRF's renderers and parsers - Martin Angelov

They started with just plain django. Then they added DRF (django rest framework). Plus react.

Then it became a single page app with django, drf, react, redux, sagas, routers, etc. (Well, there not quite there yet).

Python and javascript are different. Python uses snake_case, javascript uses camelCase. What comes out of django rest framework also often is snake_case, which looks weird in javascript.

They tried to fix it.

  • JS utils. On-the-fly translation. Hard when debugging.
  • React middleware.
  • Then he used a custom CamelCase renderer in django rest framework :-)

Git Menorahs considered harmful - Shai Berger

A bit of Jewish tradition: a menorah is a 7-armed candle holder (a temple menorah) or 9 armed (hanukkah menorah).

What do we use version control for?

  • To coordinate collaborative work.
  • To keep a record of history.

But why do we need the history? To fix mistakes. Figure out when something was broken. Reconsider decisions. Sometimes you need to revert changes.

What do we NOT need in the history? The actual work process. Commits for typo fixes, for instance.

A git menorah is a git repo with many branches and changes. Bad. Use git rebase to reshape the history of your work before pushing. Follow the example of the django git repository.

5 minutes for your mental health - Ducky

What is really important for your mental health?

  • Sleep!
  • Exercise.

As weird as it sounds, you can 'feel' in your body how you're feeling. There are some exercises you can do. Close your eyes, put your mind in your fingertips. Slowly move inside your body. Etc. "Body scan".

Meta programming system, never have syntax errors again - Ilja Bauer

MPS is a system for creating new custom languages. It has a "projectional editor".

A normal editor shows the code, which is converted to an AST (abstract syntax tree) and it gets byte-compiled.

A projectional editor has the AST as the basis and "projects" it as .py/java/whatever files purely as output. The advantage is that renaming stuff is easier.

Save ponies: reduce the resource consumption of your django server - Raphaël Barrois

He works on a big django project. 300 models. Startup time is 12 seconds... Memory consumption 900MB. The first query afterwards after 4 seconds.

How dows it work? They use uwsgi. There is a simple optimization. By default, uwsgi has lazy = true. This loads apps in workers instead of in the master. He disabled it. It started up much slower, but the first query could then be handled much quicker.

Another improvement. They generate a fake (test) request and fire it at the server upon startup. So before the workers are started, everything is already primed and active. This helps a lot.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl