Install docker 1.8 on CentOS 7

Remember to install docker-selinux manually due to the issue(?)

yum install docker-selinux

Docker 1.9 has fixed this issue

The full list of commands

sudo yum update
curl -sSL https://get.docker.com/ | sh
service docker start

chkconfig docker on

Start with No?

Here's one chapter in the book "Getting Real" (37 signal)
full book: https://basecamp.com/books/Getting%20Real.pdf

Start With No

https://gettingreal.37signals.com/ch05_Start_With_No.php

...

Don't be a yes-man

Make each feature work hard to be implemented. Make each feature prove itself and show that it's a survivor. It's like "Fight Club." You should only consider features if they're willing to stand on the porch for three days waiting to be let in.

That's why you start with no. Every new feature request that comes to us — or from us — meets a no. We listen but don't act. The initial response is "not now." If a request for a feature keeps coming back, that's when we know it's time to take a deeper look. Then, and only then, do we start considering the feature for real.

And what do you say to people who complain when you won't adopt their feature idea? Remind them why they like the app in the first place. "You like it because we say no. You like it because it doesn't do 100 other things. You like it because it doesn't try to please everyone all the time."

---

Don't say so in an interview, interviewer may not understand the point without sufficient context >.<
They may misunderstand you to be a lazy, always-no man.

---

I've burned the book....

USCIS case tracking service upgraded

service: http://case-monitoring.appspot.com/
code: https://github.com/feifangit/uscis-case-track

Now, you can get notification if adjacent case status changed.

dj-sso-server and dj-sso-client: Django applications provide SSO feature

Github:
https://github.com/feifangit/dj-sso-server
https://github.com/feifangit/dj-sso-client

SSO provider (server side) demo: https://dj-sso-sample.herokuapp.com/

SSO client side: https://github.com/feifangit/dj-sso-client/tree/master/example/ssoclient

---

dj-sso-server

dj-sso-server is a Django application that provides Single Sign-on feature for your project.

The dj-sso-server application works as a SSO provider , you can use dj-sso-client (https://github.com/feifangit/dj-sso-client) as the SSO client in other projects need SSO.

Installation

Install by command pip install dj-sso-server

The dependent package dj-api-auth (https://github.com/feifangit/dj-api-auth) will be installed automatically.

How it works

• Based on the dj-api-auth module, we can create an API key with SSO related APIs initially included. All the API communications between dj-sso-server and dj-sso-client are protected by dj-api-auth

• The API key will also be bind with a host which is used to limit the origin of SSO requests.

• SSO work flow with dj-sso-client

  1. Firstly, dj-sso-client applies a request key via API reqeusttoken/ on dj-sso-server

  2. The request key in dj-sso-server side will be kept in cache for 5 minutes, so the whole SSO login process should be done in 5 minutes.

  3. With the request key, dj-sso-client redirects user to SSO login page on SSO provider, and get auth token if login success. dj-sso-server will

     • verify the request origin
     • verify request key validity (expired?)
     • save user information in cache

  4. dj-sso-client verifies the auth token with dj-sso-server via API authtoken/, and get a SSOUser object.

  5. dj-sso-server delete the request key from cache once the authtoken/ is called.

• If there's an already logged-in account on dj-sso-server (say, the project where SSO provider is placed also provides other features, and there's a valid cookies in browser side and valid session on server side), user can select to continue with that logged account.

• SSO login through dj-sso-server with not affect the login status on dj-sso-server.

Attention

Since request keys are stored in cache waiting for verification or expiration. If you have multiple application process running in your deployment (gunicorn etc.), please use proper cache system that can be shared between processes.

Memcached and Redis are both great for caching, be aware, the Local-memory caching (django.core.cache.backends.locmem.LocMemCache) is a toy for local debugging.

Add dj-sso-server to project

1. Add djapiauth and djssoserver to INSTALLED_APPS in sttings.py
2. Assign an URL to the module

# add auth for a browser-oriented view
url(r'^sso/', include("djssoserver.urls"))
#...

Settings

• 
  

  SSO_SERVER_USER_TO_JSON_FUNC

  • optional, a path to function receives an user object and return a json string.

  • the default SSO_SERVER_USER_TO_JSON_FUNC function is djssoserver.utility.default_user_to_json

    def default_user_to_json(user):
        return json.dumps(model_to_dict(user, exclude=["password", "user_permissions"]),
            cls=DjangoJSONEncoder)
    

Scan API

In order to discover and manage APIs, after dj-sso-server is added in an accessible urls.py, run command python manage.py reloadentrypoints to collect APIs to database.

Create API key for SSO

1. From your admin site, create an API key at Single sign-on/SSO credential. All SSO related APIs will assigned to this API Key automatically.
2. After the API key for SSO is ready, you can assign more APIs for this API key at API Auth/Credential from admin site

Customize SSO login page

You can add styles to your own SSO login page. simply create djsso/ssologin.html under the templates folder. Revamp it by imitating the
original page

SSOUser object

dj-sso-client gets a SSOUser object whatever the User model is used in SSO provider project.

See detail in README file of dj-sso-client (https://github.com/feifangit/dj-sso-client)

DEMO

We have a SSO provider application running on Heroku (https://dj-sso-sample.herokuapp.com/).

Source code: under example folder

To try the demo out, check out the README file of dj-sso-client (https://github.com/feifangit/dj-sso-client)

---

---

---

dj-sso-client

dj-sso-client is the a Django application works as SSO client side of dj-sso-server (https://github.com/feifangit/dj-sso-server)

Installation

pip install dj-sso-client

Add to your project

• Modify following settings in settings.py of your project

  • AUTHENTICATION_BACKENDS, add djssoclient.authbackend.SSOAuthBackend as the backends
  • AUTH_USER_MODEL, set djssoclient.SSOUser as user model

AUTHENTICATION_BACKENDS = ('djssoclient.authbackend.SSOAuthBackend',)
AUTH_USER_MODEL = 'djssoclient.SSOUser'

• Add following dj-sso-client settings base on your demand

  • SSO_API_AUTH_SETTING: set API key, SEC key and remote SSO provider URL. This setting is used by underneath dj-api-auth module to proejct API accessing.

    SSO_API_AUTH_SETTING = {
        "apikey": "f4a05287",
        "seckey": "6a4eeaea54d54f51af703e79c6096d51",
        "url": "https://dj-sso-sample.herokuapp.com",
    }
    

  • SSO_REMOTE_URL_PREFIX (optional): SSO path in remote SSO provider. default /sso/

  • SSO_USER_STORAGE``(optional):  SSOUser storage solution, there are 2 storage backends in ``dj-sso-client already. default: SSOUserDBStorage

    • djssoclient.userstorage.SSOUserDBStorage: store user data in database
    • djssoclient.userstorage.SSOUserCacheStorage: store user data in cache. You will get better performance.

  • SSO_SETTING_CACHE (optional): if you selected SSOUserCacheStorage as your user storage backend, and you have more than one cache in settings.py, you can pick up the cache name here. default: default

Attention: SSOUserCacheStorage

The default django.core.cache.backends.locmem.LocMemCache stores data per process. In multi-process production environment (gunicorn on multi-core server), it may cause problem while using SSOUserCacheStorage as your user storage engine.

Please use dedicate cache system (Memcached or Redis) as cache backend to avoid this problem.

SSOUser

SSOUser is the user model to store user data. It can be used as database model class if you selected SSOUserDBStorage to be your user storage engine.

class SSOUser(AbstractBaseUser):
    username = models.CharField(unique=True, max_length=50)
    extras = models.TextField(default="{}")
    ...

extra user attributes : attributes not exists in the SSOUser class. (attributes except username, password, last_login etc.)

All extra user attributes can be access by getattr method or . operator. And they are stored in class member extras in JSON format.

Get your hands dirty

We already have a SSO provider (dj-sso-server) application running on Heroku: http://dj-sso-sample.herokuapp.com/ . Run the example application in folder example/ssoclient/ locally.

The API key using in the example application is binding with localhost:8000, so make sure you're accessing local application by localhost:8000 rather than the 127.0.0.1:8000.

fresh login

1. make sure you're not logged in on http://dj-sso-sample.herokuapp.com/, you should see please log in with ...
2. click sso login on local application, you will be redirected to http://dj-sso-sample.herokuapp.com/sso/....
3. you will see user information after be redirected to local application.
4. on http://dj-sso-sample.herokuapp.com/ , your status is still not logged-in

login with existing logged account

1. log in with user name user1 and password 123 on http://dj-sso-sample.herokuapp.com/
2. click sso login on local application, you will be redirected to http://dj-sso-sample.herokuapp.com/sso/...., but you will see a different login page with fresh login.
3. select continue with current user? user1
4. you will be logged in as user1 at local application

switch account

1. if you select switch account and login with user2/456 from step 3 in previous sample
2. you will be logged in as user2 at local application
3. your login status on http://dj-sso-sample.herokuapp.com/ will NOT be changed (still logged in as user1)

TODO

• example: work as an extra auth method

dj-api-auth, a Django application providing AWS-alike API protection

Github: https://github.com/feifangit/dj-api-auth

dj-api-auth

dj-api-auth is a Django application, providing an AWS-alike API auth solution.

When I was seeking a simple solution rather than intricate OAuth, I was inspired by this article
Designing a Secure REST (Web) API without OAuth.
Thanks to the author and the comments.

Features

1. API key, SEC key related forms
2. Each API key can be associated with a set of API
3. API can be associated with user, your legacy code with request.user underneath can work smoothly with dj-api-auth
4. Add auth by simply put a decorator on your view.
5. Discover API with auth enabled automatically, these auth-required APIs will display in assignable list when creating API keys
6. A Django command to scan and update API information to database.

How it works

1. Generate a pair of API key and SEC key, assign some APIs to it.

2. Client put API key and current UNIX time as apikey and timestemp in requestURL

3. Client also generate a signature by calculate a SHA256 value on the whole URL(without signature) by its known SEC key.

4. Server side will verify

   • is timestamp from client in reasonable rift compare to server time.
   • is apikey from client exists
   • is the API client trying to access allowed for the apikey
   • compare the signature with the one calculated on server side by same algorithm




5. if any verification failed, return 403 error with brief message

Add to your project

1. Add djapiauth to INSTALLED_APPS in sttings.py
2. There are two optional settings

• API_AUTH_ALLOWED_TIME_RIFT

  • optional, set the allowed time rift between server time and the timestamp parameter in coming URL.
  • format : integer, unit: second
  • default : 300, (5 minutes)




• API_AUTH_ADMIN_USER_FILTER

  • optional, when creating API keys, you can assign the API key to an user, this filter is used to filter the users showing in the API key creating form.
  • format: dictionary, the filter parameter will be passed to get_user_model().objects.filter(). e.g. {'name_startswith': 'admin'}
  • default: {}, means all users will show in the API key creating form.

Generate/Manage API and SEC key

If you have admin enabled for your project, you can find these features in admin site. Otherwise, you can import forms from djapiauth.forms or write your own form based on models in djapiauth.models

Add auth for views

• For legacy views, we provide utility function url_with_auth in djapiauth.utility

# add auth for a browser-oriented view
url_with_auth(r'^hello/$', 'djapp.views.index'),
#...

• For API views, simply add @api_auth for the view after from djapiauth.auth import api_auth

@api_auth
def api_whoami(request):
        return JsonResponse({"user": "feifan", "boss": "lidan zhou"})

Scan API

we have a Django command reloadentrypoints to help you to collect and save all auth-required APIs to database.

Error messages

• parameter missing, any of apikey, timestamp or signature missing in URL
• timestamp rift xxx, check your local time and server time. You can implement an API to return server time
• entry point not allowed for the API key, check the assigned API for this API key in admin site or anywhere else you manage API keys
• signature error, obviously, signature mismatch

DEMO

• Source code under example/djapp folder.
• Test code is under example/test/, we have python and javascript test code ready.

Server application provides 2 APIs

• /hello/ : reused the code of index view, add an auth layer on it
• /goodbye/ : a view you must access it by the signature stuff

DIY:

• Start the djapp
• there's already one pair of API+SEC keys: 483a570a, d7228d70cd7f456d9bfdc35ed8fee375
• modify variable URL in test.py, or URL in test.js
• Generate API key and SEC key from localhost:8000/admin/xxx, modify variable API_KEY and SEC_KEY in test.py or test.js
• Run python test.py or node test.js
• login admin site with admin user: admin/123, remove all APIs associated with 483a570a, try to run the test code again, you should see 403 errors __main__.APIException: (403, '{"error": "entry point not allowed for the API key"}')
• modify the API key to an invalid one
• modify the SEC key to an invalid one
• modify your local time to one hour ago

Thanks

Thanks for the Javascript test code from Neil Chen (neil.chen.nj@gmail.com)

TODO

• performance improvement for entry point matching in API permission check.

dj-mongo-reader: A Django application for fast implementing MongoDB-based data rending

Github: https://github.com/feifangit/dj-mongo-reader


ReadMe and example appliction: http://dj-mongo-reader.herokuapp.com/

dj-mongo-reader

dj-mongo-reader is a Django application can be used to query MongoDB via AJAX requests and render result with several customization options.

Installation

• Install from PyPi pip install dj-mongo-reader

Features

1. Able to send find, count, collstats commands to MongoDB via designated URL
2. Basic HTML pages included, only few code is needed to work with your existing CSS framework
3. Pagination implemented, you can customize record number show in each page
4. Permission check, you can deny a data fetching request based on user's permission and the query(database, collection, command, criteria).
5. Pick up some keys of the record to fill the table, leave complete record in detail dialog
6. Assign display names for keys in MongoDB record
7. value transformation, you can provide Javascript callback functions to process raw value data from MongoDB to a proper text for displaying

DEMO

A Django application runs on heroku.

• Functionality: A query form and result table included.
• UI: Bootstrap 3 applied.

URL: http://dj-mongo-reader.herokuapp.com

Source code Please see example folder

Guide

http://dj-mongo-reader.herokuapp.com

Credits

• Fan Fei feifan.pub@gmail.com backend
• Neil Chen neil.chen.nj@gmail.com frontend

1, screenshot of heroku example application

2, screenshot of an application using dj-mongo-reader in production

Deploy heroku app from a Git subfolder

I have an existing Git repo on Github, with an example app inside the sub-folder. That looks like

dj-mongo-reader
├── dist
├── dj_mongo_reader.egg-info
├── djmongoreader
│   ├── static
│   │   └── dj-mongo-reader
│   │       └── js
│   └── templates
│       └── dj-mongo-reader
└── example
    └── *****sampleapp****
        ├── sampleapp
        └── templates
            └── dj-mongo-reader

The sampleapp folder under example is the app which I wanna push to Heroku.

Here's how to archive the goal:
1, enter git root folder

cd dj-mongo-reader 

2, register app on Hoerku (add remote url)

heroku git:remote -a dj-mongo-reader

3, push sub folder to heroku

git subtree push --prefix example/sampleapp heroku master

4, to force push sub folder to heroku

git push heroku `git subtree split --prefix example/sampleapp  master`:master --force

fix: tornado HTTP 599 issue / gnutls_handshake() failed

After I moved a Tornado application into docker. I noticed lots of HTTP 599 errors were popping out during async http client's fetching method. And these errors only occurs on HTTPS requests.

The HTTP code 599 itself is not specified in any RFCs, but in Tornado, it's an error stands for SSL error.

Here's the unit test code I found in Tornado source code: https://github.com/tornadoweb/tornado/blob/master/tornado/test/httpserver_test.py#L87


Here's the server log with error message

[ERROR] 2015-01-15 18:07:40,933 url fetch error: xxx.xxxx.com, HTTP 599: gnutls_handshake() failed: Illegal parameter, 599
...

This is the code I used in my application, since the curl and pycurl are installed in advance, the code will use libcurl to fetching content underneath.

try:
    import pycurl
    logging.info("py curl version:%s" % pycurl.version)

    httpclient.AsyncHTTPClient.configure(
        "tornado.curl_httpclient.CurlAsyncHTTPClient", 
        max_clients=1000
        )
    logging.info("curl found, use curl client")
except:
    logging.warning("run pip install pycurl for better performance")

The first impression I got is, the target website has a invalid SSL certificate.
Anyway, I tried to avoid SSL certificate verification by adding parameter

validate_cert=False,

in http client's fetch method.

But the problem is still there, also, I noticed that I can get correct response on my dev machine where the application is not running in docker.
So, the most suspicious cause is the curl/pycurl library comes with the docker image.



And this is the pycurl version in the docker container.

python
>>> import pycurl
>>> pycurl.version_info()
(3, '7.35.0', 467712, 'x86_64-pc-linux-gnu', 50877, 'GnuTLS/2.12.23', 0, '1.2.8', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'http', 'https', 'imap', 'imaps', 'ldap', 'ldaps', 'pop3', 'pop3s', 'rtmp', 'rtsp', 'smtp', 'smtps', 'telnet', 'tftp'), None, 0, '1.28')

On my dev machine, it's linked with OpenSSL instead of GnuTLS. So, I reinstalled pycurl with openSSL.

apt-get update
apt-get install -y curl libcurl3-openssl-dev
pip uninstall -y pycurl

export PYCURL_SSL_LIBRARY=openssl
pip install pycurl

Here's the information after I replace the SSL library used by pycurl

python
>>> import pycurl
>>> pycurl.version_info()
(3, '7.35.0', 467712, 'x86_64-pc-linux-gnu', 50877, 'OpenSSL/1.0.1f', 0, '1.2.8', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'http', 'https', 'imap', 'imaps', 'ldap', 'ldaps', 'pop3', 'pop3s', 'rtmp', 'rtsp', 'smtp', 'smtps', 'telnet', 'tftp'), None, 0, '1.28')

Finally, the HTTP 599 error code gone in my Tornado application.
Happy 2015~