Thursday, March 12, 2015

dj-sso-server and dj-sso-client: Django applications provide SSO feature

Github:
https://github.com/feifangit/dj-sso-server
https://github.com/feifangit/dj-sso-client






dj-sso-server

dj-sso-server is a Django application that provides Single Sign-on feature for your project.

The dj-sso-server application works as a SSO provider , you can use dj-sso-client (https://github.com/feifangit/dj-sso-client) as the SSO client in other projects need SSO.

Installation

Install by command pip install dj-sso-server

The dependent package dj-api-auth (https://github.com/feifangit/dj-api-auth) will be installed automatically.

How it works

  • Based on the dj-api-auth module, we can create an API key with SSO related APIs initially included. All the API communications between dj-sso-server and dj-sso-client are protected by dj-api-auth

  • The API key will also be bind with a host which is used to limit the origin of SSO requests.

  • SSO work flow with dj-sso-client

    1. Firstly, dj-sso-client applies a request key via API reqeusttoken/ on dj-sso-server

    2. The request key in dj-sso-server side will be kept in cache for 5 minutes, so the whole SSO login process should be done in 5 minutes.

    3. With the request key, dj-sso-client redirects user to SSO login page on SSO provider, and get auth token if login success. dj-sso-server will

      • verify the request origin
      • verify request key validity (expired?)
      • save user information in cache
    4. dj-sso-client verifies the auth token with dj-sso-server via API authtoken/, and get a SSOUser object.

    5. dj-sso-server delete the request key from cache once the authtoken/ is called.

  • If there's an already logged-in account on dj-sso-server (say, the project where SSO provider is placed also provides other features, and there's a valid cookies in browser side and valid session on server side), user can select to continue with that logged account.

  • SSO login through dj-sso-server with not affect the login status on dj-sso-server.

Attention

Since request keys are stored in cache waiting for verification or expiration. If you have multiple application process running in your deployment (gunicorn etc.), please use proper cache system that can be shared between processes.

Memcached and Redis are both great for caching, be aware, the Local-memory caching (django.core.cache.backends.locmem.LocMemCache) is a toy for local debugging.

Add dj-sso-server to project

  1. Add djapiauth and djssoserver to INSTALLED_APPS in sttings.py
  2. Assign an URL to the module
# add auth for a browser-oriented view
url(r'^sso/', include("djssoserver.urls"))
#...

Settings


  • SSO_SERVER_USER_TO_JSON_FUNC
    • optional, a path to function receives an user object and return a json string.

    • the default SSO_SERVER_USER_TO_JSON_FUNC function is djssoserver.utility.default_user_to_json

      def default_user_to_json(user):
          return json.dumps(model_to_dict(user, exclude=["password", "user_permissions"]),
              cls=DjangoJSONEncoder)
      


Scan API

In order to discover and manage APIs, after dj-sso-server is added in an accessible urls.py, run command python manage.py reloadentrypoints to collect APIs to database.

Create API key for SSO

  1. From your admin site, create an API key at Single sign-on/SSO credential. All SSO related APIs will assigned to this API Key automatically.
  2. After the API key for SSO is ready, you can assign more APIs for this API key at API Auth/Credential from admin site

Customize SSO login page

You can add styles to your own SSO login page. simply create djsso/ssologin.html under the templates folder. Revamp it by imitating the
original page

SSOUser object

dj-sso-client gets a SSOUser object whatever the User model is used in SSO provider project.

See detail in README file of dj-sso-client (https://github.com/feifangit/dj-sso-client)

DEMO

We have a SSO provider application running on Heroku (https://dj-sso-sample.herokuapp.com/).

Source code: under example folder

To try the demo out, check out the README file of dj-sso-client (https://github.com/feifangit/dj-sso-client)









dj-sso-client

dj-sso-client is the a Django application works as SSO client side of dj-sso-server (https://github.com/feifangit/dj-sso-server)

Installation

pip install dj-sso-client

Add to your project

  • Modify following settings in settings.py of your project

    • AUTHENTICATION_BACKENDS, add djssoclient.authbackend.SSOAuthBackend as the backends
    • AUTH_USER_MODEL, set djssoclient.SSOUser as user model
AUTHENTICATION_BACKENDS = ('djssoclient.authbackend.SSOAuthBackend',)
AUTH_USER_MODEL = 'djssoclient.SSOUser'
  • Add following dj-sso-client settings base on your demand

    • SSO_API_AUTH_SETTING: set API key, SEC key and remote SSO provider URL. This setting is used by underneath dj-api-auth module to proejct API accessing.

      SSO_API_AUTH_SETTING = {
          "apikey": "f4a05287",
          "seckey": "6a4eeaea54d54f51af703e79c6096d51",
          "url": "https://dj-sso-sample.herokuapp.com",
      }
      
    • SSO_REMOTE_URL_PREFIX (optional): SSO path in remote SSO provider. default /sso/

    • SSO_USER_STORAGE``(optional):  SSOUser storage solution, there are 2 storage backends in ``dj-sso-client already. default: SSOUserDBStorage

      • djssoclient.userstorage.SSOUserDBStorage: store user data in database
      • djssoclient.userstorage.SSOUserCacheStorage: store user data in cache. You will get better performance.
    • SSO_SETTING_CACHE (optional): if you selected SSOUserCacheStorage as your user storage backend, and you have more than one cache in settings.py, you can pick up the cache name here. default: default

Attention: SSOUserCacheStorage

The default django.core.cache.backends.locmem.LocMemCache stores data per process. In multi-process production environment (gunicorn on multi-core server), it may cause problem while using SSOUserCacheStorage as your user storage engine.

Please use dedicate cache system (Memcached or Redis) as cache backend to avoid this problem.

SSOUser

SSOUser is the user model to store user data. It can be used as database model class if you selected SSOUserDBStorage to be your user storage engine.

class SSOUser(AbstractBaseUser):
    username = models.CharField(unique=True, max_length=50)
    extras = models.TextField(default="{}")
    ...

extra user attributes : attributes not exists in the SSOUser class. (attributes except username, password, last_login etc.)

All extra user attributes can be access by getattr method or . operator. And they are stored in class member extras in JSON format.

Get your hands dirty

We already have a SSO provider (dj-sso-server) application running on Heroku: http://dj-sso-sample.herokuapp.com/ . Run the example application in folder example/ssoclient/ locally.

The API key using in the example application is binding with localhost:8000, so make sure you're accessing local application by localhost:8000 rather than the 127.0.0.1:8000.

fresh login

  1. make sure you're not logged in on http://dj-sso-sample.herokuapp.com/, you should see please log in with ...
  2. click sso login on local application, you will be redirected to http://dj-sso-sample.herokuapp.com/sso/....
  3. you will see user information after be redirected to local application.
  4. on http://dj-sso-sample.herokuapp.com/ , your status is still not logged-in

login with existing logged account

  1. log in with user name user1 and password 123 on http://dj-sso-sample.herokuapp.com/
  2. click sso login on local application, you will be redirected to http://dj-sso-sample.herokuapp.com/sso/...., but you will see a different login page with fresh login.
  3. select continue with current user? user1
  4. you will be logged in as user1 at local application

switch account

  1. if you select switch account and login with user2/456 from step 3 in previous sample
  2. you will be logged in as user2 at local application
  3. your login status on http://dj-sso-sample.herokuapp.com/ will NOT be changed (still logged in as user1)

TODO

  • example: work as an extra auth method


dj-api-auth, a Django application providing AWS-alike API protection

Github: https://github.com/feifangit/dj-api-auth



dj-api-auth

dj-api-auth is a Django application, providing an AWS-alike API auth solution.

When I was seeking a simple solution rather than intricate OAuth, I was inspired by this article
Designing a Secure REST (Web) API without OAuth.
Thanks to the author and the comments.

Features

  1. API key, SEC key related forms
  2. Each API key can be associated with a set of API
  3. API can be associated with user, your legacy code with request.user underneath can work smoothly with dj-api-auth
  4. Add auth by simply put a decorator on your view.
  5. Discover API with auth enabled automatically, these auth-required APIs will display in assignable list when creating API keys
  6. A Django command to scan and update API information to database.

How it works

  1. Generate a pair of API key and SEC key, assign some APIs to it.

  2. Client put API key and current UNIX time as apikey and timestemp in requestURL

  3. Client also generate a signature by calculate a SHA256 value on the whole URL(without signature) by its known SEC key.

  4. Server side will verify
    • is timestamp from client in reasonable rift compare to server time.
    • is apikey from client exists
    • is the API client trying to access allowed for the apikey
    • compare the signature with the one calculated on server side by same algorithm

  5. if any verification failed, return 403 error with brief message


Add to your project

  1. Add djapiauth to INSTALLED_APPS in sttings.py
  2. There are two optional settings
  • API_AUTH_ALLOWED_TIME_RIFT
    • optional, set the allowed time rift between server time and the timestamp parameter in coming URL.
    • format : integer, unit: second
    • default : 300, (5 minutes)

  • API_AUTH_ADMIN_USER_FILTER
    • optional, when creating API keys, you can assign the API key to an user, this filter is used to filter the users showing in the API key creating form.
    • format: dictionary, the filter parameter will be passed to get_user_model().objects.filter(). e.g. {'name_startswith': 'admin'}
    • default: {}, means all users will show in the API key creating form.

Generate/Manage API and SEC key

If you have admin enabled for your project, you can find these features in admin site. Otherwise, you can import forms from djapiauth.forms or write your own form based on models in djapiauth.models

Add auth for views

  • For legacy views, we provide utility function url_with_auth in djapiauth.utility
# add auth for a browser-oriented view
url_with_auth(r'^hello/$', 'djapp.views.index'),
#...
  • For API views, simply add @api_auth for the view after from djapiauth.auth import api_auth
@api_auth
def api_whoami(request):
        return JsonResponse({"user": "feifan", "boss": "lidan zhou"})

Scan API

we have a Django command reloadentrypoints to help you to collect and save all auth-required APIs to database.

Error messages

  • parameter missing, any of apikey, timestamp or signature missing in URL
  • timestamp rift xxx, check your local time and server time. You can implement an API to return server time
  • entry point not allowed for the API key, check the assigned API for this API key in admin site or anywhere else you manage API keys
  • signature error, obviously, signature mismatch

DEMO

  • Source code under example/djapp folder.
  • Test code is under example/test/, we have python and javascript test code ready.

Server application provides 2 APIs

  • /hello/ : reused the code of index view, add an auth layer on it
  • /goodbye/ : a view you must access it by the signature stuff

DIY:

  • Start the djapp
  • there's already one pair of API+SEC keys: 483a570a, d7228d70cd7f456d9bfdc35ed8fee375
  • modify variable URL in test.py, or URL in test.js
  • Generate API key and SEC key from localhost:8000/admin/xxx, modify variable API_KEY and SEC_KEY in test.py or test.js
  • Run python test.py or node test.js
  • login admin site with admin user: admin/123, remove all APIs associated with 483a570a, try to run the test code again, you should see 403 errors __main__.APIException: (403, '{"error": "entry point not allowed for the API key"}')
  • modify the API key to an invalid one
  • modify the SEC key to an invalid one
  • modify your local time to one hour ago

Thanks

Thanks for the Javascript test code from Neil Chen (neil.chen.nj@gmail.com)

TODO

  • performance improvement for entry point matching in API permission check.



dj-mongo-reader: A Django application for fast implementing MongoDB-based data rending

Github: https://github.com/feifangit/dj-mongo-reader


ReadMe and example appliction: http://dj-mongo-reader.herokuapp.com/




dj-mongo-reader

dj-mongo-reader is a Django application can be used to query MongoDB via AJAX requests and render result with several customization options.

Installation

  • Install from PyPi pip install dj-mongo-reader

    Features

    1. Able to send find, count, collstats commands to MongoDB via designated URL
    2. Basic HTML pages included, only few code is needed to work with your existing CSS framework
    3. Pagination implemented, you can customize record number show in each page
    4. Permission check, you can deny a data fetching request based on user's permission and the query(database, collection, command, criteria).
    5. Pick up some keys of the record to fill the table, leave complete record in detail dialog
    6. Assign display names for keys in MongoDB record
    7. value transformation, you can provide Javascript callback functions to process raw value data from MongoDB to a proper text for displaying

    DEMO

    A Django application runs on heroku.
    • Functionality: A query form and result table included.
    • UI: Bootstrap 3 applied.
    Source code Please see example folder

    Credits




    1, screenshot of heroku example application






    2, screenshot of an application using dj-mongo-reader in production




    Friday, March 6, 2015

    Deploy heroku app from a Git subfolder

    I have an existing Git repo on Github, with an example app inside the sub-folder. That looks like

    dj-mongo-reader
    ├── dist
    ├── dj_mongo_reader.egg-info
    ├── djmongoreader
    │   ├── static
    │   │   └── dj-mongo-reader
    │   │       └── js
    │   └── templates
    │       └── dj-mongo-reader
    └── example
        └── *****sampleapp****
            ├── sampleapp
            └── templates
                └── dj-mongo-reader
    

    The sampleapp folder under example is the app which I wanna push to Heroku.

    Here's how to archive the goal:
    1, enter git root folder
    cd dj-mongo-reader 
    

    2, register app on Hoerku (add remote url)
    heroku git:remote -a dj-mongo-reader

    3, push sub folder to heroku
    git subtree push --prefix example/sampleapp heroku master


    4, to force push sub folder to heroku
    git push heroku `git subtree split --prefix example/sampleapp  master`:master --force

    Thursday, January 15, 2015

    fix: tornado HTTP 599 issue / gnutls_handshake() failed

    After I moved a Tornado application into docker. I noticed lots of HTTP 599 errors were popping out during async http client's fetching method. And these errors only occurs on HTTPS requests.

    The HTTP code 599 itself is not specified in any RFCs, but in Tornado, it's an error stands for SSL error.

    Here's the unit test code I found in Tornado source code: https://github.com/tornadoweb/tornado/blob/master/tornado/test/httpserver_test.py#L87


    Here's the server log with error message
    [ERROR] 2015-01-15 18:07:40,933 url fetch error: xxx.xxxx.com, HTTP 599: gnutls_handshake() failed: Illegal parameter, 599
    ...
    


    This is the code I used in my application, since the curl and pycurl are installed in advance, the code will use libcurl to fetching content underneath.
    try:
        import pycurl
        logging.info("py curl version:%s" % pycurl.version)
    
        httpclient.AsyncHTTPClient.configure(
            "tornado.curl_httpclient.CurlAsyncHTTPClient", 
            max_clients=1000
            )
        logging.info("curl found, use curl client")
    except:
        logging.warning("run pip install pycurl for better performance")
    


    The first impression I got is, the target website has a invalid SSL certificate.
    Anyway, I tried to avoid SSL certificate verification by adding parameter
    validate_cert=False,
    in http client's fetch method.

    But the problem is still there, also, I noticed that I can get correct response on my dev machine where the application is not running in docker.
    So, the most suspicious cause is the curl/pycurl library comes with the docker image.



    And this is the pycurl version in the docker container.
    python
    >>> import pycurl
    >>> pycurl.version_info()
    (3, '7.35.0', 467712, 'x86_64-pc-linux-gnu', 50877, 'GnuTLS/2.12.23', 0, '1.2.8', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'http', 'https', 'imap', 'imaps', 'ldap', 'ldaps', 'pop3', 'pop3s', 'rtmp', 'rtsp', 'smtp', 'smtps', 'telnet', 'tftp'), None, 0, '1.28')
    
    On my dev machine, it's linked with OpenSSL instead of GnuTLS. So, I reinstalled pycurl with openSSL.
    apt-get update
    apt-get install -y curl libcurl3-openssl-dev
    pip uninstall -y pycurl
    
    export PYCURL_SSL_LIBRARY=openssl
    pip install pycurl
    


    Here's the information after I replace the SSL library used by pycurl
    python
    >>> import pycurl
    >>> pycurl.version_info()
    (3, '7.35.0', 467712, 'x86_64-pc-linux-gnu', 50877, 'OpenSSL/1.0.1f', 0, '1.2.8', ('dict', 'file', 'ftp', 'ftps', 'gopher', 'http', 'https', 'imap', 'imaps', 'ldap', 'ldaps', 'pop3', 'pop3s', 'rtmp', 'rtsp', 'smtp', 'smtps', 'telnet', 'tftp'), None, 0, '1.28')
    


    Finally, the HTTP 599 error code gone in my Tornado application.
    Happy 2015~

    Saturday, November 29, 2014

    Error occurred while loading visualforce page.错误:加载 Visualforce 页面时出现错误


    problem现象:
    While using the visa interview appointment system (cgifederal.secure.force.com),
    keep running into "Error occurred while loading visualforce page." after the time window selection page.
    And the error page shows up no matter what OS/browser you use.

    在使用签证面试预约系统(cgifederal.secure.force.com)时,
    总是在选择完时间的页面之后遇到"错误:加载 Visualforce 页面时出现错误"的报错页面。
    而且不管切换到什么操作系统活着浏览器都无法解决。


    cause原因:
    Someone added you as their dependent(family member who're going to take the interview together) in their unfinished appointment.
    可能有人在另外一份没有完成的预约草稿中将你添加为了一起通行签证的人(家庭成员可以一起面签).


    fix搞定:
    - Delete you from that(theirs) unfinished appointment draft at step 6.
    - Come back to your appointment, you may find many fields in your form are reseted.
    - Finish your appointment, you will see appointment confirmation and appointment invoice after the time window selection page.
    先从那一份没有完成的(别人的)面签预约程序的第六步中将你删除。回到你自己的面签程序中,选择完要面签的时间窗口,你会看到面签确认页和已缴费的条形码。

    Samsung S5 can not get new Gmail or Hangout message while using mobile data

    Problem description: In Mobile network(3G/4G data), you cannot receive "server pushed message", applications such as Gmail, Google hangouts whose who heavily relay on this "server push" technology(Google Cloud Message) can not work properly, no new message can be received.
    - but, you can manually pull email by "refresh" button in Gmail app.
    - after switch to Wi-Fi, no no problem at all.


    Fix: Settings -> Data usage -> upper right corner(a 3-dotted button) ->
    - make sure the "restrict background data" is unchecked,
    - if it's already unchecked when you seen it, check it, then uncheck it. (UI and underneath setting may mismatched)