Monday, December 23, 2013

MongoDB GridFS performance test

Test code and test result: https://github.com/feifangit/MongoDB-GridFS-test

Purpose

MongoDB GridFS comes with some natural advantages such as scalability(sharding) and HA(replica set). But as it stores file in ASCII string chunks, there's no doubt a performance loss.
I'm trying 3 different deployments (different MongoDB drivers) to read from GridFS. And compare the results to classic Nginx configuration.

Credits

Configurations

1, Nginx

location /files/ {
  alias /home/ubuntu/;
} 
open_file_cache kept off during the test.

2, Nginx_GridFS

It's a Nginx plugin based on MongoDB C driver. https://github.com/mdirolf/nginx-gridfs

Compile code & install

I made a quick install script in this repo, run it with sudo. After Nginx is ready, modify the configration file under /usr/local/nginx/conf/nginx.conf (if you didn't change the path).

Configuration

location /gridfs/{
   gridfs test1 type=string field=filename;
}
Use /usr/local/nginx/sbin/nginx to start Nginx. And use parameter -s reload if you changed the configuration file again.

3, Python

library version

  • Flask 0.10.1
  • Gevent 1.0.0
  • Gunicorn 0.18.0
  • pymongo 2.6.3

run application

cd flaskapp/
sudo chmod +x runflask.sh
bash runflask.sh
Script runflask.sh will start gunicorn with gevnet woker mode. Gunicorn configuration file here

4, Node.js

library version

  • Node.js 0.10.4
  • Express 3.4.7
  • mongodb(driver) 1.3.23

run application

cd nodejsapp/
sudo chmod +x runnodejs.sh
bash runnodejs.sh

Test

Test items:

  1. file served by Nginx directly
  2. file served by Nginx_gridFS + GridFS
  3. file served by Flask + pymongo + gevent + GridFS
  4. file served by Node.js + GridFS

Files for downloading:

Run script insert_file_gridfs.py from MongoDB server to insert 4 different size of file to databasetest1(pymongo is required)
  • 1KB
  • 100KB
  • 1MB

Test Environment

2 servers:
  • MongoDB+Application/Nginx
  • tester(Apache ab/JMeter)
hardware:

Concurrency

100 concurrent requests, total 500 requests.
ab -c 100 -n 500 ...

Result

Throughput

Time per request (download)
File sizeNginx+Hard driveNginx+GridFS pluginPython(pymongo+gevent)Node.js
1KB0.1741.1241.9821.679
100KB1.0141.5723.1033.708
1MB9.5829.56715.97318.317
You can get Apache ab report in folder: testresult

Server load

The server load is be monitored by command: vmstat 2
Nginx:
Nginx
Nginx_gridfs
Nginx
gevent+pymongo
Nginx
Node.js
Nginx

Conclusion

  • Files served by Nginx directly
  • No doubt it's the most efficient one, whether performance or server load.
  • Support cache. In real world, the directive open_file_cache should be configured well for better performance.
  • And must mention, it's the only one support pause and resume during the download(HTTP range support).
  • For the rest 3 test items, files are stored in MongoDB, but served by different drivers.
  • serve static files by application is really not an appropriate choice. They drains CPU too much and the performance is not good.
  • nginx_gridfs (MongoDB C driver): downloading requests will be processed at Nginx level, which is in front of web applications in most deployments. Web application can focus on processing dynamic contents instead of static content.
  • nginx_gridfs got the best performance comparing to other applications written in script languages. - The performance differences between Nginx and nginx_gridfs getting small after file size increased. But you can not turn a blind eye on the server load.
  • pymongo and node.js driver: it's a draw game. Static files should be avoid to be served in productive applications.

Advantages of GridFS

  • Put files in database make static content management much easier. We can omit maintain the consistency between files and its meta data in database.
  • Scalable and HA advantages come with MongoDB

Drawbacks of GridFS

  • bad performance

When should I use MongoDB GridFS

There are rare use cases I can imagine, especially in a performance sensitive system. But I may taste it in some prototype projects.
Here goes the answer from MongoDB official website, hope this will help.http://docs.mongodb.org/manual/faq/developers/#faq-developers-when-to-use-gridfs

52 comments:

  1. Thanks for sharing such informative article on Loadrunner Automation testing tool. This load testing tool will provide most precise information about the quality of software. Loadrunner training in Chennai| Loadrunner training institute in Chennai

    ReplyDelete
  2. Thankful for sharing such educational article Hadoop Training Chennai on Loadrunner Automation testing gadget.Oracle Training in Chennai This load testing gadget will give most correct information about the way of programming.

    ReplyDelete
  3. I get a lot of great information from your blog. Thank you for your sharing this informative blog. I have bookmarked this page for my future reference.

    Regards...
    Ethical Hacking Course in Chennai

    ReplyDelete
  4. Cloud computing is storing and accessing the large data sets over the internet instead of your PC computer. So that you can manage the data and program anywhere through the internet.
    Regards..
    Best institute for Cloud Computing in Chennai


    ReplyDelete
  5. Thanks for sharing this information. Java is one of the popular object oriented programming language used for many of the multinational corporation. So learning Java Training in Chennai is really helpful to make a bright future.

    ReplyDelete
  6. Oracle is one of the most traditional and promising CRM used for maintain their customers. If you want to know more about this crm and this features, reach us FITA. Rated as No.1 Oracle Training Institutes in Chennai.


    ReplyDelete
  7. Excellent information. HTML5 is a markup language used for designing responsive website and it is also used for structuring and presenting the website content.

    Regards..
    HTML5 Training Institutes in Chennai | PHP Training Institute in Chennai

    ReplyDelete

  8. Nice information. Salesforce is a most powerful CRM tool used for manage all customers including potential customers. Salesforce Training in Chennai offering this course with experienced professionals.

    ReplyDelete
  9. Really awesome blog. Software testing is a method of executing the application or program with the intent of searching the software errors. Software Testing Training in Chennai offering this course at reasonable cost.



    ReplyDelete
  10. Heads-up on repairs for bigger DBs however - another framework we're creating, mongo didn't neatly exit, and repairing the 7TB GridFS appears as though it will take 130 hrs.

    As a result of this, I think I'll take a gander at changing to OpenStack Swift or Ceph. Still, until then it was
    great. What's more, the nginx-gridfs module is sweet.

    Best DB Training in Chennai

    ReplyDelete
  11. my system show this messange
    PCI: BIOS BUGS: MCFG AREA AT e0000000
    PCI: NOT useing MMCONFIG.
    welcome to the NICLIINUX LIVE niclINUX
    scanning for usb/firewire device......
    can't find NICLINUX FILESYSTEM, SORRY
    DROPPING YOU TOA(VERY LIMITED ) SHELL
    PRESS RESET BUTTON TO QUIT
    Best Linux Training in Chennai

    ReplyDelete
  12. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this.

    ReplyDelete
  13. Great post.!! Your post is really useful for me to enlarge my knowledge in a right way.
    Loadrunner Training in Chennai

    ReplyDelete
  14. Thank you for taking time to provide us the useful information with us. This provides me the useful information which very useful in requesting the post.
    Loadrunner Training in Chennai

    ReplyDelete
  15. It has more interesting information.Nice to read it.
    Java Training in Chennai

    ReplyDelete
  16. Nice article share!!!Thanks for sharing this information with us.In-case if any body wants to undergone salesforce training course,can Enroll from our best salesforce training institute.Here we offered world class training with placement support.For more course details,just visit here @http://www.salesforcetraining-chennai.in/salesforce-training-in-chennai.php

    ReplyDelete
  17. all the new versions of the software are shared. provide more usefull information. very interesting things are explained.
    Informatica Training in Chennai

    ReplyDelete
  18. This blog provides useful information. It provides knowleddge about the GridFs for the beginners Loadrunner Testing Training in Chennai

    ReplyDelete
  19. Thank for sharing your valuable thoughts with us.It's very useful for us to understand more about it..Thanks for sharing.
    Android Training in Chennai

    ReplyDelete
  20. excellent,your way of explantion is good.noone can feel difficult,ur step by step information s are useful to us.It improves our knowledge to got new ideas for develop our career.


    CCNA Training in Chennai

    ReplyDelete
  21. more updatable block.... new versions of the software are introduced well and more informative.VMWare Training in Chennai

    ReplyDelete
  22. It is new to me i like this concept thanks for sharing it is helpful for in future.
    sharepoint training in chennai

    ReplyDelete
  23. It is nice article.I am very happy to read this content.
    Sharepoint admin training in chennai

    ReplyDelete
  24. thanks for shared about MongoDB GridFS performance test. easily can understand by every one and really such a informative post.
    thanks.

    ReplyDelete
  25. More informative about new software.Its more useful.
    Linux training in chennai

    ReplyDelete
  26. Great post.Thanks for sharing useful information.Really it's very useful for everyone.Some people will be eager to learn Microstrategy training course in best institute.We are providing best Microstrategy training in Chennai with placement support.We are providing world class training for that kind of people.

    ReplyDelete
  27. Thanks for sharing this article, nice one. join our Datawarehousing Training in Chennai

    ReplyDelete
  28. different versions of software are posted. thanks for this great post. CCNA Training in Chennai

    ReplyDelete
  29. It is a unique contents.I will thank for post this article.It is very useful information.
    Websphere MQ Training in Chennai

    ReplyDelete
  30. This blog is nice.unique article and useful information ios training in chennai

    ReplyDelete
  31. Thanks for sharing your blog.Its very useful and informative.Its very useful
    weblogic training in chennai

    ReplyDelete
  32. Really great work thanks for useful sharing with amazing content project management blog

    ReplyDelete
  33. Excellent posts..
    SAS Institute introduced the SAS Certified Professional Program,training proper understanding of how the SAS software works. Among the five certification programs that SAS Institute has come up with, SAS training can be considered as the entry point into the big data and the data analytics industry.
    SAS online training in hyderabad

    ReplyDelete
  34. monggodb gridfs performance nice posts..

    Hadoop online training .All the basic and get the full knowledge of hadoop.
    hadoop online training


    ReplyDelete
  35. Nice post. Glad to find your blog. Keep sharing.

    php training in chennai

    ReplyDelete
  36. hi welcome to this blog. really you have post an informative blog.it will be really helpful to many peoples.
    codedUI training

    ReplyDelete
  37. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharng this information,this is useful to me...
    Android training in chennai
    Ios training in chennai

    ReplyDelete

  38. Thanks for posting useful information.You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...Really it was an awesome article...very interesting to read..please sharing like this information......
    Web Design Development Company
    Mobile App Development Company

    ReplyDelete
  39. Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving..
    Android App Development Company

    ReplyDelete
  40. Very good idea you've shared here, from here I can be a very valuable new experience.
    Java Training in Chennai
    Java Training Course in Chennai

    ReplyDelete
  41. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    web design Company
    web development Company
    web design Company in chennai
    web development Company in chennai
    web design Company in India
    web development Company in India

    ReplyDelete
  42. it is really amazing...thanks for sharing....provide more useful information...
    Mobile app development company

    ReplyDelete
  43. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Fitness SMS
    Salon SMS
    Investor Relation SMS

    ReplyDelete
  44. These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
    iOS App Development Company

    ReplyDelete
  45. H Good post. This is a very nice blog I wonder why the other of this sector do not notice this. Am sure this post has helped me save many hours of browsing other similar posts just to find what I was looking for. Thanks! Website Stress Test

    ReplyDelete
  46. great and nice blog thanks sharing..I just want to say that all the information you have given here is awesome...Thank you very much for this one.
    web design Company
    web development Company
    web design Company in chennai
    web development Company in chennai
    web design Company in India
    web development Company in India

    ReplyDelete