The test result may misleading or even incorrect.
You can reproduce the test cases in your own environment easily.
Some benchmark articles:
http://nbonvin.wordpress.com/2011/03/14/apache-vs-nginx-vs-varnish-vs-gwan/http://nbonvin.wordpress.com/2011/03/24/serving-small-static-files-which-server-to-use/
All the reports show G-WAN is mush faster than Nginx, but that's the comparison more than 1 year ago, the version it used( Nginx 0.7 & G-WAN 2.1) had been out of date.
I made my tests with current popular version: Nginx 1.1.19 & G-WAN 3.3.28
Hardware and Software:
CPU: 4 core, Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
Memory: 5GB
OS: Ubuntu server 12.04.1 x64
OS Kernel: 3.2.0-34-generic #53-Ubuntu SMP
**I made a mistake in first several tests, because I put server and client in different machines, although they are connected in a 1Gb LAN environment, but in this kind of heavy load test, the 1Gb network become the bottleneck. In those tests, Nginx and G-WAN, even Apache2 got almost the same score. That confused me for a while, finally I figured out this problem.
And retested the cases by running server and client on same machine.
Test cases:
3 static files in different size: 17KB, 3.5MB, 653MBcase 1: use ab.c (http://gwan.ch/source/ab.c) provided by G-WAN to download the 17KB file in different concurrency
case 2: download 100Byte file in different concurrency
case 3: 100 clients concurrent download a 623MB size file.
Preparation:
1, Tune up system* </etc/security/limits.conf> append
* soft nofile 200000
* hard nofile 200000
root soft nofile 200000
root hard nofile 200000
*</etc/pam.d/common-session> append
session required pam_limits.so
* in </etc/sysctl.conf> append or modify
fs.file-max = 5000000
net.core.netdev_max_backlog = 400000
net.core.optmem_max = 10000000
net.core.rmem_default = 10000000
net.core.rmem_max = 10000000
net.core.somaxconn = 100000
net.core.wmem_default = 10000000
net.core.wmem_max = 10000000
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_congestion_control = bic
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_max_syn_backlog = 12000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_mem = 30000000 30000000 30000000
net.ipv4.tcp_rmem = 30000000 30000000 30000000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_wmem = 30000000 30000000 30000000
2, Static Files on server
copy testing files to G-WAN & Nginx folders.
for G-WAN, put the files under gwan installation path, like gwan_linux64-bit/0.0.0.0_8080/#0.0.0.0/www/
for Nginx, put files anywhere, then alias the path to URL in nginx configuration.
3, Configurations
for G-WAN, there's no necessary additional configurations at all. Let it works on default port 8080.
for Nginx, increase the acceptable connections, and let the app serve on port 8989
* </etc/nginx/nginx.conf>
user www-data;
worker_processes 8;
pid /var/run/nginx.pid;
events {
worker_connections 10000;
multi_accept off; <-----MUST be OFF in this test, othervise, you will see lots of non-2xx failures.
use epoll;
}
* application configuration </etc/nginx/sites-enable/ANYNAME>
server {
listen 8989;
server_name localhost;
location / {
alias /var/tf/;
}
access_log off;
error_log off;
}
for Apache2, make it work on port 8888, and add some extra settings in </etc/apache2/apache2.conf>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestsPerChild 0
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestsPerChild 0
4, Reboot & verify
reboot the server to make sure all changed configuration become effective.
after reboot, run cmd "ulimit -a" to check "max open file number"
start G-WAN by simply run ./gwan in its installation folder.
nginx should already started up as a service.
Try to download the file by browsers:
http://10.1.2.229:8080/17k.rar <---G-WAN work
http://10.1.2.229:8989/17k.rar <---Nginx work
http://10.1.2.229:8888/17k.rar <---Apache work
<ab.c>
first, we have to modify some macros in <ab.c> to fit my needs.
1, select bench-marking tool
#define IBM_APACHEBENCH
//#define HP_HTTPERF
//#define LIGHTY_WEIGHTTP
2, indicate target server
#define IP "10.1.2.229"
#define PORT "8080" //"8989" for Nginx, "8888" for Apache2
3, number of concurrent clients, and increasing steps,
#define FROM 0 // range to cover (1 - 1,000 concurrent clients)
#define TO 1000 // range to cover (1 - 1,000 concurrent clients)
#define STEP 200 // number of concurrency steps we actually skip
4, indicate target URL
#define URL "/17k.rar" //Or other file names
gcc ab.c -O2 -o testapp -lpthread
./testapp
Test case 1:
download 17KB file
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 2.97/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l
ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d "http://127.0.0.1:8080/17k.rar"
Client Requests per second CPU
----------- ------------------------------- ---------------- -------
Concurrency min ave max user kernel MB RAM
----------- --------- --------- --------- ------- ------- -------
1, 9420, 10636, 13093,
200, 12987, 13035, 13095,
400, 12222, 12254, 12299,
600, 11701, 11882, 12022,
800, 11717, 11808, 11934,
1000, 11407, 11499, 11627,
-------------------------------------------------------------------------------
min:69454 avg:71114 max:74070 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------
Nginx Result:
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.55/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l
ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d "http://127.0.0.1:8989/17k.rar"
Client Requests per second CPU
----------- ------------------------------- ---------------- -------
Concurrency min ave max user kernel MB RAM
----------- --------- --------- --------- ------- ------- -------
1, 8722, 9065, 9569,
200, 16120, 16299, 16506,
400, 14652, 14987, 15110,
600, 14092, 14220, 14314,
800, 13540, 13759, 13986,
1000, 13356, 13492, 13722,
-------------------------------------------------------------------------------
min:80482 avg:81822 max:83207 Time:61 second(s) [00:01:01]
-------------------------------------------------------------------------------
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.54/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l
ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d "http://127.0.0.1:8888/17k.rar"
Client Requests per second CPU
----------- ------------------------------- ---------------- -------
Concurrency min ave max user kernel MB RAM
----------- --------- --------- --------- ------- ------- -------
1, 5512, 5639, 5761,
200, 13004, 14481, 15457,
400, 11569, 13759, 14052,
600, 13420, 13485, 13532,
800, 11723, 12788, 13341,
1000, 5570, 10521, 12729,
-------------------------------------------------------------------------------
min:60798 avg:70673 max:74872 Time:61 second(s) [00:01:01]
-------------------------------------------------------------------------------
Really out of expectation, RPS of Apache2 is close to G-WAN, while Nginx serves 10000 more requests per second !!!
The result is different to what I read from Internet.
A friend of mine, asked me to test the performance when JBoss serving static files.
here's the result of JBoss serving this 17KB file
JBOSS 7.1
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 0.41/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l
ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d "http://127.0.0.1:8080/17k.rar"
Client Requests per second CPU
----------- ------------------------------- ---------------- -------
Concurrency min ave max user kernel MB RAM
----------- --------- --------- --------- ------- ------- -------
1, 6459, 6550, 6666,
200, 8089, 9262, 9704,
400, 8427, 9287, 9581,
600, 8433, 9208, 9373,
800, 7625, 9020, 9544,
1000, 7407, 9206, 9559,
-------------------------------------------------------------------------------
min:46440 avg:52533 max:54427 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------
Obviously, application server is not good at serving static files.
Test case 2:
100 Byte size file downloading test.
G-WAN
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.37/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l
ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d "http://127.0.0.1:8080/100.html"
Client Requests per second CPU
----------- ------------------------------- ---------------- -------
Concurrency min ave max user kernel MB RAM
----------- --------- --------- --------- ------- ------- -------
1, 10649, 13385, 15629,
200, 16514, 17171, 17398,
400, 15559, 15838, 16155,
600, 15231, 15444, 15582,
800, 14978, 15133, 15360,
1000, 14700, 14895, 15101,
-------------------------------------------------------------------------------
min:87631 avg:91866 max:95225 Time:61 second(s) [00:01:01]
-------------------------------------------------------------------------------
NGINX
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.34/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l
ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d "http://127.0.0.1:8989/100.html"
Client Requests per second CPU
----------- ------------------------------- ---------------- -------
Concurrency min ave max user kernel MB RAM
----------- --------- --------- --------- ------- ------- -------
1, 9343, 9776, 10148,
200, 17679, 17964, 18336,
400, 16568, 16991, 17304,
600, 16132, 16385, 16582,
800, 15955, 16168, 16350,
1000, 15580, 15787, 16083,
-------------------------------------------------------------------------------
min:91257 avg:93071 max:94803 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------
Apache2
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.35/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l
ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d "http://127.0.0.1:8888/100.html"
Client Requests per second CPU
----------- ------------------------------- ---------------- -------
Concurrency min ave max user kernel MB RAM
----------- --------- --------- --------- ------- ------- -------
1, 3524, 3570, 3604,
200, 8280, 8606, 9162,
400, 8246, 8466, 8665,
600, 5381, 7998, 8482,
800, 1503, 8043, 10969,
1000, 0, 6814, 11594,
-------------------------------------------------------------------------------
min:26934 avg:43497 max:52476 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------
well, Nginx is still the fastest one. But G-WAN is close to it. Apache seems very slow in this test, just half of request Nginx served.
Test case 3:
623MB size file downloading test. By command:ab -c 100 -n 100 http://127.0.0.1:PORT/623mb.zip
Server Software: G-WAN
Requests per second: 2.01 [#/sec] (mean)
Time per request: 49747.564 [ms] (mean)
Time per request: 497.476 [ms] (mean, across all concurrent requests)
Transfer rate: 1283066.55 [Kbytes/sec] received
Apache2
Server Software: Apache/2.2.22
Requests per second: 2.59 [#/sec] (mean)
Time per request: 38638.219 [ms] (mean)
Time per request: 386.382 [ms] (mean, across all concurrent requests)
Transfer rate: 1651976.61 [Kbytes/sec] received
Nginx
Server Software: nginx/1.1.19
Requests per second: 1.69 [#/sec] (mean)
Time per request: 59245.971 [ms] (mean)
Time per request: 592.460 [ms] (mean, across all concurrent requests)
Transfer rate: 1077363.21 [Kbytes/sec] received
It shocked me again... RPS rank:
Apache > G-WAN > Nginx
No comments:
Post a Comment