Tuesday, December 4, 2012

G-WAN, Nginx, Apache2 benchmark (serving static files)

It's just a personal, informal, not rigorous experiment.
The test result may misleading or even incorrect. 

You can reproduce the test cases in your own environment easily.

Some benchmark articles:

http://nbonvin.wordpress.com/2011/03/14/apache-vs-nginx-vs-varnish-vs-gwan/
http://nbonvin.wordpress.com/2011/03/24/serving-small-static-files-which-server-to-use/

All the reports show G-WAN is mush faster than Nginx, but that's the comparison more than 1 year ago, the version it used( Nginx 0.7 & G-WAN 2.1) had been out of date.

I made my tests with current popular version: Nginx 1.1.19 & G-WAN 3.3.28

Hardware and Software:

CPU: 4 core, Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz
Memory: 5GB
OS: Ubuntu server 12.04.1 x64 
OS Kernel: 3.2.0-34-generic #53-Ubuntu SMP
**I made a mistake in first several tests, because I put server and client in different machines, although they are connected in a 1Gb LAN environment,  but in this kind of heavy load test, the 1Gb network become the bottleneck. 
In those tests, Nginx and G-WAN, even Apache2 got almost the same score. That confused me for a while, finally I figured out this problem. 
And retested the cases by running server and client on same machine.

Test cases:

3 static files in different size: 17KB, 3.5MB, 653MB
case 1: use ab.c (http://gwan.ch/source/ab.c) provided by G-WAN to download the 17KB file  in different concurrency
case 2: download 100Byte file in different concurrency
case 3: 100 clients concurrent download a 623MB size file.


Preparation:

1, Tune up system
* </etc/security/limits.conf> append
* soft nofile 200000
* hard nofile 200000
root soft  nofile 200000
root hard  nofile 200000


*</etc/pam.d/common-session> append
session required pam_limits.so

* in </etc/sysctl.conf> append or modify
fs.file-max = 5000000
net.core.netdev_max_backlog = 400000
net.core.optmem_max = 10000000
net.core.rmem_default = 10000000
net.core.rmem_max = 10000000
net.core.somaxconn = 100000
net.core.wmem_default = 10000000
net.core.wmem_max = 10000000
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_congestion_control = bic
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_max_syn_backlog = 12000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_mem = 30000000 30000000 30000000
net.ipv4.tcp_rmem = 30000000 30000000 30000000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_wmem = 30000000 30000000 30000000 


2, Static Files on server
copy testing files to G-WAN & Nginx folders.
for G-WAN, put the files under gwan installation path, like  gwan_linux64-bit/0.0.0.0_8080/#0.0.0.0/www/

for Nginx, put files anywhere, then alias the path to URL in nginx configuration.

3, Configurations
for G-WAN, there's no necessary additional configurations at all. Let it works on default port 8080.
for Nginx, increase the acceptable connections, and let the app serve on port 8989
* </etc/nginx/nginx.conf>

user www-data;
worker_processes 8;
pid /var/run/nginx.pid;

events {
        worker_connections 10000;
        multi_accept off; <-----MUST be OFF in this test, othervise, you will see lots of non-2xx failures.
       use epoll;
}
* application configuration </etc/nginx/sites-enable/ANYNAME>
server {
  listen 8989;
  server_name localhost;
        location / {
                alias /var/tf/;
        }
        access_log off;
        error_log off;
}

for Apache2, make it work on port 8888, and add some extra settings in </etc/apache2/apache2.conf>

StartServers          2
MaxClients          150
MinSpareThreads      25
MaxSpareThreads      75
ThreadLimit          64
ThreadsPerChild      25
MaxRequestsPerChild   0


4, Reboot & verify
reboot the server to make sure all changed configuration become effective.
after reboot, run cmd "ulimit -a" to check "max open file number"

start G-WAN by simply run ./gwan  in its installation folder.
nginx should already started up as a service.

Try to download the file by browsers:
http://10.1.2.229:8080/17k.rar <---G-WAN work
http://10.1.2.229:8989/17k.rar <---Nginx work
http://10.1.2.229:8888/17k.rar <---Apache work

<ab.c>


first, we have to modify some macros in <ab.c> to fit my needs.
1, select bench-marking tool

#define IBM_APACHEBENCH 
//#define HP_HTTPERF 
//#define LIGHTY_WEIGHTTP

2, indicate target server
#define IP   "10.1.2.229"
#define PORT "8080" //"8989" for Nginx, "8888" for Apache2

3, number of concurrent clients, and increasing steps,
#define FROM     0 // range to cover (1 - 1,000 concurrent clients)
#define TO      1000 // range to cover (1 - 1,000 concurrent clients)
#define STEP      200 // number of concurrency steps we actually skip

4, indicate target URL
#define URL "/17k.rar" //Or other file names

5, Build
gcc ab.c -O2 -o testapp -lpthread
./testapp




Test case 1:

download 17KB file

G-WAN Result:
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 2.97/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l


ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d  "http://127.0.0.1:8080/17k.rar"

  Client           Requests per second               CPU
-----------  -------------------------------  ----------------  -------
Concurrency     min        ave        max      user     kernel   MB RAM
-----------  ---------  ---------  ---------  -------  -------  -------
         1,      9420,     10636,     13093,
       200,     12987,     13035,     13095,
       400,     12222,     12254,     12299,
       600,     11701,     11882,     12022,
       800,     11717,     11808,     11934,
      1000,     11407,     11499,     11627,
-------------------------------------------------------------------------------
min:69454   avg:71114   max:74070 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------



Nginx Result:

===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.55/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l


ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d  "http://127.0.0.1:8989/17k.rar"

  Client           Requests per second               CPU
-----------  -------------------------------  ----------------  -------
Concurrency     min        ave        max      user     kernel   MB RAM
-----------  ---------  ---------  ---------  -------  -------  -------
         1,      8722,      9065,      9569, 
       200,     16120,     16299,     16506, 
       400,     14652,     14987,     15110, 
       600,     14092,     14220,     14314, 
       800,     13540,     13759,     13986, 
      1000,     13356,     13492,     13722, 
-------------------------------------------------------------------------------
min:80482   avg:81822   max:83207 Time:61 second(s) [00:01:01]
-------------------------------------------------------------------------------


Apache Result:

===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.54/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l


ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d  "http://127.0.0.1:8888/17k.rar"

  Client           Requests per second               CPU
-----------  -------------------------------  ----------------  -------
Concurrency     min        ave        max      user     kernel   MB RAM
-----------  ---------  ---------  ---------  -------  -------  -------
         1,      5512,      5639,      5761,
       200,     13004,     14481,     15457,
       400,     11569,     13759,     14052,
       600,     13420,     13485,     13532,
       800,     11723,     12788,     13341,
      1000,      5570,     10521,     12729,
-------------------------------------------------------------------------------
min:60798   avg:70673   max:74872 Time:61 second(s) [00:01:01]
-------------------------------------------------------------------------------


Really out of expectation, RPS of Apache2 is close to G-WAN, while Nginx serves 10000 more requests per second !!!
The result is different to what I read from Internet. 
A friend of mine, asked me to test the performance when JBoss serving static files.
here's the result of JBoss serving this 17KB file
JBOSS 7.1
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 0.41/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l


ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d  "http://127.0.0.1:8080/17k.rar"

  Client           Requests per second               CPU
-----------  -------------------------------  ----------------  -------
Concurrency     min        ave        max      user     kernel   MB RAM
-----------  ---------  ---------  ---------  -------  -------  -------
         1,      6459,      6550,      6666, 
       200,      8089,      9262,      9704, 
       400,      8427,      9287,      9581, 
       600,      8433,      9208,      9373, 
       800,      7625,      9020,      9544, 
      1000,      7407,      9206,      9559, 
-------------------------------------------------------------------------------
min:46440   avg:52533   max:54427 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------
Obviously, application server is not good at serving static files. 





Test case 2:

100 Byte size file downloading test.


G-WAN
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.37/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l


ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d  "http://127.0.0.1:8080/100.html"

  Client           Requests per second               CPU
-----------  -------------------------------  ----------------  -------
Concurrency     min        ave        max      user     kernel   MB RAM
-----------  ---------  ---------  ---------  -------  -------  -------
         1,     10649,     13385,     15629, 
       200,     16514,     17171,     17398, 
       400,     15559,     15838,     16155, 
       600,     15231,     15444,     15582, 
       800,     14978,     15133,     15360, 
      1000,     14700,     14895,     15101, 
-------------------------------------------------------------------------------
min:87631   avg:91866   max:95225 Time:61 second(s) [00:01:01]
-------------------------------------------------------------------------------

NGINX
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.34/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l


ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d  "http://127.0.0.1:8989/100.html"

  Client           Requests per second               CPU
-----------  -------------------------------  ----------------  -------
Concurrency     min        ave        max      user     kernel   MB RAM
-----------  ---------  ---------  ---------  -------  -------  -------
         1,      9343,      9776,     10148, 
       200,     17679,     17964,     18336, 
       400,     16568,     16991,     17304, 
       600,     16132,     16385,     16582, 
       800,     15955,     16168,     16350, 
      1000,     15580,     15787,     16083, 
-------------------------------------------------------------------------------
min:91257   avg:93071   max:94803 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------

Apache2
===============================================================================
G-WAN ab.c ApacheBench wrapper, http://gwan.ch/source/ab.c.txt
-------------------------------------------------------------------------------
Machine: 1 x 4-Core CPU(s) Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz
RAM: 4.35/4.77 (Free/Total, in GB)
Linux x86_64 v#53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 3.2.0-34-generic
Ubuntu 12.04.1 LTS \n \l


ab -n 1000000 -c [0-1000 step:200 rounds:10] -S -d  "http://127.0.0.1:8888/100.html"

  Client           Requests per second               CPU
-----------  -------------------------------  ----------------  -------
Concurrency     min        ave        max      user     kernel   MB RAM
-----------  ---------  ---------  ---------  -------  -------  -------
         1,      3524,      3570,      3604, 
       200,      8280,      8606,      9162, 
       400,      8246,      8466,      8665, 
       600,      5381,      7998,      8482, 
       800,      1503,      8043,     10969, 
      1000,         0,      6814,     11594, 
-------------------------------------------------------------------------------
min:26934   avg:43497   max:52476 Time:62 second(s) [00:01:02]
-------------------------------------------------------------------------------

well, Nginx is still the fastest one. But G-WAN is close to it. Apache seems very slow in this test, just half of request Nginx served.






Test case 3:

623MB size file downloading test. By command:
ab  -c 100 -n 100 http://127.0.0.1:PORT/623mb.zip

G-WAN
Server Software:        G-WAN
Requests per second:    2.01 [#/sec] (mean)
Time per request:       49747.564 [ms] (mean)
Time per request:       497.476 [ms] (mean, across all concurrent requests)
Transfer rate:          1283066.55 [Kbytes/sec] received

Apache2
Server Software:        Apache/2.2.22
Requests per second:    2.59 [#/sec] (mean)
Time per request:       38638.219 [ms] (mean)
Time per request:       386.382 [ms] (mean, across all concurrent requests)
Transfer rate:          1651976.61 [Kbytes/sec] received


Nginx
Server Software:        nginx/1.1.19
Requests per second:    1.69 [#/sec] (mean)
Time per request:       59245.971 [ms] (mean)
Time per request:       592.460 [ms] (mean, across all concurrent requests)
Transfer rate:          1077363.21 [Kbytes/sec] received


It shocked me again... RPS rank:
Apache > G-WAN > Nginx


No comments:

Post a Comment