my @a = (qw(Lorem ipsum dolor sit amet consectetuer adipiscing elit
	     sed diam nonummy nibh euismod tincidunt ut laoreet dolore
	     magna aliquam erat volutpat), "\n");

 sub print_many {
  for (@a) { print $_ };
  return 1;
 }

 sub print_join {
   print join "", @a;
   return 1;
 }

 sub print_list {
   print @a;
   return 1;
 }

Use the Benchmark module to find out!

                Rate print_many print_list print_join
 print_many  78668/s         --       -60%       -76%
 print_list 194313/s       147%         --       -41%
 print_join 329448/s       319%        70%         --

print join "", @list is MUCH faster

Boring!

Who cares?!

Show more pages

With less resource usage

Faster

Design the arcitecture right

Design the code right

Optimize the code

(in that order!)

Server performance:

 Requests per second:
     Concurrent requests 
   * Response time
   = Requests per second

User performance:

  Response time:
     Time for document to start serving
   + Download time
   = Response time
 

One server with one httpd installation running mod_perl

Ouch; the machine is being killed by swap!

 252 processes: 2 running, 245 sleeping, 5 zombie
 CPU states:  1.8% user,  0.0% nice, 13.2% system,  0.9% interrupt, 84.1% idle
 Mem: 396M Active, 50M Inact, 52M Wired, 1272K Cache, 61M Buf, 992K Free
 Swap: 2048M Total, 351M Used, 1697M Free, 12% Inuse, 416K In, 6812K Out

   PID USER PRI NICE  SIZE    RES STATE  C   TIME   WCPU    CPU COMMAND
 88477 web    2   0 29508K 17720K select 0   0:02  0.00%  0.00% httpd
 88475 web    2   0 29492K 18104K select 1   0:02  0.00%  0.00% httpd
 88478 web    2   0 29508K 19352K select 0   0:02  0.00%  0.00% httpd
 88467 web   28   0 29516K  3144K pfault 1   0:02  0.00%  0.00% httpd
 88470 web    2   0 29500K 19436K select 0   0:02  0.00%  0.00% httpd
 88472 web   28   0 29500K  8088K pfault 0   0:02  0.00%  0.00% httpd
 88476 web    2   0 29512K  3320K select 0   0:02  0.00%  0.00% httpd
 88471 web    2   0 29484K 20364K select 0   0:02  0.00%  0.00% httpd
 88460 web    2   0 29504K 20300K select 0   0:02  0.00%  0.00% httpd
 88469 web    2   0 29508K  2836K select 1   0:02  0.00%  0.00% httpd
 88468 web    2   0 29504K 19320K select 1   0:02  0.00%  0.00% httpd
 88465 web    2   0 29500K  3232K select 1   0:02  0.00%  0.00% httpd
 88464 web    2   0 29500K 18636K select 1   0:02  0.00%  0.00% httpd
 88450 web   28   0 26696K 14476K pfault 0   0:02  0.00%  0.00% httpd
 88466 web   28   0 29500K 19200K pfault 1   0:02  0.00%  0.00% httpd
 88461 web   28   0 26656K 14180K pfault 0   0:02  0.00%  0.00% httpd
 88457 web   28   0 26976K 13796K pfault 0   0:02  0.00%  0.00% httpd
 88456 web   28   0 26912K  9936K pfault 0   0:02  0.00%  0.00% httpd
 88449 web   28   0 26856K 13540K pfault 0   0:02  0.00%  0.00% httpd
 88453 web   28   0 26944K 13776K pfault 1   0:01  0.00%  0.00% httpd
 88462 web   28   0 27008K 15948K pfault 0   0:01  0.00%  0.00% httpd
 88459 web   28   0 26952K 15328K pfault 1   0:01  0.00%  0.00% httpd
 88454 web   28   0 26952K 15448K pfault 1   0:01  0.00%  0.00% httpd
 88474 web   28   0 26632K  8612K pfault 1   0:01  0.00%  0.00% httpd

Default httpd.conf

  MaxClients 150

N connections = N fat mod_perl processes

    15 connections per second
  + Each request takes 4 seconds to
    write to the network
  = 60 active mod_perl processes
  + 20 spare processes for peaks
  = 80 active mod_perl processes

  * 25MB-15MB shared = 10MB memory
    per process
  = 800MB memory usage

Mostly wasted on waiting for slow clients!

Set the max number of httpd processes

  MaxClients 15

in httpd.conf

  10MB * 20 processes = 200MB memory

No more swapping

The machine is not crashing

But it's just the same!

    15 processes available

    Each request takes *4* seconds to
    write to the network

    RPS (requests per second) is below 4! 

200ms or 500ms to generate a document doesn't matter much for server performance

CPU's are almost 100% idle

Setup images.perl.org and make all image references point to there

The "static server" can serve all not dynamic embedded content

Buffer everything through a proxy

• What will it do?

  Offload the buffering

  Serve static content

  Caching

  Distribute requests to different backend servers

  All in a "slim" process!

• Apache 2.0

  Very simple to plug into your environment

   LoadModule proxy_module modules/mod_proxy.so
   LoadModule rewrite_module modules/mod_rewrite.so

   ProxyPreserveHost On
   RewriteEngine On 
   RewriteCond  %{REQUEST_URI}    !^/images/
   RewriteCond  %{REQUEST_URI}    !^/proxy-status
   RewriteRule (.*) http://localhost:8933$1 [P]

• ProxyPreserveHost On

  Preserves the Host: header from the original request for seamless VirtualHosts

• $r->connection->remote_ip

 RewriteEngine On 
 RewriteCond  %{REQUEST_URI}    !^/images/
 RewriteCond  %{REQUEST_URI}    !^/proxy-status
 RewriteRule (.*) http://localhost:8933$1 [P]

• Pre-1.3.25

  To get $r->connection->remote_ip to work, install the mod_perl_add_forward.c module

  http://develooper.com/code/mpaf/

• VirtualHosts

  VirtualHosts will not work out of the box

  Use different ports

• Proxy Setup

   <VirtualHost *>
      servername dev.perl.org
      RewriteRule ^/(.*) http://localhost:8932/$1 [P,L]
   </VirtualHost>

   <VirtualHost *>
     servername develooper.com
     serveralias www.develooper.dk
     RewriteRule ^/(.*) http://localhost:8933/$1 [P,L]
   </VirtualHost>

• mod_perl setup Port 80 Listen 127.0.0.1:8932 Listen 127.0.0.1:8933

   <VirtualHost *:8932>
     ServerName dev.perl.org
     ...
   </VirtualHost>

   <VirtualHost *:8933>
     ServerName develooper.com
     ...
   </VirtualHost>

Use different backend servers for different "virtual hosts"

images.perl.org served by proxy

Use different backend servers for different urls

Can be physically different boxes

Use different backend servers for different urls

 RewriteRule ^/(foo.*) http://localhost:8933/$1 [P,L]
 RewriteRule ^/(.*)    http://localhost:8934/$1 [P,L]

Or different filetypes

 RewriteRule ^/(.*\.tt)$ http://localhost:8933/$1 [P,L]
 RewriteRule ^/(.*)      http://localhost:8934/$1 [P,L]

Proxy to different backend servers

 RewriteRule ^/(.*\.asp) http://win32box/$1 [P,L]
 RewriteRule ^/(.*)$     http://modperl/$1 [P,L]

Each backend can be run by a different user

  functionality
  developers
  customers
  sites

• Squid

  Good caching engine

  Not as flexible for "reverse proxying" as apache

• mod_backhand

  Load balancing proxy module

• lingerd

  tcp/ip "tweak"

Handle many front-end requests with little memory usage

mod_perl processes gets freed up as fast as possible

Memory issue is eliminated

Requests Per Seconds compare to load testing on a local LAN

  MaxRequestsPerChild  2000

  MinSpareServers         1
  MaxSpareServers        10
  MaxClients             10
  StartServers           10

Always keep at least one server ready for a new request

Allow up to 10 idle servers

Allow up to 10 servers

Start 10 servers at startup time

Let each server handle 2000 requests before it gets "respawned"

proxy and mod_perl instances sharing each physical box

Indispensable for monitoring how many processes are running and what they are doing

 ExtendedStatus On

 <Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 1.2.3.5
 </Location>

 http://dev.perl.org/server-status

 http://dev.perl.org/server-status?auto
 

• Browsers/end user proxies can cache from servers

  Set the right headers, Expires, Content-Length, Last-Modified

• Reverse proxies

  You set the rules

  Only when complete documents can be cached

• Application can cache from other parts of the system (eg database)

  Cache database queries

  Temporary denormalized tables

  Cache remote data

  Complicated computations

Make caches Expire the document in 48 hours

 $r->header_out('Expires', 
                 HTTP::Date::time2str(time + 60*60*24*2))

Modern caches will honor this

 $r->header_out('Cache-Control', "max-age=" . 60*60*24*2);

Don't cache

 $r->header_out('Cache-Control', "no-cache, private");

• Server sends with document request

   Last-Modified: Mon, 24 Jun 2002 09:07:15 GMT

• Browser or cache sends with later request for the same document

   If-Modified-Since: Mon, 24 Jun 2002 09:07:15 GMT

• Server checks something like the following

   $r->header_out("Last-Modified", ...);

   if ((my $rc = $r->meets_conditions) != OK) {
       return $rc;
   }

  If document hasn't been modified since last time, return

   HTTP/1.1 304 Not Modified

  Otherwise generate the document as usual

Always include the content_type header

  $r->content_type("image/png");

Caching and Keep-Alive requires Content-Length

  $r->header_out('Content-Length', $length);

Etags

http://perl.apache.org/docs/general/correct_headers.html

 $ ab -c 3 -n 100 http://nntp.x.perl.org/group/

Does a slow request to a remote server to fetch data

 Time taken for tests:   10.12586 seconds
 Requests per second:    9.99 [#/sec] (mean)
 Time per request:       300.378 [ms] (mean)
 Time per request:       100.126 [ms] (mean, across all concurrent requests)
 Transfer rate:          75.90 [Kbytes/sec] received

Add mason caching

  $m->cache_self(expire_in => "12h");

 Time taken for tests:   3.978180 seconds
 Requests per second:    25.14 [#/sec] (mean)
 Time per request:       119.345 [ms] (mean)
 Time per request:       39.782 [ms] (mean, across all concurrent requests)
 Transfer rate:          192.30 [Kbytes/sec] received

Cache the whole document. Really really efficient.

• Apache 2.0

  Flexible storage modules

  Make your own backend storage module (in C)

  Disk and memory implemented

  Marked experimental, and it is!

   <VirtualHost *>
     ServerName nntp.x.perl.org

     # mod_cache settings
     CacheOn On
     CacheIgnoreCacheControl on
     CacheIgnoreNoLastMod on
     CacheDefaultExpire 43200

     # mod_disk_cache settings
     LoadModule disk_cache_module modules/mod_disk_cache.so
     CacheRoot /home/web/front/cache/
     CacheSize 50000
     CacheEnable disk /
     CacheDirLevels 5
     CacheDirLength 3

     RewriteEngine On
     RewriteRule (.*) http://localhost:8222$1 [P]
   </VirtualHost>

• Apache 2.0, ... continued

  If it worked, a benchmark would show several hundreds of requests per second.

  Works without the Content-Length header

  Only the proxy will log cached requests

  Last-Modified and Expires headers controls the expire time

Databases are hard(er) to scale

Reverse proxy minimizes the need for many concurrent database connections

Apache::DBI minimizes the number of new connections made

Summary tables can make the lookups faster

Use faster databases for caching (MySQL as cache for Oracle)

• Lookup tables

  Frequent "name" to "id" lookups

  If the list is small, cache them in memory

   my $cache;
   my $cache_refreshed = 0;

   sub id_to_data {
     my $id = shift;

     return $cache->{$id}
       if ($cache and $cache_refreshed > time-600);

     my $dbh = db_open;
   
     $cache = $dbh->selectall_hashref('select * from table', 'id');
     $cache_refreshed = time;
     return $cache->{$id};
   }

• Cache::Cache

  Really easy to use.

  Caching "infrastructure"

     use Cache::FileCache;

     my $cache = new Cache::FileCache;

     my $customer = $cache->get($name);

     unless ($customer) {
       $customer = get_customer_from_db( $name );
       $cache->set( $name, $customer, "10 minutes" );
     }
     return $customer;

• When and what to cache!

  Be sure to benchmark that it's a good idea first!

  Measure cache hit ratios

  
• DBI::Profile

  Gives you list of statements and methods

  How often and how long they take

Divide pages in smaller fragments and cache each fragment

Benchmark!

Sometimes use regular expressions to hack components into "super components"

  $output = $cache->get('foo');
  $dynamic_stuff = get_dynamic_data($r);
  $output =~ s{<!-- DYNAMIC_DATA_HERE -->}
	      {$dynamic_stuff}s;

Caching is good

Multiple levels of caching better

Be careful not to spend more time managing the cache than you save caching

Caching minimizes the number of database lookups

Do your own!

• Perl tools

  Benchmark.pm - Benchmark snippets

  Apache::DProf & Devel::DProf - Profile code

  DBI::Profile - Profile database usage

• HTTP testing

  Run them on a separate box!

  ab - really simple http tester

  http_load - slightly more advanced, list of urls

  Apache Flood - advanced profile driven http tester

• Why
  

  Divide load between servers

  Redundancy between servers

• How

  "Hardware" load balancers

  LVN (Linux software solution)

No real world experience yet

(Can be) multithreaded

Run more http threads than "perl threads"

Don't have to use the proxy setup. Maybe.

Lots of stuff doesn't work with threads yet

Architecture more important than code

Architecture much more important than code

Some architecture improvements are really easy to implement

Premature optimization is a waste of time

Lots of optimizations are a waste of time

• These slides

  http://develooper.com/modperl/

• mod_perl guide

  http://perl.apache.org/docs/1.0/guide/

• lingerd

  http://www.iagora.com/about/software/lingerd/

• Perrin Harkins eToys paper

  http://perl.apache.org/docs/tutorials/apps/scale_etoys/etoys.html

• mod_perl list

  modperl-subscribe@perl.apache.org

• Perrin's "Efficient Shared Data with mod_perl"

  Thursday 4.30pm

Thank you for listening

TicketMaster/CitySearch

ValueClick

Viridiana

the mod_perl list