CF.Objective Notes -- Maintaining Healthy ColdFusion Servers
July 02, 2012
Maintaining Healthy ColdFusion Servers -- Rob Brooks Bilson
ColdFusion server can be perfect, but if the network or other things are bad, you'll still have problems.
Step through each phase of the ecosystem
Web server, app code, app server, os, hardware, external systems, network, etc.
Where do we start with all of this?
Get organized
Are you virtualized?
Virtualization to adopted everywhere
but it gets you the ability to set something up ONCE, save that VM, and now all of the sudden I can bring up another CF server.
Just spin up another copy of the VM I've already done.
In a virtual environment, disaster recovery is way easier
No longer concerned with physical hardware
Pre-installation checklist
latest drivers?
OS patched?
Current web server version and patches?
Separate drives/partitions
C: OS
D: CF Install
E: CFML and website assets
Don't want app server to compete with OS for disk IO, etc.
Installation considerations
set up service accounts (don't want to run CF under "root" account, etc)
Only install what you need now
-- solr, .NET integration, examples, etc.
If you're not going to use those things, don't install them
You can always rerun the installer later
If you don't need them, your'e just tying up additional disk space, potentially leaving server open to new vulnerabilities.
Multi-tier architecture (will CF run on the same box as the web server? what if i want to scale out?)
CF10 Secure Profile
in installer
if you choose secure profile, will install with Adobe's recommended security settings.
May be unhappy with those default settings, read the docs, can make the assessment if that's a good starting point for you or not.
Post-install checklist
Update JVM
Secure the CF Administrator
For CF9 make sure the JRun build in web server is not running (if applicable)
-- check JRun4/servers/instance/server-inf/jrun.xml
CF Admin settings
settings, request timeout after N seconds, etc
request tuning
caching
memory variable timeouts
turn off debugging
logging
-log slow pages
-log scheduled tasks
System Probes
Logging
Do as much logging as possible
in the CF Admin
- customize your log location
set a max log file
specify the number of log archives to keep
in your code
- consider segmenting your logs by application or module
- aggressively log within your application
3rd party tools for logging and analysis
-- Splunk, both open source and commercial versions
Security -- where do you start
CF9, Pete Freitag -- CF Lockdown Guide
"Security" tab in CF Admin
If you don't use RDS, turn it OFF
if you DO use it, make sure it has a password
Don't forget about the web server
Code Review early and often
-- write secure code
-- new security features in CF10
-- new toolset for cryptography in CF10
Crucible -- 5 user license for 10 bucks
Can go in and comment code for code reviews
CF10 Audit logging
logs most admin changes
5 servers, different settings in each, now we can tell who changed what, when, etc
(as long as you use individual usernames for each person that has access to CF Admin)
ESAPI -- included in CF9 and 10 (but not implemented natively until CF10, but the files are there in 9)
Server Management
AIR app
gets a consolidated view of all your servers
Can "diff" server configurations
start/stop instances
etc
Hotfix Installer in CF10
(and there's talk of rolling this back into CF9, tho no more details on that right now)
Monitoring
Proactive vs reactive
Logigng
-- custom log files
Probes
Server monitor
3rd party apps
- fusion reactor
- seefusion
- cftracker
- new relic
- pingdom
Sending an email when the system errors
- bad ideal, you get USED to the way your system breaks, and start ignoring the emails
Don't' want to address on a "problem by problem" basis, want to be more proactive than that
Performance ramifications
- yes, everything has a performance side effect
- really comes down to understanding the types of things you're doing
- i.e. writing log files is a small performance hit
"A fool with a tool is still a fool."
really easy to run this stuff and not understand it
take the time to learn what the metrics mean
CF Server Monitor
JMeter
Managing Robots and Spiders
robots.txt
-- not all bots/spiders honor this
no-index, no follow directives
-- again, not always respected
be mindful of the effect on CF sessions
Tuning and Optimization
the importance of load testing
-- how will your app react with 10 users? 100? 1000?
-- need to understand the capacity of your app and server
-- are there things i can do to tune my app, squeeze more performance out?
-- am i database constrained? network?
-- what happens if 50 users log in at the same time? or create PDFs at the same time? that's why it's important to load-test all of your app, not just "yeah i logged in and clicked a couple pages"
check out JMeter, easy to setup and walk through
GZip compression
caching
When garbage collection happens...everything STOPS
As more garbage gets in the system, the pauses for GC tale longer and longer, more frequently
can end up with a "spiral" that brings down the server
recommended: instead of "1 big CF instance", run multiple smaller ones
doesn't ALWAYS work, but might work for your server depending on what you are running
Resources:
CF Lockdown Guide
Apache JMeter
etc
@cftracker
@cfwhisperer
@fusion_reactor
@carehart
full slide deck for this press is on Rob's blog:
blog.brooks-bilson.com
ColdFusion server can be perfect, but if the network or other things are bad, you'll still have problems.
Step through each phase of the ecosystem
Web server, app code, app server, os, hardware, external systems, network, etc.
Where do we start with all of this?
Get organized
Are you virtualized?
Virtualization to adopted everywhere
but it gets you the ability to set something up ONCE, save that VM, and now all of the sudden I can bring up another CF server.
Just spin up another copy of the VM I've already done.
In a virtual environment, disaster recovery is way easier
No longer concerned with physical hardware
Pre-installation checklist
latest drivers?
OS patched?
Current web server version and patches?
Separate drives/partitions
C: OS
D: CF Install
E: CFML and website assets
Don't want app server to compete with OS for disk IO, etc.
Installation considerations
set up service accounts (don't want to run CF under "root" account, etc)
Only install what you need now
-- solr, .NET integration, examples, etc.
If you're not going to use those things, don't install them
You can always rerun the installer later
If you don't need them, your'e just tying up additional disk space, potentially leaving server open to new vulnerabilities.
Multi-tier architecture (will CF run on the same box as the web server? what if i want to scale out?)
CF10 Secure Profile
in installer
if you choose secure profile, will install with Adobe's recommended security settings.
May be unhappy with those default settings, read the docs, can make the assessment if that's a good starting point for you or not.
Post-install checklist
Update JVM
Secure the CF Administrator
For CF9 make sure the JRun build in web server is not running (if applicable)
-- check JRun4/servers/instance/server-inf/jrun.xml
CF Admin settings
settings, request timeout after N seconds, etc
request tuning
caching
memory variable timeouts
turn off debugging
logging
-log slow pages
-log scheduled tasks
System Probes
Logging
Do as much logging as possible
in the CF Admin
- customize your log location
set a max log file
specify the number of log archives to keep
in your code
- consider segmenting your logs by application or module
- aggressively log within your application
3rd party tools for logging and analysis
-- Splunk, both open source and commercial versions
Security -- where do you start
CF9, Pete Freitag -- CF Lockdown Guide
"Security" tab in CF Admin
If you don't use RDS, turn it OFF
if you DO use it, make sure it has a password
Don't forget about the web server
Code Review early and often
-- write secure code
-- new security features in CF10
-- new toolset for cryptography in CF10
Crucible -- 5 user license for 10 bucks
Can go in and comment code for code reviews
CF10 Audit logging
logs most admin changes
5 servers, different settings in each, now we can tell who changed what, when, etc
(as long as you use individual usernames for each person that has access to CF Admin)
ESAPI -- included in CF9 and 10 (but not implemented natively until CF10, but the files are there in 9)
Server Management
AIR app
gets a consolidated view of all your servers
Can "diff" server configurations
start/stop instances
etc
Hotfix Installer in CF10
(and there's talk of rolling this back into CF9, tho no more details on that right now)
Monitoring
Proactive vs reactive
Logigng
-- custom log files
Probes
Server monitor
3rd party apps
- fusion reactor
- seefusion
- cftracker
- new relic
- pingdom
Sending an email when the system errors
- bad ideal, you get USED to the way your system breaks, and start ignoring the emails
Don't' want to address on a "problem by problem" basis, want to be more proactive than that
Performance ramifications
- yes, everything has a performance side effect
- really comes down to understanding the types of things you're doing
- i.e. writing log files is a small performance hit
"A fool with a tool is still a fool."
really easy to run this stuff and not understand it
take the time to learn what the metrics mean
CF Server Monitor
JMeter
Managing Robots and Spiders
robots.txt
-- not all bots/spiders honor this
no-index, no follow directives
-- again, not always respected
be mindful of the effect on CF sessions
Tuning and Optimization
the importance of load testing
-- how will your app react with 10 users? 100? 1000?
-- need to understand the capacity of your app and server
-- are there things i can do to tune my app, squeeze more performance out?
-- am i database constrained? network?
-- what happens if 50 users log in at the same time? or create PDFs at the same time? that's why it's important to load-test all of your app, not just "yeah i logged in and clicked a couple pages"
check out JMeter, easy to setup and walk through
GZip compression
caching
When garbage collection happens...everything STOPS
As more garbage gets in the system, the pauses for GC tale longer and longer, more frequently
can end up with a "spiral" that brings down the server
recommended: instead of "1 big CF instance", run multiple smaller ones
doesn't ALWAYS work, but might work for your server depending on what you are running
Resources:
CF Lockdown Guide
Apache JMeter
etc
@cftracker
@cfwhisperer
@fusion_reactor
@carehart
full slide deck for this press is on Rob's blog:
blog.brooks-bilson.com