[Orca-users] Announcement: Devilator 1 alpha for FreeBSD

Tue Sep 11 04:10:14 PDT 2007

Hello,

I'm happy to announce the first public release (consider it an alpha)  
of Devilator, a performance
data collector for FreeBSD systems that generates Orca compatible  
files. The README file follows.

************************************************************************ 
*******
WARNING!!!!

This is the first public evaluation release of a program developed  
for internal
use. Please exercise extreme care when using it, and report any  
issues to the
author to have them fixed in future releases.
************************************************************************ 
*******

==================================
           DEVILATOR
==================================

"Why does my system run slowly?" "What can I do to make it run  
faster?" "The
CPU seems to be idle. Why don't I get better performance?" "Will a  
bigger CPU
make it run faster?" "Will I need a better CPU next year?" "Was the  
system
slower last Thursday, or is it just my imagination?" These questions  
are made probably hundreds or perhaps even thousands of times every  
day, and the answer
is not always easy.

A system's performance is a function of its CPU power, bus bandwidth,
memory capacity, disk I/O bandwidth (often neglected) and software.  
Nowadays,
with the usage of concurrent programs growing, software performance  
problems
get increasingly important as well. We can have the fastest and  
greatest hardware available, but a poor locking decision in a  
concurrent program
can make it crawl, no matter what we do. A bad mix of workloads can  
cause
these problems as well.

Performance tuning is often a black art, and historical data is reall
valuable, a need that the classical commands such as top, vmstat,  
gstat, etc
don't address adequately (in my opinion).

Devilator is a system performance data collector for FreeBSD. It's  
designed
to work with Blair Zajac's Orca system (http://www.orcaware.com/ 
orca/) and
it aims to be a FreeBSD equivalent of the "Orcallator" data collector  
for
Solaris.

This program has been growing for a long time, supplying performance  
data for
our in-house systems. As this program has been written as a series of  
quick
hacks, it is actually quite ugly, but effective anyway. The main goal  
when
writing it was to gather enough performance data from the OS, always  
avoiding
typical scripts that launch many other utilities. This program reads  
OS data
directly, either through the sysctl interface or from GEOM (in the  
case of
disk I/O performance data). As the documentation on some of these  
issues is
not great, I have borrowed code from the GEOM utility "gstat", from  
"top",
and "orcallator.se", part of the Orca package.

The data collected by Devilator can be divided among the following  
groups:

1- Processor/s usage. This includes the classical "system load"  
values, and the percentage of CPU time spent on user processes,  
system, interrupt and nice.

2- OS activity. Includes a graph of forks/s, system calls/s, system  
processes and interrupt activity. The system processes and interrupt  
information show
where the system is spending its share of CPU time. The system  
processes shown
vary from version to version, with more information offered by the  
most recent
FreeBSD releases. Hopefully this can be useful for system  
administrators and
system developers alike, keeping historic data on the activity of tasks
like "vmdaemon", "syncer", etc.

3- Process activity. Includes the number of processes in the system,  
how many
of them are in a given state (running, sleeping, stopped, zombies,  
waiting or
locked), and a graph of sleep situations. How many processes are  
sleeping on
select(), waiting for network I/O, waiting for the disk subsystem,  
filesystem,
etc? This is really useful when evaluating the performance of  
programs such
as MySQL, trying different filesystem configurations or threading  
models. These
data will be hopefully useful to FreeBSD developers and porters as well.

4- Network activity, offering the classical information on  
established connections, TCP and UDP traffic, network interface  
traffic and errors, TCP
connection rate, etc.

5- Virtual memory, including swap and paging activity, memory usage,  
and page
scanner activity.

6- Disk I/O data (available on FreeBSD 5+, as it depends on GEOM),  
including
the percentage of busy time for disks, partitions and slices, together
with the volume of data transfers in MB/s and the average service  
time in ms.

---------------
USING DEVILATOR
---------------

In order to use this program it's a good idea to be familiarised with  
Orca
first. Orca (http://www.orcaware.com/orca/) is a general purpose  
graphing
program based on RRDtool.

This program can be used to monitor a single machine, running both  
Devilator
and Orca on it, or it can be installed at dozens of servers, where it  
will just
gather performance data and write it to text files, while a different  
server
downloads those performance data files periodically, running Orca and
generating the HTML files and RRD graphs for all of them.

Depending on the number of machines, a single machine can monitor  
dozens of
servers, and the impact of Devilator is negligible. It just wakes up at
5 minute intervals, reads some OS statistics using sysctl(3) and  
GEOM, and
writes a line to a text file.

In order to have Orca use the data provided by Devilator, the supplied
Orca configuration file, devilator.cfg, must be used. The relevant
variables to set up are:

--------------
# base_dir is the directory where the RRD files will be stored.
base_dir                /var/orca/rrd/orcallator

# html_dir is the directory where the web pages (HTML files and PNG  
graphs)
# will be stored. This directory will be made available with a suitable
# web server such as Apache.

html_dir                /var/www/orca

# find_files specifies the directory under which the text files with
# performance data reside. It is possible to monitor a set of  
machines with a
# single Orca instance, with each machine having its own directory under
# /var/orca/orcallator, such as /var/orca/orcallator/server1,
# /var/orca/orcallator/server2...

find_files              /var/orca/orcallator/(.*)/(?:(?:orcallator)| 
(?:percol))-
\d{4}-\d{2}-\d{2}(?:-\d{3,})?(?:\.(?:Z|gz|bz2))?
---------------

Please read the Orca documentation and become familiar with it.

-------------------
FUTURE WORK
-------------------

The graphs are a bit messy right now, although they are easy to  
customize
editing the .cfg file. This program generates a lot of information  
that might
not be necessary for everyone. Unnecessary graphs can be turned off  
just by
commenting out the relevant sections in the configuration file.

Some enhancements would probably be useful. Tracking resources usage by
process name would allow the administrator to know which CGI scripts are
using more resources, etc.

I will be happy to hear about other interesting datasets. Simple sysctl
fetches are trivial to do. But obviously it would be insane to include
all the available data.

--------------------
PORTABILITY
--------------------

This program has been developed under FreeBSD, and some of the bits
(like its reliance on GEOM to gather disk I/O statistics) are pretty
FreeBSD-specific.
So far I have tested it and used it on FreeBSD 4.9 (x86), FreeBSD 6.x on
sparc64 and x86, and FreeBSD 7 on amd64.

--------------------
SUPPORT
--------------------

This program is supplied as is, and there's absolutely no guarantee
of any kind, including suitability for any particular purpose.  
There's no
guarantee of technical support either.

This program started its life as an internal development, and I'm  
polishing
it for public distribution. Most of the development time has been  
payed for
by my employer, Sarenet, and it's being released as a contribution to  
the
FreeBSD community.

Please send bugs, ideas, flames, etc to the following address:

Borja Marcos.
Sarenet S.A.

borjamar at sarenet.es
borjam at gmail.com (Preferred for Devilator issues)

Devilator can be downloaded from:

http://homepage.mac.com/borjam/.Public/devilator-1.0a.tar.gz
MD5 (devilator-1.0a.tar.gz) = 5868761c2ce0f8dc3ff51167fd06400a

And the PGP signature for the file follows:

-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.5.3 (Build 5003)

qANQR1DEDQMAAhGlPtFtULJLjAHCPwMFAEbmbwOlPtFtULJLjBECYlEAnj1HejG2
rBJVYur8VvZ51MlSYVX1AJ9Aak05xRRkQ/BKn8Asl9nWsxD6rA==
=rKMg
-----END PGP SIGNATURE-----

----------------
"The thing he realised about the windows was this: because they had  
been converted into openable windows after they had first been  
designed to be impregnable, they were, in fact, much less secure than  
if they had been designed as openable windows in the first place."
    Douglas Adams, "Mostly Harmless"