collect to STDOUT as well as to a binary file set interval [process interval] to I [PI] seconds (default: 10 seconds) This can be a floating-point value. exclude subsystem(s) from collection/playback. subsystems are: Proc,Mem,Disk,Tape,Lsm-Vol,Net,Cpu,Filesys,ttY,Header select subsystem(s) for collection/playback (default: all subsystems) sort processes by %CPU usage select only top X processes (useful with -S) collect data only for disks in list (Regular Expression allowed) collect data only for processes in list ('%' = collect process) collect data only for processes whose Parent PID (PPID) is in list OR who are members of a Process Group (PGID) with the same ID. collect data only for processes whose Process Names contain above string. This can be a partial string, but must match exactly (no Regular Expressions). collect data only for processes owned by the specified users. UIDs (numeric) are also allowed. collect data only for LSM volumes in list. In order to make the names unique, the format is the diskgroup name, a slash ('/') and the volume name, for example: "my_dg/vol01". (Regular Expression allowed) sum MB/sec for all disks and tapes (turns off all subsystems by default) prefixes a 'tag' to all data lines to facilitate data-crunching in scripts Chop out a series of samples from a binary file. Format is [+]Year:Month:Day:Hour:Minute:Second. The optional '+' sign at the front indicates that time time is relative to the beginning of collection. No plus means absolute time. Every field except 'Second' can be left off. For absent fields, in absolute format, the values from the beginning of collection are used (see DESCRIPTION). If <start-time> is omitted, the start of the collection period is used. if <end-time> is omitted, the end of the collection period is used. display collect executable and datafile versions. Also, if used with the -p <playback-file> flag, will display the version of the given datafile. seek to last valid record and print it. This is primarily used by collgui to get the ending time of the collection period. print usage summary. Print 'full' process lines which are longer that 80 columns. The process priorities are shown and the RSS and VSZ values are always in KB, rather than formatted to fit into 4 columns. Set the duration for collect to (R)un. This can have one of two forms. The first can be sequence of <number><letter>[<number><letter>[...]] where number is an integer and letter is one of w,d,h,m,s where w=week, d=day, h=hour, m=minute, and s=second. For example: 4w2d5h would be 4 weeks, 2 days, and 5 hours. The second form is the same as that used by the -C option, except that a + (plus-sign) indicates the value should be relative to the current time. With no +, the value is an absolute time at which collect should exit. Set addition options. The t option tells collect to show the system and user time in the process subsystem as an absolute value, the way ps does. The default is to show a 1-second-normalized delta since the last sample, thus making graphs of these values more useful. The m option tells collect to show all the absolute memory values as Pages (8192 bytes) instead of in Megabytes. The f option causes collect not to ask before overwriting an output datafile. The n option causes collect not to set its priority higher. The z option causes collect not to write a compressed output datafile. The l option causes collect not to lock its pages in memory. The q option causes collect to use instantaneously measured queue lengths, instead of calculated averages.
collect is a tool to collect operating system and process data. Although it was originally intended for benchmarking situations, it was also designed to be as flexible as possible. Any subset of the Filesystems, message Queue, ttY), and Header can be defined for which data is to be collected. This subset is contstructed by either selecting (-s...) or excluding (-e...) particular subsystems, using the first letter, except in the cases of message Queue and ttY.
Because collect is meant to be as reliable as possible, it takes some steps to insure that it delivers reliable statistics: it locks itself into memory using plock(), so that it cannot be swapped out by the system, and it raises its priority using nice(). These measures should not have any impact on a system under normal load, and they should have only a minimal impact on a system under extremely high load. The locking of pages can be disabled using the -ol switch. The raising of the priority can be disabled using the -on switch.
Playing back multiple data files
collect can accept multiple binary data files using the -p switch and play them back as one stream, with monotonically increasing sample numbers. It is also possible to combine multiple binary input files into one binary output file, simply by supplying the input files with the -p switch and the output file with the -f switch. collect will combine arbitrary input files in whatever order you give on the command line!. That means the input files must be in chronological order if you want to do anything with them later. collect does NOT check this. It is up to YOU! Also, collect will allow you to combine binary input files from different systems, made at different times, with differing subsets of subsystems for which data has been collected. I let common sense guide you is using this option. The usual filtering options (-s<ss>, -e<ss>, -P, -D, etc) can be used here.
Normalization of data
Where it makes sense, data is presented in units/second. That is, for a disk, data such as KiloBytes transferred or the number of transfers is always "normalized" for 1 second. This happens no matter what interval is chosen. The same is true for cpu interrupts, system calls, and context switches; memory pages out, pages in, pages zeroed, reactivated, and copied on write; network packets in and out, collisions, KiloBytes in and out, and process user and system time consumed. On the other hand, things like free memory pages are (obviously) snapshot values, as are cpu states, disk queue lengths, or process memory (to name a few).
A collection interval can be specified using -i followed by an integer, optionally followed (without spaces) by comma and another integer. If the optional second integer is given, this is a separate interval which applies only to the process subsystem. The process-interval must be a multiple of the regular interval. Collecting process information is more expensive than everything else, and furthermore is not generally needed at the same frequency as, for example, disk I/O. Also, process data is the single largest space-hog in the binary data-file. Generally, specifying a process-interval of anything greater than 1 will significantly decrease the load the collector places on the system being monitored (< 2 % of a cpu).
Specifying what data to collect
To address space problems, the -S -nX options (mnemonics: sort,num) can be used to sort by percent CPU usage and and save only X processes, or specific processes can be targeted using -P<list>, where the list is comma-separated without blanks. Also, if there are many (> 100) disks connected to the system being monitored, the -D<list> option can be used to monitor only a particular list of disks. The form for specifying a disk is exactly the same as the way collect display disk names. That is, for versions 3 and 4 of the operating system, 'rzX', where X is the unit number (examples: rz0, rz22, rz109), and for version 5 (Steel) 'dskX', where X is a sequentially assigned number not related to SCSI IDs. Because of the naming ambiguity under V4 when a disk has a LUN of zero, both -Drz23 and -Drza23 will match both 'rz23' and 'rza23'. There is no ambiguity in Steel.
collect can now read and write gnuzip format compressed datafiles, thanks to the authors of the zlib compression library. Compressed output is by default enabled. It can be disabled using the -oz switch. The gnuzip compression format is used, so output files can be decompressed with the gnu tools, and older, uncompressed datafiles can be compressed with gzip, and the resulting files can be read by collect in their compressed form. Compression during collection does not appear to generate any additional CPU load. It appears that because compression uses buffers and therefore does not write to disk after every sample, and therefore makes fewer system calls, its overall impact is negligible. There is one aspect of compression that deserves mention: because the output is buffered, if collect should be terminated abnormally (due to a system crash, for example) more samples stand to be lost than if compression is not used. This should not, however, be an important consideration for most people. The extension ".gz" will be appended to the output filename, unless it already has this extension.
Specifying a time-range from a playback file
It may be useful to select samples from a sub-period of the time that the collector ran. This can be done using the -C<start>,<end> (mnemonic: chop) option. The format is [+]Year:Month:Day:Hour:Minute:Second. The plus sign ('+') indicates that the time should be interpreted as relative to the beginning of the collection period. If any of the fields Year, Month, Day, Hour or Minute are left out, the corresponding values from the start time are used in their place. To put it another way, the time-value is parsed from right to left: the first field is interpreted as Second, the second (if there is one), as Minute, and so on. For example, if the collection period is from October 21, 1996, 16:44:03 to October 21, 1996, 16:54:55, all but minutes and seconds can be left out: -C46:00,47:00 (from 16:46:00 to 16:47:00). However, if the collection has run overnight, it is necessary to specify the day as well. For example, if the period were Oct 21 16:44 to Oct 22 9:30, to specify a time-slice from 23:00 to 1:00, -C21:23:00:00,22:1:00:00 would be necessary.
If you want ascii output while collecting to a file, -a will do the trick.
-t (mnemonic: tags) will prefix each real data line with a unique tag. This in intended for make it easier for script-writers to extract data. (tags are superfluous if the perl script 'cfilt' is used).
-T (mnemonic: total) will shut off collection for all subsystems besides 'disk', and only display a total MB/sec across all disks in system. Of course, with -s, data for subsystems can be collected (or displayed in playback mode) in spite of this.
-R (mnemonic: run) will cause collect to terminate after a specified amount of time.
All flags that can reasonably be applied to collection AND playback do. That is, -P<list> during collection will collect data ONLY for the processes specified. However during playback it will only display data for the corresponding processes. So, if you're trying to save space in the binary data file, you can limit your collection to specific processes or specific disks, or specific subsystems, but if you want to look at lots of data, different bits at a time, you should collect everything, and then use the flags during playback to see just what you want. Obviously -sh (select Header subsystem) won't be very effective during collection, but you can do it if you want :-).
On Digital UNIX some disk statistics are problematic. On all versions of the operating system KBytes read ("RKB/S") and written ("WKB/S") , and the read ("R/S") and write ("W/S") operations are absolutely reliable (except on V4.0D if you're using collect V1.08d and haven't patched it :-). On V3 only the previous four statistics are meaningful. The rest should always be zero. On V4, data is presented for average service time ("AVS"), the queue length ("ACTQ"), and the percent busy ("%BSY"). Of these, only the "ACTQ" and "WTQ" are reliable. The other two will usually be more-or-less" accurate, but can under certain circumstances, can become wildly inaccurate. The "AVW", "WTQ", and "%WT" should always be zero. In V5, data is presented for all statistics except "%BSY", which is zero. "ACTQ" and "WTQ" are absolutely accurate. I *think* the others, with the possible exception of "%WT", are accurate (that is, "AVS" and "AVW"). I'm still testing V5. (It hasn't, after all, been released yet!)
Starting with datafile version 7, collect automatically reads older datafile versions when playing back. collect can also be used to 'convert' an older version datafile to the current version. This is done by specifying a playback file using -p flag and specifying a file to be written using the -f flag. Most of the flags the can be used to collect or extract particular kinds of data also work during conversion, for example: -s and -e to select data only from particular subsystems, -nX and -S to take only X processes and sort them by cpu usage, -D to select disks, -L to select LSM Volumes, and -P, -PC, -PU, and -PP to select processes. The timeslice, -C mechanism also does what you would expect.
Processes The process ID The username The percent of the cpu(s) the process is currently (more or less) using Resident Set Size - Physical memory used by process; includes shared memory. When the -F flag is used, this value is in KB, otherwise it is displayed in a compact format using 4 columns. The suffixes 'K', 'M', and 'G' are DECIMAL! That is, 'K' means x1000, 'M' x1000000, and 'G' x1000000000. Virtual memory used by process. The format is the same as described above. This is the user-mode cpu time being consumed by the process. It has two modes, depending on whether -ot has been used. In the default mode, the value is a normalized delta, that is, how much user time has been consumed since the last sample, normalized over 1 second. If -ot has been specified, the value is the absolute amount of user time the process has accumulated since birth, in the form Minutes:Seconds. The cpu time in kernel-mode being consumed by the process (see the description of UsrTime above). The Unix priority of the process. Only shown when the -F (full) flag is used. Input Block Operations - actual filesystem blocks read or written Output Block Operations Major faults - faults that were satisfied by doing I/O (going to disk) Minor faults - faults that were satisfied from cache The name of the running program; arguments are not retrieved.
Disks This is an index into the table that collect spits out - for scripting The name of the disk: rz<Unit-Number> in V3,4 or dsk<num> in V5. If this is a SCSI disk, the Bus/Target/Lun IDs, otherwise a "-". Reads per second KiloBytes read per second Writes per second KiloBytes written per second Average service time; time spent actually servicing the request - no wait time (in milliseconds) Average wait time; time spent in the wait queue. (in milliseconds) The number of requests in the active queue (that is, being serviced by the disk). The number of requests in the wait queue (have not yet been submitted to disk). Percent Busy - time spent servicing requests in interval divided by interval Percent Wait - time spent waiting in interval divided by interval
Tapes an index for scripting the device name, rmt<UNIT>, where UNIT is bus * 8 + target. This does NOT correspond to the device name in /dev, which is arbitrary. In V5 (steel), the device name is tape<num>. The Bus/Target/Lun IDs Reads per second KiloBytes read per second Writes per second KiloBytes written per second
LSM volumes Index for scripting Name in the form "Diskgroup/Volume" to insure uniqueness Reads per second KiloBytes read per second Average service time for reads with respect to LSM driver (includes disk driver wait time) Writes per second KiloBytes written per second Average service time for writes with respect to LSM driver (includes disk driver wait time)
CPU SUMMARY CPU states, averaged over all CPUs Interrupts per second System calls per second Context switches per second Number of processes in the run queue Load average over the last 5, 30, and 60 seconds Number of forks/Second Number of vforks/Second
SINGLE CPUs Index for scripts percent time (ticks) spent in user-level code. This includes NICE ticks. percent time (ticks) spent in kernel percent time (ticks) spent doing nothing Idle ticks while waiting for I/O to happen
Memory Number of Megabytes (or Pages -- see -tm switch) available Number of Megabytes (or Pages...) available on swap device(s) Amount of "active" memory in MB (or Pages) Amount of "Inactive" memory in MB (or Pages) - allocated to a process, but marked as not used in > X seconds Non-swappable kernel memory in MB (or Pages) Megabytes (or Pages) of memory used by Bufcache Pages paged in per second Pages paged out per second Pages zeroed per second - overwritten with zeroes before handing to a process Pages reactivated - status changed from inactive to active Copies-on-write per second Processes swapped per second UBC hits per second UBC pages pushed (written to disk) per second Pages allocated by UBC per second
Filesystem Index for scripting Name of Filesystem (or Domain#Fileset in the case of AdvFS) In MB In MB
Network Index for scripting Name of Network adaptor Packets received per second Input error packets per second Packets sent per second Output error packets per second Collisions per second KiloBytes received per second KiloBytes sent per second Percent of theoretical bandwidth being used (ethernet = 10Mbits/sec)
Message Queues this is the ID according to ipcs the key according to ipcs the owner UID of the Message Queue the number of bytes is use for all messages in this queue the number of messages in queue the PID of the last process to send a message on this queue the PID of the last process to read a message from this queue the time (in epoch seconds) of the last send the time of the last receive the creation time of this queue
Terminal I/O number of characters input number of characters output portion of input chars on CANNON queue portion of input chars on RAW queue
display to stdout, show data for all subsystems, interval is 10 seconds. (this is like running 'vmstat 10', 'iostat 10', 'netstat 10', 'volstat -i 10', and doing a 'while true; do ps ax; sleep 10; done') simultaneously. collect process info to data-file "foo.data", sort by cpu usage, and save top ten. exclude filesystem data, collect every second, except for process data, which gets collected every 5 seconds. print info header from binary data-file "foo.data" and exit. select only disk subsystem, and then only disks "rz0", "rz1", and "rz8" convert V7 datafile /tmp/olddata.col to V8 datafile /tmp/oldconverted.col write a new datafile, selecting only Disk, Cpu, Memory, and LSM subsystems, and only for samples between 4:22:00 and 5:22:00.
The average service time for raid devices is not available.
collect is not capable of dynamically recognising new devices or hardware. That is, if you run collect, and then shove a new disk onto a bus and start using it, collect will not see the disk until it is run again. The same is true of tape drives and LSM volumes created.
Statistics for ISDN PPP connections are not available.
Thanks to Jean-loup Gailly ([email protected]) and Mark Adler ([email protected]) for making zlib good and freely available!