projects / exim.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
Fix exigrep for "Completed" lines that have additional data (as a result
[exim.git] / src / src / eximstats.src
CommitLineData
059ec3d9 1#!PERL_COMMAND -w
8974000d 2# $Cambridge: exim/src/src/eximstats.src,v 1.3 2004/12/20 13:12:32 steve Exp $
059ec3d9
PH		 3
4# Copyright (c) 2001 University of Cambridge.
5# See the file NOTICE for conditions of use and distribution.
6
7# Perl script to generate statistics from one or more Exim log files.
8
9# Usage: eximstats [<options>] <log file> <log file> ...
10
11# 1996-05-21: Ignore lines not starting with valid date/time, just in case
12# these get into a log file.
13# 1996-11-19: Add the -h option to control the size of the histogram,
14# and optionally turn it off.
15# Use some Perl 5 things; it should be everywhere by now.
16# Add the Perl -w option and rewrite so no warnings are given.
17# Add the -t option to control the length of the "top" listing.
18# Add the -ne, -nt options to turn off errors and transport
19# information.
20# Add information about length of time on queue, and -q<list> to
21# control the intervals and turn it off.
22# Add count and percentage of delayed messages to the Received
23# line.
24# Show total number of errors.
25# Add count and percentage of messages with errors to Received
26# line.
27# Add information about relaying and -nr to suppress it.
28# 1997-02-03 Merged in some of the things Nigel Metheringham had done:
29# Re-worded headings
30# Added received histogram as well as delivered
31# Added local senders' league table
32# Added local recipients' league table
33# 1997-03-10 Fixed typo "destinationss"
34# Allow for intermediate address between final and original
35# when testing for relaying
36# Give better message when no input
37# 1997-04-24 Fixed bug in layout of error listing that was depending on
38# text length (output line got repeated).
39# 1997-05-06 Bug in option decoding when only one option.
40# Overflow bug when handling very large volumes.
41# 1997-10-28 Updated to handle revised log format that might show
42# HELO name as well as host name before IP number
43# 1998-01-26 Bugs in the function for calculating the number of seconds
44# since 1970 from a log date
45# 1998-02-02 Delivery to :blackhole: doesn't have a T= entry in the log
46# line; cope with this, thereby avoiding undefined problems
47# Very short log line gave substring error
48# 1998-02-03 A routed delivery to a local transport may not have <> in the
49# log line; terminate the address at white space, not <
50# 1998-09-07 If first line of input was a => line, $thissize was undefined;
51# ensure it is zero.
52# 1998-12-21 Adding of $thissize from => line should have been adding $size.
53# Oops. Should have looked more closely when fixing the previous
54# bug!
55# 1999-11-12 Increased the field widths for printed integers; numbers are
56# bigger than originally envisaged.
57# 2001-03-21 Converted seconds() routine to use Time::Local, fixing a bug
58# whereby seconds($timestamp) - id_seconds($id) gave an
59# incorrect result.
60# Added POD documentation.
61# Moved usage instructions into help() subroutine.
62# Added 'use strict' and declared all global variables.
63# Added '-html' flag and resultant code.
64# Added '-cache' flag and resultant code.
65# Added add_volume() routine and converted all volume variables
66# to use it, fixing the overflow problems for individual hosts
67# on large sites.
68# Converted all volume output to GB/MB/KB as appropriate.
69# Don't store local user stats if -nfl is specified.
70# Modifications done by: Steve Campbell (<steve@computurn.com>)
71# 2001-04-02 Added the -t_remote_users flag. Steve Campbell.
72# 2001-10-15 Added the -domain flag. Steve Campbell.
73# 2001-10-16 Accept files on STDIN or on the command line. Steve Campbell.
74# 2001-10-21 Removed -domain flag and added -bydomain, -byhost, and -byemail.
75# We now generate our main parsing subroutine as an eval statement
76# which improves performance dramatically when not all the results
77# are required. We also cache the last timestamp to time convertion.
78#
79# NOTE: 'Top 50 destinations by (message count|volume)' lines are
80# now 'Top N (host|email|domain) destinations by (message count|volume)'
81# where N is the topcount. Steve Campbell.
82#
83# 2001-10-30 V1.16 Joachim Wieland.
84# Fixed minor bugs in add_volume() when taking over this version
85# for use in Exim 4: -w gave uninitialized value warnings in
86# two situations: for the first addition to a counter, and if
87# there were never any gigabytes, thereby leaving the $gigs
88# value unset.
89# Initialized $last_timestamp to stop a -w uninitialized warning.
90# Minor layout tweak for grand totals (nitpicking).
91# Put the IP addresses for relaying stats in [] and separated by
92# a space from the domain name.
93# Removed the IPv4-specific address test when picking out addresses
94# for relaying. Anything inside [] is OK.
95#
96# 2002-07-02 Philip Hazel
97# Fixed "uninitialized variable" message that occurred for relay
98# messages that arrived from H=[1.2.3.4] hosts (no name shown).
99# This bug didn't affect the output.
100#
101# 2002-04-15 V1.17 Joachim Wieland.
102# Added -charts, -chartdir. -chartrel options which use
103# GD::Graph modules to create graphical charts of the statistics.
104#
105# 2002-04-15 V1.18 Steve Campbell.
106# Added a check for $domain to to stop a -w uninitialized warning.
107# Added -byemaildomain option.
108# Only print HTML header links to included tables!
109#
110# 2002-08-02 V1.19 Steve Campbell.
111# Changed the debug mode to dump the parser onto STDERR rather
112# than STDOUT. Documented the -d flag into the help().
113# Rejoined the divergent 2002-04-15 and 2002-07-02 releases.
114#
115# 2002-08-21 V1.20 Steve Campbell.
116# Added the '-merge' option to allow merging of previous reports.
117# Fixed a missing semicolon when doing -bydomain.
118# Make volume charts plot the data gigs and bytes rather than just bytes.
119# Only process log lines with $flag =~ /<=|=>|->|==|\*\*|Co/
120# Converted Emaildomain to Edomain - the column header was too wide!
121# This changes the text output slightly. You can revert to the old
122# column widths by changing $COLUMN_WIDTHS to 7;
123#
124# 2002-09-04 V1.21 Andreas J Mueller
125# Local deliveries domain now defaults to 'localdomain'.
126# Don't match F=<From> when looking for the user.
127#
128# 2002-09-05 V1.22 Steve Campbell
129# Fixed a perl 5.005 incompatibility problem ('our' variables).
130#
131# 2002-09-11 V1.23 Steve Campbell
132# Stopped -charts option from throwing errors on null data.
133# Don't print out 'Errors encountered' unless there are any.
134
135# 2002-10-21 V1.23a Philip Hazel - patch from Tony Finch put in until
136# Steve's eximstats catches up.
137# Handle log files that include the timezone after the timestamp.
138# Switch to assuming that log timestamps are in local time, with
139# an option for UTC timestamps, as in Exim itself.
140#
141# 2003-02-05 V1.24 Steve Campbell
142# Added in Sergey Sholokh's code to convert '<' and '>' characters
143# in HTML output. Also added code to convert them back with -merge.
144# Fixed timestamp offsets to convert to seconds rather than minutes.
145# Updated -merge to work with output files using timezones.
146# Added cacheing to speed up the calculation of timezone offsets.
147#
148# 2003-02-07 V1.25 Steve Campbell
149# Optimised the usage of mktime() in the seconds subroutine.
150# Removed the now redundant '-cache' option.
151# html2txt() now explicitly matches HTML tags.
152# Implemented a new sorting algorithm - the top_n_sort() routine.
153# Added Danny Carroll's '-nvr' flag and code.
154#
155# 2003-03-13 V1.26 Steve Campbell
156# Implemented HTML compliance changes recommended by Bernard Massot.
157# Bug fix to allow top_n_sort() to handle null keys.
158# Convert all domains and edomains to lowercase.
159# Remove preceding dots from domains.
160#
161# 2003-03-13 V1.27 Steve Campbell
162# Replaced border attributes with 'border=1', as recommended by
163# Bernard Massot.
164#
165# 2003-06-03 V1.28 John Newman
166# Added in the ability to skip over the parsing and evaulation of
167# specific transports as passed to eximstats via the new "-nt/.../"
168# command line argument. This new switch allows the viewing of
169# not more accurate statistics but more applicable statistics when
170# special transports are in use (ie; SpamAssassin). We need to be
171# able to ignore transports such as this otherwise the resulting
172# local deliveries are significantly skewed (doubled)...
173#
174# 2003-11-06 V1.29 Steve Campbell
175# Added the '-pattern "Description" "/pattern/"' option.
176#
177# 2004-02-17 V1.30 Steve Campbell
178# Added warnings if required GD::Graph modules are not available or
179# insufficient -chart* options are specified.
180#
1b4fe9dd 181# 2004-02-20 V1.31 Andrea Balzi
059ec3d9
PH		 182# Only show the Local Sender/Destination links if the tables exist.
183#
1b4fe9dd
PH		 184# 2004-07-05 V1.32 Steve Campbell
185# Fix '-merge -h0' divide by zero error.
186#
187# 2004-07-15 V1.33 Steve Campbell
188# Documentation update - I've converted the subroutine
189# documentation from POD to comments.
8974000d
SC		 190#
191# 2004-12-10 V1.34 Steve Campbell
192# Eximstats can now parse syslog lines as well as mainlog lines.
193#
194# 2004-12-20 V1.35 Andreas Metzler
195# Pie charts by volume were actually generated by count. Fixed.
059ec3d9
PH		 196
197
198=head1 NAME
199
8974000d 200eximstats - generates statistics from Exim mainlog or syslog files.
059ec3d9
PH		 201
202=head1 SYNOPSIS
203
204 eximstats [Options] mainlog1 mainlog2 ... > report.txt
1b4fe9dd 205 eximstats -html [Options] mainlog1 mainlog2 ... > report.html
059ec3d9
PH		 206 eximstats -merge [Options] report.1.txt report.2.txt ... > weekly_report.txt
207
208Options:
209
210=over 4
211
212=item B<-h>I<number>
213
214histogram divisions per hour. The default is 1, and
2150 suppresses histograms. Valid values are:
216
2170, 1, 2, 3, 5, 10, 15, 20, 30 or 60.
218
219=item B<-ne>
220
221Don't display error information.
222
223=item B<-nr>
224
225Don't display relaying information.
226
227=item B<-nr>I</pattern/>
228
229Don't display relaying information that matches.
230
231=item B<-nt>
232
233Don't display transport information.
234
235=item B<-nt>I</pattern/>
236
237Don't display transport information that matches
238
239=item B<-q>I<list>
240
241List of times for queuing information single 0 item suppresses.
242
243=item B<-t>I<number>
244
245Display top <number> sources/destinations
246default is 50, 0 suppresses top listing.
247
248=item B<-tnl>
249
250Omit local sources/destinations in top listing.
251
252=item B<-t_remote_users>
253
254Include remote users in the top source/destination listings.
255
256=item B<-byhost>
257
258Show results by sending host. This may be combined with
259B<-bydomain> and/or B<-byemail> and/or B<-byedomain>. If none of these options
260are specified, then B<-byhost> is assumed as a default.
261
262=item B<-bydomain>
263
264Show results by sending domain.
265May be combined with B<-byhost> and/or B<-byemail> and/or B<-byedomain>.
266
267=item B<-byemail>
268
269Show results by sender's email address.
270May be combined with B<-byhost> and/or B<-bydomain> and/or B<-byedomain>.
271
272=item B<-byemaildomain> or B<-byedomain>
273
274Show results by sender's email domain.
275May be combined with B<-byhost> and/or B<-bydomain> and/or B<-byemail>.
276
277=item B<-pattern> I<Description> I</Pattern/>
278
279Look for the specified pattern and count the number of lines in which it appears.
280This option can be specified multiple times. Eg:
281
282 -pattern 'Refused connections' '/refused connection/'
283
284
285=item B<-merge>
286
287This option allows eximstats to merge old eximstat reports together. Eg:
288
289 eximstats mainlog.sun > report.sun.txt
290 eximstats mainlog.mon > report.mon.txt
291 eximstats mainlog.tue > report.tue.txt
292 eximstats mainlog.wed > report.web.txt
293 eximstats mainlog.thu > report.thu.txt
294 eximstats mainlog.fri > report.fri.txt
295 eximstats mainlog.sat > report.sat.txt
296 eximstats -merge report.*.txt > weekly_report.txt
297 eximstats -merge -html report.*.txt > weekly_report.html
298
299=over 4
300
301=item *
302
303You can merge text or html reports and output the results as text or html.
304
305=item *
306
307You can use all the normal eximstat output options, but only data
308included in the original reports can be shown!
309
310=item *
311
312When merging reports, some loss of accuracy may occur in the top I<n> lists.
313This will be towards the ends of the lists.
314
315=item *
316
317The order of items in the top I<n> lists may vary when the data volumes
318round to the same value.
319
320=back
321
322=item B<-html>
323
324Output the results in HTML.
325
326=item B<-charts>
327
328Create graphical charts to be displayed in HTML output.
329
330This requires the following modules which can be obtained
331from http://www.cpan.org/modules/01modules.index.html
332
333=over 4
334
335=item GD
336
337=item GDTextUtil
338
339=item GDGraph
340
341=back
342
343To install these, download and unpack them, then use the normal perl installation procedure:
344
345 perl Makefile.PL
346 make
347 make test
348 make install
349
350=item B<-chartdir>I <dir>
351
352Create the charts in the directory <dir>
353
354=item B<-chartrel>I <dir>
355
356Specify the relative directory for the "img src=" tags from where to include
357the charts
358
359=item B<-d>
360
361Debug flag. This outputs the eval()'d parser onto STDOUT which makes it
362easier to trap errors in the eval section. Remember to add 1 to the line numbers to allow for the
363title!
364
365=back
366
367=head1 DESCRIPTION
368
8974000d 369Eximstats parses exim mainlog and syslog files to output a statistical
059ec3d9
PH		 370analysis of the messages processed. By default, a text
371analysis is generated, but you can request an html analysis
372by using the B<-html> flag. See the help (B<-help>) to learn
373about how to create charts from the tables.
374
375=head1 AUTHOR
376
377There is a web site at http://www.exim.org - this contains details of the
378mailing list exim-users@exim.org.
379
380=head1 TO DO
381
382This program does not perfectly handle messages whose received
383and delivered log lines are in different files, which can happen
384when you have multiple mail servers and a message cannot be
385immeadiately delivered. Fixing this could be tricky...
386
059ec3d9
PH		 387=cut
388
389use integer;
390use strict;
391
392# use Time::Local; # PH/FANF
393use POSIX;
394
395use vars qw($HAVE_GD_Graph_pie $HAVE_GD_Graph_linespoints);
396eval { require GD::Graph::pie; };
397$HAVE_GD_Graph_pie = $@ ? 0 : 1;
398eval { require GD::Graph::linespoints; };
399$HAVE_GD_Graph_linespoints = $@ ? 0 : 1;
400
401
402##################################################
403# Static data #
404##################################################
405# 'use vars' instead of 'our' as perl5.005 is still in use out there!
406use vars qw(@tab62 @days_per_month $gig);
407use vars qw($VERSION);
408use vars qw($COLUMN_WIDTHS);
409
410
411@tab62 =
412 (0,1,2,3,4,5,6,7,8,9,0,0,0,0,0,0, # 0-9
413 0,10,11,12,13,14,15,16,17,18,19,20, # A-K
414 21,22,23,24,25,26,27,28,29,30,31,32, # L-W
415 33,34,35, 0, 0, 0, 0, 0, # X-Z
416 0,36,37,38,39,40,41,42,43,44,45,46, # a-k
417 47,48,49,50,51,52,53,54,55,56,57,58, # l-w
418 59,60,61); # x-z
419
420@days_per_month = (0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334);
421$gig = 1024 * 1024 * 1024;
8974000d 422$VERSION = '1.35';
059ec3d9
PH		 423
424# How much space do we allow for the Hosts/Domains/Emails/Edomains column headers?
425$COLUMN_WIDTHS = 8;
426
427# Declare global variables.
428use vars qw($total_received_data $total_received_data_gigs $total_received_count);
429use vars qw($total_delivered_data $total_delivered_data_gigs $total_delivered_count);
430use vars qw(%arrival_time %size %from_host %from_address);
431use vars qw(%timestamp2time); #Hash of timestamp => time.
432use vars qw($last_timestamp $last_time); #The last time convertion done.
433use vars qw($last_date $date_seconds); #The last date convertion done.
434use vars qw($last_offset $offset_seconds); #The last time offset convertion done.
435use vars qw($localtime_offset);
436use vars qw($i); #General loop counter.
437use vars qw($debug); #Debug mode?
438use vars qw($ntopchart); #How many entries should make it into the chart?
439use vars qw($gddirectory); #Where to put files from GD::Graph
440
441$ntopchart = 5;
442
443# The following are parameters whose values are
444# set by command line switches:
445use vars qw($show_errors $show_relay $show_transport $transport_pattern);
446use vars qw($topcount $local_league_table $include_remote_users);
447use vars qw($hist_opt $hist_interval $hist_number $volume_rounding);
448use vars qw($relay_pattern @queue_times $html @user_patterns @user_descriptions);
449
450use vars qw(%do_sender); #Do sender by Host, Domain, Email, and/or Edomain tables.
451use vars qw($charts $chartrel $chartdir $charts_option_specified);
452use vars qw($merge_reports); #Merge old reports ?
453
454# The following are modified in the parse() routine, and
455# referred to in the print_*() routines.
456use vars qw($queue_more_than $delayed_count $relayed_unshown $begin $end);
457use vars qw(%received_count %received_data %received_data_gigs);
458use vars qw(%delivered_count %delivered_data %delivered_data_gigs);
459use vars qw(%received_count_user %received_data_user %received_data_gigs_user);
460use vars qw(%delivered_count_user %delivered_data_user %delivered_data_gigs_user);
461use vars qw(%transported_count %transported_data %transported_data_gigs);
462use vars qw(%remote_delivered %relayed %delayed %had_error %errors_count);
463use vars qw(@queue_bin @remote_queue_bin @received_interval_count @delivered_interval_count);
464use vars qw(@user_pattern_totals);
465
466use vars qw(%report_totals);
467
468
469
470
471##################################################
472# Subroutines #
473##################################################
474
475
1b4fe9dd
PH		 476#######################################################################
477# volume_rounded();
478#
479# $rounded_volume = volume_rounded($bytes,$gigabytes);
480#
481# Given a data size in bytes, round it to KB, MB, or GB
482# as appropriate.
483#
484# Eg 12000 => 12KB, 15000000 => 14GB, etc.
485#
486# Note: I've experimented with Math::BigInt and it results in a 33%
487# performance degredation as opposed to storing numbers split into
488# bytes and gigabytes.
489#######################################################################
059ec3d9
PH		 490sub volume_rounded {
491 my($x,$g) = @_;
492 $x = 0 unless $x;
493 $g = 0 unless $g;
494 my($rounded);
495
496 while ($x > $gig) {
497 $g++;
498 $x -= $gig;
499 }
500
501 if ($volume_rounding) {
502 # Values < 1 GB
503 if ($g <= 0) {
504 if ($x < 10000) {
505 $rounded = sprintf("%6d", $x);
506 }
507 elsif ($x < 10000000) {
508 $rounded = sprintf("%4dKB", ($x + 512)/1024);
509 }
510 else {
511 $rounded = sprintf("%4dMB", ($x + 512*1024)/(1024*1024));
512 }
513 }
514 # Values between 1GB and 10GB are printed in MB
515 elsif ($g < 10) {
516 $rounded = sprintf("%4dMB", ($g * 1024) + ($x + 512*1024)/(1024*1024));
517 }
518 else {
519 # Handle values over 10GB
520 $rounded = sprintf("%4dGB", $g + ($x + $gig/2)/$gig);
521 }
522 }
523 else {
524 # We don't want any rounding to be done.
525 $rounded = sprintf("%4d", ($g * $gig) + $x);
526 }
527
528 return $rounded;
529}
530
531
1b4fe9dd
PH		 532#######################################################################
533# un_round();
534#
535# un_round($rounded_volume,\$bytes,\$gigabytes);
536#
537# Given a volume in KB, MB or GB, as generated by volume_rounded(),
538# do the reverse transformation and convert it back into Bytes and Gigabytes.
539# These are added to the $bytes and $gigabytes parameters.
540#
541# Given a data size in bytes, round it to KB, MB, or GB
542# as appropriate.
543#
544# EG: 500 => (500,0), 14GB => (0,14), etc.
545#######################################################################
059ec3d9
PH		 546sub un_round {
547 my($rounded,$bytes_sref,$gigabytes_sref) = @_;
548
549 if ($rounded =~ /(\d+)GB/) {
550 $$gigabytes_sref += $1;
551 }
552 elsif ($rounded =~ /(\d+)MB/) {
553 $$gigabytes_sref += $1 / 1024;
554 $$bytes_sref += (($1 % 1024 ) * 1024 * 1024);
555 }
556 elsif ($rounded =~ /(\d+)KB/) {
557 $$gigabytes_sref += $1 / (1024 * 1024);
558 $$bytes_sref += ($1 % (1024 * 1024) * 1024);
559 }
560 elsif ($rounded =~ /(\d+)/) {
561 $$gigabytes_sref += $1 / $gig;
562 $$bytes_sref += $1 % $gig;
563 }
564
565 #Now reduce the bytes down to less than 1GB.
566 add_volume($bytes_sref,$gigabytes_sref,0) if ($$bytes_sref > $gig);
567}
568
569
1b4fe9dd
PH		 570#######################################################################
571# add_volume();
572#
573# add_volume(\$bytes,\$gigs,$size);
574#
575# Add $size to $bytes/$gigs where this is a number split into
576# bytes ($bytes) and gigabytes ($gigs). This is significantly
577# faster than using Math::BigInt.
578#######################################################################
059ec3d9 579sub add_volume {
1b4fe9dd
PH		 580 my($bytes_ref,$gigs_ref,$size) = @_;
581 $$bytes_ref = 0 if ! defined $$bytes_ref;
582 $$gigs_ref = 0 if ! defined $$gigs_ref;
583 $$bytes_ref += $size;
584 while ($$bytes_ref > $gig) {
585 $$gigs_ref++;
586 $$bytes_ref -= $gig;
059ec3d9
PH		 587 }
588}
589
590
1b4fe9dd
PH		 591#######################################################################
592# format_time();
593#
594# $formatted_time = format_time($seconds);
595#
596# Given a time in seconds, break it down into
597# weeks, days, hours, minutes, and seconds.
598#
599# Eg 12005 => 3h20m5s
600#######################################################################
059ec3d9
PH		 601sub format_time {
602my($t) = pop @_;
603my($s) = $t % 60;
604$t /= 60;
605my($m) = $t % 60;
606$t /= 60;
607my($h) = $t % 24;
608$t /= 24;
609my($d) = $t % 7;
610my($w) = $t/7;
611my($p) = "";
612$p .= "$w"."w" if $w > 0;
613$p .= "$d"."d" if $d > 0;
614$p .= "$h"."h" if $h > 0;
615$p .= "$m"."m" if $m > 0;
616$p .= "$s"."s" if $s > 0 || $p eq "";
617$p;
618}
619
620
1b4fe9dd
PH		 621#######################################################################
622# unformat_time();
623#
624# $seconds = unformat_time($formatted_time);
625#
626# Given a time in weeks, days, hours, minutes, or seconds, convert it to seconds.
627#
628# Eg 3h20m5s => 12005
629#######################################################################
059ec3d9
PH		 630sub unformat_time {
631 my($formated_time) = pop @_;
632 my $time = 0;
633
634 while ($formated_time =~ s/^(\d+)([wdhms]?)//) {
635 $time += $1 if ($2 eq '' || $2 eq 's');
636 $time += $1 * 60 if ($2 eq 'm');
637 $time += $1 * 60 * 60 if ($2 eq 'h');
638 $time += $1 * 60 * 60 * 24 if ($2 eq 'd');
639 $time += $1 * 60 * 60 * 24 * 7 if ($2 eq 'w');
640 }
641 $time;
642}
643
644
1b4fe9dd
PH		 645#######################################################################
646# seconds();
647#
648# $time = seconds($timestamp);
649#
650# Given a time-of-day timestamp, convert it into a time() value using
651# POSIX::mktime. We expect the timestamp to be of the form
652# "$year-$mon-$day $hour:$min:$sec", with month going from 1 to 12,
653# and the year to be absolute (we do the necessary conversions). The
654# timestamp may be followed with an offset from UTC like "+$hh$mm"; if the
655# offset is not present, and we have not been told that the log is in UTC
656# (with the -utc option), then we adjust the time by the current local
657# time offset so that it can be compared with the time recorded in message
658# IDs, which is UTC.
659#
660# To improve performance, we only use mktime on the date ($year-$mon-$day),
661# and only calculate it if the date is different to the previous time we
662# came here. We then add on seconds for the '$hour:$min:$sec'.
663#
664# We also store the results of the last conversion done, and only
665# recalculate if the date is different.
666#
667# We used to have the '-cache' flag which would store the results of the
668# mktime() call. However, the current way of just using mktime() on the
669# date obsoletes this.
670#######################################################################
059ec3d9
PH		 671sub seconds {
672 my($timestamp) = @_;
673
674 # Is the timestamp the same as the last one?
675 return $last_time if ($last_timestamp eq $timestamp);
676
677 return 0 unless ($timestamp =~ /^((\d{4})\-(\d\d)-(\d\d))\s(\d\d):(\d\d):(\d\d)( ([+-])(\d\d)(\d\d))?/o);
678
679 unless ($last_date eq $1) {
680 $last_date = $1;
681 my(@timestamp) = (0,0,0,$4,$3,$2);
682 $timestamp[5] -= 1900;
683 $timestamp[4]--;
684 $date_seconds = mktime(@timestamp);
685 }
686 my $time = $date_seconds + ($5 * 3600) + ($6 * 60) + $7;
687
688 # SC. Use cacheing. Also note we want seconds not minutes.
689 #my($this_offset) = ($10 * 60 + $11) * ($9 . "1") if defined $8;
690 if (defined $8 && ($8 ne $last_offset)) {
691 $last_offset = $8;
692 $offset_seconds = ($10 * 60 + $11) * 60;
693 $offset_seconds = -$offset_seconds if ($9 eq '-');
694 }
695
696
697 if (defined $7) {
698 #$time -= $this_offset;
699 $time -= $offset_seconds;
700 } elsif (defined $localtime_offset) {
701 $time -= $localtime_offset;
702 }
703
704 # Store the last timestamp received.
705 $last_timestamp = $timestamp;
706 $last_time = $time;
707
708 $time;
709}
710
711
1b4fe9dd
PH		 712#######################################################################
713# id_seconds();
714#
715# $time = id_seconds($message_id);
716#
717# Given a message ID, convert it into a time() value.
718#######################################################################
059ec3d9
PH		 719sub id_seconds {
720my($sub_id) = substr((pop @_), 0, 6);
721my($s) = 0;
722my(@c) = split(//, $sub_id);
723while($#c >= 0) { $s = $s * 62 + $tab62[ord(shift @c) - ord('0')] }
724$s;
725}
726
727
728
1b4fe9dd
PH		 729#######################################################################
730# calculate_localtime_offset();
731#
732# $localtime_offset = calculate_localtime_offset();
733#
734# Calculate the the localtime offset from gmtime in seconds.
735#
736# $localtime = time() + $localtime_offset.
737#
738# These are the same semantics as ISO 8601 and RFC 2822 timezone offsets.
739# (West is negative, East is positive.)
740#######################################################################
059ec3d9
PH		 741
742# $localtime = gmtime() + $localtime_offset. OLD COMMENT
743# This subroutine commented out as it's not currently in use.
744
745#sub calculate_localtime_offset {
746# # Pick an arbitrary date, convert it to localtime & gmtime, and return the difference.
747# my (@sample_date) = (0,0,0,5,5,100);
748# my $localtime = timelocal(@sample_date);
749# my $gmtime = timegm(@sample_date);
750# my $offset = $localtime - $gmtime;
751# return $offset;
752#}
753
754sub calculate_localtime_offset {
755 # Assume that the offset at the moment is valid across the whole
756 # period covered by the logs that we're analysing. This may not
757 # be true around the time the clocks change in spring or autumn.
758 my $utc = time;
759 # mktime works on local time and gmtime works in UTC
760 my $local = mktime(gmtime($utc));
761 return $local - $utc;
762}
763
764
1b4fe9dd
PH		 765#######################################################################
766# print_queue_times();
767#
768# $time = print_queue_times($message_type,\@queue_times,$queue_more_than);
769#
770# Given the type of messages being output, the array of message queue times,
771# and the number of messages which exceeded the queue times, print out
772# a table.
773#######################################################################
059ec3d9
PH		 774sub print_queue_times {
775no integer;
776my($string,$array,$queue_more_than) = @_;
777my(@chartdatanames);
778my(@chartdatavals);
779
780my $printed_one = 0;
781my $cumulative_percent = 0;
782#$queue_unknown += keys %arrival_time;
783
784my $queue_total = $queue_more_than;
785for ($i = 0; $i <= $#queue_times; $i++) { $queue_total += $$array[$i] }
786
787my $temp = "Time spent on the queue: $string";
788
789my($format);
790if ($html) {
791 print "<hr><a name=\"$string time\"></a><h2>$temp</h2>\n";
792 print "<table border=0 width=\"100%\">\n";
793 print "<tr><td>\n";
794 print "<table border=1>\n";
795 print "<tr><th>Time</th><th>Messages</th><th>Percentage</th><th>Cumulative Percentage</th>\n";
796 $format = "<tr><td align=\"right\">%s %s</td><td align=\"right\">%d</td><td align=\"right\">%5.1f%%</td><td align=\"right\">%5.1f%%</td>\n";
797}
798else
799{
800 printf("%s\n%s\n\n", $temp, "-" x length($temp));
801 $format = "%5s %4s %6d %5.1f%% %5.1f%%\n";
802}
803
804for ($i = 0; $i <= $#queue_times; $i++) {
805 if ($$array[$i] > 0)
806 {
807 my $percent = ($$array[$i] * 100)/$queue_total;
808 $cumulative_percent += $percent;
809 printf($format,
810 $printed_one? " " : "Under",
811 format_time($queue_times[$i]),
812 $$array[$i], $percent, $cumulative_percent);
813 if (!defined($queue_times[$i])) {
814 print "Not defined";
815 }
816 push(@chartdatanames,
817 ($printed_one? "" : "Under") . format_time($queue_times[$i]));
818 push(@chartdatavals, $$array[$i]);
819 $printed_one = 1;
820 }
821}
822
823if ($queue_more_than > 0) {
824 my $percent = ($queue_more_than * 100)/$queue_total;
825 $cumulative_percent += $percent;
826 printf($format,
827 "Over ",
828 format_time($queue_times[$#queue_times]),
829 $queue_more_than, $percent, $cumulative_percent);
830}
831push(@chartdatanames, "Over " . format_time($queue_times[$#queue_times]));
832push(@chartdatavals, $queue_more_than);
833
834#printf("Unknown %6d\n", $queue_unknown) if $queue_unknown > 0;
835if ($html) {
836 print "</table>\n";
837 print "</td><td>\n";
838
839 if ($HAVE_GD_Graph_pie && $charts) {
840 my @data = (
841 \@chartdatanames,
842 \@chartdatavals
843 );
844 my $graph = GD::Graph::pie->new(200, 200);
845 my $pngname;
846 my $title;
847 if ($string =~ /all/) { $pngname = "queue_all.png"; $title = "Queue (all)"; }
848 if ($string =~ /remote/) { $pngname = "queue_rem.png"; $title = "Queue (remote)"; }
849 $graph->set(
850 title => $title,
851 );
852 my $gd = $graph->plot(\@data) or warn($graph->error);
853 if ($gd) {
854 open(IMG, ">$chartdir/$pngname") or die $!;
855 binmode IMG;
856 print IMG $gd->png;
857 close IMG;
858 print "<img src=\"$chartrel/$pngname\">";
859 }
860 }
861 print "</td></tr></table>\n";
862}
863print "\n";
864}
865
866
867
1b4fe9dd
PH		 868#######################################################################
869# print_histogram();
870#
871# print_histogram('Deliverieds|Messages received',@interval_count);
872#
873# Print a histogram of the messages delivered/received per time slot
874# (hour by default).
875#######################################################################
059ec3d9
PH		 876sub print_histogram {
877my($text) = shift;
878my(@interval_count) = @_;
879my(@chartdatanames);
880my(@chartdatavals);
881my($maxd) = 0;
882for ($i = 0; $i < $hist_number; $i++)
883 { $maxd = $interval_count[$i] if $interval_count[$i] > $maxd; }
884
885my $scale = int(($maxd + 25)/50);
886$scale = 1 if $scale == 0;
887
888my($type);
889if ($text eq "Deliveries")
890 {
891 $type = ($scale == 1)? "delivery" : "deliveries";
892 }
893else
894 {
895 $type = ($scale == 1)? "message" : "messages";
896 }
897
898my($title) = sprintf("$text per %s (each dot is $scale $type)",
899 ($hist_interval == 60)? "hour" :
900 ($hist_interval == 1)? "minute" : "$hist_interval minutes");
901
902if ($html) {
903 print "<hr><a name=\"$text\"></a><h2>$title</h2>\n";
904 print "<table border=0 width=\"100%\">\n";
905 print "<tr><td><pre>\n";
906}
907else {
908 printf("%s\n%s\n\n", $title, "-" x length($title));
909}
910
911my $hour = 0;
912my $minutes = 0;
913for ($i = 0; $i < $hist_number; $i++)
914 {
915 my $c = $interval_count[$i];
916
917 # If the interval is an hour (the maximum) print the starting and
918 # ending hours as a label. Otherwise print the starting hour and
919 # minutes, which take up the same space.
920
921 my $temp;
922 if ($hist_opt == 1)
923 {
924 $temp = sprintf("%02d-%02d", $hour, $hour + 1);
925 print $temp;
926 push(@chartdatanames, $temp);
927 $hour++;
928 }
929 else
930 {
931 if ($minutes == 0)
932 { $temp = sprintf("%02d:%02d", $hour, $minutes) }
933 else
934 { $temp = sprintf(" :%02d", $minutes) }
935 print $temp;
936 push(@chartdatanames, $temp);
937 $minutes += $hist_interval;
938 if ($minutes >= 60)
939 {
940 $minutes = 0;
941 $hour++;
942 }
943 }
944 push(@chartdatavals, $c);
945 printf(" %6d %s\n", $c, "." x ($c/$scale));
946 }
947print "\n";
948if ($html)
949 {
950 print "</pre>\n";
951 print "</td><td>\n";
952 if ($HAVE_GD_Graph_linespoints && $charts) {
953 # calculate the graph
954 my @data = (
955 \@chartdatanames,
956 \@chartdatavals
957 );
958 my $graph = GD::Graph::linespoints->new(300, 300);
959 $graph->set(
960 x_label => 'Time',
961 y_label => 'Amount',
962 title => $text,
963 x_labels_vertical => 1
964 );
965 my($pngname);
966 if ($text =~ /Deliveries/) { $pngname = "histogram_del.png"; }
967 if ($text =~ /Messages/) { $pngname = "histogram_mes.png"; }
968 my $gd = $graph->plot(\@data) or warn($graph->error);
969 if ($gd) {
970 open(IMG, ">$chartdir/$pngname") or die $!;
971 binmode IMG;
972 print IMG $gd->png;
973 close IMG;
974 print "<img src=\"$chartrel/$pngname\">";
975 }
976 }
977 print "</td></tr></table>\n";
978}
979}
980
981
982
1b4fe9dd
PH		 983#######################################################################
984# print_league_table();
985#
986# print_league_table($league_table_type,\%message_count,\%message_data,\%message_data_gigs);
987#
988# Given hashes of message count and message data, which are keyed by
989# the table type (eg by the sending host), print a league table
990# showing the top $topcount (defaults to 50).
991#######################################################################
059ec3d9
PH		 992sub print_league_table {
993my($text,$m_count,$m_data,$m_data_gigs) = @_;
994my($name) = ($topcount == 1)? "$text" : "$topcount ${text}s";
995my($temp) = "Top $name by message count";
996my(@chartdatanames) = ();
997my(@chartdatavals) = ();
998my $chartotherval = 0;
999
1000my($format);
1001if ($html) {
1002 print "<hr><a name=\"$text count\"></a><h2>$temp</h2>\n";
1003 print "<table border=0 width=\"100%\">\n";
1004 print "<tr><td>\n";
1005 print "<table border=1>\n";
1006 print "<tr><th>Messages</th><th>Bytes</th><th>\u$text</th>\n";
1007
1008 # Align non-local addresses to the right (so all the .com's line up).
1009 # Local addresses are aligned on the left as they are userids.
1010 my $align = ($text !~ /local/i) ? 'right' : 'left';
1011 $format = "<tr><td align=\"right\">%d</td><td align=\"right\">%s</td><td align=\"$align\" nowrap>%s</td>\n";
1012}
1013else {
1014 printf("%s\n%s\n\n", $temp, "-" x length($temp));
1015 $format = "%7d %10s %s\n";
1016}
1017
1018my($key,$htmlkey);
1019foreach $key (top_n_sort($topcount,$m_count,$m_data_gigs,$m_data)) {
1020 if ($html) {
1021 $htmlkey = $key;
1022 $htmlkey =~ s/>/\&gt\;/g;
1023 $htmlkey =~ s/</\&lt\;/g;
1024 printf($format, $$m_count{$key}, volume_rounded($$m_data{$key},$$m_data_gigs{$key}), $htmlkey);
1025 }
1026 else {
1027 printf($format, $$m_count{$key}, volume_rounded($$m_data{$key},$$m_data_gigs{$key}), $key);
1028 }
1029 if (scalar @chartdatanames < $ntopchart)
1030 {
1031 push(@chartdatanames, $key);
1032 push(@chartdatavals, $$m_count{$key});
1033 }
1034 else
1035 {
1036 $chartotherval += $$m_count{$key};
1037 }
1038 }
1039push(@chartdatanames, "Other");
1040push(@chartdatavals, $chartotherval);
1041
1042if ($html)
1043 {
1044 print "</table>\n";
1045 print "</td><td>\n";
1046 if ($HAVE_GD_Graph_pie && $charts)
1047 {
1048 # calculate the graph
1049 my @data = (
1050 \@chartdatanames,
1051 \@chartdatavals
1052 );
1053 my $graph = GD::Graph::pie->new(300, 300);
1054 $graph->set(
1055 x_label => 'Name',
1056 y_label => 'Amount',
1057 title => 'By count',
1058 );
1059 my $gd = $graph->plot(\@data) or warn($graph->error);
1060 if ($gd) {
1061 my $temp = $text;
1062 $temp =~ s/ /_/g;
1063 open(IMG, ">$chartdir/${temp}_count.png") or die $!;
1064 binmode IMG;
1065 print IMG $gd->png;
1066 close IMG;
1067 print "<img src=\"$chartrel/${temp}_count.png\">";
1068 }
1069 }
1070 print "</td><td>\n";
1071 print "</td></tr></table>\n";
1072}
1073print "\n";
1074
1075$temp = "Top $name by volume";
1076if ($html) {
1077 print "<hr><a name=\"$text volume\"></a><h2>$temp</h2>\n";
1078 print "<table border=0 width=\"100%\">\n";
1079 print "<tr><td>\n";
1080 print "<table border=1>\n";
1081 print "<tr><th>Messages</th><th>Bytes</th><th>\u$text</th>\n";
1082}
1083else {
1084 printf("%s\n%s\n\n", $temp, "-" x length($temp));
1085}
1086
1087@chartdatanames = ();
1088@chartdatavals = ();
1089$chartotherval = 0;
8974000d 1090my $use_gig = 0;
059ec3d9 1091foreach $key (top_n_sort($topcount,$m_data_gigs,$m_data,$m_count)) {
8974000d
SC		 1092 if ($$m_data_gigs{$key}) {
1093 $use_gig = 1;
1094 }
059ec3d9
PH		 1095 if ($html) {
1096 $htmlkey = $key;
1097 $htmlkey =~ s/>/\&gt\;/g;
1098 $htmlkey =~ s/</\&lt\;/g;
1099 printf($format, $$m_count{$key}, volume_rounded($$m_data{$key},$$m_data_gigs{$key}), $htmlkey);
1100 }
1101 else {
1102 printf($format, $$m_count{$key}, volume_rounded($$m_data{$key},$$m_data_gigs{$key}), $key);
1103 }
1104
8974000d
SC		 1105 if (scalar @chartdatanames < $ntopchart) {
1106 if ($use_gig) {
1107 if ($$m_data_gigs{$key}) {
1108 push(@chartdatanames, $key);
1109 push(@chartdatavals, $$m_data_gigs{$key});
1110 }
059ec3d9 1111 }
8974000d
SC		 1112 else {
1113 push(@chartdatanames, $key);
1114 push(@chartdatavals, $$m_data{$key});
059ec3d9
PH		 1115 }
1116 }
8974000d
SC		 1117 else {
1118 $chartotherval += ($use_gig) ? $$m_data_gigs{$key} : $$m_data{$key};
1119 }
1120}
059ec3d9
PH		 1121push(@chartdatanames, "Other");
1122push(@chartdatavals, $chartotherval);
1123
1124if ($html) {
1125 print "</table>\n";
1126 print "</td><td>\n";
1127 if ($HAVE_GD_Graph_pie && $charts) {
1128 # calculate the graph
1129 my @data = (
1130 \@chartdatanames,
1131 \@chartdatavals
1132 );
1133 my $graph = GD::Graph::pie->new(300, 300);
1134 $graph->set(
1135 x_label => 'Name',
8974000d 1136 y_label => 'Volume' ,
059ec3d9
PH		 1137 title => 'By Volume',
1138 );
1139 my $gd = $graph->plot(\@data) or warn($graph->error);
1140 if ($gd) {
1141 my $temp = $text;
1142 $temp =~ s/ /_/g;
1143 open(IMG, ">$chartdir/${temp}_volume.png") or die $!;
1144 binmode IMG;
1145 print IMG $gd->png;
1146 close IMG;
1147 print "<img src=\"$chartrel/${temp}_volume.png\">";
1148 }
1149 }
1150 print "</td><td>\n";
1151 print "</td></tr></table>\n";
1152}
1153
1154print "\n";
1155}
1156
1157
1b4fe9dd
PH		 1158#######################################################################
1159# top_n_sort();
1160#
1161# @sorted_keys = top_n_sort($n,$href1,$href2,$href3);
1162#
1163# Given a hash which has numerical values, return the sorted $n keys which
1164# point to the top values. The second and third hashes are used as
1165# tiebreakers. They all must have the same keys.
1166#
1167# The idea behind this routine is that when you only want to see the
1168# top n members of a set, rather than sorting the entire set and then
1169# plucking off the top n, sort through the stack as you go, discarding
1170# any member which is lower than your current n'th highest member.
1171#
1172# This proves to be an order of magnitude faster for large hashes.
1173# On 200,000 lines of mainlog it benchmarked 9 times faster.
1174# On 700,000 lines of mainlog it benchmarked 13.8 times faster.
1175#
1176# We assume the values are > 0.
1177#######################################################################
059ec3d9
PH		 1178sub top_n_sort {
1179 my($n,$href1,$href2,$href3) = @_;
1180
1181 # PH's original sort was:
1182 #
1183 # foreach $key (sort
1184 # {
1185 # $$m_count{$b} <=> $$m_count{$a} ||
1186 # $$m_data_gigs{$b} <=> $$m_data_gigs{$a} ||
1187 # $$m_data{$b} <=> $$m_data{$a} ||
1188 # $a cmp $b
1189 # }
1190 # keys %{$m_count})
1191 #
1192
1193 #We use a key of '_' to represent non-existant values, as null keys are valid.
1194 #'_' is not a valid domain, edomain, host, or email.
1195 my(@top_n_keys) = ('_') x $n;
1196 my($minimum_value1,$minimum_value2,$minimum_value3) = (0,0,0);
1197 my $top_n_key = '';
1198 my $n_minus_1 = $n - 1;
1199 my $n_minus_2 = $n - 2;
1200
1b4fe9dd 1201 # Pick out the top $n keys.
059ec3d9
PH		 1202 my($key,$value1,$value2,$value3,$i,$comparison,$insert_position);
1203 while (($key,$value1) = each %$href1) {
1204
1205 #print STDERR "key $key ($value1,",$href2->{$key},",",$href3->{$key},") <=> ($minimum_value1,$minimum_value2,$minimum_value3)\n";
1b4fe9dd 1206
059ec3d9
PH		 1207 # Check to see that the new value is bigger than the lowest of the
1208 # top n keys that we're keeping.
1b4fe9dd 1209 $comparison = $value1 <=> $minimum_value1 ||
059ec3d9
PH		 1210 $href2->{$key} <=> $minimum_value2 ||
1211 $href3->{$key} <=> $minimum_value3 ||
1212 $top_n_key cmp $key;
1213 next unless ($comparison == 1);
1214
1215 # As we will be using these values a few times, extract them into scalars.
1216 $value2 = $href2->{$key};
1217 $value3 = $href3->{$key};
1218
1219 # This key is bigger than the bottom n key, so the lowest position we
1220 # will insert it into is $n minus 1 (the bottom of the list).
1221 $insert_position = $n_minus_1;
1222
1223 # Now go through the list, stopping when we find a key that we're
1224 # bigger than, or we come to the penultimate position - we've
1225 # already tested bigger than the last.
1226 #
1227 # Note: we go top down as the list starts off empty.
1228 # Note: stepping through the list in this way benchmarks nearly
1229 # three times faster than doing a sort() on the reduced list.
1230 # I assume this is because the list is already in order, and
1231 # we get a performance boost from not having to do hash lookups
1232 # on the new key.
1233 for ($i = 0; $i < $n_minus_1; $i++) {
1234 $top_n_key = $top_n_keys[$i];
1235 if ( ($top_n_key eq '_') ||
1b4fe9dd 1236 ( ($value1 <=> $href1->{$top_n_key} ||
059ec3d9
PH		 1237 $value2 <=> $href2->{$top_n_key} ||
1238 $value3 <=> $href3->{$top_n_key} ||
1239 $top_n_key cmp $key) == 1
1240 )
1241 ) {
1242 $insert_position = $i;
1243 last;
1244 }
1245 }
1246
1247 # Remove the last element, then insert the new one.
1248 $#top_n_keys = $n_minus_2;
1249 splice(@top_n_keys,$insert_position,0,$key);
1250
1251 # Extract our new minimum values.
1252 $top_n_key = $top_n_keys[$n_minus_1];
1253 if ($top_n_key ne '_') {
1254 $minimum_value1 = $href1->{$top_n_key};
1255 $minimum_value2 = $href2->{$top_n_key};
1256 $minimum_value3 = $href3->{$top_n_key};
1257 }
1258 }
1259
1260 # Return the top n list, grepping out non-existant values, just in case
1261 # we didn't have that many values.
1262 return(grep(!/^_$/,@top_n_keys));
1263}
1264
1265
1b4fe9dd
PH		 1266#######################################################################
1267# html_header();
1268#
1269# $header = html_header($title);
1270#
1271# Print our HTML header and start the <body> block.
1272#######################################################################
059ec3d9
PH		 1273sub html_header {
1274 my($title) = @_;
1275 my $text = << "EoText";
1276<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
1277<html>
1278<head>
1279<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-15">
1280<title>$title</title>
1281</head>
1282<body bgcolor="white">
1283<h1>$title</h1>
1284EoText
1285 return $text;
1286}
1287
1288
1289
1b4fe9dd
PH		 1290#######################################################################
1291# help();
1292#
1293# help();
1294#
1295# Display usage instructions and exit.
1296#######################################################################
059ec3d9
PH		 1297sub help {
1298 print << "EoText";
1299
1300eximstats Version $VERSION
1301
1302Usage: eximstats [Options] mainlog1 mainlog2 ... > report.txt
1303 eximstats -html [Options] mainlog1 mainlog2 ... > report.html
1304 eximstats -merge [Options] report.1.txt report.2.txt ... > weekly_rep.txt
1305 eximstats -merge -html [Options] report.1.html ... > weekly_rep.html
1306
8974000d
SC		 1307Parses exim mainlog or syslog files and generates a statistical analysis
1308of the messages processed. Valid options are:
059ec3d9
PH		 1309
1310-h<number> histogram divisions per hour. The default is 1, and
1311 0 suppresses histograms. Other valid values are:
1312 2, 3, 5, 10, 15, 20, 30 or 60.
1313-ne don't display error information
1314-nr don't display relaying information
1315-nr/pattern/ don't display relaying information that matches
1316-nt don't display transport information
1317-nt/pattern/ don't display transport information that matches
1318-nvr don't do volume rounding. Display in bytes, not KB/MB/GB.
1319-q<list> list of times for queuing information
1320 single 0 item suppresses
1321-t<number> display top <number> sources/destinations
1322 default is 50, 0 suppresses top listing
1323-tnl omit local sources/destinations in top listing
1324-t_remote_users show top user sources/destinations from non-local domains
1325
1326-byhost show results by sending host (default unless bydomain or
1327 byemail is specified)
1328-bydomain show results by sending domain.
1329-byemail show results by sender's email address
1330-byedomain show results by sender's email domain
1331
1332-pattern "Description" /pattern/
1333 Count lines matching specified patterns and show them in
1334 the results. It can be specified multiple times. Eg:
1335 -pattern 'Refused connections' '/refused connection/'
1336
1337-merge merge previously generated reports into a new report
1338
1339-html output the results in HTML
1340-charts Create charts (this requires the GD::Graph modules)
1341-chartdir <dir> Create the charts' png files in the directory <dir>
1342-chartrel <dir> Specify the relative directory for the "img src=" tags
1343 from where to include the charts in the html file
1344 -chartdir and -chartrel default to '.'
1345
1346-d Debug mode - dump the eval'ed parser onto STDERR.
1347
1348EoText
1349
1350 exit 1;
1351}
1352
1353
1354
1b4fe9dd
PH		 1355#######################################################################
1356# generate_parser();
1357#
1358# $parser = generate_parser();
1359#
1360# This subroutine generates the parsing routine which will be
1361# used to parse the mainlog. We take the base operation, and remove bits not in use.
1362# This improves performance depending on what bits you take out or add.
1363#
1364# I've tested using study(), but this does not improve performance.
1365#
1366# We store our parsing routing in a variable, and process it looking for #IFDEF (Expression)
1367# or #IFNDEF (Expression) statements and corresponding #ENDIF (Expression) statements. If
1368# the expression evaluates to true, then it is included/excluded accordingly.
1369#######################################################################
059ec3d9
PH		 1370sub generate_parser {
1371 my $parser = '
1372 my($ip,$host,$email,$edomain,$domain,$thissize,$size,$old,$new);
1373 my($tod,$m_hour,$m_min,$id,$flag);
1374 while (<$fh>) {
059ec3d9 1375
8974000d
SC		 1376 # Convert syslog lines to mainlog format.
1377 if (! /^\\d{4}/) {
1378 next unless s/^.*? exim\\b.*?: //;
1379 }
1380
1381 next if length($_) < 38;
059ec3d9
PH		 1382 next unless /^(\\d{4}\\-\\d\\d-\\d\\d\\s(\\d\\d):(\\d\\d):\\d\\d( [-+]\\d\\d\\d\\d)?)/o;
1383
1384 ($tod,$m_hour,$m_min) = ($1,$2,$3);
1385
1386 # PH
1387 my($extra) = defined($4)? 6 : 0;
1388 $id = substr($_, 20 + $extra, 16);
1389 $flag = substr($_, 37 + $extra, 2);
1390';
1391
1392 # Watch for user specified patterns.
1393 my $user_pattern_index = 0;
1394 foreach (@user_patterns) {
1395 $user_pattern_totals[$user_pattern_index] = 0;
1396 $parser .= " \$user_pattern_totals[$user_pattern_index]++ if $_;\n";
1397 $user_pattern_index++;
1398 }
1399
1400 $parser .= '
1401 next unless ($flag =~ /<=|=>|->|==|\\*\\*|Co/);
1402
1403 #Strip away the timestamp, ID and flag (which could be "Com" for completed)
1404 #This speeds up the later pattern matches.
1405 # $_ = substr($_, 40);
1406
1407 $_ = substr($_, 40 + $extra); # PH
1408
1409 # JN - Skip over certain transports as specified via the "-nt/.../" command
1410 # line switch (where ... is a perl style regular expression). This is
1411 # required so that transports that skew stats such as SpamAssassin can be
1412 # ignored.
1413 #IFDEF ($transport_pattern)
1414 if (/\\sT=(\\S+)/) {
1415 next if ($1 =~ /$transport_pattern/o) ;
1416 }
1417 #ENDIF ($transport_pattern)
1418
1419
1420 $host = "local"; #Host is local unless otherwise specified.
1421 $domain = "localdomain"; #Domain is localdomain unless otherwise specified.
1422
1423
1424 # Do some pattern matches to get the host and IP address.
1425 # We expect lines to be of the form "H=[IpAddr]" or "H=Host [IpAddr]" or
1426 # "H=Host (UnverifiedHost) [IpAddr]" or "H=(UnverifiedHost) [IpAddr]".
1427 # We do 2 separate matches to keep the matches simple and fast.
1428 if (/\\sH=(\\S+)/) {
1429 $host = $1;
1430
1431 ($ip) = /\\sH=.*?(\\s\\[[^]]+\\])/;
1432 # If there is only an IP address, it will be in $host and $ip will be
1433 # unset. That is OK, because we only use $ip in conjunction with $host
1434 # below. But make it empty to avoid warning messages.
1435 $ip = "" if !defined $ip;
1436
1437 #IFDEF ($do_sender{Domain})
1438 if ($host !~ /^\\[/ && $host =~ /^(\\(?)[^\\.]+\\.([^\\.]+\\..*)/) {
1439 # Remove the host portion from the DNS name. We ensure that we end up with
1440 # at least xxx.yyy. $host can be "(x.y.z)" or "x.y.z".
1441 $domain = lc("$1.$2");
1442 $domain =~ s/^\\.//; #Remove preceding dot.
1443 }
1444 #ENDIF ($do_sender{Domain})
1445
1446 }
1447
1448 #IFDEF ($do_sender{Email})
1449 $email = (/^(\S+)/) ? $1 : "";
1450 #ENDIF ($do_sender{Email})
1451
1452 #IFDEF ($do_sender{Edomain})
1453 $edomain = (/^\S*?\\@(\S+)/) ? lc($1) : "";
1454 #ENDIF ($do_sender{Edomain})
1455
1456 if ($tod lt $begin) {
1457 $begin = $tod;
1458 }
1459 elsif ($tod gt $end) {
1460 $end = $tod;
1461 }
1462
1463
1464 if ($flag eq "<=") {
1465 $thissize = (/\\sS=(\\d+)( |$)/) ? $1 : 0;
1466 $size{$id} = $thissize;
1467
1468 #IFDEF ($show_relay)
1469 if ($host ne "local") {
1470 # Save incoming information in case it becomes interesting
1471 # later, when delivery lines are read.
1472 my($from) = /^(\\S+)/;
1473 $from_host{$id} = "$host$ip";
1474 $from_address{$id} = $from;
1475 }
1476 #ENDIF ($show_relay)
1477
1478 #IFDEF ($local_league_table || $include_remote_users)
1479 if (/\sU=(\\S+)/) {
1480 my $user = $1;
1481
1482 #IFDEF ($local_league_table && $include_remote_users)
1483 { #Store both local and remote users.
1484 #ENDIF ($local_league_table && $include_remote_users)
1485
1486 #IFDEF ($local_league_table && ! $include_remote_users)
1487 if ($host eq "local") { #Store local users only.
1488 #ENDIF ($local_league_table && ! $include_remote_users)
1489
1490 #IFDEF ($include_remote_users && ! $local_league_table)
1491 if ($host ne "local") { #Store remote users only.
1492 #ENDIF ($include_remote_users && ! $local_league_table)
1493
1494 $received_count_user{$user}++;
1495 add_volume(\\$received_data_user{$user},\\$received_data_gigs_user{$user},$thissize);
1496 }
1497 }
1498 #ENDIF ($local_league_table || $include_remote_users)
1499
1500 #IFDEF ($do_sender{Host})
1501 $received_count{Host}{$host}++;
1502 add_volume(\\$received_data{Host}{$host},\\$received_data_gigs{Host}{$host},$thissize);
1503 #ENDIF ($do_sender{Host})
1504
1505 #IFDEF ($do_sender{Domain})
1506 if ($domain) {
1507 $received_count{Domain}{$domain}++;
1508 add_volume(\\$received_data{Domain}{$domain},\\$received_data_gigs{Domain}{$domain},$thissize);
1509 }
1510 #ENDIF ($do_sender{Domain})
1511
1512 #IFDEF ($do_sender{Email})
1513 $received_count{Email}{$email}++;
1514 add_volume(\\$received_data{Email}{$email},\\$received_data_gigs{Email}{$email},$thissize);
1515 #ENDIF ($do_sender{Email})
1516
1517 #IFDEF ($do_sender{Edomain})
1518 $received_count{Edomain}{$edomain}++;
1519 add_volume(\\$received_data{Edomain}{$edomain},\\$received_data_gigs{Edomain}{$edomain},$thissize);
1520 #ENDIF ($do_sender{Edomain})
1521
1522 $total_received_count++;
1523 add_volume(\\$total_received_data,\\$total_received_data_gigs,$thissize);
1524
1525 #IFDEF ($#queue_times >= 0)
1526 $arrival_time{$id} = $tod;
1527 #ENDIF ($#queue_times >= 0)
1528
1529 #IFDEF ($hist_opt > 0)
1530 $received_interval_count[($m_hour*60 + $m_min)/$hist_interval]++;
1531 #ENDIF ($hist_opt > 0)
1532 }
1533
1534 elsif ($flag eq "=>") {
1535 $size = $size{$id} || 0;
1536 if ($host ne "local") {
1537 $remote_delivered{$id} = 1;
1538
1539
1540 #IFDEF ($show_relay)
1541 # Determine relaying address if either only one address listed,
1542 # or two the same. If they are different, it implies a forwarding
1543 # or aliasing, which is not relaying. Note that for multi-aliased
1544 # addresses, there may be a further address between the first
1545 # and last.
1546
1547 if (defined $from_host{$id}) {
1548 if (/^(\\S+)(?:\\s+\\([^)]\\))?\\s+<([^>]+)>/) {
1549 ($old,$new) = ($1,$2);
1550 }
1551 else {
1552 $old = $new = "";
1553 }
1554
1555 if ("\\L$new" eq "\\L$old") {
1556 ($old) = /^(\\S+)/ if $old eq "";
1557 my $key = "H=\\L$from_host{$id}\\E A=\\L$from_address{$id}\\E => " .
1558 "H=\\L$host\\E$ip A=\\L$old\\E";
1559 if (!defined $relay_pattern || $key !~ /$relay_pattern/o) {
1560 $relayed{$key} = 0 if !defined $relayed{$key};
1561 $relayed{$key}++;
1562 }
1563 else {
1564 $relayed_unshown++
1565 }
1566 }
1567 }
1568 #ENDIF ($show_relay)
1569
1570 }
1571
1572 #IFDEF ($local_league_table || $include_remote_users)
1573 #IFDEF ($local_league_table && $include_remote_users)
1574 { #Store both local and remote users.
1575 #ENDIF ($local_league_table && $include_remote_users)
1576
1577 #IFDEF ($local_league_table && ! $include_remote_users)
1578 if ($host eq "local") { #Store local users only.
1579 #ENDIF ($local_league_table && ! $include_remote_users)
1580
1581 #IFDEF ($include_remote_users && ! $local_league_table)
1582 if ($host ne "local") { #Store remote users only.
1583 #ENDIF ($include_remote_users && ! $local_league_table)
1584
1585 if (my($user) = split((/\\s</)? " <" : " ", $_)) {
1586 if ($user =~ /^[\\/|]/) {
1587 my($parent) = $_ =~ /(<[^@]+@?[^>]*>)/;
1588 $user = "$user $parent" if defined $parent;
1589 }
1590 $delivered_count_user{$user}++;
1591 add_volume(\\$delivered_data_user{$user},\\$delivered_data_gigs_user{$user},$size);
1592 }
1593 }
1594 #ENDIF ($local_league_table || $include_remote_users)
1595
1596 #IFDEF ($do_sender{Host})
1597 $delivered_count{Host}{$host}++;
1598 add_volume(\\$delivered_data{Host}{$host},\\$delivered_data_gigs{Host}{$host},$size);
1599 #ENDIF ($do_sender{Host})
1600 #IFDEF ($do_sender{Domain})
1601 if ($domain) {
1602 $delivered_count{Domain}{$domain}++;
1603 add_volume(\\$delivered_data{Domain}{$domain},\\$delivered_data_gigs{Domain}{$domain},$size);
1604 }
1605 #ENDIF ($do_sender{Domain})
1606 #IFDEF ($do_sender{Email})
1607 $delivered_count{Email}{$email}++;
1608 add_volume(\\$delivered_data{Email}{$email},\\$delivered_data_gigs{Email}{$email},$size);
1609 #ENDIF ($do_sender{Email})
1610 #IFDEF ($do_sender{Edomain})
1611 $delivered_count{Edomain}{$edomain}++;
1612 add_volume(\\$delivered_data{Edomain}{$edomain},\\$delivered_data_gigs{Edomain}{$edomain},$size);
1613 #ENDIF ($do_sender{Edomain})
1614
1615 $total_delivered_count++;
1616 add_volume(\\$total_delivered_data,\\$total_delivered_data_gigs,$size);
1617
1618 #IFDEF ($show_transport)
1619 my $transport = (/\\sT=(\\S+)/) ? $1 : ":blackhole:";
1620 $transported_count{$transport}++;
1621 add_volume(\\$transported_data{$transport},\\$transported_data_gigs{$transport},$size);
1622 #ENDIF ($show_transport)
1623
1624 #IFDEF ($hist_opt > 0)
1625 $delivered_interval_count[($m_hour*60 + $m_min)/$hist_interval]++;
1626 #ENDIF ($hist_opt > 0)
1627
1628 }
1629
1630 elsif ($flag eq "==" && defined($size{$id}) && !defined($delayed{$id})) {
1631 $delayed_count++;
1632 $delayed{$id} = 1;
1633 }
1634
1635 elsif ($flag eq "**") {
1636 $had_error{$id} = 1 if defined ($size{$id});
1637
1638 #IFDEF ($show_errors)
1639 $errors_count{$_}++;
1640 #ENDIF ($show_errors)
1641
1642 }
1643
1644 elsif ($flag eq "Co") {
1645 #Completed?
1646 #IFDEF ($#queue_times >= 0)
1647 #Note: id_seconds() benchmarks as 42% slower than seconds() and computing
1648 #the time accounts for a significant portion of the run time.
1649 my($queued);
1650 if (defined $arrival_time{$id}) {
1651 $queued = seconds($tod) - seconds($arrival_time{$id});
1652 delete($arrival_time{$id});
1653 }
1654 else {
1655 $queued = seconds($tod) - id_seconds($id);
1656 }
1657
1658 for ($i = 0; $i <= $#queue_times; $i++) {
1659 if ($queued < $queue_times[$i]) {
1660 $queue_bin[$i]++;
1661 $remote_queue_bin[$i]++ if $remote_delivered{$id};
1662 last;
1663 }
1664 }
1665 $queue_more_than++ if $i > $#queue_times;
1666 #ENDIF ($#queue_times >= 0)
1667
1668 #IFDEF ($show_relay)
1669 delete($from_host{$id});
1670 delete($from_address{$id});
1671 #ENDIF ($show_relay)
1672
1673 }
1674 }';
1675
1676 # We now do a 'C preprocessor style operation on our parser
1677 # to remove bits not in use.
1678 my(%defines_in_operation,$removing_lines,$processed_parser);
1679 foreach (split (/\n/,$parser)) {
1680 if ((/^\s*#\s*IFDEF\s*\((.*?)\)/i && ! eval $1) ||
1681 (/^\s*#\s*IFNDEF\s*\((.*?)\)/i && eval $1) ) {
1682 $defines_in_operation{$1} = 1;
1683 $removing_lines = 1;
1684 }
1685
1686 $processed_parser .= $_."\n" unless $removing_lines;
1687
1688 if (/^\s*#\s*ENDIF\s*\((.*?)\)/i) {
1689 delete $defines_in_operation{$1};
1690 unless (keys %defines_in_operation) {
1691 $removing_lines = 0;
1692 }
1693 }
1694 }
1695 print STDERR "# START OF PARSER:\n$processed_parser\n# END OF PARSER\n\n" if $debug;
1696
1697 return $processed_parser;
1698}
1699
1700
1701
1b4fe9dd
PH		 1702#######################################################################
1703# parse();
1704#
1705# parse($parser,\*FILEHANDLE);
1706#
1707# This subroutine accepts a parser and a filehandle from main and parses each
1708# line. We store the results into global variables.
1709#######################################################################
059ec3d9
PH		 1710sub parse {
1711 my($parser,$fh) = @_;
1712
1713 if ($merge_reports) {
1714 parse_old_eximstat_reports($fh);
1715 }
1716 else {
1717 eval $parser;
1718 die ($@) if $@;
1719 }
1720
1721}
1722
1723
1724
1b4fe9dd
PH		 1725#######################################################################
1726# print_header();
1727#
1728# print_header();
1729#
1730# Print our headers and contents.
1731#######################################################################
059ec3d9
PH		 1732sub print_header {
1733
1734 my $title = "Exim statistics from $begin to $end";
1735
1736 if ($html) {
1737 print html_header($title);
1738 print "<ul>\n";
1739 print "<li><a href=\"#grandtotal\">Grand total summary</a>\n";
1740 print "<li><a href=\"#patterns\">User Specified Patterns</a>\n" if @user_patterns;
1741 print "<li><a href=\"#transport\">Deliveries by Transport</a>\n" if $show_transport;
1742 if ($hist_opt) {
1743 print "<li><a href=\"#Messages received\">Messages received per hour</a>\n";
1744 print "<li><a href=\"#Deliveries\">Deliveries per hour</a>\n";
1745 }
1746 if ($#queue_times >= 0) {
1747 print "<li><a href=\"#all messages time\">Time spent on the queue: all messages</a>\n";
1748 print "<li><a href=\"#messages with at least one remote delivery time\">Time spent on the queue: messages with at least one remote delivery</a>\n";
1749 }
1750 print "<li><a href=\"#Relayed messages\">Relayed messages</a>\n" if $show_relay;
1751 if ($topcount) {
1752 foreach ('Host','Domain','Email','Edomain') {
1753 next unless $do_sender{$_};
1754 print "<li><a href=\"#sending \l$_ count\">Top $topcount sending \l${_}s by message count</a>\n";
1755 print "<li><a href=\"#sending \l$_ volume\">Top $topcount sending \l${_}s by volume</a>\n";
1756 }
1757 if ($local_league_table || $include_remote_users) {
1758 print "<li><a href=\"#local sender count\">Top $topcount local senders by message count</a>\n";
1759 print "<li><a href=\"#local sender volume\">Top $topcount local senders by volume</a>\n";
1760 }
1761 foreach ('Host','Domain','Email','Edomain') {
1762 next unless $do_sender{$_};
1763 print "<li><a href=\"#\l$_ destination count\">Top $topcount \l$_ destinations by message count</a>\n";
1764 print "<li><a href=\"#\l$_ destination volume\">Top $topcount \l$_ destinations by volume</a>\n";
1765 }
1766 if ($local_league_table || $include_remote_users) {
1767 print "<li><a href=\"#local destination count\">Top $topcount local destinations by message count</a>\n";
1768 print "<li><a href=\"#local destination volume\">Top $topcount local destinations by volume</a>\n";
1769 }
1770 }
1771 print "<li><a href=\"#errors\">List of errors</a>\n" if %errors_count;
1772 print "</ul>\n<hr>\n";
1773
1774 }
1775 else {
1776 print "\n$title\n";
1777 }
1778}
1779
1780
1b4fe9dd
PH		 1781#######################################################################
1782# print_grandtotals();
1783#
1784# print_grandtotals();
1785#
1786# Print the grand totals.
1787#######################################################################
059ec3d9
PH		 1788sub print_grandtotals {
1789
1790 # Get the sender by headings and results. This is complicated as we can have
1791 # different numbers of columns.
1792 my($sender_txt_header,$sender_html_header,$sender_txt_format,$sender_html_format);
1793 my(@received_totals,@delivered_totals);
1794 foreach ('Host','Domain','Email','Edomain') {
1795 next unless $do_sender{$_};
1796 if ($merge_reports) {
1797 push(@received_totals, get_report_total($report_totals{Received},"${_}s"));
1798 push(@delivered_totals,get_report_total($report_totals{Delivered},"${_}s"));
1799 }
1800 else {
1801 push(@received_totals,scalar(keys %{$received_data{$_}}));
1802 push(@delivered_totals,scalar(keys %{$delivered_data{$_}}));
1803 }
1804 $sender_html_header .= "<th>${_}s</th>";
1805 $sender_txt_header .= " " x ($COLUMN_WIDTHS - length($_)) . $_ . 's';
1806 $sender_html_format .= "<td align=\"right\">%d</td>";
1807 $sender_txt_format .= " " x ($COLUMN_WIDTHS - 5) . "%6d";
1808 }
1809
1810 my($format1,$format2);
1811 if ($html) {
1812 print << "EoText";
1813<a name="grandtotal"></a>
1814<h2>Grand total summary</h2>
1815<table border=1>
1816<tr><th>TOTAL</th><th>Volume</th><th>Messages</th>$sender_html_header<th colspan=2>At least one addr<br>Delayed</th><th colspan=2>At least one addr<br>Failed</th>
1817EoText
1818
1819 $format1 = "<tr><td>%s</td><td align=\"right\">%s</td>$sender_html_format<td align=\"right\">%d</td>";
1820 $format2 = "<td align=\"right\">%d</td><td align=\"right\">%4.1f%%</td><td align=\"right\">%d</td><td align=\"right\">%4.1f%%</td>";
1821 }
1822 else {
1823 my $sender_spaces = " " x length($sender_txt_header);
1824 print << "EoText";
1825
1826Grand total summary
1827-------------------
1828 $sender_spaces At least one address
1829 TOTAL Volume Messages $sender_txt_header Delayed Failed
1830EoText
1831 $format1 = " %-16s %9s %6d $sender_txt_format";
1832 $format2 = " %6d %4.1f%% %6d %4.1f%%",
1833 }
1834
1835 my($volume,$failed_count);
1836 if ($merge_reports) {
1837 $volume = volume_rounded($report_totals{Received}{Volume}, $report_totals{Received}{'Volume-gigs'});
1838 $total_received_count = get_report_total($report_totals{Received},'Messages');
1839 $failed_count = get_report_total($report_totals{Received},'Failed');
1840 $delayed_count = get_report_total($report_totals{Received},'Delayed');
1841 }
1842 else {
1843 $volume = volume_rounded($total_received_data, $total_received_data_gigs);
1844 $failed_count = keys %had_error;
1845 }
1846
1847 {
1848 no integer;
1849 printf("$format1$format2\n",'Received',$volume,$total_received_count,
1850 @received_totals,$delayed_count,
1851 ($total_received_count) ? ($delayed_count*100/$total_received_count) : 0,
1852 $failed_count,
1853 ($total_received_count) ? ($failed_count*100/$total_received_count) : 0);
1854 }
1855
1856 if ($merge_reports) {
1857 $volume = volume_rounded($report_totals{Delivered}{Volume}, $report_totals{Delivered}{'Volume-gigs'});
1858 $total_delivered_count = get_report_total($report_totals{Delivered},'Messages');
1859 }
1860 else {
1861 $volume = volume_rounded($total_delivered_data, $total_delivered_data_gigs);
1862 }
1863 printf("$format1\n\n",'Delivered',$volume,$total_delivered_count,@delivered_totals);
1864 print "</table>\n" if $html;
1865}
1866
1867
1b4fe9dd
PH		 1868#######################################################################
1869# print_user_patterns()
1870#
1871# print_user_patterns();
1872#
1873# Print the counts of user specified patterns.
1874#######################################################################
059ec3d9
PH		 1875sub print_user_patterns {
1876 my($format1);
1877
1878 if ($html) {
1879 print "<hr><a name=\"patterns\"></a><h2>User Specified Patterns</h2>\n";
1880 print "<table border=0 width=\"100%\">\n";
1881 print "<tr><td>\n";
1882 print "<table border=1>\n";
1883 print "<tr><th>&nbsp;</th><th>Total</th>\n";
1884 $format1 = "<tr><td>%s</td><td align=\"right\">%d</td>";
1885 }
1886 else {
1887 print "User Specified Patterns\n";
1888 print "-----------------------";
1889 print "\n Total\n";
1890 $format1 = " %-18s %6d";
1891 }
1892
1893 my($key);
1894 if ($merge_reports) {
1895 # We are getting our data from previous reports.
1896 foreach $key (@user_descriptions) {
1897 my $count = get_report_total($report_totals{patterns}{$key},'Total');
1898 printf("$format1\n",$key,$count);
1899 }
1900 }
1901 else {
1902 # We are getting our data from mainlog files.
1903 my $user_pattern_index = 0;
1904 foreach $key (@user_descriptions) {
1905 printf("$format1\n",$key,$user_pattern_totals[$user_pattern_index]);
1906 $user_pattern_index++;
1907 }
1908 }
1909 if ($html) {
1910 print "</table>\n";
1911 }
1912 print "\n";
1913}
1914
1915
1b4fe9dd
PH		 1916#######################################################################
1917# print_transport();
1918#
1919# print_transport();
1920#
1921# Print totals by transport.
1922#######################################################################
059ec3d9
PH		 1923sub print_transport {
1924 my($format1);
1925 my(@chartdatanames);
1926 my(@chartdatavals_count);
1927 my(@chartdatavals_vol);
1928 no integer; #Lose this for charting the data.
1929
1930 if ($html) {
1931 print "<hr><a name=\"transport\"></a><h2>Deliveries by Transport</h2>\n";
1932 print "<table border=0 width=\"100%\">\n";
1933 print "<tr><td>\n";
1934 print "<table border=1>\n";
1935 print "<tr><th>&nbsp;</th><th>Volume</th><th>Messages</th>\n";
1936 $format1 = "<tr><td>%s</td><td align=\"right\">%s</td><td align=\"right\">%d</td>";
1937 }
1938 else {
1939 print "Deliveries by transport\n";
1940 print "-----------------------";
1941 print "\n Volume Messages\n";
1942 $format1 = " %-18s %6s %6d";
1943 }
1944
1945 my($key);
1946 if ($merge_reports) {
1947 # We are getting our data from previous reports.
1948 foreach $key (sort keys %{$report_totals{transport}}) {
1949 my $count = get_report_total($report_totals{transport}{$key},'Messages');
1950 printf("$format1\n",$key,
1951 volume_rounded($report_totals{transport}{$key}{Volume},$report_totals{transport}{$key}{'Volume-gigs'}),
1952 $count);
1953 push(@chartdatanames, $key);
1954 push(@chartdatavals_count, $count);
1955 push(@chartdatavals_vol, $report_totals{transport}{$key}{'Volume-gigs'}*$gig + $report_totals{transport}{$key}{Volume} );
1956 }
1957 }
1958 else {
1959 # We are getting our data from mainlog files.
1960 foreach $key (sort keys %transported_data) {
1961 printf("$format1\n",$key,
1962 volume_rounded($transported_data{$key},$transported_data_gigs{$key}),
1963 $transported_count{$key});
1964 push(@chartdatanames, $key);
1965 push(@chartdatavals_count, $transported_count{$key});
1966 push(@chartdatavals_vol, $transported_data_gigs{$key}*$gig + $transported_data{$key});
1967 }
1968 }
1969 if ($html) {
1970 print "</table>\n";
1971 print "</td><td>\n";
1972 if ($HAVE_GD_Graph_pie && $charts)
1973 {
1974 # calculate the graph
1975 my @data = (
1976 \@chartdatanames,
1977 \@chartdatavals_count
1978 );
1979 my $graph = GD::Graph::pie->new(200, 200);
1980 $graph->set(
1981 x_label => 'Transport',
1982 y_label => 'Messages',
1983 title => 'By count',
1984 );
1985 my $gd = $graph->plot(\@data) or warn($graph->error);
1986 if ($gd) {
1987 open(IMG, ">$chartdir/transports_count.png") or die $!;
1988 binmode IMG;
1989 print IMG $gd->png;
1990 close IMG;
1991 print "<img src=\"$chartrel/transports_count.png\">";
1992 }
1993 }
1994 print "</td><td>\n";
1995
1996 if ($HAVE_GD_Graph_pie && $charts) {
1997 my @data = (
1998 \@chartdatanames,
1999 \@chartdatavals_vol
2000 );
2001 my $graph = GD::Graph::pie->new(200, 200);
2002 $graph->set(
2003 title => 'By volume',
2004 );
2005 my $gd = $graph->plot(\@data) or warn($graph->error);
2006 if ($gd) {
2007 open(IMG, ">$chartdir/transports_vol.png") or die $!;
2008 binmode IMG;
2009 print IMG $gd->png;
2010 close IMG;
2011 print "<img src=\"$chartrel/transports_vol.png\">";
2012 }
2013 }
2014 print "</td></tr></table>\n";
2015 }
2016 print "\n";
2017}
2018
2019
2020
1b4fe9dd
PH		 2021#######################################################################
2022# print_relay();
2023#
2024# print_relay();
2025#
2026# Print our totals by relay.
2027#######################################################################
059ec3d9
PH		 2028sub print_relay {
2029 my $temp = "Relayed messages";
2030 print "<hr><a name=\"$temp\"></a><h2>$temp</h2>\n" if $html;
2031 if (scalar(keys %relayed) > 0 || $relayed_unshown > 0) {
2032 my $shown = 0;
2033 my $spacing = "";
2034 my($format);
2035
2036 if ($html) {
2037 print "<table border=1>\n";
2038 print "<tr><th>Count</th><th>From</th><th>To</th>\n";
2039 $format = "<tr><td align=\"right\">%d</td><td>%s</td><td>%s</td>\n";
2040 }
2041 else {
2042 printf("%s\n%s\n\n", $temp, "-" x length($temp));
2043 $format = "%7d %s\n => %s\n";
2044 }
2045
2046 my($key);
2047 foreach $key (sort keys %relayed) {
2048 my $count = $relayed{$key};
2049 $shown += $count;
2050 $key =~ s/[HA]=//g;
2051 my($one,$two) = split(/=> /, $key);
2052 printf($format, $count, $one, $two);
2053 $spacing = "\n";
2054 }
2055 print "</table>\n<p>\n" if $html;
2056 print "${spacing}Total: $shown (plus $relayed_unshown unshown)\n";
2057 }
2058 else {
2059 print "No relayed messages\n";
2060 print "-------------------\n" unless $html;
2061 }
2062 print "\n";
2063}
2064
2065
2066
1b4fe9dd
PH		 2067#######################################################################
2068# print_errors();
2069#
2070# print_errors();
2071#
2072# Print our errors. In HTML, we display them as a list rather than a table -
2073# Netscape doesn't like large tables!
2074#######################################################################
059ec3d9
PH		 2075sub print_errors {
2076 my $total_errors = 0;
2077
2078 if (scalar(keys %errors_count) != 0) {
2079 my $temp = "List of errors";
2080 my($format);
2081 if ($html) {
2082 print "<hr><a name=\"errors\"></a><h2>$temp</h2>\n";
2083 print "<ul><li><b>Count - Error</b>\n";
2084 $format = "<li>%d - %s\n";
2085 }
2086 else {
2087 printf("%s\n%s\n\n", $temp, "-" x length($temp));
2088 }
2089
2090 my($key);
2091 foreach $key (sort keys %errors_count) {
2092 my $text = $key;
2093 chomp($text);
2094 $text =~ s/\s\s+/ /g; #Convert multiple spaces to a single space.
2095 $total_errors += $errors_count{$key};
2096 if ($html) {
1b4fe9dd 2097
059ec3d9
PH		 2098 #Translate HTML tag characters. Sergey Sholokh.
2099 $text =~ s/\</\&lt\;/g;
2100 $text =~ s/\>/\&gt\;/g;
2101
2102 printf($format,$errors_count{$key},$text);
2103 }
2104 else {
2105 printf("%5d ", $errors_count{$key});
2106 while (length($text) > 65) {
2107 my($first,$rest) = $text =~ /(.{50}\S*)\s+(.+)/;
2108 last if !$first;
2109 printf("%s\n ", $first);
2110 $text = $rest;
2111 }
2112 printf("%s\n\n", $text);
2113 }
2114 }
2115 print "</ul>\n<p>\n" if $html;
2116
2117 $temp = "Errors encountered: $total_errors";
2118 print $temp,"\n";
2119 print "-" x length($temp),"\n" unless $html;
2120 }
2121
2122}
2123
2124
1b4fe9dd
PH		 2125#######################################################################
2126# parse_old_eximstat_reports();
2127#
2128# parse_old_eximstat_reports($fh);
2129#
2130# Parse old eximstat output so we can merge daily stats to weekly stats and weekly to monthly etc.
2131#
2132# To test that the merging still works after changes, do something like the following.
2133# All the diffs should produce no output.
2134#
2135# options='-bydomain -byemail -byhost -byedomain'
2136# options="$options -pattern 'Completed Messages' /Completed/"
2137# options="$options -pattern 'Received Messages' /<=/"
2138#
2139# ./eximstats $options mainlog > mainlog.txt
2140# ./eximstats $options -merge mainlog.txt > mainlog.2.txt
2141# diff mainlog.txt mainlog.2.txt
2142#
2143# ./eximstats $options -html mainlog > mainlog.html
2144# ./eximstats $options -merge -html mainlog.txt > mainlog.2.html
2145# diff mainlog.html mainlog.2.html
2146#
2147# ./eximstats $options -merge mainlog.html > mainlog.3.txt
2148# diff mainlog.txt mainlog.3.txt
2149#
2150# ./eximstats $options -merge -html mainlog.html > mainlog.3.html
2151# diff mainlog.html mainlog.3.html
2152#
2153# ./eximstats $options -nvr mainlog > mainlog.nvr.txt
2154# ./eximstats $options -merge mainlog.nvr.txt > mainlog.4.txt
2155# diff mainlog.txt mainlog.4.txt
2156#
2157# # double_mainlog.txt should have twice the values that mainlog.txt has.
2158# ./eximstats $options mainlog mainlog > double_mainlog.txt
2159#######################################################################
059ec3d9
PH		 2160sub parse_old_eximstat_reports {
2161 my($fh) = @_;
2162
2163 my(%league_table_value_entered, %league_table_value_was_zero, %table_order);
2164
2165 while (<$fh>) {
2166 if (/Exim statistics from ([\d\-]+ [\d:]+(\s+[\+\-]\d+)?) to ([\d\-]+ [\d:]+(\s+[\+\-]\d+)?)/) {
2167 $begin = $1 if ($1 lt $begin);
2168 $end = $3 if ($3 gt $end);
2169 }
2170 elsif (/Grand total summary/) {
2171 # Fill in $report_totals{Received|Delivered}{Volume|Messages|Hosts|Domains|...|Delayed|DelayedPercent|Failed|FailedPercent}
2172 my(@fields);
2173 while (<$fh>) {
8974000d 2174 $_ = html2txt($_); #Convert general HTML markup to text.
059ec3d9
PH		 2175 s/At least one addr//g; #Another part of the HTML output we don't want.
2176
2177# TOTAL Volume Messages Hosts Domains Delayed Failed
2178# Received 26MB 237 177 23 8 3.4% 28 11.8%
2179# Delivered 13MB 233 99 88
2180 if (/TOTAL\s+(.*?)\s*$/) {
2181 @fields = split(/\s+/,$1);
2182 #Delayed and Failed have two columns each, so add the extra field names in.
2183 splice(@fields,-1,1,'DelayedPercent','Failed','FailedPercent');
2184 }
2185 elsif (/(Received|Delivered)\s+(.*?)\s*$/) {
2186 print STDERR "Parsing $_" if $debug;
2187 add_to_totals($report_totals{$1},\@fields,$2);
2188 }
2189 last if (/Delivered/); #Last line of this section.
2190 }
2191 }
2192
2193 elsif (/User Specified Patterns/i) {
2194#User Specified Patterns
2195#-----------------------
2196# Total
2197# Description 85
2198
2199 while (<$fh>) { last if (/Total/); } #Wait until we get the table headers.
2200 while (<$fh>) {
2201 print STDERR "Parsing $_" if $debug;
8974000d 2202 $_ = html2txt($_); #Convert general HTML markup to text.
059ec3d9
PH		 2203 if (/^\s*(.*?)\s+(\d+)\s*$/) {
2204 $report_totals{patterns}{$1} = {} unless (defined $report_totals{patterns}{$1});
2205 add_to_totals($report_totals{patterns}{$1},['Total'],$2);
2206 }
2207 last if (/^\s*$/); #Finished if we have a blank line.
2208 }
2209 }
2210
2211 elsif (/Deliveries by transport/i) {
2212#Deliveries by transport
2213#-----------------------
2214# Volume Messages
2215# :blackhole: 70KB 51
2216# address_pipe 655KB 1
2217# smtp 11MB 151
2218
2219 while (<$fh>) { last if (/Volume/); } #Wait until we get the table headers.
2220 while (<$fh>) {
2221 print STDERR "Parsing $_" if $debug;
8974000d 2222 $_ = html2txt($_); #Convert general HTML markup to text.
059ec3d9
PH		 2223 if (/(\S+)\s+(\d+\S*\s+\d+)/) {
2224 $report_totals{transport}{$1} = {} unless (defined $report_totals{transport}{$1});
2225 add_to_totals($report_totals{transport}{$1},['Volume','Messages'],$2);
2226 }
2227 last if (/^\s*$/); #Finished if we have a blank line.
2228 }
2229 }
2230 elsif (/(Messages received|Deliveries) per/) {
2231# Messages received per hour (each dot is 2 messages)
2232#---------------------------------------------------
2233#
2234#00-01 106 .....................................................
2235#01-02 103 ...................................................
2236
2237 # Set a pointer to the interval array so we can use the same code
2238 # block for both messages received and delivered.
2239 my $interval_aref = ($1 eq 'Deliveries') ? \@delivered_interval_count : \@received_interval_count;
2240 my $reached_table = 0;
2241 while (<$fh>) {
2242 $reached_table = 1 if (/^00/);
2243 next unless $reached_table;
2244 print STDERR "Parsing $_" if $debug;
2245 if (/^(\d+):(\d+)\s+(\d+)/) { #hh:mm start time format ?
1b4fe9dd 2246 $$interval_aref[($1*60 + $2)/$hist_interval] += $3 if $hist_opt;
059ec3d9
PH		 2247 }
2248 elsif (/^(\d+)-(\d+)\s+(\d+)/) { #hh-hh start-end time format ?
1b4fe9dd 2249 $$interval_aref[($1*60)/$hist_interval] += $3 if $hist_opt;
059ec3d9
PH		 2250 }
2251 else { #Finished the table ?
2252 last;
2253 }
2254 }
2255 }
2256
2257 elsif (/Time spent on the queue: (all messages|messages with at least one remote delivery)/) {
2258#Time spent on the queue: all messages
2259#-------------------------------------
2260#
2261#Under 1m 217 91.9% 91.9%
2262# 5m 2 0.8% 92.8%
2263# 3h 8 3.4% 96.2%
2264# 6h 7 3.0% 99.2%
2265# 12h 2 0.8% 100.0%
2266
2267 # Set a pointer to the queue bin so we can use the same code
2268 # block for both all messages and remote deliveries.
2269 my $bin_aref = ($1 eq 'all messages') ? \@queue_bin : \@remote_queue_bin;
2270 my $reached_table = 0;
2271 while (<$fh>) {
8974000d 2272 $_ = html2txt($_); #Convert general HTML markup to text.
059ec3d9
PH		 2273 $reached_table = 1 if (/^\s*Under/);
2274 next unless $reached_table;
2275 my $previous_seconds_on_queue = 0;
2276 if (/^\s*(Under|Over|)\s+(\d+[smhdw])\s+(\d+)/) {
2277 print STDERR "Parsing $_" if $debug;
2278 my($modifier,$formated_time,$count) = ($1,$2,$3);
2279 my $seconds = unformat_time($formated_time);
2280 my $time_on_queue = ($seconds + $previous_seconds_on_queue) / 2;
2281 $previous_seconds_on_queue = $seconds;
2282 $time_on_queue = $seconds * 2 if ($modifier eq 'Over');
2283 my($i);
2284 for ($i = 0; $i <= $#queue_times; $i++) {
2285 if ($time_on_queue < $queue_times[$i]) {
2286 $$bin_aref[$i] += $count;
2287 last;
2288 }
2289 }
2290 # There's only one counter for messages going over the queue
2291 # times so make sure we only count it once.
2292 $queue_more_than += $count if (($bin_aref == \@queue_bin) && ($i > $#queue_times));
2293 }
2294 else {
2295 last; #Finished the table ?
2296 }
2297 }
2298 }
2299
2300 elsif (/Relayed messages/) {
2301#Relayed messages
2302#----------------
2303#
2304# 1 addr.domain.com [1.2.3.4] a.user@domain.com
2305# => addr2.domain2.com [5.6.7.8] a2.user2@domain2.com
2306#
2307#<tr><td align="right">1</td><td>addr.domain.com [1.2.3.4] a.user@domain.com </td><td>addr2.domain2.com [5.6.7.8] a2.user2@domain2.com</td>
2308
2309 my $reached_table = 0;
2310 my($count,$sender);
2311 while (<$fh>) {
2312 unless ($reached_table) {
2313 last if (/No relayed messages/);
2314 $reached_table = 1 if (/^\s*\d/ || />\d+</);
2315 next unless $reached_table;
2316 }
2317 if (/>(\d+)<.td><td>(.*?) ?<.td><td>(.*?)</) {
2318 update_relayed($1,$2,$3);
2319 }
2320 elsif (/^\s*(\d+)\s+(.*?)\s*$/) {
2321 ($count,$sender) = ($1,$2);
2322 }
2323 elsif (/=>\s+(.*?)\s*$/) {
2324 update_relayed($count,$sender,$1);
2325 }
2326 else {
2327 last; #Finished the table ?
2328 }
2329 }
2330 }
2331
2332 elsif (/Top (.*?) by (message count|volume)/) {
2333#Top 50 sending hosts by message count
2334#-------------------------------------
2335#
2336# 48 1468KB local
2337 my($category,$by_count_or_volume) = ($1,$2);
2338
2339 #As we show 2 views of each table (by count and by volume),
2340 #most (but not all) entries will appear in both tables.
2341 #Set up a hash to record which entries we have already seen
2342 #and one to record which ones we are seeing for the first time.
2343 if ($by_count_or_volume =~ /count/) {
2344 undef %league_table_value_entered;
2345 undef %league_table_value_was_zero;
2346 undef %table_order;
2347 }
2348
2349 #As this section processes multiple different table categories,
2350 #set up pointers to the hashes to be updated.
2351 my($count_href,$data_href,$data_gigs_href);
2352 if ($category =~ /local sender/) {
2353 $count_href = \%received_count_user;
2354 $data_href = \%received_data_user;
2355 $data_gigs_href = \%received_data_gigs_user;
2356 }
2357 elsif ($category =~ /sending (\S+?)s?\b/) {
2358 #Top 50 sending (host|domain|email|edomain)s
2359 #Top sending (host|domain|email|edomain)
2360 $count_href = \%{$received_count{"\u$1"}};
2361 $data_href = \%{$received_data{"\u$1"}};
2362 $data_gigs_href = \%{$received_data_gigs{"\u$1"}};
2363 }
2364 elsif ($category =~ /local destination/) {
2365 $count_href = \%delivered_count_user;
2366 $data_href = \%delivered_data_user;
2367 $data_gigs_href = \%delivered_data_gigs_user;
2368 }
2369 elsif ($category =~ /(\S+) destination/) {
2370 #Top 50 (host|domain|email|edomain) destinations
2371 #Top (host|domain|email|edomain) destination
2372 $count_href = \%{$delivered_count{"\u$1"}};
2373 $data_href = \%{$delivered_data{"\u$1"}};
2374 $data_gigs_href = \%{$delivered_data_gigs{"\u$1"}};
2375 }
2376
2377 my $reached_table = 0;
2378 while (<$fh>) {
8974000d 2379 $_ = html2txt($_); #Convert general HTML markup to text.
059ec3d9
PH		 2380 $reached_table = 1 if (/^\s*\d/);
2381 next unless $reached_table;
2382 if (/^\s*(\d+)\s+(\S+)\s*(.*?)\s*$/) {
2383 my($count,$rounded_volume,$entry) = ($1,$2,$3);
2384 #Note: $entry fields can be both null and can contain spaces.
2385
2386 #Add the entry into the %table_order hash if it has a rounded volume (KB/MB/GB).
2387 push(@{$table_order{$rounded_volume}{$by_count_or_volume}},$entry) if ($rounded_volume =~ /\D/);
2388
2389 unless ($league_table_value_entered{$entry}) {
2390 $league_table_value_entered{$entry} = 1;
2391 unless ($$count_href{$entry}) {
2392 $$count_href{$entry} = 0;
2393 $$data_href{$entry} = 0;
2394 $$data_gigs_href{$entry} = 0;
2395 $league_table_value_was_zero{$entry} = 1;
2396 }
2397
2398 $$count_href{$entry} += $count;
2399 #Add the rounded value to the data and data_gigs hashes.
2400 un_round($rounded_volume,\$$data_href{$entry},\$$data_gigs_href{$entry});
2401 print STDERR "$category by $by_count_or_volume: added $count,$rounded_volume to $entry\n" if $debug;
2402 }
2403 }
2404 else { #Finished the table ?
2405 if ($by_count_or_volume =~ /volume/) {
2406 #Add a few bytes to appropriate entries to preserve the order.
2407
2408 my($rounded_volume);
2409 foreach $rounded_volume (keys %table_order) {
2410 #For each rounded volume, we want to create a list which has things
2411 #ordered from the volume table at the front, and additional things
2412 #from the count table ordered at the back.
2413 @{$table_order{$rounded_volume}{volume}} = () unless defined $table_order{$rounded_volume}{volume};
2414 @{$table_order{$rounded_volume}{'message count'}} = () unless defined $table_order{$rounded_volume}{'message count'};
2415 my(@order,%mark);
2416 map {$mark{$_} = 1} @{$table_order{$rounded_volume}{volume}};
2417 @order = @{$table_order{$rounded_volume}{volume}};
2418 map {push(@order,$_)} grep(!$mark{$_},@{$table_order{$rounded_volume}{'message count'}});
2419
2420 my $bonus_bytes = $#order;
2421 $bonus_bytes = 511 if ($bonus_bytes > 511); #Don't go over the half-K boundary!
2422 while (@order and ($bonus_bytes > 0)) {
2423 my $entry = shift(@order);
2424 if ($league_table_value_was_zero{$entry}) {
2425 $$data_href{$entry} += $bonus_bytes;
2426 print STDERR "$category by $by_count_or_volume: added $bonus_bytes bonus bytes to $entry\n" if $debug;
2427 }
2428 $bonus_bytes--;
2429 }
2430 }
2431 }
2432
2433 last;
2434 }
2435 }
2436 }
2437 elsif (/List of errors/) {
2438#List of errors
2439#--------------
2440#
2441# 1 07904931641@one2one.net R=external T=smtp: SMTP error
2442# from remote mailer after RCPT TO:<07904931641@one2one.net>:
2443# host mail.one2one.net [193.133.192.24]: 550 User unknown
2444#
2445#<li>1 - ally.dufc@dunbar.org.uk R=external T=smtp: SMTP error from remote mailer after RCPT TO:<ally.dufc@dunbar.org.uk>: host mail.dunbar.org.uk [216.167.89.88]: 550 Unknown local part ally.dufc in <ally.dufc@dunbar.org.uk>
2446
2447
2448 my $reached_table = 0;
2449 my($count,$error,$blanks);
2450 while (<$fh>) {
2451 $reached_table = 1 if (/^( *|<li>)(\d+)/);
2452 next unless $reached_table;
2453
2454 s/^<li>(\d+) -/$1/; #Convert an HTML line to a text line.
8974000d 2455 $_ = html2txt($_); #Convert general HTML markup to text.
059ec3d9
PH		 2456
2457 if (/\t\s*(.*)/) {
2458 $error .= ' ' . $1; #Join a multiline error.
2459 }
2460 elsif (/^\s*(\d+)\s+(.*)/) {
2461 if ($error) {
2462 #Finished with a previous multiline error so save it.
2463 $errors_count{$error} = 0 unless $errors_count{$error};
2464 $errors_count{$error} += $count;
2465 }
2466 ($count,$error) = ($1,$2);
2467 }
2468 elsif (/Errors encountered/) {
2469 if ($error) {
2470 #Finished the section, so save our stored last error.
2471 $errors_count{$error} = 0 unless $errors_count{$error};
2472 $errors_count{$error} += $count;
2473 }
2474 last;
2475 }
2476 }
2477 }
2478
2479 }
2480}
2481
2482
2483
1b4fe9dd
PH		 2484#######################################################################
2485# update_relayed();
2486#
2487# update_relayed($count,$sender,$recipient);
2488#
2489# Adds an entry into the %relayed hash. Currently only used when
2490# merging reports.
2491#######################################################################
059ec3d9
PH		 2492sub update_relayed {
2493 my($count,$sender,$recipient) = @_;
2494
2495 #When generating the key, put in the 'H=' and 'A=' which can be used
2496 #in searches.
2497 my $key = "H=$sender => H=$recipient";
2498 $key =~ s/ ([^=\s]+\@\S+|<>)/ A=$1/g;
2499 if (!defined $relay_pattern || $key !~ /$relay_pattern/o) {
2500 $relayed{$key} = 0 if !defined $relayed{$key};
2501 $relayed{$key} += $count;
2502 }
2503 else {
2504 $relayed_unshown += $count;
2505 }
2506}
2507
2508
1b4fe9dd
PH		 2509#######################################################################
2510# add_to_totals();
2511#
2512# add_to_totals(\%totals,\@keys,$values);
2513#
2514# Given a line of space seperated values, add them into the provided hash using @keys
2515# as the hash keys.
2516#
2517# If the value contains a '%', then the value is set rather than added. Otherwise, we
2518# convert the value to bytes and gigs. The gigs get added to I<Key>-gigs.
2519#######################################################################
059ec3d9
PH		 2520sub add_to_totals {
2521 my($totals_href,$keys_aref,$values) = @_;
2522 my(@values) = split(/\s+/,$values);
2523 my(@keys) = @$keys_aref; #Make a copy as we destroy the one we use.
2524 my($value);
2525 foreach $value (@values) {
2526 my $key = shift(@keys) or next;
2527 if ($value =~ /%/) {
2528 $$totals_href{$key} = $value;
2529 }
2530 else {
2531 $$totals_href{$key} = 0 unless ($$totals_href{$key});
2532 $$totals_href{"$key-gigs"} = 0 unless ($$totals_href{"$key-gigs"});
2533 un_round($value, \$$totals_href{$key}, \$$totals_href{"$key-gigs"});
2534 print STDERR "Added $value to $key - $$totals_href{$key} , " . $$totals_href{"$key-gigs"} . "GB.\n" if $debug;
2535 }
2536 }
2537}
2538
1b4fe9dd
PH		 2539#######################################################################
2540# get_report_total();
2541#
2542# $total = get_report_total(\%hash,$key);
2543#
2544# If %hash contains values split into Units and Gigs, we calculate and return
2545#
2546# $hash{$key} + 1024*1024*1024 * $hash{"${key}-gigs"}
2547#######################################################################
059ec3d9
PH		 2548sub get_report_total {
2549 no integer;
2550 my($hash_ref,$key) = @_;
2551 if ($$hash_ref{"${key}-gigs"}) {
2552 return $$hash_ref{$key} + $gig * $$hash_ref{"${key}-gigs"};
2553 }
2554 return $$hash_ref{$key} || 0;
2555}
2556
1b4fe9dd
PH		 2557#######################################################################
2558# html2txt();
2559#
2560# $text_line = html2txt($html_line);
2561#
2562# Convert a line from html to text. Currently we just convert HTML tags to spaces
2563# and convert &gt;, &lt;, and &nbsp; tags back.
2564#######################################################################
059ec3d9
PH		 2565sub html2txt {
2566 ($_) = @_;
2567
2568 # Convert HTML tags to spacing. Note that the reports may contain <Userid> and
2569 # <Userid@Domain> words, so explicitly specify the HTML tags we will remove
2570 # (the ones used by this program). If someone is careless enough to have their
2571 # Userid the same as an HTML tag, there's not much we can do about it.
2572 s/<\/?(html|head|title|body|h\d|ul|li|a\s+|table|tr|td|th|pre|hr|p|br)\b.*?>/ /og;
2573
2574 s/\&lt\;/\</og; #Convert '&lt;' to '<'.
2575 s/\&gt\;/\>/og; #Convert '&gt;' to '>'.
2576 s/\&nbsp\;/ /og; #Convert '&nbsp;' to ' '.
2577 return($_);
2578}
2579
1b4fe9dd
PH		 2580#######################################################################
2581# get_next_arg();
2582#
2583# $arg = get_next_arg();
2584#
2585# Because eximstats arguments are often passed as variables,
2586# we can't rely on shell parsing to deal with quotes. This
2587# subroutine returns $ARGV[1] and does a shift. If $ARGV[1]
2588# starts with a quote (' or "), and doesn't end in one, then
2589# we append the next argument to it and shift again. We repeat
2590# until we've got all of the argument.
2591#
2592# This isn't perfect as all white space gets reduced to one space,
2593# but it's as good as we can get! If it's esential that spacing
2594# be preserved precisely, then you get that by not using shell
2595# variables.
2596#######################################################################
059ec3d9
PH		 2597sub get_next_arg {
2598 my $arg = '';
2599 my $matched_pattern = 0;
2600 while ($ARGV[1]) {
2601 $arg .= ' ' if $arg;
2602 $arg .= $ARGV[1]; shift(@ARGV);
2603 if ($arg !~ /^['"]/) {
2604 $matched_pattern = 1;
2605 last;
2606 }
2607 if ($arg =~ s/^(['"])(.*)\1$/$2/) {
2608 $matched_pattern = 1;
2609 last;
2610 }
2611 }
2612 die "Mismatched argument quotes - <$arg>.\n" unless $matched_pattern;
2613 return $arg;
2614}
2615
2616
2617
2618##################################################
2619# Main Program #
2620##################################################
2621
2622
2623$last_timestamp = '';
2624$last_date = '';
2625$show_errors = 1;
2626$show_relay = 1;
2627$show_transport = 1;
2628$topcount = 50;
2629$local_league_table = 1;
2630$include_remote_users = 0;
2631$hist_opt = 1;
2632$volume_rounding = 1;
2633$localtime_offset = calculate_localtime_offset(); # PH/FANF
2634
2635$charts = 0;
2636$charts_option_specified = 0;
2637$chartrel = ".";
2638$chartdir = ".";
2639
2640@queue_times = (60, 5*60, 15*60, 30*60, 60*60, 3*60*60, 6*60*60,
2641 12*60*60, 24*60*60);
2642
2643$last_offset = '';
2644$offset_seconds = 0;
2645
2646# Decode options
2647
2648while (@ARGV > 0 && substr($ARGV[0], 0, 1) eq '-')
2649 {
2650 if ($ARGV[0] =~ /^\-h(\d+)$/) { $hist_opt = $1 }
2651 elsif ($ARGV[0] =~ /^\-ne$/) { $show_errors = 0 }
2652 elsif ($ARGV[0] =~ /^\-nr(.?)(.*)\1$/)
2653 {
2654 if ($1 eq "") { $show_relay = 0 } else { $relay_pattern = $2 }
2655 }
2656 elsif ($ARGV[0] =~ /^\-q([,\d\+\-\*\/]+)$/)
2657 {
2658 @queue_times = split(/,/, $1);
2659 my($q);
2660 foreach $q (@queue_times) { $q = eval($q) + 0 }
2661 @queue_times = sort { $a <=> $b } @queue_times;
2662 @queue_times = () if ($#queue_times == 0 && $queue_times[0] == 0);
2663 }
2664 elsif ($ARGV[0] =~ /^-nt$/) { $show_transport = 0 }
2665 elsif ($ARGV[0] =~ /^\-nt(.?)(.*)\1$/)
2666 {
2667 if ($1 eq "") { $show_transport = 0 } else { $transport_pattern = $2 }
2668 }
2669 elsif ($ARGV[0] =~ /^-t(\d+)$/) { $topcount = $1 }
2670 elsif ($ARGV[0] =~ /^-tnl$/) { $local_league_table = 0 }
2671 elsif ($ARGV[0] =~ /^-html$/) { $html = 1 }
2672 elsif ($ARGV[0] =~ /^-merge$/) { $merge_reports = 1 }
2673 elsif ($ARGV[0] =~ /^-charts$/) {
2674 $charts = 1;
2675 warn "WARNING: CPAN Module GD::Graph::pie not installed. Obtain from www.cpan.org\n" unless $HAVE_GD_Graph_pie;
2676 warn "WARNING: CPAN Module GD::Graph::linespoints not installed. Obtain from www.cpan.org\n" unless $HAVE_GD_Graph_linespoints;
2677 }
2678 elsif ($ARGV[0] =~ /^-chartdir$/) { $chartdir = $ARGV[1]; shift; $charts_option_specified = 1; }
2679 elsif ($ARGV[0] =~ /^-chartrel$/) { $chartrel = $ARGV[1]; shift; $charts_option_specified = 1; }
8974000d 2680 elsif ($ARGV[0] =~ /^-cache$/) { } #Not currently used.
059ec3d9
PH		 2681 elsif ($ARGV[0] =~ /^-byhost$/) { $do_sender{Host} = 1 }
2682 elsif ($ARGV[0] =~ /^-bydomain$/) { $do_sender{Domain} = 1 }
2683 elsif ($ARGV[0] =~ /^-byemail$/) { $do_sender{Email} = 1 }
2684 elsif ($ARGV[0] =~ /^-byemaildomain$/) { $do_sender{Edomain} = 1 }
2685 elsif ($ARGV[0] =~ /^-byedomain$/) { $do_sender{Edomain} = 1 }
2686 elsif ($ARGV[0] =~ /^-nvr$/) { $volume_rounding = 0 }
2687 elsif ($ARGV[0] =~ /^-d$/) { $debug = 1 }
2688 elsif ($ARGV[0] =~ /^--?h(elp)?$/){ help() }
2689 elsif ($ARGV[0] =~ /^-t_remote_users$/) { $include_remote_users = 1 }
2690 elsif ($ARGV[0] =~ /^-pattern$/)
2691 {
2692 push(@user_descriptions,get_next_arg());
2693 push(@user_patterns,get_next_arg());
2694 }
2695 elsif ($ARGV[0] =~ /^-utc$/)
2696 {
2697 # We don't need this value if the log is in UTC.
2698 $localtime_offset = undef;
2699 }
2700 else
2701 {
2702 print STDERR "Eximstats: Unknown or malformed option $ARGV[0]\n";
2703 help();
2704 }
2705 shift;
2706 }
2707
2708 # Check that all the charts options are specified.
2709 warn "-charts option not specified. Use -help for help.\n" if ($charts_option_specified && ! $charts);
2710
2711 # Default to display tables by sending Host.
2712 $do_sender{Host} = 1 unless ($do_sender{Domain} || $do_sender{Email} || $do_sender{Edomain});
2713
2714
2715for (my $i = 0; $i <= $#queue_times; $i++) {
2716 $queue_bin[$i] = 0;
2717 $remote_queue_bin[$i] = 0;
2718}
2719
2720# Compute the number of slots for the histogram
2721
2722if ($hist_opt > 0)
2723 {
2724 if ($hist_opt > 60 || 60 % $hist_opt != 0)
2725 {
2726 print "Eximstats: -h must specify a factor of 60\n";
2727 exit 1;
2728 }
2729 $hist_interval = 60/$hist_opt; #Interval in minutes.
2730 $hist_number = (24*60)/$hist_interval; #Number of intervals per day.
2731 @received_interval_count = (0) x $hist_number;
2732 @delivered_interval_count = (0) x $hist_number;
2733 }
2734
2735#$queue_unknown = 0;
2736
2737$total_received_data = 0;
2738$total_received_data_gigs = 0;
2739$total_received_count = 0;
2740
2741$total_delivered_data = 0;
2742$total_delivered_data_gigs = 0;
2743$total_delivered_count = 0;
2744
2745$queue_more_than = 0;
2746$delayed_count = 0;
2747$relayed_unshown = 0;
2748$begin = "9999-99-99 99:99:99";
2749$end = "0000-00-00 00:00:00";
2750my($section,$type);
2751foreach $section ('Received','Delivered') {
2752 foreach $type ('Volume','Messages','Delayed','Failed','Hosts','Domains','Emails','Edomains') {
2753 $report_totals{$section}{$type} = 0;
2754 }
2755}
2756
2757# Generate our parser.
2758my $parser = generate_parser();
2759
2760
2761
2762if (@ARGV) {
2763 # Scan the input files and collect the data
2764 foreach my $file (@ARGV) {
2765 if ($file =~ /\.gz/) {
2766 unless (open(FILE,"gunzip -c $file |")) {
2767 print STDERR "Failed to gunzip -c $file: $!";
2768 next;
2769 }
2770 }
2771 elsif ($file =~ /\.Z/) {
2772 unless (open(FILE,"uncompress -c $file |")) {
2773 print STDERR "Failed to uncompress -c $file: $!";
2774 next;
2775 }
2776 }
2777 else {
2778 unless (open(FILE,$file)) {
2779 print STDERR "Failed to read $file: $!";
2780 next;
2781 }
2782 }
2783 #Now parse the filehandle, updating the global variables.
2784 parse($parser,\*FILE);
2785 close FILE;
2786 }
2787}
2788else {
2789 #No files provided. Parse STDIN, updating the global variables.
2790 parse($parser,\*STDIN);
2791}
2792
2793
2794if ($begin eq "9999-99-99 99:99:99") {
2795 print "**** No valid log lines read\n";
2796 exit 1;
2797}
2798
2799# Output our results.
2800print_header();
2801print_grandtotals();
2802
2803# Print counts of user specified patterns if required.
2804print_user_patterns() if @user_patterns;
2805
2806# Print totals by transport if required.
2807print_transport() if $show_transport;
2808
2809# Print the deliveries per interval as a histogram, unless configured not to.
2810# First find the maximum in one interval and scale accordingly.
2811if ($hist_opt > 0) {
2812 print_histogram("Messages received", @received_interval_count);
2813 print_histogram("Deliveries", @delivered_interval_count);
2814}
2815
2816# Print times on queue if required.
2817if ($#queue_times >= 0) {
2818 print_queue_times("all messages", \@queue_bin,$queue_more_than);
2819 print_queue_times("messages with at least one remote delivery",\@remote_queue_bin,$queue_more_than);
2820}
2821
2822# Print relay information if required.
2823print_relay() if $show_relay;
2824
2825# Print the league tables, if topcount isn't zero.
2826if ($topcount > 0) {
2827 foreach ('Host','Domain','Email','Edomain') {
2828 next unless $do_sender{$_};
2829 print_league_table("sending \l$_", $received_count{$_}, $received_data{$_},$received_data_gigs{$_});
2830 }
2831
2832 print_league_table("local sender", \%received_count_user,
2833 \%received_data_user,\%received_data_gigs_user) if ($local_league_table || $include_remote_users);
2834 foreach ('Host','Domain','Email','Edomain') {
2835 next unless $do_sender{$_};
2836 print_league_table("\l$_ destination", $delivered_count{$_}, $delivered_data{$_},$delivered_data_gigs{$_});
2837 }
2838 print_league_table("local destination", \%delivered_count_user,
2839 \%delivered_data_user,\%delivered_data_gigs_user) if ($local_league_table || $include_remote_users);
2840}
2841
2842# Print the error statistics if required.
2843print_errors() if $show_errors;
2844
2845if ($html) {
2846 print "</body>\n</html>\n"
2847}
2848
2849# End of eximstats
Unnamed repository; edit this file 'description' to name the repository.