Provided by: spamassassin_3.2.3-0ubuntu1_all bug
 

NAME

        sa-learn - train SpamAssassin’s Bayesian classifier
 

SYNOPSIS

        sa-learn [options] [file]...
 
        sa-learn [options] --dump [ all | data | magic ]
 
        Options:
 
         --ham                 Learn messages as ham (non-spam)
         --spam                Learn messages as spam
         --forget              Forget a message
         --use-ignores         Use bayes_ignore_from and bayes_ignore_to
         --sync                Syncronize the database and the journal if needed
         --force-expire        Force a database sync and expiry run
         --dbpath <path>       Allows commandline override (in bayes_path form)
                               for where to read the Bayes DB from
         --dump [all|data|magic]  Display the contents of the Bayes database
                               Takes optional argument for what to display
          --regexp <re>        For dump only, specifies which tokens to
                               dump based on a regular expression.
         -f file, --folders=file  Read list of files/directories from file
         --dir                 Ignored; historical compatibility
         --file                Ignored; historical compatibility
         --mbox                Input sources are in mbox format
         --mbx                 Input sources are in mbx format
         --showdots            Show progress using dots
         --progress            Show progress using progress bar
         --no-sync             Skip synchronizing the database and journal
                               after learning
         -L, --local           Operate locally, no network accesses
         --import              Migrate data from older version/non DB_File
                               based databases
         --clear               Wipe out existing database
         --backup              Backup, to STDOUT, existing database
         --restore <filename>  Restore a database from filename
         -u username, --username=username
                               Override username taken from the runtime
                               environment, used with SQL
         -C path, --configpath=path, --config-file=path
                               Path to standard configuration dir
         -p prefs, --prefspath=file, --prefs-file=file
                               Set user preferences file
         --siteconfigpath=path Path for site configs
                               (default: /etc/spamassassin)
         --cf=’config line’    Additional line of configuration
         -D, --debug [area=n,...]  Print debugging messages
         -V, --version         Print version
         -h, --help            Print usage message
 

DESCRIPTION

        Given a typical selection of your incoming mail classified as spam or
        ham (non-spam), this tool will feed each mail to SpamAssassin, allowing
        it to ’learn’ what signs are likely to mean spam, and which are likely
        to mean ham.
 
        Simply run this command once for each of your mail folders, and it will
        ’’learn’’ from the mail therein.
 
        Note that csh-style globbing in the mail folder names is supported; in
        other words, listing a folder name as "*" will scan every folder that
        matches.  See "Mail::SpamAssassin::ArchiveIterator" for more details.
 
        SpamAssassin remembers which mail messages it has learnt already, and
        will not re-learn those messages again, unless you use the --forget
        option. Messages learnt as spam will have SpamAssassin markup removed,
        on the fly.
 
        If you make a mistake and scan a mail as ham when it is spam, or vice
        versa, simply rerun this command with the correct classification, and
        the mistake will be corrected.  SpamAssassin will automatically ’for‐
        get’ the previous indications.
 
        Users of "spamd" who wish to perform training remotely, over a network,
        should investigate the "spamc -L" switch.
 

OPTIONS

        --ham
            Learn the input message(s) as ham.   If you have previously learnt
            any of the messages as spam, SpamAssassin will forget them first,
            then re-learn them as ham.  Alternatively, if you have previously
            learnt them as ham, it’ll skip them this time around.  If the mes‐
            sages have already been filtered through SpamAssassin, the learner
            will ignore any modifications SpamAssassin may have made.
 
        --spam
            Learn the input message(s) as spam.   If you have previously learnt
            any of the messages as ham, SpamAssassin will forget them first,
            then re-learn them as spam.  Alternatively, if you have previously
            learnt them as spam, it’ll skip them this time around.  If the mes‐
            sages have already been filtered through SpamAssassin, the learner
            will ignore any modifications SpamAssassin may have made.
 
        --folders=filename, -f filename
            sa-learn will read in the list of folders from the specified file,
            one folder per line in the file.  If the folder is prefixed with
            "ham:type:" or "spam:type:", sa-learn will learn that folder appro‐
            priately, otherwise the folders will be assumed to be of the type
            specified by --ham or --spam.
 
            "type" above is optional, but is the same as the standard for
            ArchiveIterator: mbox, mbx, dir, file, or detect (the default if
            not specified).
 
        --mbox
            sa-learn will read in the file(s) containing the emails to be
            learned, and will process them in mbox format (one or more emails
            per file).
 
        --mbx
            sa-learn will read in the file(s) containing the emails to be
            learned, and will process them in mbx format (one or more emails
            per file).
 
        --use-ignores
            Don’t learn the message if a from address matches configuration
            file item "bayes_ignore_from" or a to address matches
            "bayes_ignore_to".  The option might be used when learning from a
            large file of messages from which the hammy spam messages or spammy
            ham messages have not been removed.
 
        --sync
            Syncronize the journal and databases.  Upon successfully syncing
            the database with the entries in the journal, the journal file is
            removed.
 
        --force-expire
            Forces an expiry attempt, regardless of whether it may be necessary
            or not.  Note: This doesn’t mean any tokens will actually expire.
            Please see the EXPIRATION section below.
 
            Note: "--force-expire" also causes the journal data to be synchro‐
            nized into the Bayes databases.
 
        --forget
            Forget a given message previously learnt.
 
        --dbpath
            Allows a commandline override of the bayes_path configuration
            option.
 
        --dump option
            Display the contents of the Bayes database.  Without an option or
            with the all option, all magic tokens and data tokens will be dis‐
            played.  magic will only display magic tokens, and data will only
            display the data tokens.
 
            Can also use the --regexp RE option to specify which tokens to dis‐
            play based on a regular expression.
 
        --clear
            Clear an existing Bayes database by removing all traces of the
            database.
 
            WARNING: This is destructive and should be used with care.
 
        --backup
            Performs a dump of the Bayes database in machine/human readable
            format.
 
            The dump will include token and seen data.  It is suitable for
            input back into the --restore command.
 
        --restore=filename
            Performs a restore of the Bayes database defined by filename.
 
            WARNING: This is a destructive operation, previous Bayes data will
            be wiped out.
 
        -h, --help
            Print help message and exit.
 
        -u username, --username=username
            If specified this username will override the username taken from
            the runtime environment.  You can use this option to specify users
            in a virtual user configuration when using SQL as the Bayes back‐
            end.
 
            NOTE: This option will not change to the given username, it will
            only attempt to act on behalf of that user.  Because of this you
            will need to have proper permissions to be able to change files
            owned by username.  In the case of SQL this generally is not a
            problem.
 
        -C path, --configpath=path, --config-file=path
            Use the specified path for locating the distributed configuration
            files.  Ignore the default directories (usually "/usr/share/spamas‐
            sassin" or similar).
 
        --siteconfigpath=path
            Use the specified path for locating site-specific configuration
            files.  Ignore the default directories (usually "/etc/spamassassin"
            or similar).
 
        --cf=     config line     
            Add additional lines of configuration directly from the com‐
            mand-line, parsed after the configuration files are read.
            Multiple --cf arguments can be used, and each will be considered a
            separate line of configuration.
 
        -p prefs, --prefspath=prefs, --prefs-file=prefs
            Read user score preferences from prefs (usually "$HOME/.spamassas‐
            sin/user_prefs").
 
        --progress
            Prints a progress bar (to STDERR) showing the current progress.  In
            the case where no valid terminal is found this option will behave
            very much like the --showdots option.
 
        -D [area,...], --debug [area,...]
            Produce debugging output. If no areas are listed, all debugging
            information is printed. Diagnostic output can also be enabled for
            each area individually; area is the area of the code to instrument.
            For example, to produce diagnostic output on bayes, learn, and dns,
            use:
 
                    spamassassin -D bayes,learn,dns
 
            For more information about which areas (also known as channels) are
            available, please see the documentation at:
 
                    C<http://wiki.apache.org/spamassassin/DebugChannels>
 
            Higher priority informational messages that are suitable for log‐
            ging in normal circumstances are available with an area of "info".
 
        --no-sync
            Skip the slow synchronization step which normally takes place after
            changing database entries.  If you plan to learn from many folders
            in a batch, or to learn many individual messages one-by-one, it is
            faster to use this switch and run "sa-learn --sync" once all the
            folders have been scanned.
 
            Clarification: The state of --no-sync overrides the
            bayes_learn_to_journal configuration option.  If not specified, sa-
            learn will learn to the database directly.  If specified, sa-learn
            will learn to the journal file.
 
            Note: --sync and --no-sync can be specified on the same command‐
            line, which is slightly confusing.  In this case, the --no-sync
            option is ignored since there is no learn operation.
 
        -L, --local
            Do not perform any network accesses while learning details about
            the mail messages.  This will speed up the learning process, but
            may result in a slightly lower accuracy.
 
            Note that this is currently ignored, as current versions of SpamAs‐
            sassin will not perform network access while learning; but future
            versions may.
 
        --import
            If you previously used SpamAssassin’s Bayesian learner without the
            "DB_File" module installed, it will have created files in other
            formats, such as "GDBM_File", "NDBM_File", or "SDBM_File".  This
            switch allows you to migrate that old data into the "DB_File" for‐
            mat.  It will overwrite any data currently in the "DB_File".
 
            Can also be used with the --dbpath path option to specify the loca‐
            tion of the Bayes files to use.
 

MIGRATION

        There are now multiple backend storage modules available for storing
        user’s bayesian data. As such you might want to migrate from one back‐
        end to another. Here is a simple procedure for migrating from one
        backend to another.
 
        Note that if you have individual user databases you will have to per‐
        form a similar procedure for each one of them.
 
        sa-learn --sync
            This will sync any outstanding journal entries
 
        sa-learn --backup > backup.txt
            This will save all your Bayes data to a plain text file.
 
        sa-learn --clear
            This is optional, but good to do to clear out the old database.
 
        Repeat!
            At this point, if you have multiple databases, you should perform
            the procedure above for each of them. (i.e. each user’s database
            needs to be backed up before continuing.)
 
        Switch backends
            Once you have backed up all databases you can update your configu‐
            ration for the new database backend. This will involve at least the
            bayes_store_module config option and may involve some additional
            config options depending on what is required by the module. (For
            example, you may need to configure an SQL database.)
 
        sa-learn --restore backup.txt
            Again, you need to do this for every database.
 
        If you are migrating to SQL you can make use of the -u <username>
        option in sa-learn to populate each user’s database. Otherwise, you
        must run sa-learn as the user who database you are restoring.
        (Thanks to Michael Bell for this section!)
 
        For a more lengthy description of how this works, go to
        http://www.paulgraham.com/ and see "A Plan for Spam". It’s reasonably
        readable, even if statistics make me break out in hives.
 
        The short semi-inaccurate version: Given training, a spam heuristics
        engine can take the most "spammy" and "hammy" words and apply proba‐
        bilistic analysis. Furthermore, once given a basis for the analysis,
        the engine can continue to learn iteratively by applying both the non-
        Bayesian and Bayesian rulesets together to create evolving "intelli‐
        gence".
 
        SpamAssassin 2.50 and later supports Bayesian spam analysis, in the
        form of the BAYES rules. This is a new feature, quite powerful, and is
        disabled until enough messages have been learnt.
 
        The pros of Bayesian spam analysis:
 
        Can greatly reduce false positives and false negatives.
            It learns from your mail, so it is tailored to your unique e-mail
            flow.
 
        Once it starts learning, it can continue to learn from SpamAssassin and
        improve over time.
 
        And the cons:
 
        A decent number of messages are required before results are useful for
        ham/spam determination.
        It’s hard to explain why a message is or isn’t marked as spam.
            i.e.: a straightforward rule, that matches, say, "VIAGRA" is easy
            to understand. If it generates a false positive or false negative,
            it is fairly easy to understand why.
 
            With Bayesian analysis, it’s all probabilities - "because the past
            says it is likely as this falls into a probabilistic distribution
            common to past spam in your systems". Tell that to your users!
            Tell that to the client when he asks "what can I do to change
            this". (By the way, the answer in this case is "use whitelisting".)
 
        It will take disk space and memory.
            The databases it maintains take quite a lot of resources to store
            and use.
        Still interested? Ok, here’s the guidelines for getting this working.
 
        First a high-level overview:
 
        Build a significant sample of both ham and spam.
            I suggest several thousand of each, placed in SPAM and HAM directo‐
            ries or mailboxes.  Yes, you MUST hand-sort this - otherwise the
            results won’t be much better than SpamAssassin on its own. Verify
            the spamminess/haminess of EVERY message.  You’re urged to avoid
            using a publicly available corpus (sample) - this must be taken
            from YOUR mail server, if it is to be statistically useful.  Other‐
            wise, the results may be pretty skewed.
 
        Use this tool to teach SpamAssassin about these samples, like so:
                    sa-learn --spam /path/to/spam/folder
                    sa-learn --ham /path/to/ham/folder
                    ...
 
            Let SpamAssassin proceed, learning stuff. When it finds ham and
            spam it will add the "interesting tokens" to the database.
 
        If you need SpamAssassin to forget about specific messages, use the
        --forget option.
            This can be applied to either ham or spam that has run through the
            sa-learn processes. It’s a bit of a hammer, really, lowering the
            weighting of the specific tokens in that message (only if that mes‐
            sage has been processed before).
 
        Learning from single messages uses a command like this:
                    sa-learn --ham --no-sync mailmessage
 
            This is handy for binding to a key in your mail user agent.  It’s
            very fast, as all the time-consuming stuff is deferred until you
            run with the "--sync" option.
 
        Autolearning is enabled by default
            If you don’t have a corpus of mail saved to learn, you can let Spa‐
            mAssassin automatically learn the mail that you receive.  If you
            are autolearning from scratch, the amount of mail you receive will
            determine how long until the BAYES_* rules are activated.
        Learning filters require training to be effective.  If you don’t train
        them, they won’t work.  In addition, you need to train them with new
        messages regularly to keep them up-to-date, or their data will become
        stale and impact accuracy.
 
        You need to train with both spam and ham mails.  One type of mail alone
        will not have any effect.
 
        Note that if your mail folders contain things like forwarded spam, dis‐
        cussions of spam-catching rules, etc., this will cause trouble.  You
        should avoid scanning those messages if possible.  (An easy way to do
        this is to move them aside, into a folder which is not scanned.)
 
        If the messages you are learning from have already been filtered
        through SpamAssassin, the learner will compensate for this.  In effect,
        it learns what each message would look like if you had run "spamassas‐
        sin -d" over it in advance.
 
        Another thing to be aware of, is that typically you should aim to train
        with at least 1000 messages of spam, and 1000 ham messages, if possi‐
        ble.  More is better, but anything over about 5000 messages does not
        improve accuracy significantly in our tests.
 
        Be careful that you train from the same source -- for example, if you
        train on old spam, but new ham mail, then the classifier will think
        that a mail with an old date stamp is likely to be spam.
 
        It’s also worth noting that training with a very small quantity of ham,
        will produce atrocious results.  You should aim to train with at least
        the same amount (or more if possible!) of ham data than spam.
 
        On an on-going basis, it is best to keep training the filter to make
        sure it has fresh data to work from.  There are various ways to do
        this:
 
        1. Supervised learning
            This means keeping a copy of all or most of your mail, separated
            into spam and ham piles, and periodically re-training using those.
            It produces the best results, but requires more work from you, the
            user.
 
            (An easy way to do this, by the way, is to create a new folder for
            ’deleted’ messages, and instead of deleting them from other fold‐
            ers, simply move them in there instead.  Then keep all spam in a
            separate folder and never delete it.  As long as you remember to
            move misclassified mails into the correct folder set, it is easy
            enough to keep up to date.)
 
        2. Unsupervised learning from Bayesian classification
            Another way to train is to chain the results of the Bayesian clas‐
            sifier back into the training, so it reinforces its own decisions.
            This is only safe if you then retrain it based on any errors you
            discover.
 
            SpamAssassin does not support this method, due to experimental
            results which strongly indicate that it does not work well, and
            since Bayes is only one part of the resulting score presented to
            the user (while Bayes may have made the wrong decision about a
            mail, it may have been overridden by another system).
 
        3. Unsupervised learning from SpamAssassin rules
            Also called ’auto-learning’ in SpamAssassin.  Based on statistical
            analysis of the SpamAssassin success rates, we can automatically
            train the Bayesian database with a certain degree of confidence
            that our training data is accurate.
 
            It should be supplemented with some supervised training in addi‐
            tion, if possible.
 
            This is the default, but can be turned off by setting the SpamAs‐
            sassin configuration parameter "bayes_auto_learn" to 0.
 
        4. Mistake-based training
            This means training on a small number of mails, then only training
            on messages that SpamAssassin classifies incorrectly.  This works,
            but it takes longer to get it right than a full training session
            would.
 

FILES

        sa-learn and the other parts of SpamAssassin’s Bayesian learner, use a
        set of persistent database files to store the learnt tokens, as fol‐
        lows.
 
        bayes_toks
            The database of tokens, containing the tokens learnt, their count
            of occurrences in ham and spam, and the timestamp when the token
            was last seen in a message.
 
            This database also contains some ’magic’ tokens, as follows: the
            version number of the database, the number of ham and spam messages
            learnt, the number of tokens in the database, and timestamps of:
            the last journal sync, the last expiry run, the last expiry token
            reduction count, the last expiry timestamp delta, the oldest token
            timestamp in the database, and the newest token timestamp in the
            database.
 
            This is a database file, using "DB_File".  The database ’version
            number’ is 0 for databases from 2.5x, 1 for databases from certain
            2.6x development releases, and 2 for all more recent databases.
 
        bayes_seen
            A map of Message-Id and some data from headers and body to what
            that message was learnt as. This is used so that SpamAssassin can
            avoid re-learning a message it has already seen, and so it can
            reverse the training if you later decide that message was learnt
            incorrectly.
 
            This is a database file, using "DB_File".
 
        bayes_journal
            While SpamAssassin is scanning mails, it needs to track which
            tokens it uses in its calculations.  To avoid the contention of
            having each SpamAssassin process attempting to gain write access to
            the Bayes DB, the token timestamps are written to a ’journal’ file
            which will later (either automatically or via "sa-learn --sync") be
            used to synchronize the Bayes DB.
 
            Also, through the use of "bayes_learn_to_journal", or when using
            the "--no-sync" option with sa-learn, the actual learning data will
            take be placed into the journal for later synchronization.  This is
            typically useful for high-traffic sites to avoid the same con‐
            tention as stated above.
 

EXPIRATION

        Since SpamAssassin can auto-learn messages, the Bayes database files
        could increase perpetually until they fill your disk.  To control this,
        SpamAssassin performs journal synchronization and bayes expiration
        periodically when certain criteria (listed below) are met.
 
        SpamAssassin can sync the journal and expire the DB tokens either manu‐
        ally or opportunistically.  A journal sync is due if --sync is passed
        to sa-learn (manual), or if the following is true (opportunistic):
 
        - bayes_journal_max_size does not equal 0 (means don’t sync)
        - the journal file exists
 
        and either:
 
        - the journal file has a size greater than bayes_journal_max_size
 
        or
 
        - a journal sync has previously occurred, and at least 1 day has passed
        since that sync
 
        Expiry is due if --force-expire is passed to sa-learn (manual), or if
        all of the following are true (opportunistic):
 
        - the last expire was attempted at least 12hrs ago
        - bayes_auto_expire does not equal 0
        - the number of tokens in the DB is > 100,000
        - the number of tokens in the DB is > bayes_expiry_max_db_size
        - there is at least a 12 hr difference between the oldest and newest
        token atimes
 
        EXPIRE LOGIC
 
        If either the manual or opportunistic method causes an expire run to
        start, here is the logic that is used:
 
        - figure out how many tokens to keep.  take the larger of either
        bayes_expiry_max_db_size * 75% or 100,000 tokens.  therefore, the goal
        reduction is number of tokens - number of tokens to keep.
        - if the reduction number is < 1000 tokens, abort (not worth the
        effort).
        - if an expire has been done before, guesstimate the new atime delta
        based on the old atime delta.  (new_atime_delta = old_atime_delta *
        old_reduction_count / goal)
        - if no expire has been done before, or the last expire looks "wierd",
        do an estimation pass.  The definition of "wierd" is:
            - last expire over 30 days ago
            - last atime delta was < 12 hrs
            - last reduction count was < 1000 tokens
            - estimated new atime delta is < 12 hrs
            - the difference between the last reduction count and the goal
            reduction count is > 50%
 
        ESTIMATION PASS LOGIC
 
        Go through each of the DB’s tokens.  Starting at 12hrs, calculate
        whether or not the token would be expired (based on the difference
        between the token’s atime and the db’s newest token atime) and keep the
        count.  Work out from 12hrs exponentially by powers of 2.  ie: 12hrs *
        1, 12hrs * 2, 12hrs * 4, 12hrs * 8, and so on, up to 12hrs * 512
        (6144hrs, or 256 days).
 
        The larger the delta, the smaller the number of tokens that will be
        expired.  Conversely, the number of tokens goes up as the delta gets
        smaller.  So starting at the largest atime delta, figure out which
        delta will expire the most tokens without going above the goal expira‐
        tion count.  Use this to choose the atime delta to use, unless one of
        the following occurs:
 
        - the largest atime (smallest reduction count) would expire too many
        tokens.  this means the learned tokens are mostly old and there needs
        to be new tokens learned before an expire can occur.
        - all of the atime choices result in 0 tokens being removed. this means
        the tokens are all newer than 12 hours and there needs to be new tokens
        learned before an expire can occur.
        - the number of tokens that would be removed is < 1000.  the benefit
        isn’t worth the effort.  more tokens need to be learned.
 
        If the expire run gets past this point, it will continue to the end.  A
        new DB is created since the majority of DB libraries don’t shrink the
        DB file when tokens are removed.  So we do the "create new, migrate old
        to new, remove old, rename new" shuffle.
 
        EXPIRY RELATED CONFIGURATION SETTINGS
 
        "bayes_auto_expire" is used to specify whether or not SpamAssassin
        ought to opportunistically attempt to expire the Bayes database. The
        default is 1 (yes).
        "bayes_expiry_max_db_size" specifies both the auto-expire token count
        point, as well as the resulting number of tokens after expiry as
        described above.  The default value is 150,000, which is roughly equiv‐
        alent to a 6Mb database file if you’re using DB_File.
        "bayes_journal_max_size" specifies how large the Bayes journal will
        grow before it is opportunistically synced.  The default value is
        102400.
 

INSTALLATION

        The sa-learn command is part of the Mail::SpamAssassin Perl module.
        Install this as a normal Perl module, using "perl -MCPAN -e shell", or
        by hand.
        spamassassin(1) spamc(1) Mail::SpamAssassin(3) Mail::SpamAssas‐
        sin::ArchiveIterator(3)
 
        <http://www.paulgraham.com/> Paul Graham’s "A Plan For Spam" paper
 
        <http://radio.weblogs.com/0101454/stories/2002/09/16/spamDetec‐
        tion.html> Gary Robinson’s f(x) and combining algorithms, as used in
        SpamAssassin
 
        <http://www.bgl.nu/~glouis/bogofilter/> ’Training on error’ page.  A
        discussion of various Bayes training regimes, including ’train on
        error’ and unsupervised training.
 

PREREQUISITES

        "Mail::SpamAssassin"
 

AUTHORS

        The SpamAssassin(tm) Project <http://spamassassin.apache.org/>
 
 

Powered by the Ubuntu Manpage Repository generator
Maintained by Dustin Kirkland