News: Sensation!!! Bad credit home purchase loan Compare credit card offer Bad card credit credit deposit no Well fargo secured credit card Best gas reward credit card Card company counseling credit Home equity loan calculator Card credit payment Best gas credit card Consolidation loan personal student Student credit card application Bad credit mortgage canada Personal debt consolidation loan 1000 loan no payday telecheck Card credit fixed interest low rate Free credit score repair Card credit offer Florida bad credit mortgage Card consolidation credit debt free Free credit check without credit card Credit report rating Cheap apr credit card Free online credit report without credit card Best credit card deal Debt free with credit counseling Adjustable rate home loan Citizen bank credit card Florida home loan online Annual credit equifax free report Bad credit car refinance Broker california home loan mortgage Approval fast loan payday Bankruptcy equity home loan Credit card and student loan consolidation Free credit bureau report Credit repair canada Credit card reader Free bad credit personal loan Home equity mortgage refinancing loan Citibank home equity loan Credit repair counseling The best credit card .com annual credit report Student credit card Well fargo home loan Credit card debt in america Credit card high limit instant approval Card consolidate credit loan student Card consolidation credit help Credit free missouri report Military bad credit personal loan Va home equity loan Consolidation loan loan private student Real credit card numbers Aspen card card credit gold master Consolidation loan student travel Advance loan payday quick Student loan refinancing Credit card for online shop Consolidation credit debt management Credit score chart Application card college credit student National foundation for consumer credit Personal loan canada Loan repayment student 0 interest rate credit card Credit card machine paper Family credit counseling corporation Best credit card for college student Advance cash loan payday Web credit card processing Bad card credit credit no Computer financing bad credit Payday advance loan illinois Trans union credit report history Credit report score Payday loan application Bank of america business credit card Personal loan with collateral Personal loan for people with bad credit Free credit score no credit card Home loan manufactured va Center consolidation loan student First national bank credit card Loan for personal debt Credit free repair report Personal loan uk Best airline reward credit card Debt consolidation loan without owning a home Bad credit debt consolidation 2006 loan mt personal tb trackback Apply for wal mart credit card Card counseling credit debt services Consolidation debt graduate loan student Mobile home refinancing loan Aspire credit card company Equity home home loan mortgage refinance Bad credit refinancing Credit card online ireland Credit card counseling debt consolidation Bad credit loan mortgage ohio Manufactured home equity loan Trans union credit report rating Low credit score loan Free web cam no credit card needed Bad credit Free credit card Credit instant report Apply bad card credit credit Bad car credit loan used Loan refinancing student Credit card number generator Shell gasoline credit card Shell gas station credit card Credit card processing for business Minnesota debt consolidation loan Debt consolidation home equity loan Personal loan lender Second debt consolidation mortgage Mortgage broker home loan refinance View credit report online Poor credit home improvement loan Article card consolidation credit debt Bank of america government credit card Business line of credit Manufactured home loan Bad credit home loan washington Card consolidation credit debt get Credit card account online Personal debt consolidation loan Personal student loan Personal secured loan finance California home loan refinancing Best business card credit small Credit card debt law Secured credit card in canada 0 fixed apr credit card Countrywide home loan mortgage Bad credit credit card no fee Business credit financing small Guaranteed no fax payday loan Home equity loan houston texas Credit report rating score Card credit depot home payment Credit repair book Free consumer credit counseling Illinois free credit report Approval canada card credit instant Debt consolidation quote Personal loan Low fixed apr credit card Citibank student credit card Accept business card credit small Card credit mbna Credit fix report Home improvement loan florida Credit score explanation Best home equity loan rate quote Credit counseling debt management agency Credit score rating scale Personal loan interest rate Bad credit fix report Credit free report trans union Accept major credit card American card com credit express Credit free indiana report California mobile home loan Lexington credit repair Online credit card service Direct merchant bank credit card application 3 bureau credit report Consolidating loan student Establishing business credit Credit card debt elimination Apply card credit virgin Consumer credit debt consolidation Payday loan personal Alternative student loan for bad credit Cash fast loan payday Instant approval payday loan 0 intrest free credit card Credit free obtain report Monogram credit card bank Pennsylvania debt consolidation loan Free debt consolidation company Card credit credit free report using without Apply for student credit card Credit report repair services Free credit card consolidation Card consolidation credit debt debt North carolina home equity loan Bad consolidation credit debt help 0 apr credit card offer Settle credit card debt Business credit rating Payday loan store Hawaii home equity loan Capital one credit card Card credit payment providian Card credit customer discover service Eligibility home loan va Countrywide home loan customer service Bad card credit credit history Consolidation debt free information Card credit unsecured visa Consumer credit debt consolidation Credit repair debt consolidation Prepaid credit card application Va home improvement loan New hampshire home equity loan Second mortgage loan bad credit Secured credit card application Instant approved credit card Bad credit jersey loan mortgage new Online application visa credit card California bad credit mortgage Paying off credit card debt Check credit in report Citibank business credit card Home equity loan uk Bad credit mortgage broker Card consolidation credit debt loan Annual credit free report Nurit credit card machine Opt out of credit card offer Loan sc student Card card credit master tribute 24 hour payday loan Hsbc credit card philippine California equity home in loan No apr credit card Card consolidation credit debt service Consolidation federal loan private student Canadian student credit card Broker consolidation debt lead mortgage Accept credit card merchant account Canadian international loan program student Application card credit interest low Credit card merchant terminal Prepaid gift credit card Card debt consolidation Card consolidation credit loan Bureau credit online report Airline mileage credit card Instant credit card application America card credit mbna 1000 advance loan payday Best rate credit card balance transfer Bad credit need a personal loan Credit repair lawyer Visa business credit card Free debt consolidation program Bad credit financing for notebook computer Alternative student loan for bad credit California construction home loan Business communication credit guffey refusal Federal home improvement loan Reduce credit card debt Best credit card online Instant faxless payday loan Bad credit home loan Debt consolidation uk Default loan student Home land loan manufactured Nova credit card processing Citibank credit card application Auto bad bankruptcy credit loan Free equifax credit score Bad credit loan mortgage very Credit debt consolidation loan 3 credit free report Bad credit mortgage loan in florida Best travel reward credit card New jersey home mortgage loan Get credit report online Credit card debt statue of limitation Bad card charge credit Bad credit loan mortgage rate Card college credit student visa Bad credit home loan mortgage se Apply bank bank card credit credit online orchard orchard Government loan student Home federal savings and loan College consolidation debt loan Bank credit card Credit card uk high risk Consolidation loan program student Consolidation debt loan student Bad credit cash advance Federal government free credit report College hazlewood hinson loan student Consumer credit free report No credit check payday loan Online debt consolidation program Best personal loan Quick personal loan Best california home loan Accept credit card no merchant account Chase bank one credit card Credit fico free report score Equifax consumer credit report Bad credit school loan Low apr student credit card Consolidation credit debt loan Get unsecured credit card bad credit Credit score 650 Credit repair info Personal debt consolidation loan uk Day loan payday quick same Brunswick home loan new 3 credit report and score Best rated credit card Personal loan application Apply for a business credit card Student loan default Credit card merchant account Business card credit prepaid Credit get report Sample credit report dispute letter Application approval card credit instant Business credit card with no personal guarantee Credit report trans union Prepaid credit card deal Discount credit card processing American card credit express payment Access card com credit mbna net 1000 advance cash loan payday Home equity loan indiana Apply card credit online secured Bad credit home mortgage loan quote Ez payday loan Credit card depot and debt consolidation Government debt consolidation Card chase credit na Consolidation debt lead mortgage Ingenico credit card machine Marriott reward credit card Direct merchant credit card Bad credit bill consolidation loan Credit card for student with no credit Credit free from government report Clean credit report Gasoline company credit card Construction financing home loan new Florida home loan south Bad card company credit debt Credit card for people with no credit Consumer credit counseling of atlanta Company loan student uk Terminate credit card debt Internet credit card fraud Loan pay payday quick Bad bankruptcy credit loan mortgage Capital card credit one service Approval business card credit instant Free phone no credit card required Tennessee home loan Credit card with no interest for a year Loan nz personal Great lake student loan consolidation Bad credit automobile loan Christian consolidation counseling credit debt Online credit counseling Payday loan in georgia Countrywide home loan payment Business card credit deal Bad credit loan mortgage va Balance card credit transfer uk Debt credit services Advise bad consolidation credit debt loan Credit rating company Application loan student Credit card application for college student En language loan personal Car loan credit score Continental airline credit card Debt consolidation online quote College student loan consolidation Credit report dispute letter Free instant credit card approval Gas credit card Government student loan Approval bad card credit credit instant online Low interest credit card uk Free personal credit report online Corp loan sc student Home equity loan minnesota Accept credit card payment online Eon bank credit card Fast loan payday 0 application card credit interest New home purchase loan Apply card credit macys Household bank credit card Low rate home equity loan Low rate personal loan Card credit gas hess Personal debt consolidation loan uk Card credit first premier secured Batch credit card processing Debt management credit counseling Free equifax credit report score Home loan mortgage new online quote Consolidation loan private student Green dot prepaid credit card Improving credit score Bad credit guarantee loan mortgage Credit card secured master card Free debt consolidation loan Credit repair florida Free gay porn no credit card Bad credit mortgage rating Bad credit guaranteed loan unsecured Company credit rating Consolidation debt online service Personal loan online Bad credit business loan Letter to credit card company California bad credit loan Personal loan for people with bad credit Bad credit no credit credit card Online credit card processor Credit card and check processing Pay capital one credit card bill No credit needed credit card Increase credit score Home equity loan refinancing Business card credit machine small Credit free new report york Card consolidation credit debt America bank card credit payment Travel reward credit card Card credit mbna uk Christian credit counseling services 0 business card credit interest Consolidation credit free Log home construction loan Texaco shell credit card Debt reduction solution credit card Guaranteed loan personal Bad credit computer loan Ohio home loan Bad credit home loan need Washington mutual bank home loan Advice card consolidate credit debt financial off pay Connecticut home equity loan rate Credit or bankruptcy counseling Hsbc credit card india Card consolidation credit debt depot No credit card needed Bad credit debt consolidation loan Low interest rate student credit card California free credit report Bad card credit unsecured Canadian credit report Colorado home mortgage loan Credit card debt consolidation service New york home equity loan Online credit report Apply for personal loan online Credit equifax free report Accept credit card Accepting card credit payment Canadian business credit card Accept credit card online merchant account Bad credit repair services Alaska home loan Free credit card for people with bad credit Lowest fixed rate credit card Consolidation interest loan rate student Miami home loan Lipman credit card machine Average credit card debt Loan home mortgage rate in california Instant credit card for bad credit Credit card terminal repair Sears credit card account Bad business card credit credit Bad credit home loan washington Credit equifax report sample Shell credit card online California equity home loan rate Credit equifax free online report Setup online credit card processing Card citibank credit online payment Christian debt consolidation Bad consolidation credit loan people Free annual credit score Card credit mutual providian washington Consumer counseling credit services Home loan for people with bad credit Card credit low processing rate Debt consolidation loan for home owner Bad credit refinancing home loan Free 3 bureau credit report Oklahoma home equity loan No credit credit card application Do it yourself credit repair kit Credit card debt loan Credit card terminal system Card card continental credit finance master Credit equifax identity report theft Debt consolidation calculator Uk credit card company California home equity loan rate Really bad credit personal loan No telecheck payday loan Second mortgage bad credit Buyer california first home loan time Bad credit home loan mortgage Best card credit rate transfer Trafalgar home loan Mortgage vs home equity loan Accept credit card hardware Auto bad credit loan virginia Best buy card credit services Cibc secured credit card Bad credit manufactured home loan Cash back credit card application Loan mae sallie student Wireless credit card terminal Approval loan online personal Bad credit score Home equity loan lowest rate Bad credit lo personal signature Raising credit score Community credit counseling Consolidation debt loan personal unsecured Wal mart discover credit card Dispute credit report online Credit fico report score Credit card for bad credit in canada Consolidation credit debt poor Credit card vending machine Fixed rate home loan Orchard bank credit card status Equifax credit report trans union Wamu home loan Credit card debt consolidation services Loan online personal uk Current home equity loan rate Free credit rating Bad credit mortgage financing Consolidate credit card bill Wal mart credit card Credit and debt counseling service Refinance fha home loan Consolidation credit non profit Credit dispute experian report Bank card card credit master tribute 3 bureau credit report with score Federal student loan application Apply business card credit small Bad credit used car loan Bad bankruptcy credit mortgage Auto bad car credit loan used Discover student credit card No deposit credit card 0 apr card credit student No credit card web cam Loan for people with bad credit rating Federal direct student loan consolidation Car loan for low credit score Consumer credit law Capital one visa credit card Aid direct federal loan student Fight credit card company Bureau credit score Bad card credit credit secured Home loan mobile va Credit card reward program Bad credit auto refinancing loan Gas credit card application Keyword free debt consolidation Michigan debt consolidation loan Home loan for home improvement Credit card depot and debt consolidation Reporting credit card fraud Direct merchant credit card bank na Credit report online equifax Retail credit card processing After bankruptcy home loan San diego home loan Instant online credit card Bad credit personal loan lender Aes student loan Card credit guaranteed unsecured Credit equifax from report Acs student loan consolidation Best card credit online rate Guaranteed faxless payday loan Home loan orange county Balance transfer credit card for bad credit Best airline credit card New jersey bad credit mortgage Debt reduction credit card consolidation Debt consolidation for people with bad credit Interest free credit card Eliminating credit card debt Payday loan online Student loan consolidation credit rating Kmart credit card application Fraud credit card company Credit score ranking Equity florida home loan rate Personal credit rating Long term bad credit personal loan Free lesbian porn no credit card Sallie mae student loan Capital one bank credit card Bad credit loan mortgage tennessee Home owner loan corp Instant approval unsecured credit card Countrywide home loan servicing Uk credit card apply American express delta credit card Fixed rate home equity loan Credit free pennsylvania report Card credit loan unsecured Alabama home loan Accept credit card internet Personal loan calculator Bad consolidation credit debt loan Home loan Student credit card debt Christian consumer credit counseling Consumer credit counseling debt consolidation California home loan veteran Secured corporate credit card Bad credit loan credit card American express black credit card Instant student credit card Clean credit report up Consolidation credit debt poor Quickly improve my credit score Improve credit score fast Student loan consolidation center Card credit merchant service Bank one online credit card International loan student Free music download no credit card needed Payday loan business Card chase credit disney from visa Mortgage loan refinance and debt consolidation Business card credit gm Credit repair forum Free debt consolidation service Fitch credit rating Online credit score Arizona home mortgage loan Credit counseling los angeles Instant online credit report No credit card Bureau credit equifax report Bad car credit history loan Credit counseling agency Verifone credit card machine Free credit card machine Bad credit personal signature loan Free nonprofit debt consolidation quote Best card credit low rate Approval card credit credit instant Unsecured personal loan bank Debt consolidation loan online Credit card debt consolidation Bc loan student Consolidations loan student Personal secured consolidation loan uk Applicant bad credit loan mortgage Personal loan for non home owner Check credit rating 0 balance card credit transfer uk�?? �?? Instant approval 0 credit card Maryland home loan Christian consolidation counseling credit debt Florida bad credit car loan Cheap credit report Consolidation debt defaulted loan student Consolidate defaulted loan student Bad credit down home loan money no Free credit score check Lexington law credit repair Mobile credit card processing Best secure credit card Washington mutual home equity loan Chase credit card online banking 0 credit card for 12 month on balance transfer Business credit card online Credit union master card online Airline miles credit card Bill consolidation debt free Debt consolidation counseling services Best home loan mortgage rate Get out of credit card debt Home improvement loan missouri Bad business credit en language loan Check credit report online Washington home loan Bad credit mortgage repair Alternative loan student California bad credit car loan Apply canada card credit jersey new 0 apr credit card Bad credit debt consolidation mortgage Free copy of credit report Credit card services Bad credit florida in loan mortgage Federal government student loan Secured credit card instant approval Connecticut debt consolidation loan Government home loan va National student loan canada Unsecured debt consolidation Home loan uk 3 in 1 credit report Delaware home equity loan rate Free credit score Cash loan student Bad credit payday loan

The Easy Way to Extract Useful Text from Arbitrary HTML

By alexjc | April 5, 2007

[Statistical Text Mining]

You’ve finally got your hands on the diverse collection of HTML documents you needed. But the content you’re interested in is hidden amidst adverts, layout tables or formatting markup, and other various links. Even worse, there’s visible text in the menus, headers and footers that you want to filter out. If you don’t want to write a complex scraping program for each type of HTML file, there is a solution.

This article shows you how to write a relatively simple script to extract text paragraphs from large chunks of HTML code, without knowing its structure or the tags used. It works on news articles and blogs pages with worthwhile text content, among others…

Do you want to find out how statistics and machine learning can save you time and effort mining text?

The concept is rather simple: use information about the density of text vs. HTML code to work out if a line of text is worth outputting. (This isn’t a novel idea, but it works!) The basic process works as follows:

  1. Parse the HTML code and keep track of the number of bytes processed.
  2. Store the text output on a per-line, or per-paragraph basis.
  3. Associate with each text line the number of bytes of HTML required to describe it.
  4. Compute the text density of each line by calculating the ratio of text to bytes.
  5. Then decide if the line is part of the content by using a neural network.

You can get pretty good results just by checking if the line’s density is above a fixed threshold (or the average), but the system makes fewer mistakes if you use machine learning — not to mention that it’s easier to implement!

Let’s take it from the top…

Converting the HTML to Text

What you need is the core of a text-mode browser, which is already setup to read files with HTML markup and display raw text. By reusing existing code, you won’t have to spend too much time handling invalid XML documents, which are very common — as you’ll realise quickly.

As a quick example, we’ll be using Python along with a few built-in modules: htmllib for the parsing and formatter for outputting formatted text. This is what the top-level function looks like:

def extract_text(html):
    # Derive from formatter.AbstractWriter to store paragraphs.
    writer = LineWriter()
    # Default formatter sends commands to our writer.
    formatter = AbstractFormatter(writer)
    # Derive from htmllib.HTMLParser to track parsed bytes.
    parser = TrackingParser(writer, formatter)
    # Give the parser the raw HTML data.
    # Filter the paragraphs stored and output them.
    return writer.output()

The TrackingParser itself overrides the callback functions for parsing start and end tags, as they are given the current parse index in the buffer. You don’t have access to that normally, unless you start diving into frames in the call stack — which isn’t the best approach! Here’s what the class looks like:

class TrackingParser(htmllib.HTMLParser):
    """Try to keep accurate pointer of parsing location."""
    def __init__(self, writer, *args):
        htmllib.HTMLParser.__init__(self, *args)
        self.writer = writer
    def parse_starttag(self, i):
        index = htmllib.HTMLParser.parse_starttag(self, i)
        self.writer.index = index
        return index
    def parse_endtag(self, i):
        self.writer.index = i
        return htmllib.HTMLParser.parse_endtag(self, i)

The LineWriter class does the bulk of the work when called by the default formatter. If you have any improvements or changes to make, most likely they’ll go here. This is where we’ll put our machine learning code in later. But you can keep the implementation rather simple and still get good results. Here’s the simplest possible code:

class Paragraph:
    def __init__(self):
        self.text = ''
        self.bytes = 0
        self.density = 0.0
class LineWriter(formatter.AbstractWriter):
    def __init__(self, *args):
        self.last_index = 0
        self.lines = [Paragraph()]
    def send_flowing_data(self, data):
        # Work out the length of this text chunk.
        t = len(data)
        # We've parsed more text, so increment index.
        self.index += t
        # Calculate the number of bytes since last time.
        b = self.index - self.last_index
        self.last_index = self.index
        # Accumulate this information in current line.
        l = self.lines[-1]
        l.text += data
        l.bytes += b
    def send_paragraph(self, blankline):
        """Create a new paragraph if necessary."""
        if self.lines[-1].text == '':
        self.lines[-1].text += 'n' * (blankline+1)
        self.lines[-1].bytes += 2 * (blankline+1)
    def send_literal_data(self, data):
    def send_line_break(self):

This code doesn’t do any outputting yet, it just gathers the data. We now have a bunch of paragraphs in an array, we know their length, and we know roughly how many bytes of HTML were necessary to create them. Let’s see what emerge from our statistics.

Examining the Data

Luckily, there are some patterns in the data. In the raw output below, you’ll notice there are definite spikes in the number of HTML bytes required to encode lines of text, notably around the title, both sidebars, headers and footers.

Graph of Text Output vs. HTML Bytes

While the number of HTML bytes spikes in places, it remains below average for quite a few lines. On these lines, the text output is rather high. Calculating the density of text to HTML bytes gives us a better understanding of this relationship.

Graph of Text Density per Line

The patterns are more obvious in this density value, so it gives us something concrete to work with.

Filtering the Lines

The simplest way we can filter lines now is by comparing the density to a fixed threshold, such as 50% or the average density. Finishing the LineWriter class:

    def compute_density(self):
        """Calculate the density for each line, and the average."""
        total = 0.0
        for l in self.lines:
            l.density = len(l.text) / float(l.bytes)
            total += l.density
        # Store for optional use by the neural network.
        self.average = total / float(len(self.lines))
    def output(self):
        """Return a string with the useless lines filtered out."""
        output = StringIO.StringIO()
        for l in self.lines:
            # Check density against threshold.
            # Custom filter extensions go here.
            if l.density > 0.5:
	return output.getvalue()

This rough filter typically gets most of the lines right. All the headers, footers and sidebars text is usually stripped as long as it’s not too long. However, if there are long copyright notices, comments, or descriptions of other stories, then those are output too. Also, if there are short lines around inline graphics or adverts within the text, these are not output.

To fix this, we need a more complex filtering heuristic. But instead of spending days working out the logic manually, we’ll just grab loads of information about each line and use machine learning to find patterns for us.

Supervised Machine Learning

Here’s an example of an interface for tagging lines of text as content or not:

Training From News Articles

The idea of supervised learning is to provide examples for an algorithm to learn from. In our case, we give it a set documents that were tagged by humans, so we know which line must be output and which line must be filtered out. For this we’ll use a simple neural network known as the perceptron. It takes floating point inputs and filters the information through weighted connections between “neurons” and outputs another floating point number. Roughly speaking, the number of neurons and layers affects the ability to approximate functions precisely; we’ll use both single-layer perceptrons (SLP) and multi-layer perceptrons (MLP) for prototyping.

To get the neural network to learn, we need to gather some data. This is where the earlier LineWriter.output() function comes in handy; it gives us a central point to process all the lines at once, and make a global decision which lines to output. Starting with intuition and experimenting a bit, we discover that the following data is useful to decide how to filter a line:

  • Density of the current line.
  • Number of HTML bytes of the line.
  • Length of output text for this line.
  • These three values for the previous line,
  • … and the same for the next line.

For the implementation, we’ll be using Python to interface with FANN, the Fast Artificial Neural Network Library. The essence of the learning code goes like this:

from pyfann import fann, libfann
# This creates a new single-layer perceptron with 1 output and 3 inputs.
obj = libfann.fann_create_standard_array(2, (3, 1))
ann = fann.fann_class(obj)
# Load the data we described above.
patterns = fann.read_train_from_file('training.txt')
ann.train_on_data(patterns, 1000, 1, 0.0)
# Then test it with different data.
for datin, datout in validation_data:
    result =
    print 'Got:', result, ' Expected:', datout

Trying out different data and different network structures is a rather mechanical process. Don’t have too many neurons or you may train too well for the set of documents you have (overfitting), and conversely try to have enough to solve the problem well. Here are the results, varying the number of lines used (1L-3L) and the number of attributes per line (1A-3A):

Neural Network Comparison Chart

The interesting thing to note is that 0.5 is already a pretty good guess at a fixed threshold (see first set of columns). The learning algorithm cannot find much better solution for comparing the density alone (1 Attribute in the second column). With 3 Attributes, the next SLP does better overall, though it gets more false negatives. Using multiple lines also increases the performance of the single layer perceptron (fourth set of columns). And finally, using a more complex neural network structure works best overall — making 80% less errors in filtering the lines.

Note that you can tweak how the error is calculated if you want to punish false positives more than false negatives.


Extracting text from arbitrary HTML files doesn’t necessarily require scraping the file with custom code. You can use statistics to get pretty amazing results, and machine learning to get even better. By tweaking the threshold, you can avoid the worst false positive that pollute your text output. But it’s not so bad in practice; where the neural network makes mistakes, even humans have trouble classifying those lines as “content” or not.

Now all you have to figure out is what to do with that clean text content!

Tags: , , , , ,
Category: tutorial |

7 Responses to “The Easy Way to Extract Useful Text from Arbitrary HTML”

  1. Cenny Wenenr Says:
    April 10th, 2007 at 7:40 am

    For the simple greedy immidietly local threshold, we could introduce a few assumptions to find a perhaps better estimate than the average. The average of some set is frequently used because it is the point with the least square distance over all values in the set. In this case however, there are two separate sets: text items and markup items.
    If we assume that each follows a normal distribution (which is common) and there is some probability to pick from one of them, we may estimate these parameters and find the values Thete for which the probability of either classification is 0.5.
    Solving this we receive the threshold:
    theta* = ( m_y t_y - m_z t_z +- (m_y - m_z) sqrt(t_y t_z) ) / (t_y - t_z)
    where m_y is the average of the text items, m_z the average of the markup items and t_z and t_y are given by:
    t_y = s_y ln p
    t_z = s_z ln(1-p)
    where s_y is the estimated standard deviation, which is
    sum of all i: (y_i - m_y)^2
    and similar for z.
    p is the ratio of text items, i.e. |y| / (|y| + |z|).

    A LaTeX formula can be found here:

    I can add additional details if anyone wishes. Reservations for errors, did this hastily and it is not double checked in any way.

    It should yield a bit better estimate than just the average but only relies on the immidietely local features and not for instance the relation to neighbors and global features. These should have a great impact. Hence, ANNs should work better if feasible.

  2. Anand Muthu Says:
    April 19th, 2007 at 7:02 am

    Great thought ! Superb :)
    Thinking to implement this in Java too.

  3. Phil Says:
    June 23rd, 2007 at 3:41 pm

    This approach to extract text for html page is interesting. Good idea.

  4. Lawrence Says:
    October 15th, 2007 at 11:23 am

    That is quite interesting. I want to find a similar approach only with C++. In my opinion, this is probably the only way to do an AI capable of learning from the internet or books.

  5. Betty Says:
    December 11th, 2007 at 9:27 pm

    good idea!similar to the 4th floor,I want to find a approach with C++.

  6. veelion Says:
    December 25th, 2007 at 9:42 pm

    good idea! but what’s the content of the file ‘training.txt’ while using FANN? Thanks. If possible, please send an example to veelion@gmail. It is appreciated.

  7. Einar Vollset’s Blog. » Blog Archive » Extracting human text from web pages using Ruby Says:
    February 12th, 2008 at 10:40 am

    […] number of available options out there, including using NLTK (a pretty good Python NLP library), and this approach using machine learning (also in Python - it seems to be the language of choice for AI these […]