Html Agility Pack

New Post: Project license vs LGPL

EmptyDoor — Sat, 05 Jul 2008 10:45:41 GMT

Hello,

I work for a company who is interested in using the Agility Pack for a website parser, but I'm a bit unsure about the licensing terms. If I create a parser using the agility pack, will it have to be covered by the agility pack license as well? I.e., is the license similar to GPL or LGPL?
I'd really like to use this nice software :)

Alexander

New Post: Stripping harmful HTML from user input, but allowing other HTML?

VijayKarla — Mon, 30 Jun 2008 14:57:43 GMT

Have a look into this thread... it might help you
http://www.codeplex.com/htmlagilitypack/Thread/View.aspx?ThreadId=16092

New Post: Stripping harmful HTML from user input, but allowing other HTML?

VijayKarla — Mon, 30 Jun 2008 13:44:13 GMT

This piece code works excellent.

New Post: User Name and Password for tfs01.codeplex.com

chandruvkumar — Thu, 26 Jun 2008 08:33:05 GMT

Hi,

what is the username & password for User Name and Password for tfs01.codeplex.com. i used my codeplex username & password to open it. but can't open it.

Please help. its urgent.

Thanks in advance.

Chandru.V

New Post: DHTML Content

chrisg229 — Wed, 25 Jun 2008 17:26:10 GMT

Is there a way to use Html Agility Pack to get HTML nodes that were dynamically generated by Javascript. These are not visible in the output of HtmlWeb.load nor when I directly view the page's source. The only way I can even see these nodes is when I us Firefox's "View Selection Source". Is there a way to automate this? Or any suggestions to get this in my code?

Many Thanks,
Chris

New Post: XPath expression to find heading tags

bonson — Wed, 25 Jun 2008 09:30:32 GMT

The forums seems to be *very* quiet, but incase anyone is listening - I'm having very little success with htmlagilitypack as it doesnt seem to work that well. For example I pointed the sample code 'GetDocLinks' at 'http://www.w3schools.com/TAGS/tag_hn.asp' and all it returned was:

Linked urls:
/favicon.ico
/stdtheme.css
Referenced urls:

It also completely failed to pick out any other tags. SO - does anyone have recommendations for what's happning or perhaps another product with better support material.

Thanks

New Post: XPath expression to find heading tags

bonson — Tue, 24 Jun 2008 16:00:05 GMT

Hi,

I'd be grateful if someone could nudge me in the right direction here. I have just downloaded the Agility Pack and am playing, however I'm a little stuff. I want to find all "<h1>" tags, for example, on a page so I thought the best way was to adapt one of the examples.

So, I tried this line

HtmlNodeCollection

atts = _doc.DocumentNode.SelectNodes("//h1");

However it always returns null - the document definately contains heading tags and that is the correct XPath expression.

So I've missed something - but what?

Thanks for any help

JC

New Post: Proxy settings

erhard — Tue, 17 Jun 2008 05:45:14 GMT

How can I specify proxy settings for HtmlWeb or does it use the same default IE proxy settings?

New Post: HTTP Authentication

erhard — Tue, 17 Jun 2008 05:44:26 GMT

How can I specify a user name and password for the HtmlWeb class to perform http authentication on a password protected directory?

New Post: How do I use htmlweb.load with proxy ?

erann — Tue, 10 Jun 2008 11:23:32 GMT

Hi there ,
I want to use htmlweb.load with proxy credentials ,
How can I do it ?

regards,
Eran.

NEW POST: Get Text and Anchor Text?

geolast — Mon, 02 Jun 2008 11:20:42 GMT

Hello there everyone! First of all thanks for this great pack. It has really helped me a lot!

I have a couple of questions and if someone would be willing to help me it would be great!

I am doing a project in which I have to extract three things from a webpage.
1. All of the links (urls) which are inside it.
2. It's pure text (which is the page's html without the tags etc.) and
3. After i've found the links I need to go there in the html file and get 10 words before and 10 words after each one of them.

The first one isn't that hard.With: HtmlNodeCollection myanchors = htmlData.DocumentNode.SelectNodes("//a[@href]"); i can find the links. But what about the other 2 parts?

Thanks in advance
George

COMMENTED ISSUE: "End tag was not found" when using short notation for tag closing

simonm — Mon, 02 Jun 2008 06:13:59 GMT

Here is a sample HTML file to illustrate the problem:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script type="text/javascript" src="../scripts/general.js" /><title>T24</title>
</head>
<form name="compositeScreenData"><input type="hidden" name="someName" value="0" ></form>
</html>

The parsing of the HTML does not work correctly when the short notation for tag closing is used (i.e. /> instead of </script> ) :

Doesn't work: <script type="text/javascript" src="../scripts/general.js" />
Works: <script type="text/javascript" src="../scripts/general.js"></script>

When the parser is in state ParseState.PcData (used for CDATA elements like script, style, noxhtml (see HtmlNode.cs) ) then it always expects the verbose tag closing (i.e. </tag>)

How to fix:
a) Removing ElementsFlags.Add("script", HtmlElementFlag.CData); in HtmlNode.cs might be an easy workaround, but it might have some side effects.
b) Implementing a proper fix in the Parse() method
Comments: ** Comment from web user: simonm **

Note that some browsers don't support the empty notation (<script ... />) anyway.

COMMENTED ISSUE: Unclosed paragraph lost

simonm — Mon, 02 Jun 2008 06:12:10 GMT

When inputting this html:

<p>Paragraph1<p>Paragraph2</p>

The InnerHtml property of DocumentNode will contain:

<p />Paragraph 1<p>Paragraph 2</p>

This essentially means the first paragraph is lost. I don't know if this is a rule or just a suggestion of W3C, but iirc they state that a <p> with no closing tag should be implicitly closed by the next block-level tag. In other words, the second <p> should close the first in this example, which means I'd the InnerHtml to be something like this:

<p>Paragraph1</p><p>Paragraph2</p>
Comments: ** Comment from web user: simonm **

This was a deliberate choice because:

1) it's easier to implement :-)
2) its renderered properly in IE and FF (AFAIK, but someone can correct me if I am wrong)
3) it's correct for an XML or XHTML parser

Note the HTML agility pack was never designed to be compliant with W3C specs, on purpose, but to minimise the changes in the original HTML, with a final rendering as close as possible to the original HTML in known browsers.

NEW POST: Stripping harmful HTML from user input, but allowing other HTML?

frogbody — Thu, 29 May 2008 20:55:39 GMT

Cool - thanks for that.

NEW POST: html agility pack with IE automation

annddrew — Thu, 29 May 2008 20:33:54 GMT

Hi,

I just learned how to automate the internet explorer browser through Microsoft.mshtml and SHDocVw. Now my next step is to parse the html document held by the browser with agility pack. Does anybody know where to start ? Thanks

NEW POST: Stripping harmful HTML from user input, but allowing other HTML?

mg_45 — Thu, 29 May 2008 15:42:23 GMT

I needed to do the same thing, but couldn't find any example code, so here's mine - it's not perfect, but it works well enough for my purposes...

public string ScrubHTML(string html)
{
    HtmlDocument doc = new HtmlDocument();
    doc.LoadHtml(html);

    //Remove potentially harmful elements
    HtmlNodeCollection nc = doc.DocumentNode.SelectNodes("//script|//link|//iframe|//frameset|//frame|//applet|//object");
    if (nc != null)
    {
        foreach (HtmlNode node in nc)
        {
            node.ParentNode.RemoveChild(node, false);

        }
    }

    //remove hrefs to java/j/vbscript URLs
    nc = doc.DocumentNode.SelectNodes("//a[starts-with(@href, 'javascript')]|//a[starts-with(@href, 'jscript')]|//a[starts-with(@href, 'vbscript')]");
    if (nc != null)
    {

        foreach (HtmlNode node in nc)
        {
            node.SetAttributeValue("href", "protected");
        }
    }

    //remove img with refs to java/j/vbscript URLs
    nc = doc.DocumentNode.SelectNodes("//img[starts-with(@src, 'javascript')]|//img[starts-with(@src, 'jscript')]|//img[starts-with(@src, 'vbscript')]");
    if (nc != null)
    {
        foreach (HtmlNode node in nc)
        {
            node.SetAttributeValue("src", "protected");
        }
    }

    //remove on<Event> handlers from all tags
    nc = doc.DocumentNode.SelectNodes("//*[@onclick or @onmouseover or @onfocus or @onblur or @onmouseout or @ondoubleclick or @onload or @onunload]");
    if (nc != null)
    {
        foreach (HtmlNode node in nc)
        {
            node.Attributes.Remove("onFocus");
            node.Attributes.Remove("onBlur");
            node.Attributes.Remove("onClick");
            node.Attributes.Remove("onMouseOver");
            node.Attributes.Remove("onMouseOut");
            node.Attributes.Remove("onDoubleClick");
            node.Attributes.Remove("onLoad");
            node.Attributes.Remove("onUnload");
        }
    }

   return doc.DocumentNode.WriteTo();
}

NEW POST: Html Agility Pack to LINQ to XML Converter

CVertex — Wed, 28 May 2008 23:56:37 GMT

You can get the patch form from the source code tab in the patches area.

Alot of websites dont validate as XHTML because they're probably not. they're HTML, which isn't exactly XML. In addition to that, the parsers have evolved to be very lenient to malformed X/HTML. This allows browsers to open a wider range of sketchy files, but makes scraping harder without a browsers parser.

Enter, htmlagility pack.

This is a gem IMHO in the C# OSS world. It brings a very lenient html parser and offers a set of external format converters, XML being one of them.

My patch simply uses the XML converter to stream data into the linq2xml XDocument parser. Very simple.
Check out my post at,
http://vijay.screamingpens.com/archive/2008/05/26/linq-amp-lambda-part-3-html-agility-pack-to-linq.aspx

I chose to use linq because my team needed to do some scraping work tasks and weren't very profficient in XPath, but had enough linq skills to parse xml. I prefer linq instead of xpath because it's easier to read in my opinion. It may be a bit slower, but perf is rarely something I particularly care about until we stress test and performance benchmark our apps. That said, sometimes from the start we're sure performance is going to be an issue (like on embedded devices), and we take care to design for performance early and give ourselves enough time for optimization. But I digress.

-CV

NEW POST: Html Agility Pack to LINQ to XML Converter

dscruggs — Wed, 28 May 2008 21:07:40 GMT

+1

How can I get a copy of this patch?

I've noticed that a lot of web sites won't validate as XHTML. Does this fix this? If not, would setting up the HTML Agility Pack to directly support LINQ be a better idea?

NEW POST: nested lists and OptionFixNestedTags

russau — Mon, 19 May 2008 12:46:30 GMT

Hi all,

I'm loading a HTML file which looks like this:
<html>
<head>
    <ul>
        <li>a
            <ol>
                <li>b</li>
                <li>c</li>
                <li>d</li>
            </ol>
        </li>
    </ul>
</body>
</html>

After I've loaded it I look at the DocumentNode.OuterHtml, and it looks like this:
<html>
<head>
        <ul>
                <li>a
                        <ol>
                                </ol></li><li>b</li>
                                <li>c</li>
                                <li>d</li>

        </ul>

</head></html>

Had a look at the code, and noticed that loading a URL sets a property 'OptionFixNestedTags' to true, if I change this to false everything works.

Is there a problem with HtmlAgilityPack, or am I missing a setting somewhere?

Thanks,
Russ

CREATED TASK: Add class diagram

Jessynoo — Mon, 19 May 2008 04:10:38 GMT

As a piece of code documentation and a tool for further designing.

Needs custom layout for achieving any purpose