Html Agility Pack

New Post: Where is FormsProcessor / AddOns

kersny — Fri, 03 Jul 2009 21:07:48 GMT

I recently had this same problem and was able to finally resolve it today, by creatively using Google's Cache.

Here is the original article.

And Here is the direct link.

Hope I could help!

Reviewed: 1.3.0.0 Production (Jul 02, 2009)

nilang — Thu, 02 Jul 2009 21:14:34 GMT

Rated 5 Stars (out of 5) - Thanks a lot. Just awesome tool, saved my couple of days !!!

Reviewed: 1.3.0.0 Production (Jul 02, 2009)

nilang — Thu, 02 Jul 2009 21:14:00 GMT

Rated 5 Stars (out of 5) - Just awesome tools, saved my couple of days :)

Reviewed: 1.3.0.0 Production (jul 02, 2009)

gustavoali — Thu, 02 Jul 2009 10:52:51 GMT

Rated 5 Stars (out of 5) - Genial thanks boy!!!

New Post: InnerText problems

DarthObiwan — Tue, 30 Jun 2009 01:06:16 GMT

There should be #text node after the select node inside the td node's children

<HTML>
<HEAD>

</HEAD>
<BODY BGCOLOR="#FFFFFF" TOPMARGIN="0" LEFTMARGIN="0" MARGINWIDTH="0" MARGINHEIGHT="0" TEXT="#000000" onLoad="initPopup()" onUnload="exitPopup()">
<table><tr><td>
<select>
<option>option1</option>
</select>
text
</td>
    </tr>
    </table>
</BODY>
</HTML>

I found it at

doc.DocumentNode.ChildNodes[2].ChildNodes["body"].ChildNodes["table"].ChildNodes["tr"].ChildNodes["td"].ChildNodes["select"].NextSibling.InnerText

Any text found within a tag is contained in a #text node. This includes the newlines before/after tags. It does make it a bit hard

New Post: InnerText problems

lehel — Tue, 30 Jun 2009 00:26:56 GMT

Not sure if this is a bug or feature.

I'm trying to parse for text within an HTML page with a form and when I use the InnerText property for let's say:

<td>
<select>
<option>option1</option>
</select>
text
</td>

if the node is select the InnerText property returns 'option1'. or the td node will return option1text. What I'm trying to parse is just for 'text' for instance.

Is there a way to make InnerText property not fetch the innertext of nodes?

New Post: Selecting a specific bolded text

DarthObiwan — Mon, 29 Jun 2009 20:42:45 GMT

The Descendants() method should be added to the HtmlNode.cs file.

If you don't want to edit the html agility pack source code, you can add an extension method.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace HtmlAgilityPack
{
    public static class Extensions
    {
        public static IEnumerable<HtmlNode> Descendants(this HtmlNode theNode)
        {
            List<HtmlNode> list = new List<HtmlNode>();

            if (theNode.HasChildNodes)
                list.AddRange(theNode.ChildNodes);

            foreach (var node in theNode.ChildNodes)
                if (node.HasChildNodes)
                    list.AddRange(node.Descendants());

            return list;
        }
    }
}

Though all of this may need the HtmlCollection.cs to be modified. Here's my copy (I'm hoping to get these changes added to Html Agility Pack once I get get in touch with simonm)

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;

namespace HtmlAgilityPack
{
    /// <summary>
    /// Represents a combined list and collection of HTML nodes.
    /// </summary>
    public class HtmlNodeCollection : IList<HtmlNode>
    {
        private List<HtmlNode> items = new List<HtmlNode>();
        private HtmlNode _parentnode;

        public HtmlNodeCollection(HtmlNode parentnode)
        {
            _parentnode = parentnode; // may be null
        }

        /// <summary>
        /// Gets the number of elements actually contained in the list.
        /// </summary>
        public int Count
        {
            get
            {
                return items.Count;
            }
        }

        public void Clear()
        {
            foreach (HtmlNode node in items)
            {
                node._parentnode = null;
                node._nextnode = null;
                node._prevnode = null;
            }
            items.Clear();
        }

        public void RemoveAt(int index)
        {
            HtmlNode next = null;
            HtmlNode prev = null;
            HtmlNode oldnode = (HtmlNode)items[index];

            if (index > 0)
            {
                prev = (HtmlNode)items[index - 1];
            }

            if (index < (items.Count - 1))
            {
                next = (HtmlNode)items[index + 1];
            }

            items.RemoveAt(index);

            if (prev != null)
            {
                if (next == prev)
                {
                    throw new InvalidProgramException("Unexpected error.");
                }
                prev._nextnode = next;
            }

            if (next != null)
            {
                next._prevnode = prev;
            }

            oldnode._prevnode = null;
            oldnode._nextnode = null;
            oldnode._parentnode = null;
            
        }

        public void Replace(int index, HtmlNode node)
        {
            HtmlNode next = null;
            HtmlNode prev = null;
            HtmlNode oldnode = (HtmlNode)items[index];

            if (index > 0)
            {
                prev = (HtmlNode)items[index - 1];
            }

            if (index < (items.Count - 1))
            {
                next = (HtmlNode)items[index + 1];
            }

            items[index] = node;

            if (prev != null)
            {
                if (node == prev)
                {
                    throw new InvalidProgramException("Unexpected error.");
                }
                prev._nextnode = node;
            }

            if (next != null)
            {
                next._prevnode = node;
            }

            node._prevnode = prev;
            if (next == node)
            {
                throw new InvalidProgramException("Unexpected error.");
            }
            node._nextnode = next;
            node._parentnode = _parentnode;

            oldnode._prevnode = null;
            oldnode._nextnode = null;
            oldnode._parentnode = null;
        }

        public void Insert(int index, HtmlNode node)
        {
            HtmlNode next = null;
            HtmlNode prev = null;

            if (index > 0)
            {
                prev = (HtmlNode)items[index - 1];
            }

            if (index < items.Count)
            {
                next = (HtmlNode)items[index];
            }

            items.Insert(index, node);

            if (prev != null)
            {
                if (node == prev)
                {
                    throw new InvalidProgramException("Unexpected error.");
                }
                prev._nextnode = node;
            }

            if (next != null)
            {
                next._prevnode = node;
            }

            node._prevnode = prev;

            if (next == node)
            {
                throw new InvalidProgramException("Unexpected error.");
            }

            node._nextnode = next;
            node._parentnode = _parentnode;
        }

        public void Append(HtmlNode node)
        {
            HtmlNode last = null;
            if (items.Count > 0)
            {
                last = (HtmlNode)items[items.Count - 1];
            }

            items.Add(node);
            node._prevnode = last;
            node._nextnode = null;
            node._parentnode = _parentnode;
            if (last != null)
            {
                if (last == node)
                {
                    throw new InvalidProgramException("Unexpected error.");
                }
                last._nextnode = node;
            }
        }

        public void Prepend(HtmlNode node)
        {
            HtmlNode first = null;
            if (items.Count > 0)
            {
                first = (HtmlNode)items[0];
            }

            items.Insert(0, node);

            if (node == first)
            {
                throw new InvalidProgramException("Unexpected error.");
            }
            node._nextnode = first;
            node._prevnode = null;
            node._parentnode = _parentnode;
            if (first != null)
            {
                first._prevnode = node;
            }
        }

        public void Add(HtmlNode node)
        {
            items.Add(node);
        }

        /// <summary>
        /// Gets the node at the specified index.
        /// </summary>
        public HtmlNode this[int index]
        {
            get
            {
                return items[index] as HtmlNode;
            }
            set
            {
                items[index] = value;
            }
        }
      
        public int GetNodeIndex(HtmlNode node)
        {
            // TODO: should we rewrite this? what would be the key of a node?
            for (int i = 0; i < items.Count; i++)
            {
                if (node == ((HtmlNode)items[i]))
                {
                    return i;
                }
            }
            return -1;
        }

        /// <summary>
        /// Gets a given node from the list.
        /// </summary>
        public int this[HtmlNode node]
        {
            get
            {
                int index = GetNodeIndex(node);
                if (index == -1)
                {
                    throw new ArgumentOutOfRangeException("node", "Node \"" + node.CloneNode(false).OuterHtml + "\" was not found in the collection");
                }
                return index;
            }
           
        }
        public HtmlNode this[string name]
        {
            get
            {
                return items.SingleOrDefault(x=>x.Name.Equals(name.ToLower()));
            }
        }
      
        public int IndexOf(HtmlNode item)
        {
            return items.IndexOf(item);
        }

 
        public bool Contains(HtmlNode item)
        {
           return items.Contains(item);
        }

        public void CopyTo(HtmlNode[] array, int arrayIndex)
        {
            items.CopyTo(array, arrayIndex);
        }

        public bool IsReadOnly
        {
            get { return false; }
        }

        public bool Remove(HtmlNode item)
        {
            int i = items.IndexOf(item);
             RemoveAt(i);
             return true;
        }
        public bool Remove(int i)
        {
            RemoveAt(i);
            return true;
        }
        IEnumerator<HtmlNode> IEnumerable<HtmlNode>.GetEnumerator()
        {
            return items.GetEnumerator();
        }


        IEnumerator IEnumerable.GetEnumerator()
        {
            return items.GetEnumerator();
        }

        public HtmlNode FindFirst(string name)
        {
            return FindFirst(this, name);
        }

        public static HtmlNode FindFirst(HtmlNodeCollection items, string name)
        {
            foreach (var node in items)
            {
                if (node.Name.ToLower().Contains(name))
                    return node;
                if (node.HasChildNodes)
                {
                    var returnNode = FindFirst(node.ChildNodes, name);
                    if (returnNode != null)
                        return returnNode;
                }
            }
            return null;
        }

    }

}

New Post: Selecting a specific bolded text

rcrosenb — Mon, 29 Jun 2009 20:34:21 GMT

I'm getting errors on HasChildNodes, ChildNodes, and the call to the Descendant method. Any help on how to solve these errors would be greatly appreciated.

  public IEnumerable<HtmlNode> Descendants()
        {
            List<HtmlNode> list = new List<HtmlNode>();

            if (HasChildNodes)
                list.AddRange(ChildNodes);

            foreach (HtmlNode node in ChildNodes)
                if (node.HasChildNodes)
                    list.AddRange(node.Descendants());

            return list;
        }

New Post: Selecting a specific bolded text

DarthObiwan — Mon, 29 Jun 2009 20:00:19 GMT

list would be

IEnumerable<HtmlNode> list = document.DocumentNode.Descendants();

That is doing a LINQ statement to grab every node where tag name is b, skip the first 2 and then grab the next three. You'll need to be compiling in .NET 3.5 and have included System.Linq.

Below is basically what that one line does

int count = 0;
List<HtmlNode> boldTags = new List<HtmlNode>
foreach(HtmlNode node in list)
{
   if(node.Name.Equals("b"))
   {
      count++;
      if(count>6)
        break;
 
      if(count>=3 && count<=6)
        boldTags.Add(node);
      
   }
}

New Post: Selecting a specific bolded text

rcrosenb — Mon, 29 Jun 2009 19:49:58 GMT

I'm sorry to bother you Darth, but I'm a beginner at the whole HTML Agility Pack thing. I was wondering if you could be a little more specific on how to do the second step of your explanation.

"After that do a list.Where(x=>x.Name.Equals("b")).Skip(2).Take(3); (note code may not be exact but damn close)."

Thanks

New Post: removing string formatting

rcrosenb — Mon, 29 Jun 2009 19:36:48 GMT

That worked beautifully sir. Thanks for the help.

New Post: removing string formatting

DarthObiwan — Mon, 29 Jun 2009 19:19:49 GMT

var unformated = node.InnerText.Replace(Environment.Newline," ");

This will remove all \n\r combinations in the string. To do them individually just chain the replace node.InnerText.Replace("\n"," ").Replace("\r"," "); This will give you double spaces where they were together.

var unformated = node.InnerText.Replace(Environment.Newline," ").Replace("\n"," ").Replace("\r"," ");

Note: this is not the most performant way to do it, if you are processing lots of text (1meg+) it will be slow.

If you are resaving it and HTML Agility Pack is adding the newlines back in, then it is probably making a new #text node for each one and you'll need to modify the save routine.

New Post: removing string formatting

rcrosenb — Mon, 29 Jun 2009 18:46:54 GMT

I'm trying to figure out how to remove the "\n" and "\r" characters in a string that is pulled off a web page. I would greatly appreciate any help on this matter.

New Post: Where is FormsProcessor / AddOns

DonMcNamara — Mon, 29 Jun 2009 16:23:54 GMT

Hi,

I'm new to the HTML agility pack and I'm looking for a way to use the it with HTML pages that contain forms. I have found a couple mentions of an add on library that has a FormsProcessor (HtmlAgilityPack.AddOns.FormProcessor) but I haven't been able to find it anywhere. Can someone point me in the right direction?

Thanks,

Don

New Post: Selecting a specific bolded text

Darthobiwan — Mon, 29 Jun 2009 15:50:45 GMT

build a normalized list of all the nodes (recursive foreach building a List<HtmlNode>)

After that do a list.Where(x=>x.Name.Equals("b")).Skip(2).Take(3); (note code may not be exact but damn close).

Or if you're interested in performance over maintainability you can keep track of the bold tags as you find them while recursing over the nodes.

If the text is bold via CSS then there isn't a way to do it without writing a CSS rendering engine. If strong and b are being used then you'll have to run a few queries.

The normalizing of the list will be a bit difficult in the current version of Html Agility Pack. Here's the code I use after I converted the HtmlCollection to implement IList<T> . This is a method on the HtmlNode object

        public IEnumerable<HtmlNode> Descendants()
        {
            List<HtmlNode> list = new List<HtmlNode>();

            if (HasChildNodes)
                list.AddRange(ChildNodes);

            foreach (HtmlNode node in ChildNodes)
                if (node.HasChildNodes)
                    list.AddRange(node.Descendants());

            return list;
        }

New Post: Selecting a specific bolded text

rcrosenb — Mon, 29 Jun 2009 15:35:09 GMT

I have an application which I want to read a HTML page and store the third, fourth, and fifth occurrences of bolded text in a string. Does anyone have a good idea how to do this?

New Post: Selecting Input elements

lehel — Sat, 27 Jun 2009 01:42:54 GMT

Maybe I'm still too novice to XPath to figure this out, but I'm having trouble. Not sure if this is a bug or what.

I have a webpage I want to parse through that has two forms on it. I want to be able to use HtmlAgilityPack to select all the inputs, selects, etc for each form in my app.

When I do SelectNodes("//form[1]/input") I get a null.

Also, if I do SelectNodes("//form[1]") I get a Collection of my two forms.

Subsequently, I commented out the Empty flag for the form element, but it didn't help.

New Post: parsing form tags

lehel — Fri, 26 Jun 2009 22:52:19 GMT

I have a question on this. I'm wondering why the form nodes cannot contain the inputs as child nodes since the inputs are always going to be children, even if the form overlaps other html.

I didn't understand by what was meant here:

[QUOTE] It will force the form to be closed properly.[/QUOTE] for parsing out the form element.

My problem is that I want to process pages with more than one form and need to differentiate between inputs as to which form they belong to. What is my best course of action?

Commented Issue: Odd series of '--' produce not well-formed xml output

justingrant — Wed, 24 Jun 2009 14:33:05 GMT

When input html contains comment with long odd series of minuses '-', library produces not well-formed xml output.
HTML fragment:

Produced output:

To fix it I changed HtmlNode.GetXmlComment() function body:
string s = comment.Comment.Substring(4, s.Length-7);
while(s.Contains("--"))
s = s.Replace("--", " - -");
return s;
Comments: ** Comment from web user: justingrant **

Unfortunately, the W3C XML spec says that double-dashes in XML comments are not allowed. See http://social.msdn.microsoft.com/forums/en-US/xmlandnetfx/thread/4593258d-7445-4dcc-ad24-72cc5c3b530f/ for more info.

So there's not much that the HtmlAgilityPack can do to fix this- instead of changing the HTML library, you probably want to change the underlying source HTML instead.

Reviewed: 1.3.0.0 Production (giu 24, 2009)

GiorgioGatta — Wed, 24 Jun 2009 14:15:20 GMT

Rated 5 Stars (out of 5) - Excellent work! Great man, very very good