DMOZ Data Importer

CREATED ISSUE: Could not find file 'H:\travail en cours\Perso\Henduoda\Tools\DMOZ\structure.rdf

Ussenaiah — Tue, 22 Apr 2008 12:57:34 GMT

Ouch! an error : System.IO.FileNotFoundException: Could not find file 'H:\travai
l en cours\Perso\Henduoda\Tools\DMOZ\structure.rdf'.
File name: 'H:\travail en cours\Perso\Henduoda\Tools\DMOZ\structure.rdf'
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, I
nt32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions o
ptions, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access,
FileShare share, Int32 bufferSize)
at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials)

at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofO
bjectToReturn)
at System.Xml.XmlTextReaderImpl.OpenUrlDelegate(Object xmlResolver)
at System.Threading.CompressedStack.runTryCode(Object userData)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCl
eanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.CompressedStack.Run(CompressedStack compressedStack, Cont
extCallback callback, Object state)
at System.Xml.XmlTextReaderImpl.OpenUrl()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlTextReader.Read()
at Dmoz.Importer.DmozRdfHelper.Parse()
at Dmoz.Importer.DBHelper.Process(Uri pathToRDF)
at Dmoz.Importer.Program.Main()PRESS A KEY TO QUIT.

COMMENTED ISSUE: Security Exception

pedrito68 — Wed, 19 Mar 2008 13:26:01 GMT

I got around the path problem by setting up a mapped network drive to the folder and then creating the whole folder nesting indicated by the previous error.

Now I'm seeing:

Ouch! an error : System.Security.SecurityException: Request failed.
at Dmoz.Importer.DBHelper.Process(Uri pathToRDF)
at Dmoz.Importer.Program.Main()
The action that failed was:
InheritanceDemand
The type of the first permission that failed was:
System.Security.PermissionSet
The Zone of the assembly that failed was:
InternetPRESS A KEY TO QUIT.

Any suggestions?
Comments: ** Comment from web user: pedrito68 **

The problem isn't folder security. The problem is it's not using the DMOZ_Importer.exe.config file. It retrieves the paths from settings.settings in the in the Properties folder.

The connection string used in the SQLBulkCopy constructor is not the connection string used in the NetTiers classes. This is simple enough to test: Hard-code a string into the SQLBulkCopy constructor (this is supposedly the one that comes form the DMOZ_Importer.exe.config, but it really comes from settings.settings. But when it gets to actually doing work in the NetTiers level, it's getting its connection string from yet another location (the app.config file, I believe).

As a simple test of this, as the developer, simply rename your H:\travail en cours\Perso\Henduoda\Tools\DMOZ\ folder (so that it's not available to the app) and then try using it in another directory following the instructions you provide. I strongly suspect it won't work.

COMMENTED ISSUE: Bad path?

pedrito68 — Wed, 19 Mar 2008 00:28:09 GMT

It appears that there is a path hard coded into the app (for the import.txt file), something that should be set in the config file. I get the following error regardless of the settings in the DMOZ_importer.exe.config. The path 'H:\travail en cours\Perso\Henduoda\Tools\DMOZ\Import.txt' is fixed.

Ouch! an error : System.IO.DirectoryNotFoundException: Could not find a part of
the path 'H:\travail en cours\Perso\Henduoda\Tools\DMOZ\Import.txt'.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, I
nt32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions o
ptions, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access,
FileShare share)
at System.Xml.XmlTextWriter..ctor(String filename, Encoding encoding)
at Dmoz.Importer.DBHelper.Process(Uri pathToRDF)
at Dmoz.Importer.Program.Main()PRESS A KEY TO QUIT.

Is there a solution to this problem?
Comments: ** Comment from web user: pedrito68 **

There is, in fact, a problem with the program. I'm not sure why yet, but I had the same problem as the others who posted here.

This is the section from my DMOZ_Importer.exe.config:

<DMOZ_Importer.Properties.Settings>
<setting name="DMOZ_structure_path" serializeAs="String">
<value>C:\Software Development\DMOZ_Importer\structure.rdf</value>
</setting>
<setting name="DMOZ_content_path" serializeAs="String">
<value>C:\Software Development\DMOZ_Importer\content.rdf</value>
</setting>
<setting name="OutputPath" serializeAs="String">
<value>C:\Software Development\DMOZ_Importer\DMOZ</value>
</setting>
<setting name="BatchRowsNumber" serializeAs="String">
<value>10000</value>
</setting>
</DMOZ_Importer.Properties.Settings>

But when I run it under the debugger, in DBHelper.Process(), for the line:

String FullOutputPath = Settings.Default.OutputPath;

FullOutputPath = "H:\\travail en cours\\Perso\\Henduoda\\Tools\\DMOZ"

I couldn't figure out how to get it to read the settings from the exe.config. It's clearly getting the values from the settings.settings file.

I ended up hardcoding the values in the code...

COMMENTED ISSUE: Bad path?

dpie — Thu, 21 Feb 2008 13:57:57 GMT

I modify the path at the configuration file but the problem has not been solved?
Any ideas?

COMMENTED ISSUE: Security Exception

bodzebod — Mon, 11 Feb 2008 14:12:19 GMT

Hi all, thanks for using DMOZ Importer !

In the program configuration file (in source files, it's "app.config", and in binary package, it's "dmoz_importer.exe.config"), you may locate this section :

<DMOZ_Importer.Properties.Settings>
<setting name="DMOZ_structure_path" serializeAs="String">
<value>H:\travail en cours\Perso\Henduoda\Tools\DMOZ\structure.rdf</value>
</setting>
<setting name="DMOZ_content_path" serializeAs="String">
<value>H:\travail en cours\Perso\Henduoda\Tools\DMOZ\content.rdf</value>
</setting>
<setting name="OutputPath" serializeAs="String">
<value>H:\travail en cours\Perso\Henduoda\Tools\DMOZ</value>
</setting>
<setting name="BatchRowsNumber" serializeAs="String">
<value>10000</value>
</setting>
</DMOZ_Importer.Properties.Settings>

For each setting ("DMOZ_structure_path", "DMOZ_content_path" and "OutputPath"), you should adjust the right path. The 2 first settings are the paths to the DMOZ files, and the setting "OutputPath" is the path to the folder where the program can generate its intermediate "import.txt" file, so a place where the program is allowed to read and write.

I think your security exception might be because the "Import.txt" folder is not a trusted folder for .NET. It might happen if it's a network shared folder for instance.
So there are 2 possibilities :
* you change the folder to a place located on your local drive, ensuring you have the permissions to read and write
* you add this path to the trusted paths (you can find details on how to add fulltrust rights to a remote share here : http://support.microsoft.com/kb/320268/en-us )

Hope this will work for you ! /Stephane Baudet

COMMENTED ISSUE: Bad path?

bodzebod — Mon, 11 Feb 2008 14:06:55 GMT

Hi all, thanks for using DMOZ Importer ! And sorry for my delay to reply.

Actually, I think you missed one setting. In the program configuration file (in source files, it's "app.config", and in binary package, it's "dmoz_importer.exe.config"), you may locate this section :

Hope this will work for you ! /Stephane Baudet

CREATED ISSUE: Security Exception

drgdrg — Tue, 11 Dec 2007 14:23:07 GMT

COMMENTED ISSUE: Bad path?

duckyaya — Fri, 30 Nov 2007 14:07:44 GMT

I have also met this problem, maybe a little bug.
A small trick is that I made a directory just like the default parameter : 'H:\travail en cours\Perso\Henduoda\Tools\DMOZ\Import.txt'.
so without changing the DMOZ_Importer.exe.config file, hope this will help you.

CREATED ISSUE: Bad path?

tripnosis — Tue, 20 Nov 2007 21:23:40 GMT

UPDATED RELEASE: DMOZImporter - Release 20071015 (oct. 12, 2007)

Mon, 15 Oct 2007 11:03:16 GMT

! DMOZ Data Importer
!! Release 20071015

This is the +first release+.
Basically, it only imports the DMOZ structure, i.e all the categories and their links between each other.
And it does a pretty good job of bulk importing unicode data ! Hope you'll enjoy ;)
/Stephane Baudet - BACKPACKMAN.

UPDATED RELEASE: DMOZImporter - Release 20071015 (oct. 11, 2007)

Mon, 15 Oct 2007 10:59:59 GMT

UPDATED RELEASE: DMOZImporter - Release 20071015 (oct. 10, 2007)

Mon, 15 Oct 2007 10:58:58 GMT

UPDATED RELEASE: DMOZImporter - Release 20071015

Mon, 15 Oct 2007 10:44:47 GMT

UPDATED RELEASE: DMOZImporter - Release 20071015

Mon, 15 Oct 2007 10:39:04 GMT

UPDATED RELEASE: DMOZImporter - Release 20071015

Mon, 15 Oct 2007 08:58:04 GMT

Source code checked in

bodzebod — Mon, 15 Oct 2007 08:55:54 GMT

UPDATED WIKI: Home

bodzebod — Mon, 15 Oct 2007 08:10:14 GMT

Project Description
Import the RDF data from Open Directory Project (ODP) (http://rdf.dmoz.org) straight
into a Microsoft SQL Server 2005 database.

DMOZ Data Importer License
--------------------------
DMOZ Data Importer is released under the GNU LESSER GENERAL PUBLIC LICENSE. You can always find the latest GNU LGPL license at:
http://www.gnu.org/licenses/lgpl.html

System Requirements for runtime only
------------------------------------
SQL Server Management Studio/SQL Server Management Studio Express
Microsoft SQL Server2005+ and about 4 Gb of diskspace for the database and the temporary import file
Microsoft .Net 2.0
ODP RDF dump files

System Requirements for developers (source code)
------------------------------------------------
Visual Studio 2005
SQL Server Management Studio/SQL Server Management Studio Express
Microsoft SQL Server2005+ and about 4 Gb of diskspace for the database and the temporary import file
Microsoft .Net 2.0
ODP RDF dump files

-----------------------------
Installing & Getting Started
-----------------------------
1. First download the ODP RDF dump files at http://rdf.dmoz.org/.
There are 2 files compressed with GZip :
Structure.rdf.u8.gz (68 Mb compressed, 664 Mb uncompressed), and content.rdf.u8.gz (317 Mb compressed).

Right now, only the structure is imported, so no need to download the content file.
Uncompress the .gz file.

2. In DMOZ Data Importer directory, edit the file configuration file "DMOZ_Importer.exe.config" using a text editor.

Locate the <DMOZ_Importer.Properties.Settings> section at the end of the file.
Modify the settings "DMOZstructurepath" and "OutputPath" which respectively contains the full path to the structure file,
and the path to the folder that will contain the intermediate import file (Import.txt).

Locate the line "<add name="netTiersConnectionString" ..." and modify the connection string with your SQL Server instance name
as the data source.

Save the file and close the editor.

3. Open the SQL Server Management Studio and open the file "Script DMOZ Database.sql". Execute it to create the database "dmoz".
Then open the file "Data for DMOZ Database.sql" and execute it.

4. Now, you're ready to go! Just launch the executable "DMOZ_Importer.exe" and wait a little.

On October 6th, 2007, there are 738165 topics (table "Category"), and 2890983 resources of different types (narrow, editor,
altlang, newsgroup, related, symbolic)(the resources types are in the table "resourcetype").

From altlang elements, I extracted too 2954 referenced languages (table "Language") but there are many languages which have
their names described in different languages. For example, "swedish" appears also as "suédois" in french, "ruotsi" in finnish,
or "svensk" in swedish.

All of these information takes about 40 minutes to be processed, optimization process included, on a dual-core laptop
having 2Gb of memory and SQL Server 2005 Express SP2.

--------------------------
Description of the process
--------------------------

First of all, the program parses the RDF file to create a more adequat XML import file dubbed as "Import.txt".
The challenge here was about being able to process this big file as quick as possible without having to load it
entirely into memory.

So I used the .NET class XmlTextReader which acts like a forward-only cursor on the file.

Why do I create an XML import file and not a simple plain-text file ? Well, I first used a plain text file to bulk load it,
and found some bugs from SQL Server 2005 bulk functions that deals with unicode encoded files.

As I said in the previous chapter, there are a lot of different languages in these data, and so a lot of different characters
sets such as russian, chinese, japanese, arabian and so on.

It appears that the bulk load functionalities of SQL Server 2005 are older than the unicode standard, and they seem to need
a refresh update.
There is an article in MS Knowledge base about this problem : http://support.microsoft.com/?scid=kb%3Ben-us%3B942660&x=8&y=12
A patch is scheduled but not yet available.

So I decided to perform the bulk import using the .NET 2.0 new class : SqlBulkCopy (http://support.microsoft.com/?scid=kb%3Ben-us%3B942660&x=8&y=12).
This class needs to deal with an IDataReader interface as input. Usually, it's described to be used with a SqlDataReader instance.
So I implemented an XmlTextReader inherited class which implements the IDataReader interface. This allows me to have an efficient bulk import
with the advantage of the XmlTextReader to read my import file.

Before importing the data, I execute the stored procedure "importtruncateMainTables" which will remove data from all the tables,
except the table "resourcetype" which is a static referential.
This procedure also switches the database recovery model to "BULK_LOGGED" in order to optimize the bulk import logs.

Then the data are imported to the table "Import", using the SqlBulkCopy class used with the XmlImportDataReader class.

Once the data are imported, the stored procedure "ImportPostProcess" is executed.
This procedure will put the data from the table "Import" to the tables "Language", "LocalizedName" that will keep all the distinct labels associated
to their language ("english" by default if not specified), "Category" for topics, "Resource" for the other resources,
"resourcepercategory" that is the link between "category" and "resource".
Then it calculates the parent/son relation between topics and fills the "parentid" column in the table "Category".

Eventually, it optimizes the indexes, rebuild them if necessary, updates the statistics, shrinks the files after having emptied the "Import" table
and sets the recovery model to "full".
So you can start working with a "ready to go" optimized unicode database!

More Information
--------------------
There is a pretty good data access model object generated by NetTiers templates (http://www.nettiers.com) and CodeSmith.
You can use it by referencing the assemblies dmoz.dll, dmoz.Data.dll and dmoz.Data.Sqlclient.dll. Then use the namespaces dmoz and dmoz.Data.
For more information about using the NetTiers code, please consult the sections "Getting Started", "Database and SQL", "Entity Layer",
"Data Layer" in the NetTiers documentation at http://docs.nettiers.com .

You may also have a look at the database schema in the file "DMOZ database diagram.jpg".