DuplicateFinder

Created Issue: Redirect output

flavor_18 — Wed, 10 Sep 2008 15:16:27 GMT

Another nice feature to have is the ability to redirect the output to a more readable file format.

Created Issue: Ability to restrict by file suffix

flavor_18 — Wed, 10 Sep 2008 15:13:50 GMT

It would be nice to be able to restrict the search by suffix as well.

For example: -s ".designer.cs"

Created Issue: Performance - Dupfinder is slow on lots of files

AnthonySteele — Fri, 08 Aug 2008 10:10:13 GMT

The program takes a long time to run for large numbers of input files. The increase in run time is more than linear, though I haven't managed to characterise it as O(N^2) or what.
I don't have any ideas for speedups on large input datasets.
A workaround would be to partition the checks performed - compare data layer code only to other datalayer code, Ui to Ui, etc.

Updated Wiki: Home

AnthonySteele — Sat, 12 Jul 2008 14:25:03 GMT

Project Description
Duplicate Finder is a simple utility to look though files and find identical lines, which indicates duplicate code or cut-and-paste coding. It is written in C# using some .net 2.0 features. It is aimed at detecting duplicate statements in source code, but uses simple text comparison techniques that should work on other kinds of text files.

I wrote this program for people like myself: C# programmers with an interest in code quality and automated tools to find code quality problems. However you may find other uses for it.

The source code program consists of:
DuplicateFinderLib - the engine for finding duplicates
DuplicateFinder.exe - the command-line tool that uses the engine
DuplicateFinder.Tasks - the MSBuild task for the engine. This allows integration with C# automated builds.
And unit test cases

DuplicateFinder 1.4 is out.

Usage of the command-line tool is as follows, eg:

>DupFinder.exe -t4 test5*.txt
Processing in C:\Code\DuplicateFinder\TestData
2 files read
Duplicate of length 5 at:
 Line 2-6 in C:\Code\DuplicateFinder\TestData\Test5Lines1.txt
 Line 2-6 in C:\Code\DuplicateFinder\TestData\Test5Lines2.txt
1 duplicate found

A more realistic example for C# code, looking through all files in the source tree, for duplicates of 9 lines or more, excluding the generated files called AssemblyInfo.cs:

> C:\Code\DuplicateFinder>DupFinder.exe -t9 -r -eAssemblyInfo.cs *.cs
 
Processing in C:\Code\DuplicateFinder
11 files read
Duplicate of length 11 at:
 Line 1-11 in C:\Code\DuplicateFinder\TestLibrary\TestAllFiles.cs
 Line 1-11 in C:\Code\DuplicateFinder\TestLibrary\TestFiles.cs
 Line 1-11 in C:\Code\DuplicateFinder\TestLibrary\TestProgramFile.cs
1 duplicate found

The duplicate finder has found that the test cases have the same using and namespace lines at the top.

Duplicate Finder Commandline arguments explained
The MSBuild Task

RELEASED: DuplicateFinder 1.4 (Jul 12, 2008)

Sat, 12 Jul 2008 14:23:59 GMT

No Major changes from version 1.3
There is 1 bugfix in the MSBuild tasks
And several code cleanups as prompled by the tools Fxcop, StyleCop and resharper.

UPDATED RELEASE: DuplicateFinder 1.4 (Jul 12, 2008)

Sat, 12 Jul 2008 14:23:59 GMT

No Major changes from version 1.3
There is 1 bugfix in the MSBuild tasks
And several code cleanups as prompled by the tools Fxcop, StyleCop and resharper.

Source code checked in, #19468

AnthonySteele — Sat, 12 Jul 2008 14:17:25 GMT

Ready fro V 1.4

Source code checked in, #19467

AnthonySteele — Sat, 12 Jul 2008 14:17:09 GMT

ready for v 1.4

Source code checked in, #18971

AnthonySteele — Sun, 06 Jul 2008 19:33:14 GMT

version numbers at 1.4

Source code checked in, #18709

AnthonySteele — Wed, 18 Jun 2008 11:05:34 GMT

Refactorings as recommended by resharper

Source code checked in, #18639

AnthonySteele — Fri, 13 Jun 2008 22:14:06 GMT

fixed some warnings from fxcop

Source code checked in

AnthonySteele — Mon, 02 Jun 2008 14:30:16 GMT

StyleCop compliance is now complete

Source code checked in

AnthonySteele — Mon, 02 Jun 2008 09:55:47 GMT

More styleCop compliance.

Source code checked in

AnthonySteele — Sun, 01 Jun 2008 18:30:14 GMT

more stylecop compliance work

Source code checked in

AnthonySteele — Sat, 31 May 2008 11:13:40 GMT

ms build task - syle cop compliance

Source code checked in

AnthonySteele — Sat, 31 May 2008 10:49:55 GMT

stylecop compliance in main dupfinder engine lib

Source code checked in

AnthonySteele — Sun, 04 May 2008 15:51:26 GMT

UPDATED WIKI: MSBuild Task

AnthonySteele — Sun, 04 May 2008 15:48:09 GMT

In order to use the MSBuild tasks for DuplicateFinder, you will need the libraries DuplicateFinderLib.dll and DuplicateFinder.Tasks.dll.
The simple way to use them is to place them in your $(MSBuildExtensionsPath), which is typically C:\Program Files\MSBuild

Then you can use it in MSBuild scripts, as in this simple example:

<Project DefaultTargets="RunTest"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    
  <UsingTask TaskName="DuplicateFinder" 
	AssemblyFile="$(MSBuildExtensionsPath)\DuplicateFinder.Tasks.dll" />
 
  <ItemGroup>
    <TestFiles Include="..\..\TestData\*.txt"/>
  </ItemGroup>
 
  <Target Name="RunTest">
    <DuplicateFinder Files="@(TestFiles)" />
  </Target>
</Project>

Parameters to the DuplicateFinder task are as follows:

Files: The files to inspect. Required. Note that the ExcludedFiles option is not present, since an MSBuild file group can both include and exclude files.

DuplicateThreshold: As per the commandline arguments.

FocusFile: As per the command line arguments.

FirstLineMinWidth: As per the command line arguments.

ExcludedLinePrefixes: as per the commandline arguments. Multiple prefixes can be specified on multiple lines inside the quotes as per the example below. These lines are trimmed for leading and trailing white space so indentation is unimportant.

CountThresholdForError: The task will fail if this number or more duplicates are found.

LengthThresholdForError: The task will fail if this length or longer duplicates are found.

CountThresholdForError and LengthThresholdForError work together. e.g. if CountThresholdForError =3 and LengthThresholdForError = 10, the task will fail on three or more duplicates of length 10 lines or more, but will not fail on two duplicates of length 12, or on 4 duplicates of length 9.

CountThresholdForError2 and LengthThresholdForError2 are optional, and set a second threshold for error. Likewise CountThresholdForError3 and LengthThresholdForError3.

e.g.

		CountThresholdForError="10"
		LengthThresholdForError="6"
		CountThresholdForError2="1"
		LengthThresholdForError2="16"

This will fail the build on 10 more more duplicates of length 6 or more, or on any duplicates of length 16 or more.

Here is a more complex example, which uses all the optional parameters:

<Project DefaultTargets="RunTest"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    
  <UsingTask TaskName="DuplicateFinder" 
	AssemblyFile="$(MSBuildExtensionsPath)\DuplicateFinder.Tasks.dll" />
 
  <ItemGroup>
    <TestFiles 
	  Include="..\..\**\*.cs"  
	  Exclude="..\..\**\AssemblyInfo.cs"  
	/>
  </ItemGroup>
  <Target Name="RunTest">
    <DuplicateFinder 
      Files="@(TestFiles)"
      DuplicateThreshold="3"
      FocusFile="Text1.txt"
      FirstLineMinWidth="5"
      CountThresholdForError="3"
      LengthThresholdForError="10"
      CountThresholdForError2="2"
      LengthThresholdForError2="20"
      CountThresholdForError3="1"
      LengthThresholdForError3="30"	
      ExcludedLinePrefixes="using"
     />
  </Target>
</Project>

Here is an example with multiple line prefix exclusions specified on multiple lines:

<DuplicateFinder 
    ExcludedLinePrefixes="
    using
    return
    private" 
    Files="@(TestFiles)" />

UPDATED WIKI: MSBuild Task

AnthonySteele — Sun, 04 May 2008 15:46:16 GMT

<Project DefaultTargets="RunTest"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    
  <UsingTask TaskName="DuplicateFinder" 
	AssemblyFile="$(MSBuildExtensionsPath)\DuplicateFinder.Tasks.dll" />
 
  <ItemGroup>
    <TestFiles Include="..\..\TestData\*.txt"/>
  </ItemGroup>
 
  <Target Name="RunTest">
    <DuplicateFinder Files="@(TestFiles)" />
  </Target>
</Project>

		CountThresholdForError="10"
		LengthThresholdForError="6"
		CountThresholdForError2="1"
		LengthThresholdForError2="16"

This will fail the build on 10 more more duplicates of length 6 or more, or on any duplicates of length 16 or more.

Here is a more complex example, which uses all the optional parameters:

<Project DefaultTargets="RunTest"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    
  <UsingTask TaskName="DuplicateFinder" 
	AssemblyFile="$(MSBuildExtensionsPath)\DuplicateFinder.Tasks.dll" />
 
  <ItemGroup>
    <TestFiles 
	  Include="..\..\**\*.cs"  
	  Exclude="..\..\**\AssemblyInfo.cs"  
	/>
  </ItemGroup>
  <Target Name="RunTest">
    <DuplicateFinder 
      Files="@(TestFiles)"
      DuplicateThreshold="3"
      FocusFile="Text1.txt"
      FirstLineMinWidth="5"
      CountThresholdForError="3"
      LengthThresholdForError="10"	
      ExcludedLinePrefixes="using"
     />
  </Target>
</Project>

Here is an example with multiple line prefix exclusions specified on multiple lines:

<DuplicateFinder 
    ExcludedLinePrefixes="
    using
    return
    private" 
    Files="@(TestFiles)" />

UPDATED WIKI: MSBuild Task

AnthonySteele — Sun, 04 May 2008 15:44:55 GMT

<Project DefaultTargets="RunTest"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    
  <UsingTask TaskName="DuplicateFinder" 
	AssemblyFile="$(MSBuildExtensionsPath)\DuplicateFinder.Tasks.dll" />
 
  <ItemGroup>
    <TestFiles Include="..\..\TestData\*.txt"/>
  </ItemGroup>
 
  <Target Name="RunTest">
    <DuplicateFinder Files="@(TestFiles)" />
  </Target>
</Project>

		CountThresholdForError="10"
		LengthThresholdForError="6"
		CountThresholdForError2="1"
		LengthThresholdForError2="16"

This will fail the build on 10 more more duplicates of length 6 or more. or on any duplicates of length 16 or more.
Here is a more complex example, which uses all the optional parameters:

<Project DefaultTargets="RunTest"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    
  <UsingTask TaskName="DuplicateFinder" 
	AssemblyFile="$(MSBuildExtensionsPath)\DuplicateFinder.Tasks.dll" />
 
  <ItemGroup>
    <TestFiles 
	  Include="..\..\**\*.cs"  
	  Exclude="..\..\**\AssemblyInfo.cs"  
	/>
  </ItemGroup>
  <Target Name="RunTest">
    <DuplicateFinder 
      Files="@(TestFiles)"
      DuplicateThreshold="3"
      FocusFile="Text1.txt"
      FirstLineMinWidth="5"
      CountThresholdForError="3"
      LengthThresholdForError="10"	
      ExcludedLinePrefixes="using"
     />
  </Target>
</Project>

Here is an example with multiple line prefix exclusions specified on multiple lines:

<DuplicateFinder 
    ExcludedLinePrefixes="
    using
    return
    private" 
    Files="@(TestFiles)" />