FlickrMetadataSynchr

Commented Feature: Command Line Scriptable Interface [4962]

awojtas — Wed, 07 Sep 2011 01:23:06 GMT

It would be nice if the code were refactored a little to create a command style interface in addition to the user interface. Ideally this would be in the form of PowerShell commandlets. This would open up the door to allow me to sync my entire photo collection in one go.
Comments: ** Comment from web user: awojtas **

What about instead the UI altered to have a gridview-like interface where you could map local folders to Flickr sets. These mappings would be stored as XML which would allow the mappings to be backed up and transported to other computers.

New Post: TIFF Tags

erwyn — Sun, 04 Sep 2011 15:15:24 GMT

Hi Ken, Sounds like a great contribution. I don't see the attachment. Please send me a private message (via CodePlex) and I will reply with my e-mailaddress so you can send me the code. Regards, Erwyn

New Post: TIFF Tags

kenmurrayx3 — Sat, 03 Sep 2011 12:11:30 GMT

Hi Erwyn

Please find attached your code with changes for allowing TIFF images to be synchronised. Hopefully you'll be able to use it to adapt your program to allow TIFFs to be handled along with JPGs.

Kind Regards
Ken

On 06/01/2011 17:12, erwyn wrote:

From: erwyn

Hi Ken,

The class Yorrick.FlickrMetadataSynchr.Local.LocalPicturesHelper contains the code that reads/writes metadata in the image files on your hard disk.

Regards,
Erwyn

Updated Wiki: Home

Erwyn — Fri, 26 Aug 2011 19:47:28 GMT

Blog of Erwyn van der Meer

Project Description

FlickrMetadataSynchr is a tool that allows you to synchronize metadata added to images stored on Flickr with the original versions of those images stored locally on your hard drive. If you don't have the original version on your hard drive, the application can download the missing pictures for you. The app can sync the metadata two-way on a picture-by-picture and metadata field-by-field basis.
Relevant metadata on Flickr is:

Title
Description
Tags
Author
GPS location info
Date and time at which the image was taken
Last updated date and time

The synchronization is done two-way. The embedded metadata in XMP, EXIF and IPTC sections in locally stored images can be updated with the metadata of matching pictures on Flickr. And vice versa. See also my blog entry with some background info on the metadata problem.

This is a screenshot of the latest version of the app:

Status

The project is currently at version 1.5.0.0. The application runs on both 32 and 64-bit versions of Windows XP and Vista and Windows 7, but requires the Microsoft .NET Framework 4 (Client Profile) to be installed (download here if you don't have it yet).

Download the latest release here. Source code can be found here.

How-to-install the app

Make sure you have the .NET Framework 4 installed. Only the Client Profile is needed, but the full version is fine.
Download the ZIP file with the application.
Unzip it to a folder.
Run the FlickrMetadataSynchr.exe executable.

How-to-use

If you run the application for the first time on a computer, you first have to authorize the application to access your Flickr pictures on your behalf. Click the "Authorize application button" to start the authorization process. Follow the on-screen instructions.
After you have authorized the application or have run the application before, you can connect to Flickr to fetch a list of all your photosets. The application automatically tries to connect when it starts. If this fails, for example when your Internet connection is down or if a firewall is blocking the program, you can try to connect again by using the "Connect to Flickr" button.
When you have successfully connected to Flickr, the application shows your name and Flickr avatar. You can select a photoset that you want to sync from the drop down list. You need to have at least one photoset on Flickr to be able to use the application.
After you have selected a Flickr photoset, you can select a folder on a local drive by using the "Browse for local folder" button.
If you like, you can change the synchronization settings. You can choose if subfolders of the local folder should be included and if pictures that are missing in the local folder should be downloaded from Flickr.
If you are happy with what you have selected, hit the "Sync" button and the synchronization process will start. This process is fully automatic. The app will show you what it is doing in the status area. The status area is the gray area at the bottom of the window. If anything goes wrong during the synchronization, the sync process will stop and the application will show you an error message in the status area.
The synchronization process can be cancelled at any time by hitting the "Cancel Sync" button.
If you want to see more detail about what is going on, select the "Show activity log window" option. This will open another window which shows you a log of all activity.

The application performs the following steps during the synchronization process:

Download metadata from Flickr pictures in the selected Flickr set.
Extract metadata from JPEG images in the selected local folder.
Match pictures into pairs based on Flickr id, date and time taken and/or title and filename.
Merge the metadata two-way (see details below).
Optionally download pictures from the selected Flickr set that do not have a matching picture in the local folder for users with a Flickr Pro account.

The metadata is merged from both sides. Apart from tags, if both sides contain data for a metadata field, the most recently updated side wins for that field. For tag synchronization, you can choose what behavior you want: combine tags (i.e., do not delete tags, just add them), most recently updated side wins (to enable deletion of tags) or leave tags alone. Finally the metadata for the Flickr picture, the local picture or both are updated if the synchronization procedure has determined they should be updated.

The application will embed the Flickr id of a matching picture as ImageUniqueID in the XMP and EXIF metadata of a local picture. This makes it possible to match pictures during future sync operations even if the date time taken and/or the title and filename of the picture have changed or are not unique. Usually upload tools set the title on Flickr equal to the filename. If that is the case, it is recommended to run a sync operation immediately after upload, before any metadata changes, because then the pictures can still be matched based on title/filename.

The application saves and remembers your Flickr account, the local folder, the selected Flickr set and the synchronization options when you restart the application. If you want the app to forget your Flickr account, e.g., when you want to log into another Flickr account or you are using a shared computer, use the "Forget Me" button.

The application has a simulation mode in which no real updates take place. Use this mode first, if you are not sure if the automatic synchronization will do the right thing.

The status area shows very limited information during the synchronization process. Full information about what the app is doing and detailed error information if something goes wrong is written to the so called Windows debug output. There are multiple options for viewing this information. It is written to a file called SyncActivity.log in the application directory, it can be viewed in a separate window, and finally it is viewable with the DebugView tool). This is helpful if you want to check the proposed changes in simulation mode before actually applying them.

Download of non-JPEG pictures like PNGs and GIFs is supported, but synching metadata only works for JPEG pictures.

How-to-remove the app

Click the "Forget Me" button in the running application if you are concerned about leaving behind data about your Flickr account.
Just delete the folder that contains the application.

Possible future enhancements

Show the user the match that was made by the application to give the user the opportunity to review for any matching errors.
Let the user deselect matched image pairs that he or she does not want to process.
Let the user be able to overrule the metadata merge on a picture-by-picture and field-by-field basis.

Reuse or contributions

Code adapted from Matthijs Dubbeldam's Vista Flickr Uploader
Code adapted from "TimK"'s WindowSettings class on The Code Project. See also my blog entry about remembering windows position and state in WPF.
Code adapted from Robert A. Wlodarczyk's blog.
Code contributed by Timo Proescholdt.
Code adapted from improvement sent in by Luther Miller.

Dependencies

Flickr account. For downloading pictures you need a Flickr Pro account.
Windows XP, Windows Vista, Windows 7 or Windows Server 2003/2003 R2/2008/2008 R2
.NET Framework 4, especially Windows Presentation Foundation for the User Interface and Windows Imaging Component to read and write metadata in image files.
FlickrNet as the API to access Flickr (http://www.codeplex.com/FlickrNet).

Development tools

Visual Studio 2010 Professional, Premium or Ultimate
Visual Studio Team Explorer 2010 (optional, for connecting to CodePlex source control)

Updated Release: v1.5.0.0 Robustness (Aug 26, 2011)

erwyn — Fri, 26 Aug 2011 19:39:13 GMT

This release adds extra robustness by being able to deal with a larger variety of metadata in pictures. If you encountered stability issues while syncing this is a must have release. Otherwise the upgrade is optional as no new features were added.

Released: v1.5.0.0 Robustness (Aug 26, 2011)

Fri, 26 Aug 2011 19:39:13 GMT

Released: v1.4.3.1 Bugfix (May 27, 2011)

Fri, 26 Aug 2011 19:36:47 GMT

New release to fix a crashing issue. Previous versions will crash on start up due to changed output from the Flickr API. This version deals better with unexpected data.

Updated Release: v1.4.3.1 Bugfix (May 27, 2011)

erwyn — Fri, 26 Aug 2011 19:36:47 GMT

New release to fix a crashing issue. Previous versions will crash on start up due to changed output from the Flickr API. This version deals better with unexpected data.

Source code checked in, #78902

Erwyn — Fri, 26 Aug 2011 19:17:58 GMT

Changed version number to 1.5.0.0.

Source code checked in, #78901

Erwyn — Fri, 26 Aug 2011 19:17:07 GMT

Added extra retry option to clone a picture without adding padding to the metadata. This helps with improving robustness by being able to handle a larger variety of pictures.

New Post: FlickrMetaDataSynchr crash

mwormgoor — Fri, 26 Aug 2011 19:11:00 GMT

Yup, that worked like a charm. It synced all the pictures that it skipped last time.

New Post: FlickrMetaDataSynchr crash

erwyn — Fri, 26 Aug 2011 18:43:36 GMT

Created a second preview version with the additional change. On my computer this version successfully synced your picture. Please give it a try: https://skydrive.live.com/#!/?cid=288ade681ec6e002&sc=documents&uc=1&id=288ADE681EC6E002%212445

New Post: FlickrMetaDataSynchr crash

erwyn — Fri, 26 Aug 2011 18:01:58 GMT

Close but not quite the problem. The problem is indeed with the padding. Padding is extra bytes that are added on top of the real metadata. This facilitates future in place updates (which are more efficient) by having extra legroom.

Somehow in your case adding padding to the metadata gives problems. Even when there are no tags to add. In my test of your picture I only added a short title and description on Flickr and no tags at all and I still get the same error as you encountered.

I have been able to change metadata in your file if I don't add padding. So my new approach is to first try padding (for the usual case that mostly works) and if that fails retry without adding padding. I am currently coding this new approach and will send you an updated version when finished for you to try.

New Post: FlickrMetaDataSynchr crash

mwormgoor — Fri, 26 Aug 2011 17:48:42 GMT

My guess would be that the problem is in this code:

private const uint _MetadataPaddingInBytes = 2048;
bitmapMetadata.SetQuery("/app1/ifd/PaddingSchema:Padding", _MetadataPaddingInBytes);
bitmapMetadata.SetQuery("/app1/ifd/exif/PaddingSchema:Padding", _MetadataPaddingInBytes);
bitmapMetadata.SetQuery("/xmp/PaddingSchema:Padding", _MetadataPaddingInBytes);

I know that this comes straight from the example. The problem with this code is that your adding a hardcoded 2048 bytes as padding. If the metadata that you're trying to add to the picture is more than the 2048 bytes, it will fail. In my case, I put all the tags on Flickr, so I don't have them locally. So, it's trying to add all the tags to the local image. This could potentially fail if it's more than 2048 bytes. It could easily be more than 2048 bytes, with the author info, flickr id, tags and geo info.

New Post: FlickrMetaDataSynchr crash

erwyn — Fri, 26 Aug 2011 17:09:03 GMT

My app already uses the InPlaceBitmapMetadataWriter first to try to update the metadata in the original file. That often fails. It then falls back to creating a copy of the picture with new metadata. That most often works.

The code in the app is derived from the sample in the blog post you mentioned, so it’s certainly not a stupid suggestion ;)

In the case of your picture the problem happens in the second step. I was already doing some additional investigation and I think I am on to something. I'll keep you posted.

New Post: FlickrMetaDataSynchr crash

mwormgoor — Fri, 26 Aug 2011 17:01:34 GMT

BTW, could this be the problem: http://blogs.msdn.com/b/rwlodarc/archive/2007/07/18/using-wpf-s-inplacebitmapmetadatawriter.asp.
Originally from this discussion: http://social.msdn.microsoft.com/forums/en-US/wpf/thread/8f297b11-99ba-459c-b085-1ff3ae526487.

(disclaimer: I'm not a .NET programmer - so don't blame me for stupid suggestions).

Source code checked in, #78893

Erwyn — Fri, 26 Aug 2011 16:51:09 GMT

Updated version number to 1.4.4.0. Fixed crashing issue by handling FileFormatException and skipping over files with troublesome metadata that triggers this exception.

New Post: FlickrMetaDataSynchr crash

mwormgoor — Fri, 26 Aug 2011 16:49:21 GMT

Yup, that works. I still see the occasional error, but at least the programme continues now.

Thanks for the quick response!

New Post: FlickrMetaDataSynchr crash

erwyn — Fri, 26 Aug 2011 15:14:45 GMT

Thanks for providing me a copy of the picture that will not sync. I can reproduce your issue with that picture, and with that I've been able to implement a workaround.

The sync process will now skip the picture after this particular type of error occurs and will not crash anymore. Unfortunately I have no clue as to the underlying cause in the metadata of this picture that triggers the error, so I cannot successfully sync the metadata if this error occurs.

You can download a preview of the fixed version from SkyDrive: https://skydrive.live.com/#!/?cid=288ade681ec6e002&sc=documents&id=288ADE681EC6E002%212445. Please give it a try.

New Post: FlickrMetaDataSynchr crash

erwyn — Fri, 26 Aug 2011 13:20:36 GMT

Thanks for reporting the issue including a trace of the exception. To better help me solve the problem, could you e-mail me the troublesome picture? You can send me a private message via CodePlex so you and I don't have to post our e-mailaddresses here ;)