A photo transcription benchmark for AI

Sometimes I get carried away with a tangential project and then right when I finish it I get bored with it and forget to tie a bow on it. This blog suffers from that tendency. There are a few things over the years I should have probably written about, but I never got to it. You can see that from my last post ending on somewhat of a cliffhanger. Maybe I’ll get to that later.

In the meantime, my current tangent in my genealogical journey is photo transcription. I have scanned over 10,000 photos and documents for my family and while I started manually going through them and transcribing them, I have kids and a job. I wanted to see if there was a faster and better way. And there is. For a price.

AI has been very controversial in genealogical research. It does have a tendency to hallucinate and there is a lot of inconsistency in model performance. Even within the same model over time. But when it works, it is kind of like magic. I saw potential and I started to experiment with AI generated captioning and transcription of my photo library.

In my effort to manually transcribe and caption all of my photos and documents, I realized that the reason I was going down this path wasn’t just for the sake of transcription, but so I could better search and compile the photos in the future. To that end, 100% perfect wasn’t needed; false positives are better than false negatives. If I search for my ancestor and 5% of the photos that come back are not of my ancestor, that is better to me than missing 5% of them. Honestly, given that some of my photos are 100+ years old with questionable cursive notes on them, even my manual efforts were not going to be perfect. So going into this project, I was willing to accept some level of imperfection in exchange for fast, searchable results.

In the end (at least so far), I was impressed with the results and I hope some of you out there find this project useful also.

The setup

I started this project as a Lightroom plugin where I could select photos, send them to an LLM and have it automatically tag and add metadata to the photos, but I hate being locked into vendors, so the whole backend was rebuilt out as a python library with Lightroom just as an interface. It turned out this was a better model anyway because Lightroom is very limited in how it manages metadata whereas with Python, (through ExifTool), I could update anything.

Side Note: I decided for my work I wanted the metadata to live on the photos themselves and not in sidecars or in a database. My reasoning is that the information about a thing (in this case a photo or document) should live as close to the thing as possible. I have lost xmp sidecars in the past. I have had corrupted photo databases when I have had to reinstall software. Keeping all the metadata with the files is part of a larger archive philosophy I have been working on and hopefully that will be my next post.

For models, I started with ChatGPT, but that quickly grew to Anthropic, Google and models available on OpenRouter. I tested 25 models in total. It was probably overkill, but there is a lot of buzz online around all the different models that popup every week, so I wanted to be thorough. I curated a 1,000+ line prompt to get the most consistent results across models and photo sets. And built the python framework around it to intelligently work with multi-page documents, photo variations, fronts/backs, negatives etc.

I then hand picked 43 files (26 unique “artifacts”) to run the test with. I tried to capture a wide range of artifact types. I have a multi-page guestbook with handwritten cursive notes from half a dozen different people, I have a photo with multiple variations, each with their own back that I wanted to consolidate info on. I have a simple illustration from a band I was part of for a short time as well as an old newspaper clipping and travel photos with no context.

Below, is a sampling of the wide range of files I worked with.

File Names and Organization

Because I am doing this as part of a family archive, I wanted the system to be aware of the various types and versions of files I work with. When scanning photos, you may want to scan the front of the photo and the back of the photo. You may also have a document with multiple pages. You may have the photo negative and the photo itself.

In order for this system to give the best results, most aligned with how I as a archivist works, I needed the system to be aware of these types of file variations so that the LLM is given the FULL context of the document. To do that I devised a simple file naming convention that I apply to my archive.

1. If you have a photo and the back of that photo scanned, the file name format is
   • photo.jpg and photo-back.jpg
2. If you have multiple copies of the same photo AND each one has a different scanned back (sometimes one will have some context and another will have something else), the format is:
   • [version 1] = photo.jpg photo-back.jpg (the a is optional for the first copy)
   • [version 2] = photo-b.jpg + photo-b-back.jpg (and then c,d,e,f etc for additional version)
3. If you have multiple pages of a document the format is simply:
   • document-page1.jpg | document-page2.jpg _ document-page3.jpg …
4. If you have a negative, the naming format is:
   • photo-negative.jpg

So when I say 26 artifacts or 43 files, I mean that some of those 43 files are associated with the same physical photo or document. And for the sake of my classification, I want all versions and pages of a photo or document analyzed together to make sure we are getting the full context. 43 individual files. Representing 26 unique artifacts.

Side Note: Using the above format is not required. It is only required IF you want to take advantage of the versioning features. If all your photos are named different things, they will all by default be treated as different photos with no additional effort needed.

ALSO, this is a feature that you can turn on and off in photokin. For my purposes, I wanted all versions of a photo analysed and then tagged together, so for example if I have two copies of the same photo but the writing has different context on each, I treated them as one shared context and then also shared tags writing back to all of them. So if one photo has text, they all got the transcription of that text in that set. This is also a feature you can turn on and off.

Expected Output

In building the python library and running the test, my test cases were as follows.

Given a folder of document and photo files:

• Create an exact transcription of all text on the photo or documents. There should be no hallucination and anything that is uncertain should be called out as uncertain with [square braces]
• Provide a detailed description of the photo inferring accurate details of the photo (such as this is a photo of Rome even if it doesn’t say Rome on it), without making anything up.
• Provide a title for the photo if there is one that makes logical sense given the transcript and caption.
• Infer a location of the photo if possible, but only if there is a clear argument for why it would infer a specific location.
• Infer a date range for the photo based on historic signals or captions. Additionally provide the confidence level for the date.
• Provide a list of keywords for the photo or document from a list of set keywords. Give it flexibility to add keywords to the list if and only if there are no related ones already in the list. (such as it shouldn’t use Animals if Animal is in the pre-determined list)

In terms of actual metadata fields that would get written back to the photo:

• Photo Transcription ➜ Description
• Photo Caption ➜ UserComment
  • I wanted to keep this separate than the description field. This was harder to find a field for this because most software only covers basic metadata fields. However, this field is supported by Lightroom and philosophically is to be used for general comments about a photo or context about a photo. This is separate than what is actually written on the photo.
• Title ➜ Title
• Location ➜ Location
  • Can be any level of location from country down to city. Only populates to the level of detail that we can be confident about.
• Inferred date range ➜ Original DateTime + Keyword for confidence
  • If it isn’t an exact date, I used a system with a custom keyword to highlight the confidence level for the Year, Month, and Day individually. A known year is different than a best guess year is different than a wild guess.
• Keywords ➜ Keywords
  • The list of keywords the LLM generated for the photo. Most of which should come from a pre-existing list.

This list was a pretty standard set of metadata that most software would support. The most esoteric field is UserComment.

The Test

The first step was to create a baseline. How would I transcribe these documents manually? What would I infer about their location and date? I manually reviewed the images and wrote what I thought were good versions of the transcript, captions etc. for each photo. This served as the comparison point for each model.

I then ran all 25 models through an analysis flow for the entire data set using the same prompt from above. This was messier than I thought it would be: I hit API limits, I had to adjust token limits for some models and others just refused to run.

With the results of all 25 models, I then ran the results through a separate blind judging workflow with Sonnet 4.6 (three times to test for variability in testing), as well as GPT-5.6 Sol and Gemini 3.1 Pro Preview, to control for variability in testing results and bias for their own models.

I provided the models with a rubric breaking down the scoring into a few buckets with different weights:

Dimension	Weight
Transcription	30%
Groundedness (did it make anything up?)	20%
Caption	20%
Keyword recall	16%
Location	7%
Date	7%
Title	0%

Surprisingly, the three judge models were all pretty consistent in their ranking of the performance of each model. They each had their own flavor of giving points (ChatGPT tended to be pretty generous and gave higher scores consistently across the board), but the leaderboard ranking was in agreement across all models. Going forward I am going to align on Sonnet as the model used for general scoring as there doesn’t seem to be an impact on which model does the judging.

As with any rubric, the weights are arbitrary, but I tried to weigh it towards what I thought was the most important aspect, the actual transcription. That needed to be dead on to get a good score. The caption and location/date inference were also important, but if the caption wasn’t right, the other stuff didn’t matter.

Results

The full leaderboard

Before we get to the details, here’s every model, ranked by the blended score described above.

Figure 1 · Cost vs. composite score, all 25 models (log cost axis, bubble size = seconds per photo).

Figure 2 · All 25 models. Cost, Composite Score, Groundedness (higher means less likely to make things up), and speed.

Findings

The main headline is that Opus 5 won on composite score. It came out in the middle of me running this analysis and swept up, even beating Fable. Fable 5 came in close second and fabricates the least of any model I tested, but Opus beat it overall and at less than half the price, $0.12 a photo vs $0.27.

The surprise was Kimi K3, a reasoning model I ran through OpenRouter. It came in fourth at 7.04, behind only the three Anthropic models above it, and it was the best model in the whole field at inferring locations and picking accurate keywords. The catch is that it averaged 100 seconds per photo, roughly 4x slower than anything else I tested. It did also require a bit more tweaking to get running, the token budget had to be adjusted up quite a bit or it would just refuse to run.

On pure value, my read is Kimi K3 if you have time and Sonnet 5 if you don’t. Sonnet costs a bit more per photo, $0.08 vs $0.05, but finishes in 15 seconds instead of 100 and is slightly more accurate. Gemini 3.1 was also highly accurate and even more economical.

One artifact taught me more than the whole leaderboard. A set of prints from Christmas 1944 has “on Cape May 1944” handwritten on the back. Cape May here is a street in California, not the New Jersey shore town. That is context that I have that the model doesn’t. All the models placed the photo in New Jersey with high confidence, and I can’t really call that a hallucination; it is reasonable given the context provided.

Additional details

Figure 3 · Every model, every dimension

Anthropic took the top three spots. I’d bucket Opus 5 and Fable 5 together; they are close enough that you may as well use Opus for half the price, and it sometimes scores better anyway.

Google is the transcription budget king. Flash 3 Preview scored 6.73 overall at $0.008 a photo, and on raw transcription it matches Gemini 3.1 Pro at about a quarter of the price. If I were processing a big backlog on a budget, I’d start there.

OpenAI underperformed Anthropic across the board, and none of their models cleared my 6.5 line. Their best, o3, came in at 6.28, and their flagship GPT-5.6 Sol scored below Google’s mid-tier Flash models while costing more than 3x as much as Gemini 3.1 Pro. The GPT models lagged on transcription in particular. This was very surprising to me. I look forward to testing new models as they come out.

From the OpenRouter field, aside from Kimi which did very well, Grok 4.5 was a solid 6.76 at $0.013 a photo with a clean fabrication record. The budget models were the opposite. Haiku 4.5, GPT-5.4 Nano, both Llama 4s, Mistral Medium 3.5, and GPT-5.6 Luna underperformed in every area, each with a third or more of their checked claims contradicted by my answer key. A cheap wrong answer costs more than no answer; I wouldn’t point any of them at an archive.

Recommendations

Based on the results, this is what I would recommend:

• Best overall: Opus 5, about $0.12 a photo. Top composite, the most artifact wins, and one of only eight models that figured out my old band logo.
• Best balance: Sonnet 5. $0.08 a photo, 15 seconds a photo, and near the top at not making things up.
• If you don’t mind waiting: Kimi K3. Fourth overall at $0.05 a photo and the best in the field at locations and keywords, as long as 100 seconds a photo doesn’t bother you.
• On a budget: Flash 3 Preview at $0.008, or Grok 4.5 at $0.013 if you’re already on OpenRouter.
• Avoid: Haiku 4.5, GPT-5.4 Nano, the Llama 4s, Mistral Medium 3.5, and GPT-5.6 Luna outright, and I’d keep the rest of the OpenAI lineup on a short leash; they all hover just under the fabrication line.

Whichever you pick, the output is only as good as the context provided.

Conclusion

I think this is a useful benchmark today, but I’m not sure it will stay one. The top models are converging, and from here I suspect the competition is mostly on price. My guess is most of the field ends up about where Opus 5 is now within a year or two, and I don’t expect them to get much better than that, because I’m not sure much better exists. A perfect 10 on my rubric would mean writing exactly the caption I would have written. At the Opus 5 level, the caption isn’t the exact words I would write, but it captures the essence of everything I would write, just through a different lens. Searchable, even if imperfect. And honestly, after hand-transcribing a few thousand of these, the top models are more consistent at it than I am.

I’ll keep revising this as new models come out. Or maybe not. We’ll see.