Rendered at 15:52:07 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
svcrunch 1 days ago [-]
I'd like to mention the Little Dorrit Benchmark [1] which I have been running for a couple of years now. It has a few nice features:
1. It tests visual reasoning and structured output in a single task.
2. It seems to sort correctly on advancing general intelligence. As a counterexample, if I'm not misremembering, artificialanalysis.ai made some changes to their benchmark recently after Astra ranked below several older models.
3. While models have gotten significantly better in the past 2 years, the top model is still at 0.78 F1, so the test is not yet saturated. As a reference point, when I started, the top models were in the [0.1, 0.2] range.
Qwen 3.8 Flash-Next is really missing here (because it's small enough to run it locally on a (formerly) affordable machine).
This is what Qwen3.8-Flash-Next-UD-Q4_K_XL gives me for "an octopus operating a pipe organ": https://imgur.com/a/zHyHIqI
Seems very similar to the output of Qwen 3.8 Max to me
vova_hn2 1 days ago [-]
Website looks very cool, Fable's octopus-organist looks very cute, but I feel like this benchmark (generate an SVG by a short and slightly ridiculous description) in general has been completely Goodharted [0].
I think they all just added a bunch of similar tasks to their training sets, so we cannot judge true emergent capabilities of the models anymore.
In my experience LLMs have become generally a lot more useful if you need to create something like a company logo or even 3d scenes.
andy_ppp 21 hours ago [-]
“I think they all just added a bunch of similar tasks to their training sets, so we cannot judge true emergent capabilities of the models anymore.”
This has been the plan since the start of all this, they regurgitate code in ever better forms but they still aren’t inventing new things yet.
GaggiX 24 hours ago [-]
I don't think a bunch of similar tasks can really saturate the "create a SVG of X", because the model should have a quite good spatial understanding of the world and how everything interacts.
For example Gemini 3.8 Flash seems very impressive at first glance but the results are not actually very coherent, this shows that its "world model" is not particularly great (compare to SOTA models).
samayashar 1 days ago [-]
All models are pretty good now at generating these images. Back in the day, I remember experimenting with the pelican images and most of the models couldn't align the legs with the wheels. Right now as well, GPT messed up an octopus leg by originating it through the instrument rather than the octopus itself.
I think that intertwining two entities (living/non-living) is still challenging but overall they're pretty sound.
honeycrispy 23 hours ago [-]
> All models are pretty good now at generating these images.
That's pretty generous.
CamperBob2 24 hours ago [-]
All models are pretty good now at generating these images.
Not zebras. If you want to see how bad SVG output still is, ask for a zebra riding a scooter.
rukuu001 18 hours ago [-]
Man I thought the giraffes were bad enough
outlore 22 hours ago [-]
Anyone else surprised the generations look so remarkably similar? All of these models have “independently” generalized that the moose should roughly be standing at the same position (left) or that the giraffe should have a certain color palette.
trebligdivad 17 hours ago [-]
Yes; I found it fun they all decided telescopes should be repaired at night.
Terr_ 22 hours ago [-]
With respect to bikes (with or without pelicans) there's a strong natural bias because people displaying bikes tend to want to show off the side with the gears.
More-generally, I suspect an influence from how left-to-right languages (i.e. English) affect comic layouts. Overcoming that bias often means using vertical space to exploit the top-to-bottom habit instead. (Consider the rarity of an English-language comic panel where action is from bottom-right to top-left.)
theshrike79 8 hours ago [-]
What's interesting to me is that the SVG versions don't have the "AI Image generation hates negative space" issue as badly as generated images do. They kinda stay on point and don't fill every single empty bit with some pattern.
kennywinker 22 hours ago [-]
Would love to see Qwen3.8-27b here, since that is the model most people are running locally.
MacBook Pro 3D in SVG for me the most helpful one.
steinvakt2 1 days ago [-]
Feels like google has a different training set than the others?
sajithdilshan 1 days ago [-]
Interesting, out of all examples Gemini 3.8 is the best for me. Also the image style is different and more vibrant than others
Jordan-117 23 hours ago [-]
Better than Astra and Fable? It looks quite pretty and even impressive at times if you squint, but look closer and it falls apart in terms of coherency. And I say that as somebody who mains Gemini 3.8.
23 hours ago [-]
zaphar 23 hours ago [-]
I notice none of the octopi seem to be actually facing the organ.
miohtama 18 hours ago [-]
I hope there would be parameter iterations allowing especially cheaper models on inspect and fix their output via rendering
andy_ppp 21 hours ago [-]
Love the output from Qwen 3.8 it seems very impressive for the cost! Why does Gemini 3.8 flash blur everything? What are Google playing at!
pohl 20 hours ago [-]
Gemini 2.5 Pro is the only model with a sense of where a ferris wheel operator would be.
eleventen 19 hours ago [-]
I was picturing that fleet of playground equipment they turned into space ships in the Jimmy Neutron movie, so Astra gets my vote.
Does a test of instructions how to fold origami figures in a SVG/jpeg exist? Or could be useful?
BrokenCogs 24 hours ago [-]
Gemini 3.8 flash seems to (subjectively) be the outlier in terms of performance to cost ratio?
pixelesque 23 hours ago [-]
Its giraffe / grandfather clock one is pretty bad... (two necks? wearing a suit?)
Weird as well, it's clearly pulled out some 1884 patent on clock designs, and a quick ddg/google doesn't show it as anything to do with grandfather clocks.
It’s very likely they all add svg generation into the training data. It’s part of the reason it’s no longer a good benchmark (unless you need to generate SVGs).
simonw 19 hours ago [-]
It's about more than just training data. Every model since GPT-2 had SVGs in the training data, because they all used a scrape of the Web and the Web is full of SVGs. What's unique about Gemini is that they actively worked to get better results for their SVGs - maybe RLHF, maybe RLVF of some sort.
GaggiX 24 hours ago [-]
Gemini 3.8 Flash results are often not very coherent but it does put a lot of shading and details to hide the fact.
neilellis 24 hours ago [-]
Well that benchmark is now saturated, what next. How fast you can hack the pentagon?
dustfinger 23 hours ago [-]
It is interesting how similar the designs are across the models.
qiine 1 days ago [-]
Asking to animate it add an interesting layer of difficulty
ormax3 23 hours ago [-]
I noticed in the "A penguin juggling chainsaws" prompt that Qwen created an animated svg
input_sh 23 hours ago [-]
I'd say at least half of Qwen's 2026 runs are animated.
The only other one I've spotted is animated is Gemini 3.0's 2025 run of an elephant.
water-drummer 20 hours ago [-]
Ok the Grok ones are cute
mock-possum 23 hours ago [-]
Try asking an LLM to draw you the cool S.
amysox 20 hours ago [-]
Or something like, "Draw an S, then a more different S, close it up real good here, then using consummate V's, add teeth, and scales, and eyebrows, and legs. And then add smoke, and fire, and some wings, and one of those big beefy arms for good measure." :D :D :D
dcreater 23 hours ago [-]
Why is this a good test?
simonw 20 hours ago [-]
Because it's one of the few ways of comparing models that lets you instantly evaluate them visually. That makes it more comprehensible than a numeric score on a benchmark.
21 hours ago [-]
simonw 20 hours ago [-]
I love these.
bicepjai 1 days ago [-]
[dead]
aaron695 17 hours ago [-]
[dead]
villish 23 hours ago [-]
The 3 US models have their own style.
Qwen3.8 is very clearly distilled from Claude models.
bahmboo 9 hours ago [-]
How do you support this claim? I am curious about tell tails for distillations.
1. It tests visual reasoning and structured output in a single task.
2. It seems to sort correctly on advancing general intelligence. As a counterexample, if I'm not misremembering, artificialanalysis.ai made some changes to their benchmark recently after Astra ranked below several older models.
3. While models have gotten significantly better in the past 2 years, the top model is still at 0.78 F1, so the test is not yet saturated. As a reference point, when I started, the top models were in the [0.1, 0.2] range.
[1] https://dorrit.pairsys.ai/
I think they all just added a bunch of similar tasks to their training sets, so we cannot judge true emergent capabilities of the models anymore.
[0] https://en.wikipedia.org/wiki/Goodhart%27s_law
This has been the plan since the start of all this, they regurgitate code in ever better forms but they still aren’t inventing new things yet.
For example Gemini 3.8 Flash seems very impressive at first glance but the results are not actually very coherent, this shows that its "world model" is not particularly great (compare to SOTA models).
I think that intertwining two entities (living/non-living) is still challenging but overall they're pretty sound.
That's pretty generous.
Not zebras. If you want to see how bad SVG output still is, ask for a zebra riding a scooter.
More-generally, I suspect an influence from how left-to-right languages (i.e. English) affect comic layouts. Overcoming that bias often means using vertical space to exploit the top-to-bottom habit instead. (Consider the rarity of an English-language comic panel where action is from bottom-right to top-left.)
MacBook Pro 3D in SVG for me the most helpful one.
https://youtu.be/w0qDV2QhAlg?si=JyHS6rZLmIGAuTd3&t=46
A monkey, surely?
Weird as well, it's clearly pulled out some 1884 patent on clock designs, and a quick ddg/google doesn't show it as anything to do with grandfather clocks.
See also this Jeff Dean tweet showing off their animals-in-vehicles abilities: https://twitter.com/JeffDean/status/2024525132266688757
The only other one I've spotted is animated is Gemini 3.0's 2025 run of an elephant.
Qwen3.8 is very clearly distilled from Claude models.