Gemini 2.5 "reasoning", no real improvement on river crossings.
Gemini 2.5 "reasoning", no real improvement on river crossings.
So I signed up for a free month of their crap because I wanted to test if it solves novel variants of the river crossing puzzle.
Like this one:
You have a duck, a carrot, and a potato. You want to transport them across the river using a boat that can take yourself and up to 2 other items. If the duck is left unsupervised, it will run away.
Unsurprisingly, it does not:
https://g.co/gemini/share/a79dc80c5c6c
https://g.co/gemini/share/59b024d0908b
The only 2 new things seem to be that old variants are no longer novel, and that it is no longer limited to producing incorrect solutions - now it can also incorrectly claim that the solution is impossible.
I think chain of thought / reasoning is a fundamentally dishonest technology. At the end of the day, just like older LLMs it requires that someone solved a similar problem (either online or perhaps in a problem solution pair they generated if they do that to augment the training data).
But it outputs quasi reasoning to pretend that it is actually solving the problem live.
Take the potato and the carrot, show the carrot to the duck and make it follow you, dude can swim.
One trip, solved.
Sounds like a perfectly reasonable duck.
I'm kinda tired, but this puzzle's shoved itself into my brain. The obvious solution I can see is, roughly speaking:
-
Take the duck and carrot across
-
Take the duck back
-
Take the duck and potato across
My two solutions:
- Eat the carrot. Take the duck and potato across.
- It's a row boat. Take the carrot and potato, supervise the duck as it swims behind you.
I'm not doing three river crossings, you can't make me.
another solution:
take duck, carrot and potato at once. if boat is fine if you put duck and carrot in but will sink if you put in duck, carrot and potato then you're already on horrifyingly narrow engineering margins and probably shouldn't use it in the first place
in the worst case you can put duck on a leash if it'll run away otherwise
Yeah, exactly. There's no trick to it at all, unlike the original puzzle.
I also tested OpenAI's offerings a few months back with similarly nonsensical results: https://awful.systems/post/1769506
All-vegetables no duck variant is solved correctly now, but I doubt it is due to improved reasoning as such, I think they may have augmented the training data with some variants of the river crossing. The river crossing is one of the top most known puzzles, and various people have been posting hilarious bot failures with variants of it. So it wouldn't be unexpected that their training data augmentation has river crossing variants.
Of course, there's very many ways in which the puzzle can be modified, and their augmentation would only cover obvious stuff like variation on what items can be left with what items or spots on the boat.
It's just overtrained on the puzzle such that it mostly ignores your prompt. Changing a few words out doesn't change that it recognises the puzzle. Try writing it out in ASCII or uploading an image with it written or some other weird way that it hasn't been specifically trained on and I bet it actually performs better.
-
I just asked the same question to my preferred onprem model, Mistral-Small 3.1.
13s on a 7900xtx
justinledwards/mistral-small-3.1-Q6_K:latest
To successfully transport the duck, carrot, and potato across the river without losing any of them, you need to plan your trips carefully. Here’s a step-by-step strategy:
First Trip: Take the duck across the river first.
- Items on this side: Carrot, Potato
- Items on the other side: Duck
Return Trip: Go back alone to the original side.
- Items on this side: Carrot, Potato (you)
- Items on the other side: Duck
Second Trip: Take either the carrot or the potato across the river.
Let's say you take the carrot first for illustration.
- Items on this side: Potato
- Items on the other side: Duck, Carrot (you)
Return Trip: Go back with the duck to the original side.
-
Items on this side: Potato, Duck
-
Items on the other side: Carrot
Third Trip: Leave the duck behind and take the potato across the river.
-
Items on this side: Duck
-
Items on the other side: Carrot, Potato (you)
Return Trip: Go back alone to get the duck.
-
Items on this side: Duck (you)
-
Items on the other side: Carrot, Potato
Final Trip: Take the duck across the river. All items are now on the other side.
By following these steps, you ensure that the duck is never left unsupervised and all items make it safely across the river.
While all the comments are correct, i.e. what the fuck is wrong with you, this is also wrong since it immediately leaves the duck unsupervised after the first trip. So really not sure what you're trying to prove here.
What's your point, that your favourite LLM doesn't reason either?
no-one fucking asked you to spam us with pointless llm trash
the fuck is wrong with you