186 points by speckx 3 days ago | 75 comments | View on ycombinator
socketcluster 3 days ago |
b3ing 2 days ago |
dzink 3 days ago |
throwawayffffas 2 days ago |
TacticalCoder 2 days ago |
But that's only true if we believe the value of 0.85 in itself is correct!? For all we know it could be 0.75 or 0.9 no?
If were to make calls to something like JEV and know that the probability the answer is correct is, say, 0.85193847098713, wouldn't that be even more amazing than JEV and LLMs themselves!?
I'm not being facetious here: how do we know it's 0.85 and neither 0.75 nor 0.9?
And if 0.85 is "correct", why stop there? Why not 0.85193847098713?
Any explanation is very welcome.
lhk931122 2 days ago |
simianwords 2 days ago |
Jev can't answer this for example
What is 6^8-3&2 mod 2? 0 or 1?
You might think it is a contrived example but it is illustrative. It shows that almost most questions are reasoning shaped.nostrebored 2 days ago |
Decision models are often larger, slower, non colocated… it just seems like a strange tradeoff to make.
npteljes 2 days ago |
HackerThemAll 2 days ago |
fauria 2 days ago |
munishjindal931 1 day ago |
bryanhogan 2 days ago |
saejox 2 days ago |
aabdi 3 days ago |
Yes its expensive, ugly, etc. But...
I've seen enough IRL that people just create really bad evals that do not reflect real world. Then the product is just bad. Outright broken for the p90 customer.
People don't know statistics. Offline Evals are hard. Sampling the entire population distribution is hard. Sampling the fat tail is harder. Online evals are big ops projects that are giant balls of death.
dieselgate 3 days ago |
nandanadileep29 2 days ago |
vibebingo 2 days ago |
A harnessed LLM can look this up using tools.
Maybe LLM at least ti prepare context for decision model.
I guess if using a decision model context prep is key.
codedrift_ 1 day ago |
> Decision models are up to 444.6x cheaper and 193.6x faster than frontier models
Yes, and two lines of even the worst JavaScript runs circles around making a service call
jakeydus 3 days ago |
And what is a decision model exactly? An LLM with instructions to limit responses?
Huh.
Rapzid 2 days ago |
But using protecting PII as an example use case is a bit bonkers?
tchalla 2 days ago |
jscd 2 days ago |
undefined 3 days ago |
cognitif 2 days ago |
purusa0x6c 2 days ago |
HacklesRaised 2 days ago |
brodydung 2 days ago |
johnh-hn 2 days ago |
undefined 2 days ago |
chanhyounglee 2 days ago |
poxlr 2 days ago |
ClayCantrell 2 days ago |
shipitx 2 days ago |
pj_mukh 2 days ago |
Gotcha
gaal_dornick 3 days ago |
But it reveals something about human biases that we tend to seek out simple answers to complex questions. We tend to think that because a question is short, that it's simple. We forgot how many details are involved and how much is actually going on underneath. Reality itself is often annoyingly complex.
All living creatures, including humans, are conditioned to simplify extremely complex things into simple concepts which we can react to. We only understand things to the extent that we believe they affect us.