230 points by datascientist about 17 hours ago | 211 comments | View on ycombinator
lrvick about 15 hours ago |
giancarlostoro about 16 hours ago |
They release a new RTX series every two years, last time was 2022. (edit: I mistakenly quoted the year wrong, not sure why, but it seems obvious to most that Nvidia doesn't care about the local consumer market anymore)
Meanwhile major AI companies are investing in their own chips, if Nvidia wants to survive they really should not abandon consumers.
Hoping AMD steps up to the plate. Meanwhile the nextXbox and PlayStation consoles are going to have a rough time with their next consoles which will likely cost a thousand dollars or more is my gut feeling.
paxy about 16 hours ago |
Yes, there are a hundred valid reasons to self-host. But 99%+ of consumers aren't going to care.
People don't even care when the cost of hosting is ~free, like a simple home server. How many are ready to buy thousands of dollars worth of AI hardware, which could very well be obsolete a couple of years from now?
herf about 16 hours ago |
brendan_rules_t about 15 hours ago |
montroser about 16 hours ago |
noir_lord about 10 hours ago |
You get used to seeing things over claim what they can do these days but that genuinely does do what it says it does, models that run at 30-40t/s on llama run at 70-100 t/s on that (it seems to really like Qwen 3.8 27B) and since I have a 7900XTX (bought for gaming not AI but useful anyway).
smath about 15 hours ago |
Anecdotally, I believe the open weights local models are constantly catching up with commercial ones, but I've honestly lost track. This is a dynamic space.
cj00 about 15 hours ago |
But they're ignoring that a local LLM can still be very useful for any kind of repetitive task, things like categorization or generating embeddings, translation or summarization tasks. I'm self-hosting Karakeep and use a local LLM for the AI tasks like embedding.
And once you've got that local LLM running, you can start thinking about integrations with your own custom built apps.
drillsteps5 about 8 hours ago |
> ...my sense is that open models, whatever your definition of open is, mainly open weights, I guess, will ultimately prevail. It just seems like that makes a lot more sense to me.
I'm all for local models but struggling to understand the commercial value behind developing them. Building closed LLM (in the cloud) allows the developer to get that sweet recurring revenue, while open weights models, being published, do not.
I'm really unclear on why the open weight models are superior from the commercial point of view (ie profitable for the companies making them) as opposed to providing value to their users (which is undeniable).
1970-01-01 about 14 hours ago |
jkingsman about 16 hours ago |
Some recent tasks I've thrown it:
* Go through my blood test data (from EHR XML dumps from prior years) and give me some over-time charts for consistently-sampled measurements
* Go through my Good Articles bookmarks, epubify them, and drop one down onto my ereader each morning for my commute, then archive the offline version into cold storage
* Go through my music library, identify my most-listened low-bitrate tracks, and source or assemble a shopping cart to get high res versions
* Analyze local tax/foreclosure auctions for vacant land; grab aerial screenshots, tax data, ownership data, and compile a weekly report of things maybe worth bidding on based on some specific selection criteria (fun facts: if you're willing to accept some headaches, 5+ acres can be yours for under $5k, if you're in the right area)
* Dump my grocery store loyalty data, analyze my purchasing habits, and prepare a weekly instacart for my approval that aims to sync with usage habits. Tweak and learn volume/order adjustments over time.
* Given a KVM that can mount ISOs onto the host, reimage two old laptops of unknown status with Debian and install Minecraft so my nieces and I can have a LAN party
* Organize my music (high confidence MusicBrainz Picard matches), movies/TV shows, ebooks, and video game ROM collections with metadata and good folder organization
* Archive bookmarked fanfiction for my offline library (slowly, and respectfully, and with an actual browser. The browser is anti-bot-countermeasured to the gills, but QFN regularly solves hCaptcha and Cloudflare turnstile challenges. Once I gave it the ability to drag, it even reasonably reliably solves drag-shape captchas. A little while back it made a purchase on eBay which is NOT a bot friendly site.)
Would I personally spend time on doing most of these? Generally not. Could an LLM screw them up so badly I regret it? Also generally not. I get the luxury of a psuedo-assistant that doesn't judge how menial the tasks I send it are, and I save time. With a local LLM, my privacy concerns are far less than with frontier models, and local abliterated models are less (or just not at all) twitchy around things that frontier classifiers balk at.
I'm hopeful that big projects like Muse can help to normalize menial task offloading, and that local LLMs provide the bridge for more technically minded folks to use them even more aggressively and limitlessly (as in compute limits/subscription costs) and cheaply.
sachaa about 15 hours ago |
maxdo about 16 hours ago |
I feel like people who consider it as an option thinks of narrow scenarios vs ai is part of our life like internet is. How do you things on mobile for example with your local models?
AI is very competitive, I use various models, I like to be on cutting edge and to be productive, its the opposite from doing something permanent, and not changing, where local models can be good. I like to do things on the go, and the last thing any normal person would want is to have DC at their basement.
philipodonnell about 16 hours ago |
xnx about 16 hours ago |
schaefer about 9 hours ago |
So, I listened to the interview and started digging through the extensive nest of cross-referenced info. Here's what I got so far:
Pete Warden [1], the interviewee, had been running his startup MoonShine.ai with the mission of advancing the state of the art in voice interfaces. Moonshine.ai recently went bankrupt, but Pete was ably to buy the IP for the software in the liquidation sale and it's now open sourced[2].
I tried the live demos for moonshine[3], but found the transcription accuracy to be really, truly awful. But then I found there are many different models available specifically by this team [4]. Some of which boast amazing accuracy on paper.
At the moment, I haven't figured out how to determine which model is running in the live demo... and I haven't tried to run any of this locally.
But I am curious.
If there is one thing I am SUPER excited about it's the write up about Domain Customization[5], which allows dictionary fine tuning and retraining.
This is probably going to be a weekend project very soon.
--
I'd recently been checking out Nvidia's NeMo Voice Agent, and similar projects like Pipecat.
So if anyone has a local first stack they're using, I'm super curious.
let's assume my end goal would be something like a voice native harness for LLMs that is openrouter compatible.
Bonus points if I can putz around with my smartphone and airpods, and be talking to a live development environment either at home (tailscale??) or hosted in the web somewhere.
I really do think voice UI for LLMs with noise canceling headphones would be amazing. Imagine going for a 2 hour walk with that setup.
[1]: https://petewarden.com/ [2]: https://github.com/moonshine-ai/moonshine#readme [3]: https://www.moonshine.ai/agent-flow/ [4]: https://moonshine-voice.readthedocs.io/en/latest/models/ [5]: https://moonshine-voice.readthedocs.io/en/latest/models/doma...
rockingmeister about 15 hours ago |
dahunt33 about 16 hours ago |
martialwave8 about 13 hours ago |
traydstar about 14 hours ago |
asifniaz_ai about 14 hours ago |
Farooqiainews about 13 hours ago |
gnaneshwar about 15 hours ago |
erinnakamato about 14 hours ago |
tulkailabs about 11 hours ago |
Betelbuddy about 16 hours ago |
routebraid about 16 hours ago |
I exclusively use this setup for all AI usage from fixing obscure reproducibility bugs in software that once took me weeks, to compiler bugs, to kernel hardening and hardware drivers, to building a complete unprivileged and security hardened replacement for systemd in rust. Most of that was done using Qwen 27b back when I had only 2x r9700s.
There is literally not a single problem I have faced in 25 years of systems engineering that I cannot solve now 10x+ faster using my local LLM setup.
I turned down the free OpenAI and Anthropic tokens in my YC batch because I have absolutely no interest in being reliant on third parties to do my job.