Skip Navigation
What is a self-hosted small LLM actually good for (<= 3B)
  • Please be very careful. The python code it'll spit out will most likely be outdated, not work as well as it should (the code isn't "thought out" as if a human did it.

    If you want to learn, dive it, set yourself tasks, get stuck, and f around.

  • What is a self-hosted small LLM actually good for (<= 3B)

    I've tried coding and every one I've tried fails unless really, really basic small functions like what you learn as a newbie compared to say 4o mini that can spit out more sensible stuff that works.

    I've tried explanations and they just regurgitate sentences that can be irrelevant, wrong, or get stuck in a loop.

    So. what can I actually use a small LLM for? Which ones? I ask because I have an old laptop and the GPU can't really handle anything above 4B in a timely manner. 8B is about 1 t/s!

    24
    No Kings Protest, Minnesota
  • I like how all the big media just happened to publish articles about "peaceful protests" to pacify the protestors, "please don't be violent everyone, the police can't shoot you all in the leg with rubber bullets".

    The violent protests are the successful ones. Just ask every country, ever.

  • Must fight temptation to buy an overpriced raspberry pi
  • See here's the thing. Why would anyone want to host ALL the stuff on one pi? That is not what they were designed for. Ollama on a pi? Are you out of your mind? I'd run the biggest model I can on a modern gpu not some crappy old computer or pi....Right tool, right job. And why is dropping containers "less secure"? Do you mean "less cool"? Less easy to deploy? But you're not deploying it, you're installing it. You sound like a complete newb which is fine, but just take a step back from things and get some more experience. A pi is a tool for a purpose, not the end all. Using an old laptop is not going to save the world and arguing that it's just better than a pi (or similar alternative) is just dumb. Use a laptop for all I care, I'm not the boss of you.

    As for an arr stack, I'm really disappointed with the software and don't use it and those who do have way too much time to set it up, and then make use of it!

  • Must fight temptation to buy an overpriced raspberry pi
  • Pi’s are ARM-based, which still to this day limits the scope of their applicability.

    Untrue.

    Also, you should absolutely inspect a laptop before buying. Many, if not most, of old laptops will run just fine for the next few years.

    Until the battery needs replacing, costing more than a pi, one key on the keyboard dies, etc.

  • NSFW
    [Deleted]
  • There's a difference between using it as entertainment and using it chronically. In answer to television/ films - everything in the last twenty five to thirty years is rubbish and a waste of time. I still like a blast on a cheap Chinese hand-held emulator to fill 10-20 mins or so then I put it down and forget about it.

    When you adult, you have responsibilities. I've known children who come from successful families who have never played on a console because they're busy doing positive things in life that will help them in future.

  • Must fight temptation to buy an overpriced raspberry pi
  • ODroids don't meet European legal hazard levels on poisonous fumes. I bought one back in the day and they explained they won't apply for the test because of "the cost"... not that it uses cheap solder that don't meet lead limits.

  • Must fight temptation to buy an overpriced raspberry pi
  • I dislike posts like this. Technology moves quickly. PIs are great for hobby electronics where you need a little computer. Want a cheap computer to run a few things 24/7 and know what you're doing? Pi it is. You don't need to run containers on a pi because you have the skills to install the dependencies manually. They cost pennies to run 24/7.

    I think of pis as beefed-up calculators. I have made lots of money using a pi zero running code I needed to run 24/7. Code I developed myself.

    Having an old laptop with outdated parts taking up lots of space, weighing a lot, and having components like fans, keyboard, and mousepad most-likely soon dying and needing replacing is an additional concern you don't want.

    Someone below saying use an old laptop if you're living with parents and don't pay the electricity bill is a bit lame. Do your part for the world. Someone will be paying for it.

    Ultimately, use what you want but if you're just starting with servers, use a virtual machine on your computer and log in to it. You can dick about with it as much as you want, and reset back to a working state in seconds.

  • What can I use for an offline, selfhosted LLM client, pref with images,charts, python code execution

    I was looking back at some old lemmee posts and came across GPT4All. Didn't get much sleep last night as it's awesome, even on my old (10yo) laptop with a Compute 5.0 NVidia card.

    Still, I'm after more, I'd like to be able to get image creation and view it in the conversation, if it generates python code, to be able to run it (I'm using Debian, and have a default python env set up). Local file analysis also useful. CUDA Compute 5.0 / vulkan compatibility needed too with the option to use some of the smaller models (1-3B for example). Also a local API would be nice for my own python experiments.

    Is there anything that can tick the boxes? Even if I have to scoot across models for some of the features? I'd prefer more of a desktop client application than a docker container running in the background.

    24
    Sexism broadcast on TV? "Big D" aka "Knockers" for obvious reasons! 1976 UK TV Seaside Special S2E12

    I'm watching some retro television and this show is wild! Beauty contests with 16 year-old girls (though at the time, it was legal for 16 yo girls to pose topless for newspapers), old racist comedians from working men's clubs doing their routine, Boney M, English singers from the time, and happy dance routines!

    vid

    0
    InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)CA
    catty @lemmy.world
    Posts 3
    Comments 31