• 0 Posts
  • 11 Comments
Joined 5 months ago
cake
Cake day: April 7th, 2025

help-circle
  • One of the absolute best uses for LLMs is to generate quick summaries for massive data. It is pretty much the only use case where, if the model doesn’t overflow and become incoherent immediately [1], it is extremely useful.

    But nooooo, this is luddite.ml saying anything good about AI gets you burnt at the stake

    Some of y’all would’ve lit the fire under Jan Hus if you lived in the 15th century

    [1] This is more of a concern for local models with smaller parameter counts and running quantized. For premier models it’s not really much of a concern.


  • That is different. It’s because you’re interacting with token-based models. There has been new research on giving byte level data to LLMs to solve this issue.

    The numerical calculation aspect of LLMs and this are different.

    It would be best to couple an LLM into a tool-calling system for rudimentary numeral calculations. Right now the only way to do that is to cook up a Python script with HF transformers and a finetuned model, I am not aware of any commercial model doing this. (And this is not what Microshit is doing)









  • Inference costs are very, very low. You can run Mistral Small 24B finetunes that are better than GPT-4o and actually quite usable on your own local machine.

    As for training costs, Meta’s LLAMA team displace their emissions with environmental programs, which is more green than 99.9% of any company making any product you use

    TLDR; don’t use ClosedAI use Mistral or other foss projects

    EDIT: I recommend cognitivecomputations Dolphin 3.0 Mistral Small R1 fine tune in particular. I’ve only used it for mathematical workloads in truth, but it has been exceedingly good at my tasks thus far. The training set and the model are both FOSS and uncensored. You’ll need a custom system prompt to activate the Chain of Thought reasoning, and you’ll need a comparatively low temperature to keep the model from creating logic loops for itself (0.1 - 0.4 range should be OK)