Hacker GitHub

← All repos

microsoft/unilm

View on GitHub ↗ ★ 22,239 Python

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

17 mentions on Hacker News by 16 people: 6 stories, 11 comments · first 2021-12-25 · latest 2025-08-08

Mentions per year 17 mentions · busiest year 2024 (11)
20082026
AllStoriesComments NewestTopOldest
  1. …to the CLS token in ViT. Another approach I've seen is the "Diff transformer" from MS Research ( https://github.com/microsoft/unilm/tree/master/Diff-Transfor... ).
    comment by Scene_Cast2 · 2025-08-08 · in How attention sinks keep language models stable
  2. …slow and perhaps should considered against larger networks with more attention layers. https://github.com/microsoft/unilm/blob/master/Diff-Transfor...
    comment by f38zf5vdt · 2024-10-08 · in Differential Transformer
  3. …it looked shiny and interesting, but I haven't had a chance to put it to the test much yet. [0] - https://github.com/microsoft/unilm/tree/master/kosmos-2.5
    comment by daemonologist · 2024-06-07 · in How Does GPT-4o Encode Images?
  4. ▲ 1
    Kosmos-2.5: A Multimodal Literate Model
    story by hubraumhugo · 2024-06-03 · 0 comments
  5. …* https://github.com/nkotak/1.58BitNet See some followups that has some training advice: https://github.com/microsoft/unilm/tree/master/bitnet
    comment by lhl · 2024-05-31 · in “Imprecise” language models are smaller, speedier, and nearly as accurate
  6. …for dealing with them (with the 2 bit data types). [0] https://arxiv.org/abs/2402.17764 [1] https://github.com/microsoft/unilm/blob/master/bitnet/The-Er...
    comment by UncleOxidant · 2024-05-13 · in GPUs Go Brrr
  7. ▲ 2
    The Era of 1-Bit LLMs: Training Tips, Code And FAQ [pdf]
    story by histories · 2024-03-21 · 0 comments
  8. ▲ 2
    The Era of 1 Bit LLMs – Training, Tips, Code [pdf]
    story by netsec_burn · 2024-03-20 · 0 comments
  9. ▲ 2
    The Era of 1-Bit LLMs: Training Tips, Code and FAQ
    story by rkwasny · 2024-03-20 · 1 comment
  10. …side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764
    comment by bArray · 2024-02-28 · in The Era of 1-bit LLMs: ternary parameters for cost-effective computing
  11. …with more recent advances in unsupervised learning and multimodal learning (along the lines of https://github.com/microsoft/unilm or https://github.com/Alpha-VLLM/LLaMA2-Accessory )
    comment by johnsutor · 2024-02-16 · in I'm an Old Fart and AI Makes Me Sad
  12. e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69... they actually have custom prompts for each dataset being tested. Question would be, if…
    comment by mahjongmen1998 · 2024-01-18 · in Retro on Viberary
  13. ▲ 228
    Microsoft Kosmos-1: A Multimodal Large Language Model
    story by solarist · 2023-03-01 · 104 comments
  14. They link to this repo at the top of the paper https://github.com/microsoft/unilm although I don't see anything that actually says "Kosmos". Slightly interestingly, the last commit changes a heading from "AI" to "AGI"…
    comment by ilaksh · 2023-02-28 · in Language Is Not All You Need: Aligning Perception with Language Models
  15. …layout and also output markup.. I may have to experiment with some of these like this one maybe https://github.com/microsoft/unilm/tree/master/markuplm
    comment by ilaksh · 2023-01-20 · in General technology for enabling AI capabilities with LLMs and Generative models
  16. https://github.com/microsoft/unilm/tree/master/valle Probably soon or later they will publish the code here
    comment by GaggiX · 2023-01-06 · in VALL-E: Neural codec language models are zero-shot text to speech synthesizers
  17. ▲ 3
    Large-Scale Self-Supervised Pre-Training Across Tasks, Languages, and Modalities
    story by axiomdata316 · 2021-12-25 · 0 comments