microsoft/unilm
View on GitHub ↗ ★ 22,239 Python
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
17 mentions on Hacker News by 16 people: 6 stories, 11 comments · first 2021-12-25 · latest 2025-08-08
20082026
-
…to the CLS token in ViT. Another approach I've seen is the "Diff transformer" from MS Research ( https://github.com/microsoft/unilm/tree/master/Diff-Transfor... ).
comment by Scene_Cast2 · 2025-08-08 · in How attention sinks keep language models stable -
…slow and perhaps should considered against larger networks with more attention layers. https://github.com/microsoft/unilm/blob/master/Diff-Transfor...
comment by f38zf5vdt · 2024-10-08 · in Differential Transformer -
…it looked shiny and interesting, but I haven't had a chance to put it to the test much yet. [0] - https://github.com/microsoft/unilm/tree/master/kosmos-2.5
comment by daemonologist · 2024-06-07 · in How Does GPT-4o Encode Images? -
…* https://github.com/nkotak/1.58BitNet See some followups that has some training advice: https://github.com/microsoft/unilm/tree/master/bitnet
comment by lhl · 2024-05-31 · in “Imprecise” language models are smaller, speedier, and nearly as accurate -
…for dealing with them (with the 2 bit data types). [0] https://arxiv.org/abs/2402.17764 [1] https://github.com/microsoft/unilm/blob/master/bitnet/The-Er...
comment by UncleOxidant · 2024-05-13 · in GPUs Go Brrr -
…side-by-side. It seems that it may be published on GitHub [1] according to HuggingFace [2]. [1] https://github.com/microsoft/unilm/tree/master/bitnet [2] https://huggingface.co/papers/2402.17764
comment by bArray · 2024-02-28 · in The Era of 1-bit LLMs: ternary parameters for cost-effective computing -
…with more recent advances in unsupervised learning and multimodal learning (along the lines of https://github.com/microsoft/unilm or https://github.com/Alpha-VLLM/LLaMA2-Accessory )
comment by johnsutor · 2024-02-16 · in I'm an Old Fart and AI Makes Me Sad -
e5-mistral is essentially a distillation from gpt-4 to a smaller model. You can see here https://github.com/microsoft/unilm/blob/16da2f193b9c1dab0a69... they actually have custom prompts for each dataset being tested. Question would be, if…
comment by mahjongmen1998 · 2024-01-18 · in Retro on Viberary -
They link to this repo at the top of the paper https://github.com/microsoft/unilm although I don't see anything that actually says "Kosmos". Slightly interestingly, the last commit changes a heading from "AI" to "AGI"…
comment by ilaksh · 2023-02-28 · in Language Is Not All You Need: Aligning Perception with Language Models -
…layout and also output markup.. I may have to experiment with some of these like this one maybe https://github.com/microsoft/unilm/tree/master/markuplm
comment by ilaksh · 2023-01-20 · in General technology for enabling AI capabilities with LLMs and Generative models -
https://github.com/microsoft/unilm/tree/master/valle Probably soon or later they will publish the code here
comment by GaggiX · 2023-01-06 · in VALL-E: Neural codec language models are zero-shot text to speech synthesizers