Audio Flamingo 3
research.nvidia.com
We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thought-type reasoning before answering; (iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets.
    • General_Effort@lemmy.worldOPEnglish
      0·
      2 months ago

      To save everyone a click: It’s a non-commercial license (with a very rude yoink clause, if anyone is foolish enough to build something on it.)

      By the by, there’s a good chance that AI models are not copyrightable under US law; making the license moot in the US. In other regions, such as the EU, it likely holds.

      3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use non-commercially. Notwithstanding the foregoing, NVIDIA Corporation and its affiliates may use the Work and any derivative works commercially. As used herein, “non-commercially” means for non-commercial research and educational purposes only.

      • 🍉 Albert 🍉@lemmy.worldEnglish
        0·
        2 months ago

        i know that AI output is not copyrighteable, because it wasn’t made by a human.

        however the model itself is a product of a shit ton of work. and I doubt any court will claim them non copyrighteable.

          • hendrik@palaver.p3x.deEnglish
            0·
            2 months ago

            My idea would be to slightly modify / fine-tune a model and then redistribute that modified version. And claim the same Fair Use, AI companies use to take people’s copyrighted work. Either that makes it Fair Use as well, or the “no originality required” collapses or the entire business model.

            • General_Effort@lemmy.worldOPEnglish
              1·
              2 months ago

              I don’t see how that would be fair use or what the argument is supposed to be.

              Let me warn you that Lemmy is full of disinformation on copyright. If you picked the idea up here, then it probably is absolutely bonkers.

              In any case, fair use is a US thing. In the EU, it would still be yoink.

              • hendrik@palaver.p3x.deEnglish
                0·
                2 months ago

                I think I used a bit too much sarcasm. I wanted to take a spin on the idea how the AI industry simultaneously uses copyright, and finds ways to “circumvent” the traditional copyright that was written before we had large language models. An AI is neither a telephone-book, nor should every transformative work be Fair Use, no questions asked. And this isn’t really settled as of now. We likely need some more court cases and maybe a few new laws. But you’re right, law is complicated, there is a lot of nuance to it and it depends on jurisdiction.

                • General_Effort@lemmy.worldOPEnglish
                  1·
                  2 months ago

                  There is a lot of disinformation being spread on copyright because major rights-holders hope to gain a lot of money for nothing.

                  US fair use has always worked like this. Other countries without fair use had to make laws to enable AI training. I know about Japan and the EU.

                  It is precisely because of these new laws that AI training in the EU is possible at all (eg by Mistral AI or by various universities/research institutions). But because of lobbying by rights-holders, this is quite limited. It’s not possible to train AIs in the EU that are as capable as those from the US, where Fair Use comes directly from the constitution and can’t be easily lobbied aside by monied interests.