We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thought-type reasoning before answering; (iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets.
I know those narratives, as the humanities people call this. I don’t know if you know the term. You know commercials. They rarely give you facts. They don’t give you technical data about performance, durability, or such. Usually, a commercial is a little story, maybe just a few nice people having fun. When you see the product and think about buying, you can see yourself living that story. Maybe you see yourself in a new car speeding unhindered down an empty road; not stuck in traffic like those suckers you see every day in reality.
You don’t convince people with facts. You use psychological manipulation. If you think about history, people mostly believed religious stories about what happened in the world. That many people in developed countries defer to scientific facts is unusual. Of course, many don’t. The stories are much nicer. Let’s face it: The only reason we put up with ugly, meaningless facts is because we are reliant on technology.
We want the good life. We want to be healthy, and not having to worry about food or shelter. We want comforts, like flowing hot and cold water; an extravagant luxury for most humans throughout history and even today. In war, we want the best weapons, so that it is the other guys who do the dying.
So the question is: Do you prefer the feel-good-story or do you want a society that works for everyone?
I’m a bit in the science/facts bubble. I mean sure, advertisements and narratives are effective, and I’m not exempt. But I’d like to know the truth. And have politics based on scientific evidence. The goal is to strive and have a nice life, eveyone should be happy if possible. And then we use science to tell what kind of laws we need. Are all students delegating their homework to ChatGPT and they don’t learn anything anymore? Find ways so school achieves it’s goal. Do we confuse reality and fiction? Find ways to mitigate for that, e.g. watermarking. Do we loose all artists and creative people? Find ways so they can be part of society… I mean sometimes we can have a cake and eat it too, especially with technology. But we need to be clever.
I mean in the past we’ve adopted to new technology. One example which is often cited in context with AI is channging from horses to cars. That was very disruptive as well. I think today’s situation is a bit different. And for example copyright barely works in the digital age. But AI is likely going to have a massive impact on society. Maybe we need to re-think capitalism. That’s not necessarily good or bad or a “narrative”. But somehow things need to be addressed.
Europe has clearly chosen a path that will increase its technological dependency on either the US or China. It’s not likely to play a large role in figuring out the future economic order. We’ll see how long it can continue on this path.
Its AI policies are reminiscent of Feudalism. People create AI, but then they have to pay a levy to people who have contributed nothing. But they have rights awarded by the government. AI is not the only area where the EU is shifting to policies that facilitate wealth extraction rather than creation. I don’t think that is domestically sustainable. Sooner or later the European nations will try to extract wealth from each other and that will be the end. It doesn’t have to go that far. Maybe we will just see a stagnation and decline, as in South America.
Is your stands limited to AI or do you generally condone paying a levy? Like towards Spotify or Netflix or Hollywood, because I could as well skip that and watch the newest movies without obeying their copyright…
I mean it’s not nothing, there is some effort people put into things. Like the Wikipedia is super useful for machine learning. My computer code on Github teaches AI programming. And I can see the crawlers at my own server and today I had to update my config because it’s been hammered by Alibaba. Dozens of different IP addresses, fake user agent and they completely overloaded my database with requests. It’s not like I don’t contribute or am part of a different world?!
That’s a bit of an odd question, given my praise of American Fair Use. The USA has had copyright, including Fair Use, for longer than much of Europe. The predecessor of modern copyright law was created in the 1700s in the UK. There is a German scholar, Eckhard Höffner, who argues that this caused book production to plummet in the UK. He also says that the German-speaking lands produced more books, more different books, than the UK in the century before such laws arrived.
The American founding fathers were men of the Enlightenment. They, or some of them, understood the problems with such government sponsored monopolies. Therefore, the US Constitution limits copyrights and patents. It’s an interesting clause. Congress is empowered “To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries”. It’s about progress first; very much a product of the Enlightenment.
I don’t know if there was ever a discussion if entertainment should qualify at all for copyright protection. I have to try to look it up at some time.
In 1998, US copyright was extended by 20 years. Now it is life of the author +70 years. That has been called the Mickey-Mouse-Protection-Act, because it meant that the original Mouse enjoyed another 20 years of copyright This was roundly criticized by economists and even lead to a case before the Supreme Court. Obviously, making copyright retroactively longer does not encourage any kind of creativity. It’s in the past. Well, the case was lost, nevertheless.
For many left/liberal people, this is corruption; just the Disney company getting what it wants.
The EU countries had expanded their copyright years earlier, without resistance or even comment. Smug Europeans may feel superior when Americans rage against the corporations. But the truth is often like this, where Europeans simply quietly accept such outrages.
The original copyright in the US (and before that in the UK) was 14 years. Copyright protection required registration. It worked like the patent system. The interesting thing is that patents still work a lot like that. One must register and publish them and then they last for 20 years. Patents still have a 20-year duration. Meanwhile, copyrights have gone from 14 years to life+70 years, no registration required.
Patents are public so that people can learn from them. That has been used as an argument for patents. The alternative would be that everyone tries to keep new inventions secret. This way, people can learn and try to circumvent patents; find other ways of achieving the same thing. That’s an interesting observation in light of AI training, no?
I haven’t answered your question. In my experience, pro-copyright people will always refuse to argue over what should be covered by copyright or how long. They demand an expansion and use psychological manipulation to get it. If you do not let yourself be manipulated, they change the subject and will argue if copyright should exist at all. I have never met a single person who was able to defend copyright as it exists. Perhaps you can answer own question now.
Yes, I mainly wanted to rule out the opposite. Because the multi billion dollar companies currently do some lobbying as well. Including the same manipulation and narratives, just the other way around. They want everyone else to lose rights, while they themselves retain full rights, little to no oversight… And that’s just inherently unfair.
As I said. Copyright might not be something good or defendable. It clearly comes with many obvious flaws and issues. The video you linked is nice. I’d be alright with abolishing copyright. Preferrably after finding a suitable replacement/alternative. But I’m completely against subsidising big companies just so they can grow and manifest their own Black Mirror episode. Social scoring, making my insurance 3x more expensive on a whim and a total surveillence state should be prohibited. And the same rules need to apply to everyone. Once a book author doesn’t get copyright any longer, so does OpenAI and the big tech companies. They can invest some $100 million in training models, but it’s then not copyrighted either. I get to access the model however I like and I can sell a competing service with their model weights. That’s fair and same rules for everyone. And Höffner talks to some degree about prior work and what things are based upon. So the big companies have to let go of their closely guarded trade secrets and give me the training datasets as well. I believe that’d be roughly in the spirit of what he said in the talk. And maybe that’d be acceptable. But it really has to be same rules for everyone, including big corporations.
They want everyone else to lose rights, while they themselves retain full rights, little to no oversight
Can you back this up? They certainly do not have the same reach or influence as the copyright industry.
subsidising big companies
What do you mean by that?
They can invest some $100 million in training models, but it’s then not copyrighted either.
AI models may not be copyrightable under US law. I’m fairly sure that base models aren’t. Whether curating training data, creating new training data, RL, and so on, ever makes a copyrighted model is something that courts will eventually have to decide.
They are probably copyrightable under EU law (maybe protected as databases). That’s an EU choice.
But it really has to be same rules for everyone, including big corporations.
The rules are different in different countries. They are not different for corporations.
The current thing is Meta is very vocal about the EU AI act. Their opinion is everywhere in the tech news, this week. And they’re a very influential company. Completely dominating some markets like messengers, parts of social media. Also well-known in the AI industry.
Other companies do the same. They test what they can get away with all the time. Like stealing Scarlett Johansson’s voice, pirating books on bittorrent… And they definitely have enough influence and money to pay very good lawyers. Choose what to settle out of court and what to fight. We shouldn’t underestimate the copyright industry. But Meta for example is a very influential company with a lot of impact on society and the world.
And AI is in half the products these days. Assisting you, or harvesting your data… Whether you want it or not. That’s quite some reach, pervasive, and those are the biggest companies on earth. I’d be with you if AI were some niche thing. But it’s not.
And Meta are super strict with trademark law and parts of copyright when it’s the other way around. I lately spent some time reading how you can and cannot use or mention their trademark, embed it into your website. And they’re very strict if it’s me using their stuff. The other way around they want free reign.
subsidising big companies […] What do you mean by that?
I mean manifacturing a supply chain for them where they get things practically for free. Netflix has to pay for licenses to distribute Hollywood content. OpenAI’s product also has other people’s content going into the product, but they don’t need to do the same. It’s subsidised and they get the content practically for free for their business model.
And what do you think I do with my server and the incident last week? If I now pay $30 more for a VPS that’s able to withstand Alibaba’s crawlers… Wouldn’t that be a direct sunsidy from me towards them? I pay an extra $30 a month just so they can crawl my data?
AI models may not be copyrightable […] // They are probably copyrightable […]
We were talking about a specific lecture that questions the entire concept of copyright as we have it now. You can’t argue to abolish copyright and then in the next sentence defend it for yourself or your friends. It’s either copyright for book authors and machine learning models, or it’s none of them. But you can’t say information in the products from other people is not copyright, but the information in the products of AI companies is copyright. That doesn’t make any sense.
The current thing is Meta is very vocal about the EU AI act.
And they’re not wrong.
That doesn’t quite back up what you claimed, though. You wrote: “They want everyone else to lose rights, while they themselves retain full rights,”
Their claim of Fair Use seems straightforward. That’s not everyone else losing their rights. I am not aware where they lobby for “full rights” for themselves, whatever that means.
And Meta are super strict with trademark law and parts of copyright when it’s the other way around. I lately spent some time reading how you can and cannot use or mention their trademark, embed it into your website. And they’re very strict if it’s me using their stuff. The other way around they want free reign.
There are different kinds of intellectual property. Trademarks are different from copyright. Then there’s also trade secrets, patents, publicity rights, privacy, etc.
Generally, you can use any Trademark as long as you don’t use it for trade or harm the business that owns it. I’m not going to look it up but I’m guessing that the rules are around not giving a misleading impression of your page’s relationship with Meta.
As for copyright, when you are in the US you can make Fair Use of their materials, regardless of what the license says.
That you can’t do that in Europe is not Meta’s fault.
I mean manifacturing a supply chain for them where they get things practically for free.
Oh. You’re talking about Net Neutrality and not copyright. I’m afraid I don’t know enough about the network business to form an opinion on that.
I don’t think what happened to you was a subsidy, though. You’re offering something for free, and apparently Alibaba took advantage of you for that. That’s just how it is, sometimes.
We were talking about a specific lecture that questions the entire concept of copyright as we have it now.
I touched on a lot of subjects. In a nutshell, I am against rent-seeking. No more, no less.
I know those narratives, as the humanities people call this. I don’t know if you know the term. You know commercials. They rarely give you facts. They don’t give you technical data about performance, durability, or such. Usually, a commercial is a little story, maybe just a few nice people having fun. When you see the product and think about buying, you can see yourself living that story. Maybe you see yourself in a new car speeding unhindered down an empty road; not stuck in traffic like those suckers you see every day in reality.
You don’t convince people with facts. You use psychological manipulation. If you think about history, people mostly believed religious stories about what happened in the world. That many people in developed countries defer to scientific facts is unusual. Of course, many don’t. The stories are much nicer. Let’s face it: The only reason we put up with ugly, meaningless facts is because we are reliant on technology.
We want the good life. We want to be healthy, and not having to worry about food or shelter. We want comforts, like flowing hot and cold water; an extravagant luxury for most humans throughout history and even today. In war, we want the best weapons, so that it is the other guys who do the dying.
So the question is: Do you prefer the feel-good-story or do you want a society that works for everyone?
You cannot have both.
I’m a bit in the science/facts bubble. I mean sure, advertisements and narratives are effective, and I’m not exempt. But I’d like to know the truth. And have politics based on scientific evidence. The goal is to strive and have a nice life, eveyone should be happy if possible. And then we use science to tell what kind of laws we need. Are all students delegating their homework to ChatGPT and they don’t learn anything anymore? Find ways so school achieves it’s goal. Do we confuse reality and fiction? Find ways to mitigate for that, e.g. watermarking. Do we loose all artists and creative people? Find ways so they can be part of society… I mean sometimes we can have a cake and eat it too, especially with technology. But we need to be clever.
I mean in the past we’ve adopted to new technology. One example which is often cited in context with AI is channging from horses to cars. That was very disruptive as well. I think today’s situation is a bit different. And for example copyright barely works in the digital age. But AI is likely going to have a massive impact on society. Maybe we need to re-think capitalism. That’s not necessarily good or bad or a “narrative”. But somehow things need to be addressed.
Europe has clearly chosen a path that will increase its technological dependency on either the US or China. It’s not likely to play a large role in figuring out the future economic order. We’ll see how long it can continue on this path.
Its AI policies are reminiscent of Feudalism. People create AI, but then they have to pay a levy to people who have contributed nothing. But they have rights awarded by the government. AI is not the only area where the EU is shifting to policies that facilitate wealth extraction rather than creation. I don’t think that is domestically sustainable. Sooner or later the European nations will try to extract wealth from each other and that will be the end. It doesn’t have to go that far. Maybe we will just see a stagnation and decline, as in South America.
Is your stands limited to AI or do you generally condone paying a levy? Like towards Spotify or Netflix or Hollywood, because I could as well skip that and watch the newest movies without obeying their copyright…
I mean it’s not nothing, there is some effort people put into things. Like the Wikipedia is super useful for machine learning. My computer code on Github teaches AI programming. And I can see the crawlers at my own server and today I had to update my config because it’s been hammered by Alibaba. Dozens of different IP addresses, fake user agent and they completely overloaded my database with requests. It’s not like I don’t contribute or am part of a different world?!
That’s a bit of an odd question, given my praise of American Fair Use. The USA has had copyright, including Fair Use, for longer than much of Europe. The predecessor of modern copyright law was created in the 1700s in the UK. There is a German scholar, Eckhard Höffner, who argues that this caused book production to plummet in the UK. He also says that the German-speaking lands produced more books, more different books, than the UK in the century before such laws arrived.
The American founding fathers were men of the Enlightenment. They, or some of them, understood the problems with such government sponsored monopolies. Therefore, the US Constitution limits copyrights and patents. It’s an interesting clause. Congress is empowered “To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries”. It’s about progress first; very much a product of the Enlightenment.
I don’t know if there was ever a discussion if entertainment should qualify at all for copyright protection. I have to try to look it up at some time.
In 1998, US copyright was extended by 20 years. Now it is life of the author +70 years. That has been called the Mickey-Mouse-Protection-Act, because it meant that the original Mouse enjoyed another 20 years of copyright This was roundly criticized by economists and even lead to a case before the Supreme Court. Obviously, making copyright retroactively longer does not encourage any kind of creativity. It’s in the past. Well, the case was lost, nevertheless.
For many left/liberal people, this is corruption; just the Disney company getting what it wants.
The EU countries had expanded their copyright years earlier, without resistance or even comment. Smug Europeans may feel superior when Americans rage against the corporations. But the truth is often like this, where Europeans simply quietly accept such outrages.
The original copyright in the US (and before that in the UK) was 14 years. Copyright protection required registration. It worked like the patent system. The interesting thing is that patents still work a lot like that. One must register and publish them and then they last for 20 years. Patents still have a 20-year duration. Meanwhile, copyrights have gone from 14 years to life+70 years, no registration required.
Patents are public so that people can learn from them. That has been used as an argument for patents. The alternative would be that everyone tries to keep new inventions secret. This way, people can learn and try to circumvent patents; find other ways of achieving the same thing. That’s an interesting observation in light of AI training, no?
I haven’t answered your question. In my experience, pro-copyright people will always refuse to argue over what should be covered by copyright or how long. They demand an expansion and use psychological manipulation to get it. If you do not let yourself be manipulated, they change the subject and will argue if copyright should exist at all. I have never met a single person who was able to defend copyright as it exists. Perhaps you can answer own question now.
Yes, I mainly wanted to rule out the opposite. Because the multi billion dollar companies currently do some lobbying as well. Including the same manipulation and narratives, just the other way around. They want everyone else to lose rights, while they themselves retain full rights, little to no oversight… And that’s just inherently unfair.
As I said. Copyright might not be something good or defendable. It clearly comes with many obvious flaws and issues. The video you linked is nice. I’d be alright with abolishing copyright. Preferrably after finding a suitable replacement/alternative. But I’m completely against subsidising big companies just so they can grow and manifest their own Black Mirror episode. Social scoring, making my insurance 3x more expensive on a whim and a total surveillence state should be prohibited. And the same rules need to apply to everyone. Once a book author doesn’t get copyright any longer, so does OpenAI and the big tech companies. They can invest some $100 million in training models, but it’s then not copyrighted either. I get to access the model however I like and I can sell a competing service with their model weights. That’s fair and same rules for everyone. And Höffner talks to some degree about prior work and what things are based upon. So the big companies have to let go of their closely guarded trade secrets and give me the training datasets as well. I believe that’d be roughly in the spirit of what he said in the talk. And maybe that’d be acceptable. But it really has to be same rules for everyone, including big corporations.
Can you back this up? They certainly do not have the same reach or influence as the copyright industry.
What do you mean by that?
AI models may not be copyrightable under US law. I’m fairly sure that base models aren’t. Whether curating training data, creating new training data, RL, and so on, ever makes a copyrighted model is something that courts will eventually have to decide.
They are probably copyrightable under EU law (maybe protected as databases). That’s an EU choice.
The rules are different in different countries. They are not different for corporations.
The current thing is Meta is very vocal about the EU AI act. Their opinion is everywhere in the tech news, this week. And they’re a very influential company. Completely dominating some markets like messengers, parts of social media. Also well-known in the AI industry.
Other companies do the same. They test what they can get away with all the time. Like stealing Scarlett Johansson’s voice, pirating books on bittorrent… And they definitely have enough influence and money to pay very good lawyers. Choose what to settle out of court and what to fight. We shouldn’t underestimate the copyright industry. But Meta for example is a very influential company with a lot of impact on society and the world.
And AI is in half the products these days. Assisting you, or harvesting your data… Whether you want it or not. That’s quite some reach, pervasive, and those are the biggest companies on earth. I’d be with you if AI were some niche thing. But it’s not.
And Meta are super strict with trademark law and parts of copyright when it’s the other way around. I lately spent some time reading how you can and cannot use or mention their trademark, embed it into your website. And they’re very strict if it’s me using their stuff. The other way around they want free reign.
I mean manifacturing a supply chain for them where they get things practically for free. Netflix has to pay for licenses to distribute Hollywood content. OpenAI’s product also has other people’s content going into the product, but they don’t need to do the same. It’s subsidised and they get the content practically for free for their business model.
And what do you think I do with my server and the incident last week? If I now pay $30 more for a VPS that’s able to withstand Alibaba’s crawlers… Wouldn’t that be a direct sunsidy from me towards them? I pay an extra $30 a month just so they can crawl my data?
We were talking about a specific lecture that questions the entire concept of copyright as we have it now. You can’t argue to abolish copyright and then in the next sentence defend it for yourself or your friends. It’s either copyright for book authors and machine learning models, or it’s none of them. But you can’t say information in the products from other people is not copyright, but the information in the products of AI companies is copyright. That doesn’t make any sense.
And they’re not wrong.
That doesn’t quite back up what you claimed, though. You wrote: “They want everyone else to lose rights, while they themselves retain full rights,”
Their claim of Fair Use seems straightforward. That’s not everyone else losing their rights. I am not aware where they lobby for “full rights” for themselves, whatever that means.
There are different kinds of intellectual property. Trademarks are different from copyright. Then there’s also trade secrets, patents, publicity rights, privacy, etc.
Generally, you can use any Trademark as long as you don’t use it for trade or harm the business that owns it. I’m not going to look it up but I’m guessing that the rules are around not giving a misleading impression of your page’s relationship with Meta.
As for copyright, when you are in the US you can make Fair Use of their materials, regardless of what the license says.
That you can’t do that in Europe is not Meta’s fault.
Oh. You’re talking about Net Neutrality and not copyright. I’m afraid I don’t know enough about the network business to form an opinion on that.
I don’t think what happened to you was a subsidy, though. You’re offering something for free, and apparently Alibaba took advantage of you for that. That’s just how it is, sometimes.
I touched on a lot of subjects. In a nutshell, I am against rent-seeking. No more, no less.
BTW, that turned out to be a false.