We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning across all 3 modalities of speech, sound, and music; (ii) flexible, on-demand thinking, allowing the model to do chain-of-thought-type reasoning before answering; (iii) multi-turn, multi-audio chat; (iv) long audio understanding and reasoning (including speech) up to 10 minutes; and (v) voice-to-voice interaction. To enable these capabilities, we propose several large-scale training datasets curated using novel strategies, including AudioSkills-XL, LongAudio-XL, AF-Think, and AF-Chat, and train AF3 with a novel five-stage curriculum-based training strategy. Trained on only open-source audio data, AF3 achieves new SOTA results on over 20+ (long) audio understanding and reasoning benchmarks, surpassing both open-weight and closed-source models trained on much larger datasets.
In the US, it almost certainly wouldn’t be fair use if it meant that the author doesn’t get paid. Of course, you don’t get paid for the fair use, but there are a lot of things you don’t get money for.
You’re talking about authors not being paid at all. What’s that about?
That was about abolishing copyright altogether. Since we discussed that as an option. We’re now discussing what I called “subsidies” earlier. Authors do get paid, but for certain “uses” and not for others. And authors get financed by a different group of people.
In your example with the farmers, they’re not paid by me buying the product in the supermarket and that money gets handed down the chain to every supplier… But Nestle got the cocoa beans for free and society now gets to pay the farmer by a different method. Unless you have a specific proposal here, that’d be likely the definition of a subsidy to help Nestle and make their products look cheaper on a supermarket shelf.
A cocoa farmer is paid for some uses of their cocoa beans but not others. For example, Nestle has to pay to turn their beans into chocolate and sell it in supermarkets. On the other hand, no one has to pay to take a photo of their beans and sell it to Nestle for ads. Right? I’m with you so far.
I don’t get the next step. Because some uses are free, all uses should be free? Then Nestle gets a subsidy and we pay the farmers some other way?
Pretty much. The AI companies are Nestle in that analogy. They get their supplies for free?! While I and everyone else had to pay for the very same supplies, when I needed the textbook to study CS to become a computer programmer. The professor gets to brush up their salary, and I think it’s a bit unfair to me that I’m asked to take out 60€ from my mediocre turnover of a few hundred bucks a month as a student. I think I should have been asked to pay 30€ and a company with a billion dollar budget should be asked to pay something like 100€ since they make use of it multiple times. And they should hand that cost down to their customers. And my use was transformative as well. The information from the textbook is now modeled in my brain.
I think the analogy with the picture is kind of alright as well. I mean analogies are hard here, since it’s a labour intensive task to duplicate crops and coffee beans, while duplication is pretty much for free in case of information. And it doesn’t take away the original.
Now what is a picture? It’s kind of a summary, a depiction of the outer appearance. And snapping a picture of a book cover would make sense for Fair Use. That’s kind if what it’s made for. If you now snap a picture of each and every one of the 400 pages inside, that’s where law says Fair Use stops. And what do AI companies use for training? A picture/summary of the book? Or the content within?
Your professor abused their monopoly. That’s the sort of thing I’ve been condemning. You are basically fine with that. You just think they should adjust their price policy to income. Well, yes, that would be the profit maximizing move. You make everyone pay as much as they are able to. That’s what the copyright lobby wants. But I have to point out: There is no reason why they should lower the price for you. After all, you were able to pay. Rather, there seems to be room to raise the price.
Do you actually think this kind of monopoly abuse is a good thing?
Now what is a picture? It’s kind of a summary, a depiction of the outer appearance. And snapping a picture of a book cover would make sense for Fair Use. That’s kind if what it’s made for. If you now snap a picture of each and every one of the 400 pages inside, that’s where law says Fair Use stops.
No, that’s not what the law says. I think, the problem is that we have different ideas over how Fair Use in the US actually works. I’ll have to think about that.
Well, it’s complicated. And depends on which theoretical option we’re talking about. I for example think writing the textbook when you’re the professor and selling that to the students is a very bad thing. I’m not fine with that at all. They should be funded mainly by taxpayer money (at least that’s what we do). And the fruit of their labour should then be owned by the taxpayer. The US does similar things, like government texts, NASA pictures etc used to be owned by the people. And everyone is “the people” from a random student to a big AI company.
It’s a bit a special example though, and doesn’t translate 1:1 to the private book market.
I believe your regular book author does it the other way around. They aren’t commissioned by anyone, they generally write it and only after that does the product get monetized. And I believe that’s where your “rent-seeking” comes in. Somehow the author managed to feed themselves for the time it took them to write the book, and now they have it as an asset which they can try to turn into as much money as they can. It’s two things mushed together. Their valid desire to eat and be compensated for their labour, plus the rent from the asset which might be huge for popular books and doesn’t reflect labour cost. And all of this is very different from a university professor with a salary. It could and should be decoupled for them. But it’s straight up impossible for the majority of authors, given our current copyright model. I think that’s a fundamental limitation of capitalism.
And I wonder if those regulatory mechanisms are even applied correctly. I had that with the textbooks in university to some lesser extent. School was fine. But I heard in the US for example education is a complete rip-off and we get news articles every year on how parents can’t afford the several hundred bucks for school textbooks for their children. And that is despite a different copyright doctrine. Maybe our model here leads to better results some times, I don’t really know.
And concerning the Fair Use: Is there law which offers an option for compensation? I thought that was contradictory per definition.
Hmm. You seem to treat an economic rent as being the same as a return on investment. Any particular reason for that?
I for example think writing the textbook when you’re the professor and selling that to the students is a very bad thing. I’m not fine with that at all.
You aren’t fine with that. But why are you fine with the copyright industry doing it to everyone in the country?
I don’t think my opinion as some random dude matters here. I could uphold arbitrary stupid believes. But this is kind of a factual question. So whether I personally, as one person, am fine with something is of no concern here. The question is, how do we arrive at a consistent economy model for immaterial goods…
And I think I wrote like 5 times now that I’m NOT fine with that. I said I view it as a (necessary) evil. It is evil in the sense of bad, I’m not fine with it, it comes with severe issues, we should do better than that. However “is” and “should” are two seperate things. We happen to live on a world that came up with copyright. It exists. We made a pact with the devil to address one thing. And I’m merely acknowledging that. Since it does exist, I need to deal with it. That’s not agreement from my side. Copyright serves one legitimate purpose. It applies our capitalist economy to immaterial goods. It’s supposed to allow individuals and companies to create, and trade with more than just cocoa beans. But it’s complicated and we might have come up with a stupid way to do it. And a way that simultaneously has lots of negative side-effects.
And now what? That is the question. Do we abolish it? Do we replace it with something else that handles the one legitimate purpose a better way? Do we retrofit it and try to “patch” it? Do we do that just for AI? Or for more than just one use-case?
And I think I make a point about how return on investment and an economic rent are two distinct things. Yet they’re in practice falsely(!) mushed together, which again is bad… Or am I mistaken and I can pay an artist for their investment but not pay a rent? I don’t think there is a good way to do it with the current model. That means I get to treat both as the same. You seem to be under the impression I like it. But I don’t. It’s just that I have to abide by law and that currently mandates me to do it.
Or am I mistaken and I can pay an artist for their investment but not pay a rent?
A return on investment is not the same as an economic rent.
Let’s go back to the farmer example. You agree that a monopoly on the food supply is a bad thing. It can and will be abused.
Sidenote: You suggested that the government should produce textbooks to prevent abuse. Would that also be your solution here? Would that be preferable to the current arrangement?
Now, let’s look at the situation of a farmer more closely. A farmer has to do a lot of work before they can harvest. They also need stuff like seeds, fertilizer, pesticides, fuel, machinery, spare parts and maintenance, and so on.
In the old times, one held back part of a grain harvest as seed grain for next year. That is an investment in the economics sense. You don’t consume everything, but keep it so that you have more in the future. The finance meaning is subtly different but never mind.
Farmers gets a return on investment. They invest money and labor so that there is a harvest in the future. They could sell the equipment they already own to have more spending money now.
A ROI is part of a farmers’ income but is not economic rent.
Back to authors. An established author will get an advance before they write the next book. That’s investment by the publisher. If they don’t get an advance, then the author is making the investment, but let’s ignore that for simplicity. Investments are always risky. In this case, some books don’t sell well and don’t make back the money.
As a publisher, how much money would you invest in future books to maximize your profit? It depends on the expected payout and the cost of money.
Cost of money: You could borrow the money. Then the cost of the money is the interest on the loan. Or you could use the money for something else, eg buying safe government bonds. In that case, the cost is an opportunity cost. It’s what you miss out on by not investing elsewhere.
Expected payout: It’s the average profit/loss on each book. It is something you estimate based on experience.
The more books there are on the market, the lower the average profit. There must be a limit to how much of their income people are willing to spend on books. At some point, you have a lot of similar books chasing the same audience. That lowers the average. To maximize your profit, you invest in the production of more and more books, until the average return on each book is equal to the cost of money.
In the US, it almost certainly wouldn’t be fair use if it meant that the author doesn’t get paid. Of course, you don’t get paid for the fair use, but there are a lot of things you don’t get money for.
You’re talking about authors not being paid at all. What’s that about?
That was about abolishing copyright altogether. Since we discussed that as an option. We’re now discussing what I called “subsidies” earlier. Authors do get paid, but for certain “uses” and not for others. And authors get financed by a different group of people.
In your example with the farmers, they’re not paid by me buying the product in the supermarket and that money gets handed down the chain to every supplier… But Nestle got the cocoa beans for free and society now gets to pay the farmer by a different method. Unless you have a specific proposal here, that’d be likely the definition of a subsidy to help Nestle and make their products look cheaper on a supermarket shelf.
Let me try to follow this.
A cocoa farmer is paid for some uses of their cocoa beans but not others. For example, Nestle has to pay to turn their beans into chocolate and sell it in supermarkets. On the other hand, no one has to pay to take a photo of their beans and sell it to Nestle for ads. Right? I’m with you so far.
I don’t get the next step. Because some uses are free, all uses should be free? Then Nestle gets a subsidy and we pay the farmers some other way?
Pretty much. The AI companies are Nestle in that analogy. They get their supplies for free?! While I and everyone else had to pay for the very same supplies, when I needed the textbook to study CS to become a computer programmer. The professor gets to brush up their salary, and I think it’s a bit unfair to me that I’m asked to take out 60€ from my mediocre turnover of a few hundred bucks a month as a student. I think I should have been asked to pay 30€ and a company with a billion dollar budget should be asked to pay something like 100€ since they make use of it multiple times. And they should hand that cost down to their customers. And my use was transformative as well. The information from the textbook is now modeled in my brain.
I think the analogy with the picture is kind of alright as well. I mean analogies are hard here, since it’s a labour intensive task to duplicate crops and coffee beans, while duplication is pretty much for free in case of information. And it doesn’t take away the original.
Now what is a picture? It’s kind of a summary, a depiction of the outer appearance. And snapping a picture of a book cover would make sense for Fair Use. That’s kind if what it’s made for. If you now snap a picture of each and every one of the 400 pages inside, that’s where law says Fair Use stops. And what do AI companies use for training? A picture/summary of the book? Or the content within?
Now I’m even more confused.
Your professor abused their monopoly. That’s the sort of thing I’ve been condemning. You are basically fine with that. You just think they should adjust their price policy to income. Well, yes, that would be the profit maximizing move. You make everyone pay as much as they are able to. That’s what the copyright lobby wants. But I have to point out: There is no reason why they should lower the price for you. After all, you were able to pay. Rather, there seems to be room to raise the price.
Do you actually think this kind of monopoly abuse is a good thing?
No, that’s not what the law says. I think, the problem is that we have different ideas over how Fair Use in the US actually works. I’ll have to think about that.
Well, it’s complicated. And depends on which theoretical option we’re talking about. I for example think writing the textbook when you’re the professor and selling that to the students is a very bad thing. I’m not fine with that at all. They should be funded mainly by taxpayer money (at least that’s what we do). And the fruit of their labour should then be owned by the taxpayer. The US does similar things, like government texts, NASA pictures etc used to be owned by the people. And everyone is “the people” from a random student to a big AI company.
It’s a bit a special example though, and doesn’t translate 1:1 to the private book market.
I believe your regular book author does it the other way around. They aren’t commissioned by anyone, they generally write it and only after that does the product get monetized. And I believe that’s where your “rent-seeking” comes in. Somehow the author managed to feed themselves for the time it took them to write the book, and now they have it as an asset which they can try to turn into as much money as they can. It’s two things mushed together. Their valid desire to eat and be compensated for their labour, plus the rent from the asset which might be huge for popular books and doesn’t reflect labour cost. And all of this is very different from a university professor with a salary. It could and should be decoupled for them. But it’s straight up impossible for the majority of authors, given our current copyright model. I think that’s a fundamental limitation of capitalism.
And I wonder if those regulatory mechanisms are even applied correctly. I had that with the textbooks in university to some lesser extent. School was fine. But I heard in the US for example education is a complete rip-off and we get news articles every year on how parents can’t afford the several hundred bucks for school textbooks for their children. And that is despite a different copyright doctrine. Maybe our model here leads to better results some times, I don’t really know.
And concerning the Fair Use: Is there law which offers an option for compensation? I thought that was contradictory per definition.
Hmm. You seem to treat an economic rent as being the same as a return on investment. Any particular reason for that?
You aren’t fine with that. But why are you fine with the copyright industry doing it to everyone in the country?
I don’t think my opinion as some random dude matters here. I could uphold arbitrary stupid believes. But this is kind of a factual question. So whether I personally, as one person, am fine with something is of no concern here. The question is, how do we arrive at a consistent economy model for immaterial goods…
And I think I wrote like 5 times now that I’m NOT fine with that. I said I view it as a (necessary) evil. It is evil in the sense of bad, I’m not fine with it, it comes with severe issues, we should do better than that. However “is” and “should” are two seperate things. We happen to live on a world that came up with copyright. It exists. We made a pact with the devil to address one thing. And I’m merely acknowledging that. Since it does exist, I need to deal with it. That’s not agreement from my side. Copyright serves one legitimate purpose. It applies our capitalist economy to immaterial goods. It’s supposed to allow individuals and companies to create, and trade with more than just cocoa beans. But it’s complicated and we might have come up with a stupid way to do it. And a way that simultaneously has lots of negative side-effects.
And now what? That is the question. Do we abolish it? Do we replace it with something else that handles the one legitimate purpose a better way? Do we retrofit it and try to “patch” it? Do we do that just for AI? Or for more than just one use-case?
And I think I make a point about how return on investment and an economic rent are two distinct things. Yet they’re in practice falsely(!) mushed together, which again is bad… Or am I mistaken and I can pay an artist for their investment but not pay a rent? I don’t think there is a good way to do it with the current model. That means I get to treat both as the same. You seem to be under the impression I like it. But I don’t. It’s just that I have to abide by law and that currently mandates me to do it.
I see. I think this is the big one:
A return on investment is not the same as an economic rent.
Let’s go back to the farmer example. You agree that a monopoly on the food supply is a bad thing. It can and will be abused.
Sidenote: You suggested that the government should produce textbooks to prevent abuse. Would that also be your solution here? Would that be preferable to the current arrangement?
Now, let’s look at the situation of a farmer more closely. A farmer has to do a lot of work before they can harvest. They also need stuff like seeds, fertilizer, pesticides, fuel, machinery, spare parts and maintenance, and so on.
In the old times, one held back part of a grain harvest as seed grain for next year. That is an investment in the economics sense. You don’t consume everything, but keep it so that you have more in the future. The finance meaning is subtly different but never mind.
Farmers gets a return on investment. They invest money and labor so that there is a harvest in the future. They could sell the equipment they already own to have more spending money now.
A ROI is part of a farmers’ income but is not economic rent.
Back to authors. An established author will get an advance before they write the next book. That’s investment by the publisher. If they don’t get an advance, then the author is making the investment, but let’s ignore that for simplicity. Investments are always risky. In this case, some books don’t sell well and don’t make back the money.
As a publisher, how much money would you invest in future books to maximize your profit? It depends on the expected payout and the cost of money.
Cost of money: You could borrow the money. Then the cost of the money is the interest on the loan. Or you could use the money for something else, eg buying safe government bonds. In that case, the cost is an opportunity cost. It’s what you miss out on by not investing elsewhere.
Expected payout: It’s the average profit/loss on each book. It is something you estimate based on experience.
The more books there are on the market, the lower the average profit. There must be a limit to how much of their income people are willing to spend on books. At some point, you have a lot of similar books chasing the same audience. That lowers the average. To maximize your profit, you invest in the production of more and more books, until the average return on each book is equal to the cost of money.
I’ll leave it at that for now.