Hello everyone!
It’s time for another long-overdue update on how Fedecan and our various sites are doing. It’s been just over two years since the great Reddit migration, and in that time we’ve made some solid progress:
- Registered Fedecan.ca as a non-profit
- Moved from OVH to our own hardware in a Vancouver datacenter
- Started collaborating with the team behind sh.itjust.works
- Launched Pixelfed.ca
- Launched Piefed.ca
Finances
Here’s a look at our bank balance since we began accepting donations:
We’re currently sitting at around $2,900, with a monthly burn of about $200, which gives us roughly a year of runway. We have some additional annual costs (like domain renewals and non-profit registration), but overall we run very lean.
Fedecan still owes:
- TruckBC: $1,980
- Shadow (me): $525
These were out-of-pocket hosting and non-profit registration costs from 2023/2024. It’d be great to get those covered, but we want to keep at least a year of operating expenses in reserve.
If you’re a regular user and value what we’re doing, please consider donating! We have multiple ways to donate, you can find the comparison and donation links on our website: https://fedecan.ca/en/donate
Sh.itjust.works
Nothing major to report here - we’ve all been a bit busy lately, but collaboration is continuing slowly behind the scenes.
Fediverse Growth
We’re seeing a healthy volume of posts and communities on lemmy.ca, surging with each Reddit drama:
Infrastructure
Our server is a Dell R7515 with an EPYC 7763, 1 TB ram and 4x 7.68tb nvme data disks, which is hosted in a datacenter in Vancouver, BC.
I spun up victoriametrics + victorialogs a few weeks ago and have been ingesting all of our data, giving us the ability to put together some nice grafana dashboards.
Everything is running great on the infrastructure side of things. Our server is barely working up a sweat and we shouldn’t have to worry about scaling for a long time.
Lemmy.ca still comprises almost all of our traffic:
Lemmy.ca
Our over provisioned stack is performing well, handling the occasional lemmy / lemmy-ui dropout:
Similarly the DB is mostly running out of ram:
Our object storage is slowly climbing as expected, but we’ve got several years of capacity to figure out a long term solution:
I’m also doing some limited analytics on our web logs. As expected, lemmy.world makes up the majority of our federation traffic:
One interesting thing to see from the user-agent data is the breakdown of traffic by the different mobile clients:
The “dart” UA is just a common web library, Thunder reports as this and I suspect other clients do too. If you’re a client developer, please set your user-agent!
Out of the alternative web clients we support, tesseract is the most popular although the overall traffic volume is still low:
We only store 7 days of logs but I’m hoping to get these pulled out into metrics soon, since it would be interesting to track which clients / interfaces people use over time.
Pixelfed.ca
Not much to say on this one, due to using local storage it currently runs on a single VM without redundancy.
Piefed.ca
Piefed runs on a pair of VMs with its own database and object storage backends.
Cloudflare
If you want to compare against previous data posts, here’s our same cloudflare graphs for lemmy.ca
As always, feel free to reach out if you have any questions or ideas. Thanks for being a part of the Fediverse!
You must log in or # to comment.
As requested, here’s some additional info about donation methods / sizes / volume!
Thanks, that’s quite informative. I take it the Stripe option is new. If those trends continue, it looks like things are currently financially secure, but not necessarily paying things back soon or building much of a reserve for replacement hardware. Hopefully the added awareness gets a few more donors. Given the low costs we’re seeing, even $10/year will make a notable difference.
No we launched with stripe, but if you look at our site you’ll see we don’t push it heavily since we pay more fees there.
I have some questions 😄. What’s being used for operating systems and are these just connected straight to the internet with 1:1 NATs, or is there most balancing or firewalls involved?
What kind of virtualization technology is being used?
Are all these hosted on the same physical server?
Single physical host running proxmox. Mix of Ubuntu vms and lxc depending how much I trust the app and if I need direct access to the underlying zfs.
Opnsense + haproxy in front, nothing connected directly.
♥️ opnsense and zfs
How did we get so lucky as to get a competent team willing to put in this much work…thanks so much, really.
My thoughts exactly, thank you!
I need to show people I work with how little hardware you can use to serve this much traffic. We do around 500/second but our monthly EC2 spend is horrendous for the scale of traffic we get. But rails is what it is…
I think you have it all covered, but are there ways people can contribute with time? I’m mostly k8s focused these days though.
Not really, the tech is stable and I honestly put very little time into it. Occasionally I get bored and do something like make dashboards, but ongoing maintenance is just the occasional software update.
I’d like to move it all to k8s at some point, but it’s a really low priority. Probably doesn’t make sense until we start scaling to multiple machines.
Request for info to be added to fundraising page: which fundraising payment options are processed by American companies. If they’re taking fees, I’d like those fees to go to Canadian firms.
Fuck it, sending an e-transfer. ;)
E: sent $100. The equivalent of two one-year reddit subscriptions. I cancelled my subscription to move over here. Good luck with the fundraising :)
There’s a reddit subscription? Whatever for??
Before the enshittification took root, reddit was where I interacted the most with people online. You could buy a subscription to get an ad-free experience. It came with a bunch of Reddit gold you could hand out. Basically, help pay for server time, and get an ad-free experience without adblocking.
The amount of time I spent on Reddit I would compare to a Netflix subscription or similar.
In the ramp up to their IPO when they started to fuck with everything, my goodwill evaporated. I stopped moderating and abandoned my subs. Whenever I had new content I wanted to post, I instead post to the fediverse. So, there ya go 😅
Good idea, they all do except interac since everything routes through stripe. It’s absurd the amount of transaction data they get. Thanks!
Appreciate you sharing the data like this. It’s cool to peek behind the curtain!
Is there any regular offsite backup of data?
Yes, shipped to a server at my house.
Thank you for keeping this instance alive, Also nice transparency very appreciated
I’m a sysadmin by day and… Only one host? And it needs 1TB of RAM? Seems excessive. Probably not though. IDK how optimized Lemmy is.
There’s a lot I could say here but I’ll limit myself to this: the proof is in the pudding. Lemmy.ca is working well. You all should be very proud of what you’ve done.
Two hosts would cost more in colo fees and we don’t need the resources. The redundancy would be nice, but this is enterprise grade hardware covered under Dell prosupport. I also have an identical box running my home systems, so I could steal parts temporarily if we had an issue.
Yeah the ram is just what the hosts came with. It’s way more than we need and so everything is very over provisioned.
All fair. I’m not trying to say you’re doing it wrong at all, quite the opposite.
And yes, redundancy is nice, but it really depends on the importance of the data on the system and the budget.
To be blunt: if Lemmy.ca goes down for any length of time, that would suck for everyone here, but nobody will die, there won’t be any loss of profits or whatever… In business talk, the risk of what could be lost due to an outage is less than the cost of the hardware to prevent an outage.
I understand your position and an in warranty Dell server system isn’t cheap.
What you’re currently going is clearly working. So I don’t have any complaints.
It doesn’t need it. That hardware is just what was available.
Thank you for the great job! 🙏 🚀
Let me ask two explicit questions:
-
Considering costs and total number of users, how much should be a user’s monthly donation to keep things even or a little on the safe side?
-
Considering costs and total number of donators, how much should be a donator’s monthly donation to keep things even or a little on the safe side? This is a more realistic estimate, as there are users (say, students) who can’t pay (and of course users who simply don’t want to pay).
Many Fediverse initiatives seem too shy to give this kind of information, but I think there’s nothing wrong about it. Please tell us in time if the economy were to be going bad, nobody wants another lemm.ee event :) As Impossible Mission for the Commodore 64 used to say:
Stay awhile, stay forever!
I have no problem sharing anonymized info for things like this, it’s just difficult to pull.
We accept donations a few different ways, they can be one time or recurring, we don’t know who people are , and we often get paid weeks after they happen once it gets reconciled by stripe.
Let me see what I can do.
Re: Lem.ee that’s kinda a different problem. They weren’t financial issues afaik, but admin burnout. I don’t think that’s a concern on our side at this time.
We can get a rough estimate for your first question with the information we have. They’ve shared that it costs them about $200/month, and we can see from the sidebar that we have 3k users per 6 months (estimate for number of active users). That means approximately 7c/month per user.
Cheers! Then it’d be quite cheap if every user gave their 10c/month. Let’s see what they say about actual donor-users.
-
That’s really cool that admins can tell which clients folks are using, that would be very helpful for developers to gauge user satisfaction.
Thank you for the update!
Sent you something in addition to the recurring to help with that repaying. ☺️
I tried piefed.ca but it was pretty broken. Importing filter blocks will cause internal server errors and fail every time. Some communities including the emulation community I moderate on sh.itjust.works show up blank except stickies. Constant random connection errors similar to lemmynfsw.
Was this recently? I just pulled in some more fixes a few hours ago. It’s still under very active development.
Hello Shadow! Just chiming in that import settings is also broken for me. The json file is about 35 kb or so. It just tells me ‘this might take a few minutes’ but nothing imports. I tried a few times yesterday and again just now. It works for my Piefed.social account though.
As always, thanks for all you do. :)
Willing to email me your file so I can try? Shadow@lemmy.ca
Sorry for the late reply! I’ve been on my main and forgot I made this comment using the Piefed acc lol. I just tried importing again and it worked! Thank you for following up anyways. :)
