Community networks are pretty common in rural areas in other countries. My favorites are guifi.net and b4rn.org.uk. They both use lots of fiber running through farms.
We tried repeating LinkNYC kiosks a few years ago but it wasn't reliable enough, making it too difficult to support.
We have instructions for a DIY connection on our website and a channel on our Slack where you can ask other members about problems. The DHCP lease tends to go dead every couple of months and we have various things to try when that happens.
Volunteers donate their time because they enjoy it. We have so many volunteers it is hard to keep track of them all. Obviously it is more efficient to use volunteers if there are enough. We are considering using contractors for some huge projects we are planning.
There are about 10 fiber providers in the city. We've spoken to most of them. It's pretty much impossible for us to lay fiber, so we have to lease it. We'd like to eventually lease a fiber ring across the city so we can cut costs when connecting multiple buildings. If you don't have a ring then you have to pay for the run all the way back to a coloc.
That's a good start to having "your own" network, and then you can choose to have packets traverse your network or the peering networks. You could shift egress points around, based on cost and other factors as well. Each link you add can multiply your routing options.
Get large enough, and you can start carrying other traffic too, and eventually start buying/laying fiber. But you know this, it's probably part of your business plan.
We originally used the qMp/OpenWRT package that included mdns, which is a version of zeroconf for the internal network. It isn't scaling so we now use our own DNS server to resolve internal addresses on our 10/ network.
Mesh networks are going to be incredible when they're widespread!
It's truly superior to centralized control by ISPs/Governments and the world will be grateful for this after this whole net neutrality debacle recently.
You guys are brilliant and doing an amazing job and I'm grateful people like you guys exist!
Since it sounds like you're associated with the project, can you address some of the specific concerns mentioned, like reliability? Do you disagree that hundreds of thousands of dollars are required for five nines connectivity? Do you disagree that that level of reliability is required for your customers etc.
No you don't need hundreds of thousands. You do need tens of thousands. We've had no downtime this year at supernode1. Yes reliability is a big priority, but it is not a big priority for the big ISPs in NYC. Time Warner Cable (now Spectrum) has barely two nines. In Manhattan it would go down in the East Village after every large storm. We are already providing faster more reliable connections than this.
What frequency do the point-to-point links run at? Have you considered using some sort of lower frequency packet radio for longer distance links, perhaps to other nearby nodes?
Our recurring costs are only for the lease at supernode1, which is around $1000/month. Divide that by our member node count (158) multiplied by users of each node ~4, 1000/(158*4)gives you under $2/month per user. Members usually pay for their own routers so that isn't included. To set up the supernode was about a $10K one-off charge for install fees, servers and antennas.
DE-CIX, our IXP, donated bandwidth to us, and we have transit also donated from Packet Host and WebAir. We actually pay nothing for bandwidth. Probably one day we will pay but it is not that expensive if you do this at an IXP and use peering.
We have an AirFiber pair that is 24Ghz, but all of our sectors are 5Ghz wifi.
We get a lot of support from the NYC networking community, and we have a lot of friends at the NYNOG meetups. Everyone is very helpful and they also want the big ISPs to have some competition, so we get donations from quite a few people. Some also join our network and get nodes on their own roofs and help with other installs!
DE-CIX is the biggest IXP and they measure their network in terabits/sec. It is not a big deal for them to donate us a 1 gig connection. (it's a big deal for us!) Eventually we may upgrade it to 10 gig.
So you're sharing a 1G connection between 600-ish users - all presumably on 802.11ac 450Mbps (or 1.3Gbps if they've got three-stream?) connections?
How often do you see full saturation on that link? I'm guessing if all 600 of your users all tried to stream one stream of $TV-show-de-jour at once they'd barely get 16Mbps each? Can one user's home office full of Apple gear flood the bandwidth downloading a dozen Apple software updates simultaneously?
That is a pretty low ratio in terms of over-subscription, on many cable networks you'd have 24 downstream Docsis channels @ 38Mbps usable per channel, for 912Mbps usable across all houses on said node. A single node often supports 500 homes, and that 912Mbps of capacity carries switched digital video, voice & data traffic.
Netflix standard streaming is 3Mbps, so even with your math (which isn't how bandwidth works) everyone is happy.
We've never come anywhere near saturating our gig connection. We monitor it. Each building is limited to their own rooftop connection which is an average 100Mbps, so they can't do more than that if they tried. The thing is not everyone is downloading a file at the same time, and streaming uses much less bandwidth than downloading. From Netflix site-
0.5 Megabits per second - Required broadband connection speed
1.5 Megabits per second - Recommended broadband connection speed
3.0 Megabits per second - Recommended for SD quality
5.0 Megabits per second - Recommended for HD quality
25 Megabits per second - Recommended for Ultra HD quality
Ah, so you get free bandwidth from people that want to balance their traffic for better peering contracts. That's clever. Probably not a helpful model for other areas that aren't bandwidth centers like NYC, and should probably be more out there when you're evangelizing so people understand you're getting a large portion of your operating expenses donated...
I have to assume another cost that's not noted is roof rights. That's not free - most people either have to pay or give the property management a kickback.
NYC is an interesting place for this - if you live in Manhattan, you clearly can afford the $80 for 1Gb/s FiOS or $40 for 100Mb/s FiOS or whatever Spectrum is charging. It would be way more interesting to plop this in a rust-belt city where people are on a paycheck-to-paycheck salary...
FWIW, I have many many companies as customers for whom I've offered pricing for three, four, and five nines of reliability (for mobile app backends, not residential ISP service, but lets run with it any way).
Not a single one of them has ever signed up for anything more expensive than three nines. Only a few have even discussed the differences between 3 and 4 nines solutions.
I'm reasonably sure I'd choose the same for my home internet connection - if offered representative pricing based on costs of providing 3, 4, or 5 nines, or possibly even 2 nines - I'd choose the least expensive because losing a few minutes or even occasionally 15mins a day of connectivity at home (or maybe more realistically an ~8 hour outage per month) really isn't going to bug me greatly - not if it's two or more orders of magnitude cheaper than a four of five nines connection.
Yep, DE-CIX is great and 32 Ave of the Americas is one of the most wired buildings in NYC. Great choice. Kudos on you folks, this is a great initiative. Reading this article made me happy. And I'm glad you are getting some press!
Actually - what you should do, is have a customer pay a fee of $100 to connect to the system - and take that $100 and buy another mesh was with each customer on boarding.
I think you misunderstand the goals of this community-run mesh. The FAQ has some info on why they don't charge, who runs routers, and what they're trying to accomplish.
Digi Desert LLC, which existed in 2010? Looks like that's the AS of some entity that predates the existence of your project and current public relations effort.
You announce a whopping total of two v4 /24.
Kudos for having enough clue to know that you needed to establish a presence at a major IX, and actually doing it, because it looks like you're adding peers. But your actual network presence is minuscule.
How do you intend to compete with the six NYC based companies I can think of off the top of my head that are putting fiber fed, $9,000 to $20,000 5 to 10Gbps 71-86 GHz PTP links on rooftops to build their own backbones, when you're playing around with 1000BaseT to the roof and AF24s?
You have high uptime? Do you have any of the following, because some of your much larger competitors sure do:
sites with parallel A and B side power systems
-48VDC rectifier + battery systems sized for 24 hour runtime at load
diesel generators
propane generators
generator resupply contracts
chassis-based routers with hotswap fan, N+1 power supplies, dual redundant routing engines
pair of identical core routers
singlemode fiber to the roof
-48vdc power to the roof
ironclad rooftop lease agreements with building owners, drawn up by professional telecom/real estate lawyers that run for 5+ year terms
Your original post was interesting because it raised legitimate questions about their operation. They answered those questions.
This follow up comes across as though you've drawn a conclusion and are now arguing towards that predetermined conclusion while ignoring the additional information provided. Plus your argument has kind of devolved from talking about specific concerns to throwing criticisms at the wall to see what sticks.
I'm sharing some harsh reality with them: The ISP market in NYC is highly competitive. If they want to be serious about it, it's not going to work as a nonprofit. I'm trying to tell them bluntly about what sort of infrastructure their competitors operate so that they can get an idea of the actual capital expenditure requirements involved in architecting/engineering a MAN-scale, five nines ISP composed of point to point wireless links.
Once upon a time there was a company called Microsoft. They hated free software! It costs millions to write an OS! Why is it free?? They hated the free guys so much. Then the free software got better. It got so much better than what you would have to pay for from Microsoft! Eventually Microsoft said oh well, and built the free software into their product. They even contributed back to that free software and everyone learned that you can have free and paid and as long as we all contribute the world is a better place. Thank you.
Once upon a time there was a chipmaker named Intel. Sun decided to open source its SPARC architecture. Other folks tried to design their own FOSS CPU architectures. Intel still dominates.
FOSS zealots like to point to the success of Linux over Windows (which is restricted to the server market, I might add), but there's little evidence that the FOSS philosophy is effective when it comes to physical infrastructure. Software requires bytes and labor. Bytes are cheap and labor can be donated or paid for by companies. Hardware requires fabs and factories, which are expensive.
Hello from just across the Hudson River! Keep up the great work. You are finding a way, despite what doubters and naysayers think! You will know what is possible by trying.
You're wasting your own time and those of your "customers" unless you have at least $750,000 to establish a serious presence, in my opinion. Amateur-hour WISP stuff is fine in a rural area. You're pretending to be a highly reliable capable ISP, and will eventually either overextend yourselves, run out of rubes to fund you as a nonprofit, or once you reach a size much bigger than you are now, get stomped on and obliterated by a much larger competitor that overbuilds your entire network with carrier-grade infrastructure and takes all your revenue.
This thread has to be one of the most overtly hostile threads I've read on HN in a while. What is your concern here? Why do you seem to feel they shouldn't even be trying?
We bypass traditional ISPs by connecting directly at an internet exchange point (DE-CIX IXP) and peering with other networks. This is basically how the internet is formed, by networks peering with each other. We don't need this ISP layer.
I'm pretty sure the real secret is ensuring your customers have no alternatives. Spectrum may or may not have all of that gear and they still have given me outage and throttling issues for years. I don't want somebody to compete with the terrible ISPs out there today - they're fundamentally the wrong kind of organization. I'd rather have honestly crappy service than an opaque monopoly dictating my access.
