My Home 10G Network Setup - What Works and Lessons Learned

As someone who values fast, reliable connectivity throughout my home, I've invested significant time and resources into building a robust home network. After numerous iterations and upgrades, I've learned some valuable lessons about what works well and what I would have done differently. In this post, I'll share my current setup and the insights I've gained along the way.

The Foundation: Fiber Internet

The backbone of my setup is Bell Fiber to the home, providing a symmetric 3 Gbps connection (they offer up to 8 Gbps now in Canada.) According to speed tests on Fast.com, I actually get around 5 Gbps, which is exceptional for a residential connection.

This level of bandwidth has been transformative for specific use cases:

1. Downloading large Steam games (50GB+)
2. Pulling down Ollama AI models
3. Downloading ISOs for various operating systems
4. Managing app and OS updates across multiple devices
5. Performing media backups to my NAS

Network Backbone: 10G Switching

My network core consists of a TP-Link TL-SX3008F switch with 8 SFP+ slots, providing 10G connectivity throughout my core infrastructure. This switch connects to my Bell GigaHub fiber receiver via standard Ethernet (since the HomeHub lacks SFP+ ports).

When looking at the rack photo, you might notice the variety of cable colors. This isn't just for aesthetics—there's a practical system behind it. I purchase all my cables in bulk from Infinite-Cables, and I consistently buy each cable length in a different color. This simple organization method allows me to throw all cables into a single drawer but immediately identify which cables are which lengths when I need them.

It's a minor optimization, but has saved me a lot of time over the years when trying to find the right length cable for a particular job. No more measuring or untangling cables to figure out if they're long enough!

Key Components:

• Primary 10G Switch: TP-Link TL-SX3008F with 8 SFP+ slots
• PoE Switch for Critical Devices: TP-Link SG3210XHP-M2
• PoE Switch for Non-Critical Gear: TP-Link SG2428P 24-port Gigabit
• Network Management: Omada OC300 controller

Connection Tip: DAC Wherever Possible

One important lesson: use Direct Attach Copper (DAC) cables whenever you can. They're more affordable than optical transceivers and don't require complex encoding/decoding between SFP+ ports. Moreover, I've found that 10Gbps Ethernet switches and SFP+ adapters run extremely hot compared to their 1Gbps counterparts. DAC and fiber SFP+ adapters generate significantly less heat, which is better for equipment longevity and power consumption.

My Fiber Connection Experiment

When I initially upgraded to 10Gbps networking, I purchased fiber transceivers to connect my switches. While fiber transceivers are indeed cheaper than 10Gbps Ethernet transceivers (~$30 vs. ~$60 each) and run cooler, this experiment ultimately proved to be a waste of time and money for my short-distance needs.

After some trial and error, I established a clear hierarchy for connections:

1. DAC cables: Best choice for most switch-to-switch connections and nearby machines / storages (cheap, simple, cool-running)
2. Fiber optics: Ideal for longer runs where DAC isn't practical
3. 10G Ethernet: Use only when absolutely necessary (hottest and often most expensive option)

For my home setup with relatively short distances between equipment, DAC connections have been the clear winner in terms of cost, simplicity, and reliability.

Storage Solution: NVMe-Based NAS

After experimenting with various storage configurations, I've settled on a TrueNAS setup with predominantly NVMe storage:

Current Configuration:

• Primary Storage: 8x 4TB NVMe drives in RAIDZ1 (fast, daily-use storage)
• Archival Storage: 4x 18TB HDD drives in RAIDZ1
• Hardware: AMD Ryzen 7 5700G with 128GB RAM
• Connectivity: 10Gbps DAC connection to primary switch

Evolution of My Storage Approach

I initially tried an HDD-only NAS with 12x 8TB drives, supplemented with SSD SLOG (ZFS Intent Log) and L2ARC (Level 2 Adaptive Replacement Cache). Despite these optimizations, the performance was disappointing when browsing directories and exploring content.

This led me to switch to a predominantly NVMe-based setup, which dramatically improved responsiveness. The simplicity of not needing to manage SLOG and L2ARC configurations has been an added bonus—the system is both simpler and faster.

For future consideration, I've been eyeing the Asustor Flashstor 12 Pro FS6712X with its 12-bay NVMe setup and compact form factor. However, I also use my current NAS to host various services that might not perform well on the less powerful Asustor hardware.

Client Devices and Connectivity

To fully leverage my 10G network:

• My daughter's gaming PC has a dedicated 10Gbps SFP+ card
• Media center connected to TV uses a USB 3.2 5Gbps Ethernet adapter
• My primary machine (MacBook Air M3 with 24GB RAM) connects through an OWL Thunderbolt Pro Dock with 10Gbps Ethernet
• The gaming PC doubles as an Ollama server and Windows testing machine, accessible via Windows Remote Desktop

Having all these devices on the same high-speed network as the NAS creates a seamless experience for file transfers, media streaming, and remote computing.

Security: The Decision to Stay Internal

I used to expose various services from my home network to the internet, thinking it would be convenient to access my data and applications remotely. However, over time, nearly every service I exposed ended up with serious security vulnerability alerts.

After experiencing this pattern repeatedly, I made the decision to stop exposing any services externally. The potential security risks simply weren't worth it. The constant vigilance required to maintain, patch, and secure these exposed services became an unnecessary burden.

This decision was made easier by the fact that I already use numerous cloud services for data storage and sharing (like Google Drive and GitHub), and I don't self-host any of my public-facing services. Between these cloud solutions and VPN access when absolutely necessary, there's really no compelling need to expose my local setup directly to the internet.

The peace of mind from knowing my home network is not directly accessible from the outside has been worth any minor convenience trade-offs. For my use case, the security-first approach has proven to be the right decision.

Home Infrastructure: What I Would Have Done Differently

Our house is relatively new (built 4 years ago) and is wired with Cat 6e Ethernet, which has handled 10Gbps connections without issues. However, there are several infrastructure decisions I would reconsider if I could do it again:

Ethernet and Networking

• More basement Ethernet ports: I installed only one, but as we added a gaming station and relocated the TV, four would have been ideal
• Garage Ethernet: Should have installed at least one port for an access point
• More access points: I overestimated coverage from ceiling APs and ended up hiding additional units under furniture to ensure complete coverage

Wireless Coverage: Access Point Strategy

My wireless network consists of a mix of primary and secondary access points strategically placed throughout the property to ensure complete coverage:

Primary Access Points

I have three Omada WiFi 6 AX3600 Wireless 2.5G Access Points (EAP660 HD), which were top-of-the-line when I purchased them:

• Two are properly installed in the ceilings as intended
• One had to be placed under a dresser to fix a coverage problem I hadn't anticipated

Secondary Access Points

To complement the primary APs and eliminate dead spots, I added three EAP225-Outdoor Omada AC1200 Wireless Gigabit Outdoor Access Points:

• One in the garage (where I wish I had installed an Ethernet port during construction)
• One hidden in the storage area of the basement to fix a dead spot
• One in the backyard, operating in a mesh setup with the primary APs

All six access points are powered by my 2.5Gbps PoE switch, eliminating the need for separate power supplies and simplifying installation. The result is seamless coverage throughout the entire property, with wireless speeds exceeding 500 Mbps on mobile devices (e.g. iPhone 13, MacBook Pro, etc.).

This approach to wireless coverage highlights another lesson: it's better to have more, strategically placed access points than to rely on fewer, more powerful ones. The coverage patterns of wireless signals can be unpredictable in real-world environments with various building materials and interference sources.

Key Takeaways

1. Plan for expansion: Always install more Ethernet ports (and electrical outlets) than you think you'll need
2. Consider heat management when selecting networking equipment (especially 10G gear)
3. Storage performance matters for daily use—the move to NVMe has been worth every penny
4. Connections hierarchy: DAC first, fiber for long runs, 10G Ethernet only when necessary
5. Power over Ethernet (PoE) switches provide flexibility for adding devices without worrying about power sources
6. Keep security simple: The risks of exposing services externally often outweigh the benefits
7. More APs, strategically placed: Multiple strategically positioned access points provide better coverage than fewer high-powered ones

While I've made some missteps along the way, the current setup delivers exceptional performance for our family's needs. The combination of fast internet, 10G internal networking, and responsive NVMe storage has significantly improved our experience with everything from gaming to media consumption to working from home.