RiotACT’s bad week

johnboy 30 November 2006 13

Ummm, yes… So where were we?

Looking back it started over the weekend. RiotACT was still pushing out data at a furious rate but there were complaints from within our bunker that the internet was slow.

Gradually, in fits and starts, it got worse. In the afternoon it would get better. Because it was asymmetrical, as is our data, it was hard at times to even notice there was a problem. We push out a lot more bits than we pull in and the error seemed to match it, data could get out but not in. Eventually even simple get requests from your web-browsers couldn’t make it in to trigger the website to send you data back.

And yet… and yet… the lights all stayed green, they flickered appropriately, all indicators were normal within our systems. And it kept getting better! And then worse! Surely some external event had to be responsible?

While initially distrustful of the protestations of innocence from TransACT and our ISP (from hard-won experience) eventually there was only one thing for it, put in a PPPoE connection in front of our long serving router and see if we could then connect at a more appropriate rate.

Here’s the hard-worked suspect:

The little RT314 has been beavering away for five years without skipping a beat. So long that I had to telnet onto it to even find out what the userid is for our connection (fortunately we did have a good record of the password).

And when I jacked my laptop direct into TransACT, lo and behold, suddenly the internet was working properly.

So I set up a placeholder page on the laptop and left it there to answer your queries about where RiotACT had gone.

Meanwhile Kramer was getting our spare router configured for duty. It had failed miserably in its previous role as a wireless access point in a very complicated network but we were hopeful it would respond better to being at the top of the stack.

The first go was not promising, the range of IP addresses it was assigning wasn’t right, but luckily we had another laptop handy which we were able to plug in and convince it to change its ways.

Then we let it rip on the pipe to TransACT and had a heart wrenching 30 second delay before it fired up and started talking.

Finally we had to get the bunker’s computers back on the net, at which point we found out the new router was interfering with the bunker’s wireless network. Once we identified the problem a repositioning of routers brought everything back to the way it should be.

So, as promised, RiotACT was brought back online this afternoon. Our newish server now paired with a newish router.

We now await a new and different technological catastrophe.

We hope you were productive while we were away.

What's Your Opinion?

Please login to post your comments, or connect with
13 Responses to RiotACT’s bad week
johnboy johnboy 6:09 pm 01 Dec 06

Hmm, take out the HD risk and throw in CD’s and ancient floppies? I think you’re about even there, modern HD’s don’t break like they used to.

When my previous employer was bought out by a multinational they looked at the nested set of bridging firewalls Kramer and I had built for it (out of 200Mhz surplus computers with exrta NICs) and freaked out, they sent down a contractor to replace it with something more “standard”.

In one of the prouder moments of my life he reported back to head office that his replacement was going to be technically inferior to the system we’d built. (they still replaced it)

So in the right place, for the right reasons, I’ve a fan of the home built box as firewall.

But for just sitting at the top of the network with some NAT and port forwarding to a single box I’ll take the solid state appliance for under $200 any day.

Fans are the killer point of failure these days and by the time you source a computer with a fanless power supply and modify it to use a CF card in place of a hard drive, plus set aside a spare machine with identical NICs to act as a replacement it really gets easier to just go with the consumer appliance. I guess it depends what your time is worth.

Anyway, I may be a journalist, but I’ve always worked for the smaller organisations where if you want the IT sorted you need to do it yourself.

And personally I like to get this stuff sorted.

farq farq 5:12 pm 01 Dec 06

Good points. Never thought about the latency between the two cards, the hardware routers are more ‘efficiently’ designed for that task (been doing some reading). So under normal load, they ‘should’ be faster. But to my benchmarking my monowall pc (an under-clocked Celeron 300a) solution beats my D-Link (and old Linksys) routers for keeping latency low when I’ve got the connection throttled with heaps of bittorrent connections (maybe I need a newer hd router).

On points of failure, monowall etc don’t require a hdd, all the config files sit on a floppy disk. If the machine fails, stick the CD-ROM and Floppy in another computer and bang, you have replaced the router without having to recreate a single setting.

If you under-clock the cpu, you can do without the cooling fan. Because there is no hard drive, you don’t need a case fan. Two less points of failure.

Power supplies always fail in the end, even the one on the end of the embedded router’s power cord.

Anyway, you seem to know a lot about networking for a journalist. I could crap on all day about computers, and from the sounds of it so could you. Maybe you could pass your wisdom on next time there is a riot-act barbq or something.

Love the site, don’t care how long it takes to load.

Absent Diane Absent Diane 4:02 pm 01 Dec 06

I don’t find the site slow.. infact it loads quicker than a lot of other sites i frequent… even on my shitty home dialup connection.

johnboy johnboy 3:47 pm 01 Dec 06

in defence of the old rt314 while it’s internal cooling fans had died it still kept operating, just degrading under load.

While that complicated our debugging efforts it’s impressive in its way.

The new router is 100% passive cooling.

Now that i’ve been thinking about it another thing that could slow the site down for some is the amount of google javascript that has to execute between the ads and the analytics package.

Anyway, we’re satisfied with performance at the moment.

Absent Diane Absent Diane 3:21 pm 01 Dec 06

netgear stuff is pretty reliable in general.

johnboy johnboy 3:13 pm 01 Dec 06

Farq, back in 2000 that’s exactly what we were looking at doing. But a computer running smoothwall adds a massive number of extra points of failure.

(hard drive, cooling fans, power supply to name a few)

Plus there’s networking latency on the two cards having to run through the CPU and back out again.

The new router is a lot more powerful than a DSE branded product and is not running stock software.

Five years without trouble is almost certainly better than we would have had through an old box running smoothwall.

The page would load a lot faster if we didn’t have all the pretty large images… but they’re SOOO pretty!

farq farq 3:07 pm 01 Dec 06

I’ve even got an old PII-200mhz/128mb I could donate to the cause!

farq farq 3:06 pm 01 Dec 06

Johnboy: Why not build the router using an old computer. Just throw a second nic into any old box, download smoothwall or monowall or something and away you go (I’m a big fan of using openbsd on routers).

I’m no expert (your prob a bigger nerd than me), but the consumer routers you buy at dick smith or whatever are lucky to be even 100mhz, with bugger all memory. Not the best thing to stick infront of a high use web server. Sure they work fine with a couple of connections, but increase the load and the little routers struggles to keep up. Latency and dropped packets are the result.

The only advantage the of the little routers is they are quite.

I’ve gotta say that The-RiotAct does load pretty slowly (could be the HTML render, I know next to nothing about html/css).

BTW: I’m a database nerd, not too much into networking. I could be talking out of my ass (I often do!). Just my 2c.

Mr Evil Mr Evil 11:18 am 01 Dec 06

Interesting to see you aren’t using DSE equipment. 🙂

toriness toriness 10:06 am 01 Dec 06

johnboy, yesterday was awful. i actually did work – while checking in an obsessive compulsive manner whether RIOTACT was back up.

Maelinar Maelinar 8:21 am 01 Dec 06

it’s when you telnet to the transact network to find out where the problem is that you really look in the mirror and say ‘I’m a geek’…

Vic Bitterman Vic Bitterman 8:54 pm 30 Nov 06

Onya Johnboy. I knew your nerd-dom; or the transact network; would win out eventually 🙂

Danman Danman 5:54 pm 30 Nov 06

Th eabove story is my explination to my bosses why my work has been done earlier than normal these last 2 days

CBR Tweets

Sign up to our newsletter


Search across the site