20 September 2008

RiotACT server upgrade underway

| johnboy
Join the conversation
64

For the last few weeks we’ve been the victims of our ever-escalating popularity. Our poor server has been spitting out database error messages as often as not, groaning under the increasing weight of all your visits.

In response to this we’ve now lashed out on a dual quad core (so yes eight cores) beast from HP with redundant doodads up the wazoo and 8Gb of RAM (with room for much more). Back when we started in 2000 a machine like this would have come with a team of men in white coats.

We hope it will see us through to complete media domination in Canberra with a start made into the rest of the world.

It’s currently being brought up to speed and we hope to have it up and running in a week or so.

So just stay patient and it will all be better soon. We promise.

Join the conversation

64
All Comments
  • All Comments
  • Website Comments
LatestOldest

I just read this whole thread and my head imploded.

oh and how goes the upgrade? any news on commissioning date?

Loquaciousness said :

peter@home said :

Loquaciousness said :

peterh said :

rather the suse geeko.

Teehee. What an apt typo 🙂

L

no typo – they were calling him geeko a while ago. seeing that he is a chameleon, not a gecko…

And so I stand corrected. What an interesting little bit of geek trivia!

L

and now i feel sad that I know it. wait to see the fusa (should have been Foosa) release soon. very interesting. bit like platespin….

Holden Caulfield12:03 pm 25 Sep 08

Any news on the upgrade, farkn???

Peter, its on wordpress

peterh said :

perhaps if the dl380 g5 was configured with SAS & SATAII drives, you could go with raid 1 + 5 – mirrored O/S and Apps, and raid 5 data store with SATAII – using the WD 1Tb drives, it would give you 4tb actual storage space.

Using that as a virtual ram source, you could also create a HA buffer for ram and allow the max ram config to have a buffer in case of overflow.

SATAII drives would ensure that the spindle speed wouldn’t compromise your DDR Ram bus speed.

Just helped a reseller put together a Promise 14-Bay Raid array, using 1TB SATA drives, 14 bays populated, Raid 5, with FC linkages to a HP server – I think a 560.

Goed ver dommer. Doe je niet in engels praten op het RiotACT? Ik weet het niet wat je hier zegen.

Loquaciousness10:03 pm 22 Sep 08

peter@home said :

Loquaciousness said :

peterh said :

rather the suse geeko.

Teehee. What an apt typo 🙂

L

no typo – they were calling him geeko a while ago. seeing that he is a chameleon, not a gecko…

And so I stand corrected. What an interesting little bit of geek trivia!

L

Loquaciousness said :

peterh said :

rather the suse geeko.

Teehee. What an apt typo 🙂

L

no typo – they were calling him geeko a while ago. seeing that he is a chameleon, not a gecko…

Loquaciousness7:43 pm 22 Sep 08

peterh said :

rather the suse geeko.

Teehee. What an apt typo 🙂

L

johnboy said :

Peter when a site is being overwhelmed by the number of simultaneous requests a solution that slows down the time to service those requests is more of a problem than a solution.

You’d just increase the backlog and lock it up tighter.

Trust us, we’ve been doing this a while.

out of interest, what is the basis of the riotact environment? is it joomla?

Gungahlin Al5:28 pm 22 Sep 08

Well I’ll just be happy to see it installed, because the errors are giving me the ribbets – especially today…

Peter when a site is being overwhelmed by the number of simultaneous requests a solution that slows down the time to service those requests is more of a problem than a solution.

You’d just increase the backlog and lock it up tighter.

Trust us, we’ve been doing this a while.

Holden Caulfield said :

I’d just settle for a “previous entries” / “next entries” link at the bottom of the page like most other WordPress sites have.
(

Scroll down a bit Holden, It’s already there just labeled with the title of the story rather than next/previous (which is now on the front page too)

johnboy said :

a) we do not need nor desire a “storage solution”. Storage is not our problem.

b) Virtual RAM? I’d rather shave my balls.

JB, the mental image that conjured up was not required. A simple “no” would have been better.

I have clients (resellers) currently using Virtual ram to enable their servers to bridge temporary out of memory errors. The apps that they are running are sitting on AIX & HP-UX, the HA Virtual RAM is not used continually, it is there as a failover. I wasn’t talking about deploying a swap file, just using VRAM as a backup in case the number of users accelerates yet again. Another Option would be to create several nodes under a Virtual server pool and share the load equally across the nodes. I was assuming that you were using linux, not win for the servers?

a) we do not need nor desire a “storage solution”. Storage is not our problem.

b) Virtual RAM? I’d rather shave my balls.

caf said :

Kramer: Curious as to why you went with RAID 1 + 0, rather than a small RAID 1 boot partition and big RAID 5 partition for the data? With 4 spindles it should outperform the mirror, and you certainly have the CPU to spare…

peterh: My shot-in-the-dark guess is that the database is hitting its maximum number of simultaneous connections setting, which is largely limited by available RAM.

This is a Hypothetical…
perhaps if the dl380 g5 was configured with SAS & SATAII drives, you could go with raid 1 + 5 – mirrored O/S and Apps, and raid 5 data store with SATAII – using the WD 1Tb drives, it would give you 4tb actual storage space.

Using that as a virtual ram source, you could also create a HA buffer for ram and allow the max ram config to have a buffer in case of overflow.

SATAII drives would ensure that the spindle speed wouldn’t compromise your DDR Ram bus speed.

Just helped a reseller put together a Promise 14-Bay Raid array, using 1TB SATA drives, 14 bays populated, Raid 5, with FC linkages to a HP server – I think a 560. all under $15K for the storage solution. Just because I don’t sell it, doesn’t mean i have forgotten what to do…

Spot on about the RAM caf.

The HD’s are being configured this way because we’re going for availability rather than speed.

We’re nowhere near being limited by HD speed for data access.

Kramer: Curious as to why you went with RAID 1 + 0, rather than a small RAID 1 boot partition and big RAID 5 partition for the data? With 4 spindles it should outperform the mirror, and you certainly have the CPU to spare…

peterh: My shot-in-the-dark guess is that the database is hitting its maximum number of simultaneous connections setting, which is largely limited by available RAM.

peterh said :

OT, why does the database keep futzing out? dual 80gb drives should be ok for basic text posts, shouldn’t it?

Maybe back in 1997 with static sites. Here in 2008 with a database driven design it’s all done in RAM with the HD pretty much just a backup for restoring from after a reboot.

tylersmayhem3:23 pm 22 Sep 08

Last few days – try last few weeks! I look forward to them stopping after the upgrade.

Holden Caulfield2:52 pm 22 Sep 08

I’d just settle for a “previous entries” / “next entries” link at the bottom of the page like most other WordPress sites have.

Hope the server upgrade goes well, the Database Error messages have been a bit of a bugger these last few days. 🙁

tylersmayhem said :

Point Taken JB, will behave myself now.

@peterh: methinks that warning was for me somehow. I can’t see where you were stirring trouble.

I am sure it was the crack about comic relief. OT, why does the database keep futzing out? dual 80gb drives should be ok for basic text posts, shouldn’t it?

tylersmayhem2:35 pm 22 Sep 08

Point Taken JB, will behave myself now.

@peterh: methinks that warning was for me somehow. I can’t see where you were stirring trouble.

tylersmayhem12:21 pm 22 Sep 08

Thanks for the link Skid: I figured as much.

johnboy said :

Well if you’re just crap-stirring for your own amusement consider this your final warming prior to moderation.

kthxby

Point Taken JB, will behave myself now.

tylersmayhem12:03 pm 22 Sep 08

What does KTHXBY mean JB?

@peterh: I take your point and don’t blame you mate 😉

Well if you’re just crap-stirring for your own amusement consider this your final warming prior to moderation.

kthxby

tylersmayhem said :

Yeah PeterH I noticed all of that – and I was just stirring as I ALWAYS do. JB is so easy and not just a little bit amusing to wind up.

As for your comments about servers, and the correct storage and operation of them – your comments of “it will be installed into a METAL rack – d’uh…” are probably more worthier of JB’s personal abuse than mine.

…and yes – I am very relieved to hear that it will be stored in a data centre.

Tylers,

sorry, currently coding 7x batches of 85 price lines – starting to lose plot, needed comic relief.

And I thought I didn’t need to do tenders any more. sigh.

tylersmayhem10:47 am 22 Sep 08

Yeah PeterH I noticed all of that – and I was just stirring as I ALWAYS do. JB is so easy and not just a little bit amusing to wind up.

As for your comments about servers, and the correct storage and operation of them – your comments of “it will be installed into a METAL rack – d’uh…” are probably more worthier of JB’s personal abuse than mine.

…and yes – I am very relieved to hear that it will be stored in a data centre.

tylersmayhem said :

Ah, JB – it that a metal filing cabinet I see your new server sitting on? And, is there actually anything cooling the unit (other than the standard internal fans)? I hope you have your wallet ready for another server – unless it is actually sitting on something anti-static (that we can’t see in the photo) and your A/C is on full blast in summer.

Tylers, there aren’t any blanking panels, the server has a clear access at the back, it would be very cool – and anti static??

it will be installed into a METAL rack – d’uh…

tylersmayhem10:18 am 22 Sep 08

…and just when I was considering becoming a paid subscriber now that RA are finally upgrading their servers to stop database errors.

JB: I’d like to think that every soap-box forum has different characters on there. I happen to be an opinionated, sarcastic and skeptical poster on this site. What I would have thought by now is that you could rise above “my opinion” and just get on and run the site without getting personally upset and flinging personal insults when I direct a pointed question or opinion. According to some replies to points I raise – other posters do occasionally agree with my views – most just ignore them, which is fine by me.

johnboy said :

God TM, don’t you ever get tired of being such an unpleasant half-wit?

That’s not the final installation (d’uh), it’s off to a datacentre after configuration.

so, transact, here it comes??

God TM, don’t you ever get tired of being such an unpleasant half-wit?

That’s not the final installation (d’uh), it’s off to a datacentre after configuration.

tylersmayhem9:20 am 22 Sep 08

Ah, JB – it that a metal filing cabinet I see your new server sitting on? And, is there actually anything cooling the unit (other than the standard internal fans)? I hope you have your wallet ready for another server – unless it is actually sitting on something anti-static (that we can’t see in the photo) and your A/C is on full blast in summer.

Kramer said :

The main reason for HP is that I work with their hardware on a daily basis, and have found it to be pretty solid.

The new storage config is 4 x 73GB 15K SFF (2.5″) SAS drives in RAID 1+0 with an online spare. The current box has mirrored 80GB hard drives, so the new server with 140GB should be plenty for a while. So as much as I would like to see Riot running on a full spec EVA8100, it doesn’t really need it right now.

I believe HP sorted the issues with the SCSI cables it the DL380 G3, but not before it had burnt a few people (me included – although our resller kindly sorted us out).

The DL 380 G5 is a pretty sweet box – red fans & PSUs standard; little diag panel on the front; kick arse quad core intels; plenty of room for extra RAM; integrated dual TOE NICs; and very cheap pricing – no wonder they sell heaps (HP – I expect a cheque in the mail by the end of the week :-).

thanks kramer, especially about the cable. that was always a stumbling point for me. did they throw in a decent UPS at the same time?

The main reason for HP is that I work with their hardware on a daily basis, and have found it to be pretty solid.

The new storage config is 4 x 73GB 15K SFF (2.5″) SAS drives in RAID 1+0 with an online spare. The current box has mirrored 80GB hard drives, so the new server with 140GB should be plenty for a while. So as much as I would like to see Riot running on a full spec EVA8100, it doesn’t really need it right now.

I believe HP sorted the issues with the SCSI cables it the DL380 G3, but not before it had burnt a few people (me included – although our resller kindly sorted us out).

The DL 380 G5 is a pretty sweet box – red fans & PSUs standard; little diag panel on the front; kick arse quad core intels; plenty of room for extra RAM; integrated dual TOE NICs; and very cheap pricing – no wonder they sell heaps (HP – I expect a cheque in the mail by the end of the week :-).

rephlex said :

damnit, I meant: we want pics of penguins on boot damnit! =)

what, like Tux for all??

rather the suse geeko.

Felix the Cat said :

peter@home said :

Kramer said :

Fiona – depends which men in white coats… Are you talking about the ones who hang around in cold, noisy rooms with big computers? If you are then, I’ll let you know we wear black T-shirts or polo shirts these days… As for the other guys, I’ll continue to keep them afar.

BTW – for the hardware nerds: the new box is a HP DL380 G5, with dual quad-core 2.6GHz cpus, and 8GB RAM.

how many HDD’s? what type? any plans for a san or nas? why did you guys go with HP?

now, that is a hardware nerd response – oh and the DL380 series, did they ever fix the 2 drive cage connector cable issue? – the one that the configuraider doesn’t tell you about? I wonder…..

Giganerd! Have you got the right flux capacitor?

no idea. but i sold many DL380’s over the years, and always got caught out over a $50 cable that the configuraider a tool HP / compaq use to build before you buy would never put in the config. I got wise to it really quickly.

DL380 servers are the workhorse of the HP brand, great box with very basic expansion path. I now prefer IBM servers, the fan and power options are far superior….

and I don’t sell PC hardware at all these days, only networking and storage.

Jazz said :

We are always open to feedback and ideas & who knows, maybe we’ll be able to find a place for them in our next release.

Weighted average voting!
Oh, and more beer giveaways.

Primal said :

johnboy said :

We’re not planning on redesigning a site read by thousands on the whims of one.

How many then? Six?

It would probably take a lot more than that Primal. We collect lots of information about how people use the site from where they click and how long people stay on particular pages in addition to how I and my fellow overlords would like to see the site operate. You may recall that we opened up the research into the current design around this time 12-14 months ago to get anecdotal as well as statistical feedback on The RiotACT you’d like to see. The current design is the result of a balancing act between what we wanted to achieve and the resources available to us at the time. Too much change tends to alienate people and too little means you stagnate. We are always open to feedback and ideas & who knows, maybe we’ll be able to find a place for them in our next release.

recondite said :

I reckon the design is about right. I’m surprised to hear that Riot is hosted locally though.

Why so Recondite? We could prob host offshore but it would be a little contrary to Riot’s intent to support the local community/businesses. We’ll always try to support local businesses with our patronage where we can.

Felix the Cat3:52 pm 21 Sep 08

peter@home said :

Kramer said :

Fiona – depends which men in white coats… Are you talking about the ones who hang around in cold, noisy rooms with big computers? If you are then, I’ll let you know we wear black T-shirts or polo shirts these days… As for the other guys, I’ll continue to keep them afar.

BTW – for the hardware nerds: the new box is a HP DL380 G5, with dual quad-core 2.6GHz cpus, and 8GB RAM.

how many HDD’s? what type? any plans for a san or nas? why did you guys go with HP?

now, that is a hardware nerd response – oh and the DL380 series, did they ever fix the 2 drive cage connector cable issue? – the one that the configuraider doesn’t tell you about? I wonder…..

Giganerd! Have you got the right flux capacitor?

Kramer said :

Fiona – depends which men in white coats… Are you talking about the ones who hang around in cold, noisy rooms with big computers? If you are then, I’ll let you know we wear black T-shirts or polo shirts these days… As for the other guys, I’ll continue to keep them afar.

BTW – for the hardware nerds: the new box is a HP DL380 G5, with dual quad-core 2.6GHz cpus, and 8GB RAM.

how many HDD’s? what type? any plans for a san or nas? why did you guys go with HP?

now, that is a hardware nerd response – oh and the DL380 series, did they ever fix the 2 drive cage connector cable issue? – the one that the configuraider doesn’t tell you about? I wonder…..

All the suggestions are probably made with good intentions (though you know where good intentions will take you), but just out of interest alone (no judgement involved), are you who people making the suggesions premium members?

Just to draw a very tenuous connection, a sort-of family member of mine has been in this country for 57 years and still remains a citizen of her native land in Europe. So whenever she has a whinge about anything the Government’s doing, I think (but usually don’t say) why not become a citizen and vote.

Now I know that’s a long bow, and a direct comparison isn’t valid, but I sort of understand the ‘sniping from the peanut gallery’ comment.

recondite said :

I reckon the design is about right. I’m surprised to hear that Riot is hosted locally though.

What with our lack of a mega dual redundancy datacentre with its own backup gas fired power station.

I reckon the design is about right. I’m surprised to hear that Riot is hosted locally though.

Jesus, the last redesign was only a few months ago. Took a while to get used to, but it looks fine as is. Definitely not a fan of long pages of ultra-short article summaries and […] everywhere.

Something to do with all the sniping from the peanut gallery?

johnboy said :

As a rule it looks the way it does because it’s the best way to address the needs of readers of all types, advertisers, and ourselves.

Why do you take yourself so seriously ?!

Apologies if it was taken as rude.

We get countless well meaning suggestions on site design.

As a rule it looks the way it does because it’s the best way to address the needs of readers of all types, advertisers, and ourselves.

This means compromise I’m afraid.

Johnboy said

johnboy said :

We’re not planning on redesigning a site read by thousands on the whims of one.

It was a suggestion. Did it merit such a rude response?

johnboy said :

We’re not planning on redesigning a site read by thousands on the whims of one.

How many then? Six?

ant said :

Now to re-design the main interface page so more stories are shown for longer. I’d suggest quoting the first few lines of each story, and that’s it. That way, you could feature more new stories at a time, and hopefully have them on the “front page” for longer than an hour or so.

Much as I value and agree with 95.6% of what you post, Ant, I reckon the current mix is about right. Yeah, stories slide off and if you’re like me (God, Allah and Bob forbid) and only jump on erratically, you might miss stuff. (A classic example was when I was looking for something totally unrelated and found a comment from someone who was pondering: ‘What the folk has Overheard been smoking?!’ For the record, I smoke nothing of any vegetable, animal, mineral or herbal genus or species these days.)

Also, I very irregularly will access RiotACT from my main employer’s intranet and from there I’ll get a much reduced list as compared to what’s available from my laptop, so unlike ‘Amimal Farm’, not all access points are created equal.

In summary and to conclude, yeah, I get a fair few HTTP Error no. XXXXs and ‘Database error — connection cannot be made’, but considering I pay $50 p.a. for the privilege of this site, and many more of you pay about $50 less than that, I’ll suck it up and wait for the upgrade, and not want any major changes to the architecture (which if you’re musically aware, oft times comes from Helsinki — must replace that CD I lost access to in the most recent of my property settlements (small ‘p’, small ‘s’)).

^What ant said

damnit, I meant: we want pics of penguins on boot damnit! =)

Actually it’s just the iframe hosting the sponsor’s ads which auto-reloads.

Yes turning it off would reduce load somewhat, but not a huge deal compared to the rest of the site’s load.

We think we’re on a better deal with the new server which we have high hopes for.

I noticed the page auto-reloads on a regular basis. Turning that off might cut some unnecessary load off the box.

I know, Canberra’s RSI incidence will probably skyrocket as a result of everyone refreshing their riotact page manually.

We’re not planning on redesigning a site read by thousands on the whims of one.

Now to re-design the main interface page so more stories are shown for longer. I’d suggest quoting the first few lines of each story, and that’s it. That way, you could feature more new stories at a time, and hopefully have them on the “front page” for longer than an hour or so.

Fiona – depends which men in white coats… Are you talking about the ones who hang around in cold, noisy rooms with big computers? If you are then, I’ll let you know we wear black T-shirts or polo shirts these days… As for the other guys, I’ll continue to keep them afar.

BTW – for the hardware nerds: the new box is a HP DL380 G5, with dual quad-core 2.6GHz cpus, and 8GB RAM.

mmmmm….8 cores.

Server-licious.

Are you sure the men in white coats won’t be coming anyway?

had.to.be.said.

We publish monthly reports on reader numbers.

43,857 unique visitors in August. September is on target to get well over 50,000.

How many registered members and or hits is the site getting JB? When you get more subscribers than The Crimes you should throw a big RiotAct party!

Daily Digest

Want the best Canberra news delivered daily? Every day we package the most popular Riotact stories and send them straight to your inbox. Sign-up now for trusted local news that will never be behind a paywall.

By submitting your email address you are agreeing to Region Group's terms and conditions and privacy policy.