PDA

View Full Version : Server Problems...What Would It Cost To Upgrade


RINGLEADER
09-12-2004, 08:57 PM
I'm willing to chip in to help out, but what does the planet need and what would it cost?

Damn...there's the big run our defense gives up every game...

David.
09-12-2004, 08:59 PM
we just upgraded, We need to ban guests like we did for the draft.

RINGLEADER
09-12-2004, 09:03 PM
we just upgraded, We need to ban guests like we did for the draft.

:shrug:

Well, if you want to upgrade again to make it so you don't have to ban guests just let me know...like I said, I'm willing to break out the coin to do my part...

Sure-Oz
09-12-2004, 09:08 PM
I agree, disable the guests, of course if there was a donate thing to help then i would be for it.

RINGLEADER
09-12-2004, 09:11 PM
I agree, disable the guests, of course if there was a donate thing to help then i would be for it.


What would it cost? Are we talking $1K or $10K? Sorry, but I just have no idea.

Iowanian
09-12-2004, 09:17 PM
Money has been donated............. I don't think that can be the problem.

David.
09-12-2004, 10:22 PM
it just died again. Whatever the email problem was EVERYONE needs to fix it or be banned.

dirk digler
09-12-2004, 10:24 PM
we just upgraded, We need to ban guests like we did for the draft.


I agree on gameday ban all guests. That would help alot.

Frazod
09-12-2004, 10:26 PM
Seriously. Like tonight wasn't enfuriating enough without the f#cking board taking a night-long shit. Enough.

I'll put a f#cking check in the f#cking mail tomorrow if SOMEBODY WILL F#CKING DO SOMETHING TO FIX IT.

:banghead:

Iowanian
09-12-2004, 10:28 PM
Money has been sent before to fix this problem.......

I wonder about eliminating Sig pics and stuff like that..........

It is pissing me off though..........Its not like it hasn't been a problem since last year.

I don't want to see guests banned..............I'd prefer to see things fixed.

stanleychief
09-12-2004, 11:09 PM
Banning guests is a horrible idea. Would an online store ban new customers if their server was getting busy? I don't think so! This place started out as a small community bulletin board, but has outgrown its capacity to handle gameday traffic.

Some suggestions to the admins:

Kill or limit pictures
Prune the database of old messages
Disable email notifications on replies
Clustering.. learn it.. live it.

First step is to identify the problem.. is it a lack of bandwidth, is the CPU being overworked, low on memory.. what? It takes more to run a fast server than throwing a new cpu in a box and calling it good. What is the failure point? Spending money on the wrong problem won't fix a thing.

What is the current hardware, does anyone know? Is the server dedicated to Chiefsplanet or shared? Is the database on the same server?

David.
09-12-2004, 11:11 PM
Banning guests is a horrible idea. Would an online store ban new customers if their server was getting busy? I don't think so! This place started out as a small community bulletin board, but has outgrown its capacity to handle gameday traffic.

Some suggestions to the admins:

Kill or limit pictures
Prune the database of old messages
Disable email notifications on replies
Clustering.. learn it.. live it.

First step is to identify the problem.. is it a lack of bandwidth, is the CPU being overworked, low on memory.. what? It takes more to run a fast server than throwing a new cpu in a box and calling it good. What is the failure point? Spending money on the wrong problem won't fix a thing.

What is the current hardware, does anyone know? Is the server dedicated to Chiefsplanet or shared? Is the database on the same server?

Don't BAN guests. Make it so someone has to register if they want to hang. People watching our convos adds nothing to the BB. But the email notification has to go. I don't want avy's gone, that's how I recognize posters.

Sure-Oz
09-12-2004, 11:13 PM
Shredding some of the really old posts may help as well.

Frazod
09-12-2004, 11:13 PM
Banning guests is a horrible idea. Would an online store ban new customers if their server was getting busy? I don't think so! This place started out as a small community bulletin board, but has outgrown its capacity to handle gameday traffic.

Some suggestions to the admins:

Kill or limit pictures
Prune the database of old messages
Disable email notifications on replies
Clustering.. learn it.. live it.

First step is to identify the problem.. is it a lack of bandwidth, is the CPU being overworked, low on memory.. what? It takes more to run a fast server than throwing a new cpu in a box and calling it good. What is the failure point? Spending money on the wrong problem won't fix a thing.

What is the current hardware, does anyone know? Is the server dedicated to Chiefsplanet or shared? Is the database on the same server?
I couldn't care less if we ban guests, at least until this shit is resolved. Our server just can't handle the load. I don't buy the customer analogy, either. It's not like we're selling insurance here.

For those of us who have been here since the beginning, and have voluntarily donated funds to the operation of this BB, it sucks that the place becomes unusable during a friggin game because it's crawling with lurkers. F#ck that.

Bwana
09-12-2004, 11:15 PM
I couldn't care less if we ban guests, at least until this shit is resolved. Our server just can't handle the load. I don't buy the customer analogy, either. It's not like we're selling insurance here.

For those of us who have been here since the beginning, and have voluntarily donated funds to the operation of this BB, it sucks that the place becomes unusable during a friggin game because it's crawling with lurkers. F#ck that.

:thumb:

stanleychief
09-12-2004, 11:17 PM
Don't BAN guests. Make it so someone has to register if they want to hang. People watching our convos adds nothing to the BB. But the email notification has to go. I don't want avy's gone, that's how I recognize posters.

Sorry bad choice of wording I guess... block them from viewing. I guess my point is that you don't want to ban potential users from browsing the site. That's just a poor business model. The problem needs to be addressed, not covered up.

The suggestions I made are just that... suggestions. If avatars are the most important feature to this board, then so be it. For me personally, being able to post reliably, and browse messages comes above that. ;)

stanleychief
09-12-2004, 11:20 PM
I couldn't care less if we ban guests, at least until this shit is resolved. Our server just can't handle the load. I don't buy the customer analogy, either. It's not like we're selling insurance here.

For those of us who have been here since the beginning, and have voluntarily donated funds to the operation of this BB, it sucks that the place becomes unusable during a friggin game because it's crawling with lurkers. F#ck that.

So you signed up on this board before reading a single post? Interesting. You don't have to be selling anything to see that the same rules apply here. Most people don't take the time to come back if they get anything that resembles an error message.

I'd like to think there are other solutions to the problem. There are other boards out there with far more traffic that are able to handle the load adequately.

Valiant
09-12-2004, 11:24 PM
First ban guests...

Second ban most people from the state of Colorado...

Do not allow new registrations on Game day...

stanleychief
09-12-2004, 11:59 PM
First ban guests...

Second ban most people from the state of Colorado...

Do not allow new registrations on Game day...

Ok, so the website has been crashing repeatedly on me in the time it's taken me to post my replies. Do you think this is because of guests, or new registrations?

Seriously... it doesn't take that much to do it right. Look at the ASP.NET forums.. they get thousands of guests and hundreds of posters throughout any given day. I've seen their server down one time. Before someone tries to say that they must be running on an exotic system, here's some hardware information from the system admin:


We've got 2 front-end web servers running ASP.NET:
dual-proc 750mhz

These front-end boxes average ~15% CPU utilization

The database is a single-proc 750mhz with 1gb of memory.

The db is under much more load and averages ~70% CPU utilization, with about 500mb memory committed (much of that by SQL).

We're in the neighboorhood of 25-35k unique visitors/day. The most concurrent we've seen on the forums is ~2,350 users + anonymous.


So since the utilization on each web server box is quite low (15%) you could probably get by with less horsepower, or a single web server and seperate database server.

David.
09-13-2004, 01:40 PM
bah! we don't need no stinkin new posters.

Braincase
09-13-2004, 01:54 PM
Keep in mind that what we need and what we are worth are two different things. We need an 8-node DataCenter Cluster with a switched fabric SAN and each node needs 16 processors and 64 GB RAM. What we are worth (or deserve) is a Atari with the RAM upgraded to 128k.

David.
09-13-2004, 02:07 PM
Keep in mind that what we need and what we are worth are two different things. We need an 8-node DataCenter Cluster with a switched fabric SAN and each node needs 16 processors and 64 GB RAM. What we are worth (or deserve) is a Atari with the RAM upgraded to 128k.

ROFL pft speak for yourself. I'm PRICELESS :D

KC Jones
09-13-2004, 02:07 PM
Keep in mind that what we need and what we are worth are two different things. We need an 8-node DataCenter Cluster with a switched fabric SAN and each node needs 16 processors and 64 GB RAM.

bah - just use some cheap hardware and some free clustering technologies:

http://www.budget-ha.com/

Of course this particular design is for Java web application servers. I'm not sure what you can do for using this PHP forum application on a distributed architecture. Obviously there are some big outfits that do it but they probably use more expensive solutions.

Skip Towne
09-13-2004, 02:16 PM
bah! we don't need no stinkin new posters.
We should have done something about it before you showed up. :p

Skip Towne
09-13-2004, 02:22 PM
A couple of months ago Kyle said it was someone attacking the server. I think ®aiders®ock is involved in this.

AustinChief
09-13-2004, 03:30 PM
I have upgraded the server itself.. I just finished upgrading all the server software... I also took care of our "attack" problem... now there is only ONE issue left...

The CPU is plenty fast.. we have a ton of bandwidth and a ton of disk space... our problem is I/O... our database has just gotten too damn big.

Unlike other big sites... I don't want to trash all our archives... so I am working on a few different solutions so that we can keep our archives yet also get the damn server rolling again.

Solution #1 = upgrade the BBS software
Solution #2 = "split" the db so that the archives go into a seperate table.... this is easier said than done... this will be VERY time consuming
Solution #3 = get a dedicated MySQL box (around $300 more a month)

I won't be doing Solution #3 anytime soon... but 1 and 2 are what I am currently working on...

The problem is the POSTS table which has grown to OVER 800 MB in size.... everytime someone makes a new post... the db has to open, write and close that file...

--Kyle

jcroft
09-13-2004, 03:37 PM
I think you all need to take a step back and realize that Kyle and staff are doing this on their own time for us as a public service. We're not paying for it, and we shouldn't jump down his throat when something goes wrong. He's doing this as a favor to us and we should be grateful that it exists at all.

Calcountry
09-13-2004, 03:37 PM
I agree, disable the guests, of course if there was a donate thing to help then i would be for it.
Me too. You should have a voluntary subscriber page and then give a special Icon, like a chiefs arrow to all the subscribers. :thumb:

Calcountry
09-13-2004, 03:41 PM
Don't BAN guests. Make it so someone has to register if they want to hang. People watching our convos adds nothing to the BB. But the email notification has to go. I don't want avy's gone, that's how I recognize posters.
email notification is worthless. Most times I notice them in my hotmail account days after they have already become moot.

ENDelt260
09-13-2004, 03:42 PM
Solution #2 = "split" the db so that the archives go into a seperate table.... this is easier said than done... this will be VERY time consuming

Well, quit talkin' about it and get on it, you slacker! :D

Calcountry
09-13-2004, 03:43 PM
Ok, so the website has been crashing repeatedly on me in the time it's taken me to post my replies. Do you think this is because of guests, or new registrations?

Seriously... it doesn't take that much to do it right. Look at the ASP.NET forums.. they get thousands of guests and hundreds of posters throughout any given day. I've seen their server down one time. Before someone tries to say that they must be running on an exotic system, here's some hardware information from the system admin:



So since the utilization on each web server box is quite low (15%) you could probably get by with less horsepower, or a single web server and seperate database server.
I believe I read once, that they were having issues sometimes with denial of service attacks. IIRC.

LVNHACK
09-13-2004, 05:36 PM
We should have done something about it before you showed up. :p




quit pickin' on the tards Skip.............. ROFL

ExtremeChief
09-13-2004, 05:36 PM
email notification is worthless. Most times I notice them in my hotmail account days after they have already become moot.

Then turn it off!!!!!!

stanleychief
09-13-2004, 06:33 PM
I have upgraded the server itself.. I just finished upgrading all the server software... I also took care of our "attack" problem... now there is only ONE issue left...

The CPU is plenty fast.. we have a ton of bandwidth and a ton of disk space... our problem is I/O... our database has just gotten too damn big.

Unlike other big sites... I don't want to trash all our archives... so I am working on a few different solutions so that we can keep our archives yet also get the damn server rolling again.

Solution #1 = upgrade the BBS software
Solution #2 = "split" the db so that the archives go into a seperate table.... this is easier said than done... this will be VERY time consuming
Solution #3 = get a dedicated MySQL box (around $300 more a month)

I won't be doing Solution #3 anytime soon... but 1 and 2 are what I am currently working on...

The problem is the POSTS table which has grown to OVER 800 MB in size.... everytime someone makes a new post... the db has to open, write and close that file...

--Kyle

I'm not too familiar with vBulletin, so I'm not sure how scaleable it is.. is the upgrade you speak of to another BBS software package, or just a different version of the same? I've worked some with phpBB2 and the fact that it's completely open source makes it kind of nice, but it's certainly time consuming to migrate I'm sure. Also the lack of customer support can be a thorn in the side at times in desperate situations.

Just out of curiousity, why are you being quoted $300 a month for a MySQL box? That seems pretty damn high. I assume this is a leased (dedicated) server and not a co-located box you built. What is your SLA with this company and what do they provide for that cost? I'm sure people on here would be happy to contribute some $$$ for a stable and healthy Chiefsplanet. I also have some good leads on rackmount cases, and other server parts, plus used to run a hosting company.

Send me a smoke signal if you would like to bounce any ideas off of me, I'd love to help.

OldTownChief
09-13-2004, 06:55 PM
Wouldn't shutting off the guests just cause a lot of bogus one day registrations that would muddy the water even more? For example a bunch of Panther fans going through the reg. process just to post during the game, then we don't play them again for years.

whoman69
09-13-2004, 08:06 PM
If storage is a problem, then I would begin with eliminating accounts that have never been used. There are currently 3120 registered users on the board. Of this total 1050 + have never posted even once. That means more than one third of all accounts are useless.
Add to that total over 900 posters who have posted less than a 10 times. Many of these posters are from other sites that come by only when their team is ready to play ours. In short, they are trolls.
Add to that many of these have an avatar on our site that takes up our bandwidth. I thought that new posters had limitations on being able to set up avatars and sigs until they hit 100 posts. How much storage space do these worthless accounts take up?

stanleychief
09-14-2004, 12:52 AM
If storage is a problem, then I would begin with eliminating accounts that have never been used. There are currently 3120 registered users on the board. Of this total 1050 + have never posted even once. That means more than one third of all accounts are useless.
Add to that total over 900 posters who have posted less than a 10 times. Many of these posters are from other sites that come by only when their team is ready to play ours. In short, they are trolls.
Add to that many of these have an avatar on our site that takes up our bandwidth. I thought that new posters had limitations on being able to set up avatars and sigs until they hit 100 posts. How much storage space do these worthless accounts take up?

Believe it or not, in the grand scheme of things, inactive posters don't cause much trouble or chew up much room in the database. It is in fact the active rambling posters that bring the server to its knees. If I had to guess at the types of posts that hog the most disk space, I'd say it would be article reposts from the KC Star & various sources and heated political discussions, both of which I would miss reading here at the planet.

Avatars usually aren't that bad on database resources either. Typically an image link is a single line of text. If the avatars are hosted offsite, then the bandwidth usage on the server is even less of an issue. If they are hosted onsite, the impact is still to the bandwidth and not to the database.

The Planet is simply a victim of it's own success and I think it'll take money in the long run to fix it the right way. Maybe there are some creative revenue streams that haven't been tapped yet? I'd certainly think that with the userbase of the planet that some businesses would be more than happy to pay for some links and banners to be scattered about.

KC Jones
09-14-2004, 07:05 AM
I have no idea what the DB schema is for this thing, but if all posts are in one table - that's gotta be one big honkin table. If thread id is a foreign key or has it's own index - well maintaining that index while lots of inserts and updates go on will be incredibly expensive. No doubt about it - the old posts have to be migrated to an archive table.

In the meantime disabling search functionality and reducing the number of new threads posted might help uptime - but the real culprit is probably just the size of that file. I'd be all for migrating the old posts immediately and LOSING them until Kyle and/or others have time to get them online through an archive interface. Just move every thread that hasn't been posted on in over 6 months. Do it at 4am and take the planet offline to prevent anyone from posting while you do it.

Ari Chi3fs
09-14-2004, 07:12 AM
what i would personally do on Gameday is...

Not show avatars or signatures... that is a lot of data considering most peoples sigs are 30-50 Kb each. some larger if they dont know how to compress.

Plus you have many on the same thread. I view 10 or 20 posts on a page at a time, instead of my typical 40-50 posts per page.

That would help a ton I think. IMO

StcChief
09-14-2004, 08:02 AM
MySQL may be pushed to the limit here. It's free you get what you pay for.

From what I've read it sounds like a DB issue. Scaling up is not a MySQL strength.

Somebody needs to analyze this. Sounds like the post table needs to be split if MySQL can't handle the size,
and # of concurrent users posting.

In the interim if game day is a problem for crashing the site keep non members from posting and explain on main screen.

The trolls may deliberately be trying to bring it down.
By attacking it with posts.