EOC Folding @ Home - Stats News & Updates

First off, I would like to give a HUGE thanks to Nat from Team Anandtech for donating a pair of 300GB SAS drives. This is going to be a tremendous help in that I can now put the Folding Stats on their own dedicated disks, and also not worry about running out of space! I'm hoping I can go down to the datacenter to get these installed one day this week. If you've got the time, go over to Team Anandtech's Forum and tell the guys thanks!

Second, I've taken the CSV query offline for the time being. It is not used by many people, but it does have the capability to overload the server if misused. I am planning on re-implementing it in a slightly different fashion that won't require so much processing power. Please be patient and I will make an announcement once it is back up.

Third, I have not done the DB upgrade yet, I'm waiting until after I get these new drives installed. When I do, it will be done on the weekend, during the week is just too busy for me to try and do such an overhaul uninterrupted.

Fourth, there was a little downtime this past week. I was doing some firmware updates to various components on the server, all of which required reboots.

Fifth, I've been thinking how to maximize useful data, while minimizing the not so useful data. I do plan on restoring a bunch of the old data that got pruned. But then I'm going to go back through it and remove records where [inactive] users points stayed the same, but their rank changed [down]. I ran a query the other day, and out of the ~540,000 users being tracked, only about 3-4% are regularly active! So as you can see there is a lot of overhead tracking inactivity.

This week I also plan on making a code change related to above. For inactive users when their rank goes down, instead of adding a new row with the "current update,(same) points, and (new) rank.", instead I'm just going to update their last entry with the new info. This will have one little side-effect in the display of the stats. For inactive people it won't show the little red arrow as they move down the ranks. I'm still thinking about it if I keep track of two points in time then I can still show the red arrow and not have too much overhead in space, but translating a thought into practical code is another story!

I also am planning on moving team Google into the normal ranks. Their points total has dropped so much that when I do they will be ranked around 47-48 or so.

I know, I still need to adjust production totals for the colors too... It's on my list!

Finally, one last interesting statistic. With a code change implemented around the beginning of 2010 to track more users, the number of rows added to the user tables went from about 12 million per month to over 38 million per month!

Hope everyone had a Happy New Year! Hard to believe it's already 2012!

As many have noticed the FAH Stats have been down for the past couple days. I've been pruning old historical data in an attempt to get the database under control. The user history table grew to over 1.1 Billion rows, with the table consuming over 36GB!

I've managed to shave it down to around 767 million rows and the table size is now a little under 22GB. There's some tipping point where things really start to balloon if you compare the two pairs of numbers.

Yes I did do a backup before I started removing data, I do plan to eventually re-insert some of it once I get things better organized.

I've un-blocked all the IPs for now. A few people emailed me and I don't think I got a chance to respond back to everyone. If you are still having trouble accessing the XML please send me another email.

I haven't gotten around to the color changes yet, that will probably happen next week.

Be Warned Now... I will be upgrading the database (MySQL) probably this Saturday. Meaning the whole site will be completely down for at least 2-4 hours. I've got to do a complete DB dump / reload because I'm upgrading between versions that are not compatible with leaving the data in the tables.

I'm sure there's more that I can't think of right now, if I remember it I'll be sure to make an additional post.

P.S. for those that want to complain... Realistically this is probably only the second down-time in the past seven or eight YEARS for the stats (that have been MY fault and not Stanford's)... The last time being when I moved the stats to the current server.

First thing first. We've been having some load issues with the server and I've been trying to track down the root cause. I'm pretty sure the culprit lies in some buggy scripts that people have written that are inadvertently pounding the server with mass requests.

That being said, I've been going through the logs and have blocked some of the worst offenders over-querying the XML. If you have been blocked, but you feel your usage is legitimate, then please email me and we can work things out to figure out what needs to be done.

I have NOT gone through the regular page logs yet, I'm sure there could be some people querying regular pages and parsing that data. If you are one of those people I would HIGHLY recommend you check your scripts to make sure they aren't doing anything crazy.

Second, I'm *finally* going to be changing the color levels. I know it's been a long time coming, but as we all know with processing power increasing, so does the points. Hopefully one day this week I will be able to make the change and see how the new numbers pan out.

Third, in case you didn't notice, we have a database with well over 1.1 BILLION rows of data! What I'm planning on doing is making a complete backup (safety first), then pruning out a BUNCH of the old data. First I'm going to try and simply remove old rows where a person's points did not change, simply their rank (i.e. inactive users moving down the list). Past that I will be pruning out data to keep a maximum of one record per-day per person (i.e. reducing potentially from 8 records a day to 1). Finally, if the table is still too crazy big I'll probably just lop off the oldest data first until I can get it back down to something manageable. Realistically you shouldn't notice any change in numbers on the pages.

I'm sure I'm forgetting something else, if I remember what it is I'll be sure to make another post.

How time flies when you are having fun... NOT! Anyhow, I know I haven't made any posts in a while and I apologize for that.

I'm sure some people noticed the site was down for a bit yesterday, there was a little hiccup (pretty sure it was my fault) but everything is okay now.

I've been experimenting on my dev server with the latest MySQL 5.5 and partitioning. The good news is I can partition the existing tables, the bad news is doing an upgrade to the latest MySQL will require a complete dump / import of the data.

Partitioning will allow the data (and indexes) to be physically split into separate database files on the back end. This translates into faster updates, and also I will probably merge the old archive tables so that people can get their full history (assuming they have been folding back in like 2004).

I know I've been talking about a new version of the stats for a while, but I've just had zero free time. It's all in my head, I can visualize exactly how I want the database to look and how everything is going to be divided and processed and such, it's just sitting down and getting to actually code it is where I'm at a loss.

Besides that not a lot to report. I know Stanford has been working diligently on new clients. I've read a lot about the v7 but have not personally tried it out yet. Temperatures here in Houston have been hitting 100F+ for the highs which is ridiculous this early in the summer. To top it off we have had very little rain in the past four months (if not longer)... Maybe like 1/10" a month... that ain't much! Hot and dry means the folding clients get powered down until we get a break in the weather. ;)

Hard to believe it's already been a year. Chimp Challenge 2010 has come upon us. Several teams fold under one user name and race to 20 million points!

Scoring started 12pm PDT on Wednesday May 5, 2010. I had to manually set the start scores, which is why until the 3pm CDT update they will have some negative numbers.

The stats page can be found here.

Good luck to all the teams!

This thread over at the folding forum site has some updates about the NV GPU server issues. Basically they are aware of the problem and are working on it.

I think we've had a breakthrough (well maybe that's too strong of a term), but certainly found something that will help. People should be getting more backlogged credits soon. We have to see whether this will fix all of the problems. I'm thinking it won't fix them all, but it is a step in the right direction.

On another note, I've scheduled the CPU upgrade on the server for Thursday around 10am CST. I haven't made a decision on the SSD yet. I ordered an OCZ Vertex from Newegg and I'm going to give it a whirl on my dev server here in the office and see what kind of performance it affords. I've also been trying to tweak MySQL variables, but unfortunately there is just too much other usage on the server for me to get any sort of accurate numbers.

Okay, I finally purged out all of Team FTL from the DB and re-optimized the tables. Reduced the history table by a couple gigabytes! Also now that the index is not fragmented the processing time should drop back down to hopefully about five minutes.

Good News Everybody! AMD is sending me a couple 2.5GHz High-Efficiency 45nm Shanghai quad-core processors to replaced the current 2.0GHz 65nm Barcelona quad-cores. (I tried for six-cores, but they didn't have any available.) It might not sound like much, but 500MHz x 8 cores = 4GHz additional processing. Also the 45nm chips have 6MB of L3 cache vs the 65nm which only have 2MB. It should give a nice bump in performance. When they come in I'll call my hosting company to setup a time to install them at the data center. Hopefully downtime should only be like 10-15 minutes.

I've been trying to purge out Team FTL from the database today. There's still like 16,000 users and related historical data to delete. The server is really too busy for me to continue it, hopefully later tonight or early in the morning I'll give it another shot.

That team in case you are wondering distributed the FAH client with some game cracks. Needless to say Stanford did not find this amusing and zeroed out the whole team's points.

After I delete all related records, I'm going to have to rebuild the index on the user history table. I have no idea how many hours that's going to take. Again, I'll try to run it in the middle of the night to minimize downtime for everyone.

I've been trying to get back into the FAH groove and also try to keep up with current events.

Today, those hard working guys at Stanford have make some updates to their stats DB hardware which is great news for everyone. Here's quote from Vijay's Blog:

We're done with the bulk of our initial update to new hardware. We'll be doing some more work in the future to build up some additional capacity, namely hopefully getting to the point where the stats are never off line. For now I think we're in good shape. The stats are much faster than before, so we've turned back on a lot of the capabilities we previously turned off. Also, stats update are taking about 5 minutes and now are limited not so much by db access than by other issues.


Moreover, we have now set the third party stats to update once an hour (instead of once every 3 hours). It's set to update 10 minutes before the hour, every hour, so checking on the hour should be safe.

(Note: this part is kind of on the back-burner at the moment.) There's two big upgrades I would like to perform on the server though that will be dependent on hourly updates. First, faster CPUs... I've been talking to my AMD rep to try and get some hex-core processors, or at the very least much faster quad-cores. Second, the stats DB would really need to be moved to a SSD for maximum processing speed. I've been looking into several companies for this. I've also been considering adding a server server at the co-lo facility that would primarily be used just for stats. But that would be an added monthly cost, and right now we all know how hard economic times are.

Also, I've gotten a few emails from people asking about the "Point Bonuses" with the A3 WUs. Everything you need to know can be found over at this thread at the F@H Support Forum. Briefly, there are a few requirements before you can get the bonuses (which you may or may not already have achieved):

• Run the latest SMP client (v6.29 or above).
  
• Configure that client with a passkey.
  
• Complete 10 a2 and/or a3 work units...
  
  
  
  • within their preferred deadlines
    
  • using the same passkey and fah user name combination
    
  • on one or more of your systems.
    
  
  
• Successfully return >=80% of assigned [bonus] WU's.
  

First, I made a few changes to the HTML on the site in relation to the sidebar. For some people using fixed fonts the text was wrapping. (I swear I didn't know, if you guys tell me this sooner I can fix it.) I don't think I broke anything, but if anyone starts experiencing any layout oddities, please let me know!

Second, I updated the text on the sig image page. I guess I should have clarified the "alternate method" for images a little more. Basically, if you are on a forum/site that doesn't allow dynamic URLs and/or requires a file with an image type extension to display (i.e. .gif), then you *can* do the following:

• http://folding.extremeoverclocking.com/sigs/sigimage.php/un=HotRod/t=11314/sigimage.gif
  
• http://folding.extremeoverclocking.com/sigs/sigimage.php/u=32359/sigimage.gif
  

I haven't checked, but I don't think I have the custom colors coded in the directory-structure parsing, I'll have to check on that.

I'm hoping to sit down and work on the stats one of these days. It's amazing how time flies... At least things are still chugging along happily after all these years. One change I'm thinking of doing is dropping "Google" back into the regular team mix, they are not producing *that* much and will put them in ~16th place. Default / Team Zero will still remain with a rank of 0. If I do this then I can eliminate a lot of exceptions in the code for both teams and streamline things and reuse more code and functions.

Besides that... how come nobody told me about the -bigadv flag?!?! *sigh*