Home | Reviews | Forums | Downloads | EXTREME Deals | Comparison Shopping | Free Magazines | Folding Stats Newsletters | Contact Us
FAH Stats
Update: Current
07.28.14, 3pm CDT
Time Now: 5:29pm
Next Run: 30 min




All Donations Help With Server Expenses & Future Features
upward-identification
UPDATE: Monday, March 10th 2014 The front page has been updated with new information!

EOC Folding @ Home - Stats News & Updates

Where does the time go?03.10.14, 11:09am CST

Already March of 2014, where does the time go? Yes, I do get sidetracked from time to time... lol.

I took the folding stats down for a few days in order to clean up and shift some data around. First problem the User History table was growing insanely large and causing issues. We are talking close to TWO BILLION rows of data and around 60GB of storage (just for one table). Second problem was I needed to alter the WUs & Points fields to accommodate larger numbers mostly for the anonymous user.

The stats numbers should settle back down in a week's time, but for now they should be churning along without any more issues.

Since I've already started fixing some things I'm working on the code right now to fix some other minor issues, pretty much all behind-the-scenes type stuff. I will adjust the rank colors in a week or so, I'm waiting for the production numbers to settle back down then I can better gauge what to make each level.

I have a failed drive in the server right now and am waiting on a replacement to arrive before heading down to the data center to swap it out. I've moved all content off that raid partition to be safe. It shouldn't affect the stats at all, and there shouldn't be any downtime replacing the drive.

Other than that, everything is status-quo I suppose. I'll post some more updates in the coming days / weeks as I update and fix stuff.

Here are some query times, now you can see why having a very large table is a bad idea... lol.

Query OK, 739383379 rows affected (6 hours 39 min 20.11 sec)
Query OK, 739383379 rows affected (21 hours 9 min 39.23 sec)
Query OK, 519104842 rows affected (7 hours 4 min 24.29 sec)
Query OK, 519104842 rows affected (14 hours 54 min 37.28 sec)
Query OK, 602639803 rows affected (5 hours 36 min 4.34 sec)


So I'm a couple days late...01.29.13, 5:48pm CST

Oh what a weekend, it seemed like everyone I know managed to get that stupid fake FBI ransom-ware on their PC and needed me to clean it off. AFAIK, even the latest version of Java has possible exploitsÖ yay!

On a different note, I HAVE been working on the FAH stats on my dev box. Basically doing things I planned on doing a year ago. LOL! Yeah how time flies. Iím getting close to pushing it out onto the main server, when that happens expect the stats to be down for a full day. Itís going to be a lot of work / processing to fix things up. I will give you guys a couple days warning before I do it, it will more than likely be on the weekend or maybe a Friday. Itís not going to be this weekend because of the Super Bowl, hopefully February 9th or 10th.

Yes I plan on changing team colors too during the refresh, please just be patient. Yes Iím aware of the anonymous user not showing proper historical data, thatís one reason for this code change. Also *fingers crossed* Iíll be able to process more people & teams without adding any overhead.

For you DB guruís, amongst other things I did do an interesting test comparing times and space usage for the historical data as one table vs. individual tables for each teams. While having the thousands of individual tables overcomes the inherent MyISAM table-locking issue, you run into issues having to tune MySQL parameters to handle having so many tables open, then there is a file-system issue having so many files in a single directory. In the end it was a toss-up. All the extra data processing / manipulation takes more time and there really isnít a bottle-neck with DB table-locks. Disk usage also appears to be close to the same, there is no space savings advantage one way or another. So in the end Iím keeping the historical data as one big table like it has been (more or less) and not having to worry about as many code changes. It was a lot of work rewriting all the code for the tests, and while it didn't prove to be beneficial it was still a fun little project in itself.

On my dev box here I put in an old SSD to use for the stats database, which even an old SSD smokes any hard drive. I saw the new Samsung 840 Pro 256GB drives on sale so I pulled the trigger on that and am waiting for it to be delivered, that's going to be cool to try out and see if there is any improvement.

Now, in other news... A HUGE THANK YOU to everyone that has donated for Horatioís surgery (info & link in post below). He has had the surgery and will hopefully be going to his foster home this week to recover. She did send me a couple pictures right before & after the surgery, they are pretty graphic and I donít think are really appropriate to post. Maybe after he has healed up some and on his feet I can post something a little nicer. They were very ecstatic by all your generosity, believe me it has really helped out to defray costs.


Yes, I am still here!!!01.23.13, 11:25am CST

Howdy Everyone! Iíve received a few emails lately from people asking if Iím still alive and why I havenít made any posts lately. Itís scary to think how fast this past year has gone by, but it seems like every year is like that. Sadly most of that time is just work, work, workÖ

Anyhow, before I talk about the folding stats, there is something else I wanted to bring up that is dear and personal to me. Please bear with me as I really donít want to go into too many details because it is such a hard thing to talk about. Back on November 11, my dog Squirt passed away. She was a German Shepherd mix and would have been 14 this January. Nobody believed how old she was because she was always in such good health and very active, but I rescued her when she was just a tiny little puppy, only about a month old. She was my little shadow always following me around, and I was fortunate enough to be able to bring her to work with me, so we were always together all these years. When she passed away so suddenly my world just collapsed, I was devastated. And thatís about as much as I want to talk about that.

One day in Mid-December I saw on the local news where they had security camera footage of a lady abandoning a little puppy in front of a groomerís shop in the middle of a freezing cold night. The dog had a disorder where part of his brain didnít fully develop and thus he didnít have very good motor skills. Basically he walked around looking like he was drunk. But if you saw how his tail was wagging and how happy he was to see people and other dogs, you could tell that little guy was full of life, love, and happiness. That was kind of like a wakeup call where I realized there are so many pets out there that need loving homes and I knew I could damn sure do better than that lady that abandoned that sweet little puppy.

So I started searching on PetFinder and contacting local rescue groups. At first I found a dog that looked a LOT like Squirt, they could have been family they looked so similar. I called but found out she was already adopted. But then this very nice lady sent me a picture of a 2-yr old boy dog in a local shelter that kind of matched the description of what I was looking for, except he was male and I really wanted a female. I really thought long and hard about it, and the lady contacted me again the next day asking if maybe I would just foster a dog instead since she knew my whole back story. Well, sometimes the universe works in mysterious ways and long story short I decided to adopt the dog that was in the shelter. Only afterwards did I learn that the day I decided to adopt him was the day he was scheduled to be euthanized! I decided ďLuckyĒ would be a good name for him.

This was right before Christmas, and even with everything else going on the lady from the rescue group took the time off from her work to go pick him up (which I also discovered later was pretty far away), take him to the vet to get check out, fixed, get his shots, etc, etc, so I could legally adopt him, and even watch him for a couple days because I was extremely sick and stuck in bed for a week. On Christmas Eve morning I was finally well enough to meet her and pick up Lucky. He was a little shy at first, but quickly warmed up to everyone, and now he follows me around everywhere. He is such a loving and sweet dog and extremely well behaved, every day Iím in disbelief that this little guy wouldnít be around if all those prior events didnít take place when they did. Like I said, the universe works in mysterious ways.

Fast forward to this week. I get an email from the lady that rescued Lucky. They rescued another dog from a local shelter that unfortunately one of his legs is crushed so bad he is going to have to get it amputated. They really need donations for his surgery (I'm sure it will cost several thousand dollars!), which is why Iím turning to you guys. Iím asking that instead of the donations you make to these folding stats, instead please donate to this rescue group to help them defray the costs for all the good that they do. Iím including a picture of Horatio, the dog needing surgery below. Please help out if you can, I know you people are very generous already otherwise you wouldn't be doing F@H. I made a ChipIn widgit below. If you donate, please be sure to put in a note, "For Horatio's Surgery" so they know. ;)

Okay, I think this post is long enough. I'll post some about the folding stats this weekend. I have been working quite a bit on them behind the scenes with an update very soon.


Several New Things01.08.12, 6:08pm CST

First off, I would like to give a HUGE thanks to Nat from Team Anandtech for donating a pair of 300GB SAS drives. This is going to be a tremendous help in that I can now put the Folding Stats on their own dedicated disks, and also not worry about running out of space! I'm hoping I can go down to the datacenter to get these installed one day this week. If you've got the time, go over to Team Anandtech's Forum and tell the guys thanks!

Second, I've taken the CSV query offline for the time being. It is not used by many people, but it does have the capability to overload the server if misused. I am planning on re-implementing it in a slightly different fashion that won't require so much processing power. Please be patient and I will make an announcement once it is back up.

Third, I have not done the DB upgrade yet, I'm waiting until after I get these new drives installed. When I do, it will be done on the weekend, during the week is just too busy for me to try and do such an overhaul uninterrupted.

Fourth, there was a little downtime this past week. I was doing some firmware updates to various components on the server, all of which required reboots.

Fifth, I've been thinking how to maximize useful data, while minimizing the not so useful data. I do plan on restoring a bunch of the old data that got pruned. But then I'm going to go back through it and remove records where [inactive] users points stayed the same, but their rank changed [down]. I ran a query the other day, and out of the ~540,000 users being tracked, only about 3-4% are regularly active! So as you can see there is a lot of overhead tracking inactivity.

This week I also plan on making a code change related to above. For inactive users when their rank goes down, instead of adding a new row with the "current update,(same) points, and (new) rank.", instead I'm just going to update their last entry with the new info. This will have one little side-effect in the display of the stats. For inactive people it won't show the little red arrow as they move down the ranks. I'm still thinking about it if I keep track of two points in time then I can still show the red arrow and not have too much overhead in space, but translating a thought into practical code is another story!

I also am planning on moving team Google into the normal ranks. Their points total has dropped so much that when I do they will be ranked around 47-48 or so.

I know, I still need to adjust production totals for the colors too... It's on my list!

Finally, one last interesting statistic. With a code change implemented around the beginning of 2010 to track more users, the number of rows added to the user tables went from about 12 million per month to over 38 million per month!

Hope everyone had a Happy New Year! Hard to believe it's already 2012!


Sorry For The Down Time12.29.11, 11:16am CST

As many have noticed the FAH Stats have been down for the past couple days. I've been pruning old historical data in an attempt to get the database under control. The user history table grew to over 1.1 Billion rows, with the table consuming over 36GB!

I've managed to shave it down to around 767 million rows and the table size is now a little under 22GB. There's some tipping point where things really start to balloon if you compare the two pairs of numbers.

Yes I did do a backup before I started removing data, I do plan to eventually re-insert some of it once I get things better organized.

I've un-blocked all the IPs for now. A few people emailed me and I don't think I got a chance to respond back to everyone. If you are still having trouble accessing the XML please send me another email.

I haven't gotten around to the color changes yet, that will probably happen next week.

Be Warned Now... I will be upgrading the database (MySQL) probably this Saturday. Meaning the whole site will be completely down for at least 2-4 hours. I've got to do a complete DB dump / reload because I'm upgrading between versions that are not compatible with leaving the data in the tables.

I'm sure there's more that I can't think of right now, if I remember it I'll be sure to make an additional post.

P.S. for those that want to complain... Realistically this is probably only the second down-time in the past seven or eight YEARS for the stats (that have been MY fault and not Stanford's)... The last time being when I moved the stats to the current server.


Making Some Changes11.29.11, 2:50pm CST

First thing first. We've been having some load issues with the server and I've been trying to track down the root cause. I'm pretty sure the culprit lies in some buggy scripts that people have written that are inadvertently pounding the server with mass requests.

That being said, I've been going through the logs and have blocked some of the worst offenders over-querying the XML. If you have been blocked, but you feel your usage is legitimate, then please email me and we can work things out to figure out what needs to be done.

I have NOT gone through the regular page logs yet, I'm sure there could be some people querying regular pages and parsing that data. If you are one of those people I would HIGHLY recommend you check your scripts to make sure they aren't doing anything crazy.

Second, I'm *finally* going to be changing the color levels. I know it's been a long time coming, but as we all know with processing power increasing, so does the points. Hopefully one day this week I will be able to make the change and see how the new numbers pan out.

Third, in case you didn't notice, we have a database with well over 1.1 BILLION rows of data! What I'm planning on doing is making a complete backup (safety first), then pruning out a BUNCH of the old data. First I'm going to try and simply remove old rows where a person's points did not change, simply their rank (i.e. inactive users moving down the list). Past that I will be pruning out data to keep a maximum of one record per-day per person (i.e. reducing potentially from 8 records a day to 1). Finally, if the table is still too crazy big I'll probably just lop off the oldest data first until I can get it back down to something manageable. Realistically you shouldn't notice any change in numbers on the pages.

I'm sure I'm forgetting something else, if I remember what it is I'll be sure to make another post.



Sorry for no updates in a while!06.18.11, 4:11pm CST

How time flies when you are having fun... NOT! Anyhow, I know I haven't made any posts in a while and I apologize for that.

I'm sure some people noticed the site was down for a bit yesterday, there was a little hiccup (pretty sure it was my fault) but everything is okay now.

I've been experimenting on my dev server with the latest MySQL 5.5 and partitioning. The good news is I can partition the existing tables, the bad news is doing an upgrade to the latest MySQL will require a complete dump / import of the data.

Partitioning will allow the data (and indexes) to be physically split into separate database files on the back end. This translates into faster updates, and also I will probably merge the old archive tables so that people can get their full history (assuming they have been folding back in like 2004).

I know I've been talking about a new version of the stats for a while, but I've just had zero free time. It's all in my head, I can visualize exactly how I want the database to look and how everything is going to be divided and processed and such, it's just sitting down and getting to actually code it is where I'm at a loss.

Besides that not a lot to report. I know Stanford has been working diligently on new clients. I've read a lot about the v7 but have not personally tried it out yet. Temperatures here in Houston have been hitting 100F+ for the highs which is ridiculous this early in the summer. To top it off we have had very little rain in the past four months (if not longer)... Maybe like 1/10" a month... that ain't much! Hot and dry means the folding clients get powered down until we get a break in the weather. ;)


Chimp Challenge 2010 Is Here!05.05.10, 2:22pm CST

Hard to believe it's already been a year. Chimp Challenge 2010 has come upon us. Several teams fold under one user name and race to 20 million points!

Scoring started 12pm PDT on Wednesday May 5, 2010. I had to manually set the start scores, which is why until the 3pm CDT update they will have some negative numbers.

The stats page can be found here.

Good luck to all the teams!


Some good news for GPU users02.16.10, 1:14pm CST

This thread over at the folding forum site has some updates about the NV GPU server issues. Basically they are aware of the problem and are working on it.

I think we've had a breakthrough (well maybe that's too strong of a term), but certainly found something that will help. People should be getting more backlogged credits soon. We have to see whether this will fix all of the problems. I'm thinking it won't fix them all, but it is a step in the right direction.

On another note, I've scheduled the CPU upgrade on the server for Thursday around 10am CST. I haven't made a decision on the SSD yet. I ordered an OCZ Vertex from Newegg and I'm going to give it a whirl on my dev server here in the office and see what kind of performance it affords. I've also been trying to tweak MySQL variables, but unfortunately there is just too much other usage on the server for me to get any sort of accurate numbers.


Cleanup is complete!02.10.10, 11:08am CST

Okay, I finally purged out all of Team FTL from the DB and re-optimized the tables. Reduced the history table by a couple gigabytes! Also now that the index is not fragmented the processing time should drop back down to hopefully about five minutes.

Good News Everybody! AMD is sending me a couple 2.5GHz High-Efficiency 45nm Shanghai quad-core processors to replaced the current 2.0GHz 65nm Barcelona quad-cores. (I tried for six-cores, but they didn't have any available.) It might not sound like much, but 500MHz x 8 cores = 4GHz additional processing. Also the 45nm chips have 6MB of L3 cache vs the 65nm which only have 2MB. It should give a nice bump in performance. When they come in I'll call my hosting company to setup a time to install them at the data center. Hopefully downtime should only be like 10-15 minutes.


Folding Stats 2.3.0
Copyright © 2000-2014 EXTREME Overclocking. All rights reserved.
Disclaimer of Liability - Privacy Policy
Page generated in 0.002 seconds (86.96% PHP - 13.04% MySQL) with 5 queries. 243KB of memory used.
Secret Forum