EOC Folding@Home Stats - News & Updates

I've finally managed to get the additional Active / Total User charts up. You will notice them on the Team Summary page and the Aggregate Summary page. Besides the existing daily scale, I added hourly, weekly, and monthly to match the points / wus charts. The weekly / monthly user numbers are AVERAGES. After trying a few different things, a basic average seemed to best represent what was going on for each time period. Daily simply takes the last record for each day.

Before anyone emails me, the reason that you will see a couple drops in 'total users' is because I did some pruning previously of low-point (550 points or less) idle users.

I spent quite a bit of time validating the chart data, but if you see something that looks wrong, please let me know!

Next thing I want to get online is the OS stats. I've been storing the data for a few months now, just need to finish the pages & charts (which is next on my list). I wish I started storing the data sooner, it would have been interesting to have more data pre-covid.

Oh, for those wondering, I finally got my RAM back on May 30th I think it was (long story on that). The new RAM tested out okay and back up and running. The stick they said was scratched I have to counter and say it was a factory defect in the PCB as it is clearly not a scratch. But that's neither here nor there, they replaced just the bad stick, I have working RAM, crisis averted.

First the good news. I don't remember why, but I had a couple extra sticks of matching RAM for my old Supermicro board just sitting on the shelf. I think I bought them a few years (or more) ago on eBay because they were dirt cheap, not a lot of demand for PC2-5300 ECC Registered memory. Got those swapped out and all is well. The bad stick was dated 2007, so I guess it has had a good run. So at least my VMWare box is back up and running again.

Now for the bad news. The bad RAM in my FAH dev box I mailed back to G.Skill this past Saturday (the 2nd), and amazingly it travled half-way across the US and got delivered Monday (the 4th)! Anyhow, I just got an email this evening from their RMA department... "We recently received the memory that you sent us for a replacement. While inspecting the memory, our technicians noticed that one of your memory modules had scratches to the pins. Unfortunately, scratches to the pins are considered physically damaged and are not covered under our warranty. We apologize for the inconvenience and have arranged for the memory to be shipped back shortly."

First, that RAM has been installed and untouched in that computer for the past 7 years! Second, the module serial # they said had scratches wasn't even the one that was faulty! Third, I cleaned off the pins when I was started having issues and never noticed any scratches. *rolleyes* FML...

I feel so stick to my stomach... Frustrating, very frustrating... Sorry for the rant, and also the lack of FAH updates... This week has just been the week of putting out fires...

I'm sure you were noticing the 0 updates twice a day these past couple weeks. I talked to Joseph (the lead developer over at F@H), he reduced the frequency a while back to help reduce the load of the F@H servers (as I figured). I logged the flat-files every hour just to see what was going on, and determined it was updating every 4 hours. I asked him if he could bump that to every 3 hours, and he was kind enough to do so (since that is how often the EOC stats update). I has been a couple days now an no missed updates, so hopefully everyone is happy now!

My F@H dev box started acting quirky recently. It was having issues POSTing when you would power on or reboot. So I re-seated the four RAM modules (32GB total) and made sure everything was connected tight. Still had the same issues, so I ran memtest86 and it looks like one of the modules is bad (arghhhhhh). Still running tests to narrow down the exact module and all. But as I'm sure you all know, it's not exactly a fast process. It's G.Skill RAM, and supposedly has a lifetime warranty, so not a huge deal, just a nuisance I didn't need to deal with right now. It will still run perfectly fine on 16GB in the meantime.

Figured while I was at it I would check my VMware box (older Supermicro dual-socket xeon board). The BIOS DMI log showed a bunch of single-bit errors (arghhhhhh), so I'm running memtest86 on it too and getting errors. It's all Crucial ECC Registered RAM (24GB total), and I'm pretty sure they have lifetime warranty too.

Then I had not one, but two USB flash sticks die... I don't use them that often anymore, just when needing to install a new OS, or boot to run memtest86. Still frustrating though when you are down to just a couple and those that are good have information you don't want to delete, so you have to copy everything off before you can re-format it.

So yeah, yesterday and today have been somewhat of a boondoggle, just going between machines swapping out RAM modules. Ugh...

I did some quick calculations the other day since I was curious exactly how much of the stats I'm processing (compared to the total data in the flat-files). Turns out I'm tracking 99.7% Points & 96.2% WUs based on the teams totals.

I've had a number of people ask me about charts for Work Units (WUs) and I finally got a chance to update the chart code. Unfortunately it doesn't look too terribly interesting, almost the same as the points graphs. I'm going to try and expand the Users / Users Active charts (which are currently just daily) to include the same time frames at the Points/WUs.

I recently started tracking the OS client stats (like I've talked about in previous posts). They are checked each update, but the numbers seem to update once a day. Eventually I'll get around to making some charts / tables with the data as I'm sure people would be extremely interested to see how the total FAH performance has been changing. I even went on archive.org to dig up what old data they had! Unfortunately the numbers were kind of spotty and only sparsely available, but I manually recorded everything and can sort through it later. It looks like the numbers continue to grow for all clients, with the current estimate at almost 2.6 (x86)exaFLOPs!

I saw a neat article the other day at Bleeping Computer. There is a project posted by Microsoft on Github to fold in a sandbox. The sandbox feature is only available on Windows 10 Pro and Enterprise, build 18362 or newer. Basically it runs the F@H client in a Hyper-V isolated environment.

Been doing lots of other little fixes and tweaks, not really worth mentioning.

Literally in the span of a week, according to the FAH Client Stats you guys have DOUBLED the processing power and now have exceeded 2.3 (x86)exaFLOPs!

The new totals (compared to those I posted LESS than a week ago): 117,103 AMD GPUs (47.30%) / 535,769 Nvidia GPUs (50.47%) / 1,205,450 CPUs (102.94%)

A couple cool videos on YouTube, first is from the Financial Times interviewing Dr. Bowman and a little talk about the Folding@Home project and the science behind it. Second video is from Linus Tech Tips where Linus shows off the Folding@Home WS Server they built and is currently part of the F@H Server Pool, helping to send out all those work units to all you guys keeping those GPUs & CPUs busy!

As for recent team production, it looks like Linus Tech Tips Team is currently in the lead, with the Default Team trying to keep pace, but the big surprise is AWSFolds sneaking up from the back of the pack and putting on the heat against the other top teams! If AWSFolds continues to ramp up like they have been we might have a new top contender! There were a couple times late 2017 / early 2018 where CureCoin broke 60 BILLION points produced in a month... It looks like that record *could* get shattered this April by any (or all) of these top 3 teams if they can keep up the pace!

So, I started my day wanting to troubleshoot a specific issue. As I was doing that on one of my dev servers (they are all just VMWare images, it's just one little machine), I noticed its copy of the FAH database was from 2004-2012 (both team & user)! I totally forgot having the FAH data on that particular VM. So that's at least one more source of old data I know about.

One side-track leads to another, and next thing you know I'm back to figuring out the nginx proxy caching issue. After reading some RFC's I got it sorted out, moral of the story is you don't have to put a 'max-age=X' in your 'cache-control' header (you can have just the other directives like immutable, public, must-revalidate, etc), as long as you have an 'expires' header with a properly formatted date, both nginx and web browsers will fall back to using that.

While working on that I noticed yet another issue. Wasn't sure if it was a change I made during the day, so I was going back and undoing changes one-by-one, testing to see what was causing the issue. Eventually after I don't even know how long I *finally* found the cause. It ended up being the brotli module. Brotli & gzip are two compression schemes to shrink the content before sending to your web browser. At first I posted the issue on its github project page. After thinking about it I figured I might as well compare the code to see how different the brotli & gzip modules are, and lucky me they were quite similar. So after looking them over and figuring out what went wrong where, I managed to create a working patch that fixed the issue. After testing on my dev server I once again went back to Github and submitted a pull request with my patch, so hopefully the fix will be official. Then I recompiled nginx on the EOC server, and re-did allllll the changed I previously un-did... Gave it some more testing just to be double sure all is well (and it appears to be). Before you know it, it's now 8pm... ugh...

That's how my day goes, unfortunately far too often...

With that said, if you notice some pages loading weird, or have issues with stale pages (where you have to shift-refresh to get them to load the correct fresh page), please let me know!

Before I get any more emails, yes I am aware of all the 0 updates. The issue is purely on the Folding@Home server side. You guys are overloading their servers and the flat-file generation gets paused, but they are working hard to increase capacity and cope with the onslaught. Check the video below to understand more about just how MUCH of an increase we are talking about!

Between fixing other miscellaneous issues, I made some changes to the bottom of the aggregate page and I didn't want anyone to miss out.

First, I bumped up the list from the top 20 teams to top 25 (FYI: You can do the same thing going to the Teams List page and clicking on the "Points 24hr Avg" header to sort by that column). The "Aggregate User Info" table has been fixed, previously I had a few calculation errors. 'New' users are those that are new within the past 7 days, and the up/down change is the difference from the previous 7 days before that (hopefully that makes sense). 'Active' users is a current figure (based on 24hr average > 0), the up/down change is based on 7 days ago.

Finally, there are both User & Team totals broken down by their color-coded production 'groups'. Hovering over the numbers will tell you the percentage.

If you noticed some weird page caching issues the other day, I tried to enable some proxy caching with nginx and it did not go well. The caching part worked great, however there was an issue where it does not update the cache-control 'max-age' time (which is the number of seconds left before a page expires). Going to try and see if I can do something with the lua module... Always something...

It's Sunday again, and I managed to make another video. This one highlights WEEKLY production, starting January 2019 and runs up until yesterday (4/4/20). The "weeks" shown are based on Sunday's date for the week total, just like the tables in the stats pages.

It looks like the F@H servers have been getting hit pretty hard these past couple days. When things get overloaded, the stats feed become the low priority on the update list (hence, all the 0 updates). Rest assured, any WUs you send back to the servers are safe and will get processed eventually, just please be patient! The F@H team has been diligently bringing more servers online, but I think there are so many idle clients out there it still far exceeds all the new capacity.

In case you missed the news the other day, the F@H project has broken the (x86) exaFLOP barrier! With over a million devices currently online (79,498 AMD GPUs / 356,065 Nvidia GPUs / 593,983 CPUs), you guys are setting new records left & right!

There is some exciting new progress on COVID-19 that the F@H team has posted here on twitter, or a more detailed blog post here at folding@home.

There was also this blog post from earlier this week about a new batch of small molecule screening simulations related to COVID-19. One important bit of info is these can only run on CPUs, so make sure you have your CPUs enabled in your clients!

I've been juggling several things at once, fixing little bugs, working with a fellow folder to bring some new features online for you guys, braving venturing out of the house to run some essential errands. We are working on querying the F@H server stats to log and try to better understand what's going on and hopefully translate that into some nice graphs with trends & tables for you guys.

The video I posted below went online last Sunday, so to try and keep things interesting I'm going to try and post another (different) video this coming Sunday. People have been posting suggestions in the YouTube comments, so if you have something you would like to see be sure to post it there, or send me an email directly. I still need to load into the DB some archived data I have to see exactly what it contains, maybe I can do that tomorrow if nothing major comes up.

Here's a funny throwback pic I found the other day while looking for something else. It's one of my old folding rigs from January 2003! Back in the days of single-core CPUs (dual-core was still a couple years away), and WAY before GPU processing. I'm pretty sure those are AthlonXP CPUs, probably on cheap Biostar boards. I remember making custom Y power cables so I could use one PSU for two boards. LOL, those were the days. They were booting over the network to my main linux machine that held a common image, so at least I didn't have to worry about a bunch of hard drives and independent operating systems to maintain.

Stay safe everyone!

So, I decided to take a little break from the usual fixing bugs in the stats and decided to channel my creative energy towards something I think everyone will enjoy. Just created the channel on YouTube so I'm still setting things up. Let me know what you think and if you want to see more. :)