PDA

View Full Version : Environment Climategate Computer Codes Are the Real Story


mikey23545
11-26-2009, 03:35 AM
So far, most of the Climategate attention has been on the emails in the data dump of November 19 (see here [1], here [2], and here [3]), but the emails are only about 5 percent of the total. What does examining the other 95 percent tell us?

Here’s the short answer: it tells us that something went very wrong in the data management at the Climatic Research Unit.

We start with a file called “HARRY_READ_ME.txt.” This is a file containing notes of someone’s three-year effort to try to turn a pile of existing code and data into something useful. Who is Harry, you ask? Clearly, a skilled programmer with some expertise in data reduction, statistics, and climate science. Beyond that I won’t go. I’ve seen sites attributing this file to an identifiable person, but I don’t have any corroboration, and frankly the person who wrote these years of notes has suffered enough.

The story the file tells is of a programmer who started off with a collection of code and data — and the need to be able to replicate some results. The first entry:

1. Two main filesystems relevant to the work:

/cru/dpe1a/f014

/cru/tyn1/f014

Both systems copied in their entirety to /cru/cruts/

Nearly 11,000 files! And about a dozen assorted “read me” files addressing individual issues, the most useful being:

fromdpe1a/data/stnmon/doc/oldmethod/f90_READ_ME.txt

fromdpe1a/code/linux/cruts/_READ_ME.txt

fromdpe1a/code/idl/pro/README_GRIDDING.txt

(yes, they all have different name formats, and yes, one does begin ‘_’!)

Believe it or not, this tells us quite a bit. “Harry” is starting off with two large collections of data on a UNIX or UNIX-like system (forward slashes, the word “filesystem”) and only knows very generally what the data might be. He has copied it from where it was to a new location and started to work on it. Almost immediately, he notices a problem:

6. Temporarily abandoned 5., getting closer but there’s always another problem to be evaded. Instead, will try using rawtogrim.f90 to convert straight to GRIM. This will include non-land cells but for comparison purposes that shouldn’t be a big problem … [edit] noo, that’s not gonna work either, it asks for a “template grim filepath,” no idea what it wants (as usual) and a serach for files with “grim” or “template” in them does not bear useful fruit. As per usual. Giving up on this approach altogether.

Things aren’t going well. Harry is trying to reconstruct results that someone else obtained, using their files but without their help.

8. Had a hunt and found an identically-named temperature database file which did include normals lines at the start of every station. How handy — naming two different files with exactly the same name and relying on their location to differentiate! Aaarrgghh!! Re-ran anomdtb:

Okay, this isn’t so unusual, actually, but unless you document and describe your file structure, it’s pretty much opaque to a new reader. Still, Harry presses on:

11. Decided to concentrate on Norwich. Tim M uses Norwich as the example on the website, so we know it’s at (363,286). Wrote a prog to extract the relevant 1961-1970 series from the published output, the generated .glo files, and the published climatology. Prog is norwichtest.for. Prog also creates anomalies from the published data, and raw data from the generated .glo data. Then Matlab prog plotnorwich.m plots the data to allow comparisons. First result: works perfectly, except that the .glo data is all zeros. This means I still don’t understand the structure of the .glo files. Argh!

Poor Harry is in the first circle of programmer hell: the program runs fine; the output is wrong.

He presses on:

17. Inserted debug statements into anomdtb.f90, discovered that a sum-of-squared variable is becoming very, very negative! Key output from the debug statements:

some test output…

forrtl: error (75): floating point exception

IOT trap (core dumped)

..so the data value is unbfeasibly large, but why does the sum-of-squares parameter OpTotSq go negative?!!

This is not good — the existing program produces a serious error when it’s run on what is supposed to be the old, working data. Harry presses on, finding a solution to that bug, going through many more issues as he tried to recreate the results of these runs for the data from 1901 to 1995. Finally he gives up. He has spoken to someone about what should be done:

AGREED APPROACH for cloud (5 Oct 06).

For 1901 to 1995 – stay with published data. No clear way to replicate process as undocumented.

For 1996 to 2002:

1. convert sun database to pseudo-cloud using the f77 programs;

2. anomalise wrt 96-00 with anomdtb.f;

3. grid using quick_interp_tdm.pro (which will use 6190 norms);

4. calculate (mean9600 – mean6190) for monthly grids, using the published cru_ts_2.0 cloud data;

5. add to gridded data from step 3.

This should approximate the correction needed.

Catch that? They couldn’t recreate the results, so they’re going back to their published data for the first 95 years of the 20th century. Only …

Next problem — which database to use? The one with the normals included is not appropriate (the conversion progs do not look for that line so obviously are not intended to be used on +norm databases).

They still don’t know what to use for the next several years. Harry gives up; it’s easier to write new codes.

22. Right, time to stop pussyfooting around the niceties of Tim’s labyrinthine software suites – let’s have a go at producing CRU TS 3.0! since failing to do that will be the definitive failure of the entire project.

This kind of thing is as fascinating as a soap opera, but I want to know how it comes out. Near the bottom of the file, I find:

I am seriously close to giving up, again. The history of this is so complex that I can’t get far enough into it before by head hurts and I have to stop. Each parameter has a tortuous history of manual and semi-automated interventions that I simply cannot just go back to early versions and run the update prog. I could be throwing away all kinds of corrections – to lat/lons, to WMOs (yes!), and more.

The file peters out, no conclusions. I hope they find this poor guy, and he didn’t hang himself in his rooms or something, because this file is a summary of three years of trying to get this data working. Unsuccessfully.

I think there’s a good reason the CRU didn’t want to give their data to people trying to replicate their work.

It’s in such a mess that they can’t replicate their own results.

This is not, sadly, all that unusual. Simply put, scientists aren’t software engineers. They don’t keep their code in nice packages and they tend to use whatever language they’re comfortable with. Even if they were taught to keep good research notes in the past, it’s not unusual for things to get sloppy later. But put this in the context of what else we know from the CRU data dump:

1. They didn’t want to release their data or code, and they particularly weren’t interested in releasing any intermediate steps that would help someone else

2. They clearly have some history of massaging the data — hell, practically water-boarding the data — to get it to fit their other results. Results they can no longer even replicate on their own systems.

3. They had successfully managed to restrict peer review to what we might call the “RealClimate clique” — the small group of true believers they knew could be trusted to say the right things.

As a result, it looks like they found themselves trapped. They had the big research organizations, the big grants — and when they found themselves challenged, they discovered they’d built their conclusions on fine beach sand.

But the tide was coming in.


http://www.pjtv.com/?cmd=browse-events&event-type-id=10&event-id=1913&event-context-theme-id=1&c=10&s=coverage&r=true&p=1&t=overview

Guru
11-26-2009, 05:15 AM
So can I go back to using incandescent bulbs now?

HonestChieffan
11-26-2009, 05:30 AM
http://avionod.files.wordpress.com/2009/04/then-a-miracle-happens.gif?w=300&h=364

HonestChieffan
11-26-2009, 06:51 AM
And the Hoax continues to unwind...yet Obama is going to Copenhagen to lend support.

Phoney buying into more phoney.

http://nzclimatescience.net/index.php?option=com_content&task=view&id=550&Itemid=1

tiptap
11-26-2009, 08:27 AM
I would just like to point out that the sum of squared variables are always positive and if they are added only they can only get bigger. So until this is cleared up there is a programming error.

KCWolfman
11-26-2009, 08:37 AM
I would just like to point out that the sum of squared variables are always positive and if they are added only they can only get bigger. So until this is cleared up there is a programming error.

lol.

Pay no attention to the man behind the curtain.

bevischief
11-26-2009, 08:41 AM
Can we kick out the UN now?

HonestChieffan
11-26-2009, 09:34 AM
http://4.bp.blogspot.com/_PxZyE6Jgabo/Sw6YAcxWi_I/AAAAAAAAMJA/axWcg5GpFI8/s400/theo2.gif

mikey23545
11-26-2009, 09:49 AM
lol.

Pay no attention to the man behind the curtain.

It is somewhat disconcerting to watch a man's entire belief system crumbling right before his eyes, and in a public forum, as well...I actually feel sorry for Tiptap in a way...

Bwana
11-26-2009, 12:01 PM
I would just like to point out that the sum of squared variables are always positive and if they are added only they can only get bigger. So until this is cleared up there is a programming error.

If you believe this, I have some really really bad news regarding the Tooth Fairy.

RedNeckRaider
11-26-2009, 12:07 PM
If you believe this, I have some really really bad news regarding the Tooth Fairy.

LMAO rep

irishjayhawk
11-26-2009, 01:05 PM
If you believe this, I have some really really bad news regarding the Tooth Fairy.

:spock:

What's wrong with what he said?

KC native
11-26-2009, 01:42 PM
:spock:

What's wrong with what he said?

It's math and it's correct.

jjjayb
11-26-2009, 01:48 PM
:spock:

What's wrong with what he said?

Why would a Wookiee, an eight-foot tall Wookiee, want to live on Endor, with a bunch of two-foot tall Ewoks? That does not make sense! But more important, you have to ask yourself: What does this have to do with this case? Nothing. Ladies and gentlemen, it has nothing to do with this case! It does not make sense! Look at me. I'm an idiot defending a major global warming science study, and I'm talkin' about Chewbacca! Does that make sense? Ladies and gentlemen, I am not making any sense! None of this makes sense! And so you have to remember, when you're in that jury room deliberatin' and conjugatin' the Emancipation Proclamation, does it make sense? No! Ladies and gentlemen of this supposed jury, it does not make sense! If Chewbacca lives on Endor, you must acquit! The defense rests

StcChief
11-26-2009, 01:53 PM
It's math and it's correct.not a strong point for the "scientists" for GW either. wow. how much are they paid?

KCWolfman
11-26-2009, 01:55 PM
:spock:

What's wrong with what he said?

Only the fact that tiptap found a single COMPUTER error in all the data shown to be false over the last several days, including blatant emails from multiple HUMAN correspondents admitting to hiding and falsifying data and in a desperate attempt to be right he is drawing correlation between the two and holding off a conclusion on the human factor based upon the computer factor.

KCWolfman
11-26-2009, 01:58 PM
It's math and it's correct.

So what you are saying is that paid life long professionals taking millions of dollars from our government to study global warming data had false data that tiptap, a hobbyist at global warming, found in a matter of moments?

Sooooooo.... is it:

A. tiptap is wrong and the paid professionals taking tax dollars are doing their job correctly.

or

B. The professionals really don't know their ass from a hole in the ground and makes you wonder about the rest of the data they have accumulated is inaccurate as well.

irishjayhawk
11-26-2009, 01:58 PM
not a strong point for the "scientists" for GW either. wow. how much are they paid?

:spock:

ROFLROFLROFLROFLROFLROFL

Now math principles are under fire for being invented by biased mathematicians who receive funding from corporations?

irishjayhawk
11-26-2009, 01:59 PM
Only the fact that tiptap found a single COMPUTER error in all the data shown to be false over the last several days, including blatant emails from multiple HUMAN correspondents admitting to hiding and falsifying data and in a desperate attempt to be right he is drawing correlation between the two and holding off a conclusion on the human factor based upon the computer factor.

I'm still looking into it. So I'm not going to comment on the emails yet.

But balloon boy should convince people to be a tad more skeptical than they are currently.

Oh, and what he points out is highly relevant and has nothing to do with humans.

KCWolfman
11-26-2009, 01:59 PM
:spock:

ROFLROFLROFLROFLROFLROFL

Now math principles are under fire for being invented by biased mathematicians who receive funding from corporations?

Refer to my post under yours. One of the postulates must be true.

KCWolfman
11-26-2009, 01:59 PM
I'm still looking into it. So I'm not going to comment on the emails yet.

But balloon boy should convince people to be a tad more skeptical than they are currently.

Oh, and what he points out is highly relevant and has nothing to do with humans.

Do you need as long as the POTUS does for Afghanistan?

irishjayhawk
11-26-2009, 02:00 PM
So what you are saying is that paid life long professionals taking millions of dollars from our government to study global warming data had false data that tiptap, a hobbyist at global warming, found in a matter of moments?

Sooooooo.... is it:

A. tiptap is wrong and the paid professionals taking tax dollars are doing their job correctly.

or

B. The professionals really don't know their ass from a hole in the ground and makes you wonder about the rest of the data they have accumulated is inaccurate as well.

Or....

C. The emails aren't really what they're made out to be.

irishjayhawk
11-26-2009, 02:01 PM
Do you need as long as the POTUS does for Afghanistan?

No, but I am working on clearing my to-read queue that's still highly populated. Then I can devote my attention to the newer, recent developments.

KCWolfman
11-26-2009, 02:02 PM
Or....

C. The emails aren't really what they're made out to be.

If it were one or two emails, you may be accurate.

Unfortunately, there are way too many to be dismissed.

irishjayhawk
11-26-2009, 02:03 PM
If it were one or two emails, you may be accurate.

Unfortunately, there are way too many to be dismissed.

:spock:

Well, there's so many Prince of Nigeria emails that they must be true!

KCWolfman
11-26-2009, 02:19 PM
:spock:

Well, there's so many Prince of Nigeria emails that they must be true!

LMAO.

See, you get data and you dismiss the facts you don't like - kinda like the scientists in question.

these aren't random emails from false outside sources, they were garnered from a single company with multiple experts all covering up all kinds of data.

A cigar is sometimes just a cigar

irishjayhawk
11-26-2009, 02:43 PM
LMAO.

See, you get data and you dismiss the facts you don't like - kinda like the scientists in question.

these aren't random emails from false outside sources, they were garnered from a single company with multiple experts all covering up all kinds of data.

A cigar is sometimes just a cigar

Way to miss my point.

KCWolfman
11-26-2009, 03:00 PM
Way to miss my point.

Funny, I thought the same thing about your statement.

irishjayhawk
11-26-2009, 04:29 PM
Funny, I thought the same thing about your statement.

Weird. What was your point in your original statement then?

If it were one or two emails, you may be accurate.

Unfortunately, there are way too many to be dismissed.

I don't see any other substance here than "there's too many to ignore".

jjjayb
11-26-2009, 09:35 PM
But balloon boy should convince people to be a tad more skeptical than they are currently.



You think? Have you missed the whole point? We are man made global warming skeptics. :doh!:

morphius
11-26-2009, 10:19 PM
Would someone give me an example of why you would need to add the square root of two numbers when trying to give out temp data?

I have to wonder if this might be another case of capitalism screwing up science, there is too much money in global warming.

irishjayhawk
11-27-2009, 03:05 AM
You think? Have you missed the whole point? We are man made global warming skeptics. :doh!:

There's a differece between man-made global warming and global warming. The latter is as good as fact. The former is a fact but still up for speculation as to how much humans are affecting, which is where the controversy lies. Often, people throw out the latter entirely when their objection is that humans aren't affecting it.

Would someone give me an example of why you would need to add the square root of two numbers when trying to give out temp data?

I have to wonder if this might be another case of capitalism screwing up science, there is too much money in global warming.

Don't go around saying that else KC Titus will go full retard on you.

tiptap
11-27-2009, 09:17 AM
Here is how I see this starter quote:

We have two persons separated by time. The initial data and data computed wants to be duplicated in a new article and extend that data with up to date measurements of the present.

A scientist, sloppy in his programming, generated the original graph. There was a new programmer fellow hired to recreate the original work and extend it. The walk through is the frustration in reading the original work. You are moving from an era when memory constraints were tighter and more expensive.

The new hired hand programmer/meteologist failed to duplicate the process. They ended up just tacking the new data to the old graph. THAT IS UNFORTUNATE at the least if they did not state this in the published paper. Is that the case? Well you don't get that answer from what is posted.

When I said earlier that the thread pointed out that the data was getting negative numbers from a squaring and addition operation. That is mathematically impossible. Period. So there had to be a programming error that introduced the +/- error.

As to why anyone would square a temperature reading in reproducing models, you don't. You use the bell shape curve for the a bunch of temperatures and the differences between the mean and readings are substracted and THOSE values are squared in producing the range, the spread of the values. Remember your statistics classes on how one gets the value for the mean and the value for one standard deviation (you square it).

tiptap
11-27-2009, 09:22 AM
Would someone give me an example of why you would need to add the square root of two numbers when trying to give out temp data?

I have to wonder if this might be another case of capitalism screwing up science, there is too much money in global warming.

see above last paragraph

RINGLEADER
11-27-2009, 10:21 AM
This is all old news.

Everyone should have known there was nothing to their "science" the minute the global warming theorists became climate change theorists.

I continue to blame the sun for global warming (and cooling).

And maybe those pesky sunspots.

Damn sunspots.

irishjayhawk
11-27-2009, 01:32 PM
This is all old news.

Everyone should have known there was nothing to their "science" the minute the global warming theorists became climate change theorists.

I continue to blame the sun for global warming (and cooling).

And maybe those pesky sunspots.

Damn sunspots.

When you realize that the change between global warming and climate change was purely PR based specifically because of the "oh my god it's 0 degrees out in December it must not be warming" idiots, it'll be the day I win the lottery.

At least be honest.

RINGLEADER
11-27-2009, 02:17 PM
When you realize that the change between global warming and climate change was purely PR based specifically because of the "oh my god it's 0 degrees out in December it must not be warming" idiots, it'll be the day I win the lottery.

At least be honest.

No, I think temps are trending down in general, not up. And not just in December.

The climate change fantatics are ideologues who search for ways to mold science to fit conclusions.

That's one of the things the recent email releases underscore.

irishjayhawk
11-27-2009, 02:23 PM
No, I think temps are trending down in general, not up. And not just in December.

The climate change fantatics are ideologues who search for ways to mold science to fit conclusions.

That's one of the things the recent email releases underscore.

THERE IS NO FALSIFICATION OF DATA IN THE EMAILS.

headsnap
11-27-2009, 05:10 PM
THERE IS NO FALSIFICATION OF DATA IN THE EMAILS.

of course there isn't, the falsification of data is already out in the public, the emails point point out/describe the falsification...

headsnap
11-27-2009, 05:12 PM
of the "oh my god it's 0 degrees out in December it must not be warming" idiots...

At least be honest.

that just balances out the "oh my science it's 102 in July" alarmists...


At least be honest.

;)

KCTitus
11-27-2009, 10:02 PM
Don't go around saying that else KC Titus will go full retard on you.

LOL...actually all of 'that' money is government money...money taken from the capitalist system via taxes to fund junk science. Going full 'retard' is making statements like Capitalism is the enemy of science.

irishjayhawk
11-27-2009, 10:19 PM
of course there isn't, the falsification of data is already out in the public, the emails point point out/describe the falsification...

There's falsified data in the public realm. Can you point me to it?

that just balances out the "oh my science it's 102 in July" alarmists...


At least be honest.

;)

If there were such a thing, it would. :)

LOL...actually all of 'that' money is government money...money taken from the capitalist system via taxes to fund junk science. Going full 'retard' is making statements like Capitalism is the enemy of science.

:spock:

We're funding junk science with taxes? Can you point me to a specific instance, please?

headsnap
11-28-2009, 04:13 AM
There's falsified data in the public realm. Can you point me to it?


http://www.chiefsplanet.com/BB/showthread.php?t=218863

Baby Lee
11-28-2009, 11:35 AM
If there were such a thing, it would. :)
Are you seriously suggesting there isn't a 'I saw a polar bear on a tiny ice floe and it convinced me we all need to cut down to one square of TP per BM' contingent out there?

banyon
11-28-2009, 12:45 PM
This smells like a hoax.

Saul Good
11-28-2009, 07:31 PM
This smells like a hoax.

AGW? It certainly does.

KCTitus
11-29-2009, 12:17 AM
We're funding junk science with taxes? Can you point me to a specific instance, please?

Every bit of it...you see, if it were profitable, these so called 'evil' capitalist systems would be exploiting it.

Instead the government has to direct funds to the CRU and other 'scientific' research labs. Obviously, those labs are going to produce the results that their funding source wants.

Im not sure if you've ever heard of NASA or NOAA.

There are more funds going to various other interests including your personal favorite GE. They have been positioning themselves for a financial windfall once crap and tax made it through. This is what's called fascism, not capitalism.

Please continue to blame capitalism, however, it's almost dead.

Other examples would be governments restrictions on furthering additional oil exploration off both coasts and in Alasksa, nuclear research and building of reactors, and building of additional refineries for oil.

irishjayhawk
11-29-2009, 09:18 AM
http://www.chiefsplanet.com/BB/showthread.php?t=218863

:spock:

That's all you got?

Are you seriously suggesting there isn't a 'I saw a polar bear on a tiny ice floe and it convinced me we all need to cut down to one square of TP per BM' contingent out there?

I was taking his statement literally, BL. I'm sure there are PETA-level equivalents of GW. Just as there are PETA-level equivalents of anti-GW. Hell, a lot of those are in this very forum.

Every bit of it...you see, if it were profitable, these so called 'evil' capitalist systems would be exploiting it.

Instead the government has to direct funds to the CRU and other 'scientific' research labs. Obviously, those labs are going to produce the results that their funding source wants.

Im not sure if you've ever heard of NASA or NOAA.

NASA and NOAA are producing results the government wants?


There are more funds going to various other interests including your personal favorite GE. They have been positioning themselves for a financial windfall once crap and tax made it through. This is what's called fascism, not capitalism.

Please continue to blame capitalism, however, it's almost dead.

Other examples would be governments restrictions on furthering additional oil exploration off both coasts and in Alasksa, nuclear research and building of reactors, and building of additional refineries for oil.

Phew, I'm glad I'm not living in the paranoia world you live in. It's like a whole new reality. Almost a different dimension.

cdcox
11-29-2009, 10:18 AM
nuclear research and building of reactors

There is 1 nuclear power plant under construction, 11 planned and 19 additional ones are proposed. Nuclear engineering departments at US Universities are at capacity enrollment, new programs are starting an many schools, existing departments are adding faculty, and Government funding to do nuclear research is way up.