Datamining in Intercst SWR Study

Financial Independence/Retire Early -- Learn How!
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Datamining in Intercst SWR Study

Post by hocus »

I tried putting this post up as a response in the Raddr thread re sequencing of returns. It turned out sort of messed up, and I would like to get some responses to the question posed at the end of the post, so I thought I should try it as a separate thread. If you want to know the background discussion that led to this thread, please take a look at the two pages of posts in the Raddr thread re Sequencing of Returns. The stuff by raddr, which supplies the data at issue here, is at the top of Page One of that thread from a few days ago.


The new post starts here:

At the T .Rowe Price site, the result is 2.64% for a 99% survival rate for 40 years. This would be the absolute worst case based on historical data.

One of the criticisms that has been raised re the intercst SWR study (but not by me) is that it is produced through DataMining, and therefore the results are not reliable. Intercst acknowledges that the study is an exercise in datamining, but defends his use of this statistical technique in this particular case on grounds that what he is doing is really an example of reverse datamining.

The claim is that he is not looking for a particularly good result (as was the case with the Foolish Four strategy that DataSnooper exposed at the TMF site) but a particularly bad result, the worst case scenario of all those that turned up in the 130 years of historical data examined. I personally believe that this defense is a valid one, so long as intercst makes every effort possible to identify the true worst possible case.

But this is precisely what her failed to do. There are all sorts of worst case scenarios that intercst failed to take into account in his study. One is the possibility that someday stocks might reach a level of overvaluation higher than they ever had in the 130 years he examined. That in fact happened in the late 1990s, and the failure to take this reality into account means that the study does not in fact consider the worst case scenario. Bernstein's analysis does factor in the overvaluation levels of the late 1990s, and that is why he came up with a SWR of 2 percent.

Another worst-case scenario not examined is the one pointed to by Raddr at the open of this thread--the fact that returns in the future may be the same as in the past, but the process by which those returns are achieved may not play out in the precise same way. Stocks may provide an average return of x, but Year Six may get switched with Year Five. Those little switches can cause a big drop in the safe withdrawal rate, as raddr shows with data. What he is describing is not a future worse than what we have seen in the past. The only investors who will perceive it as being "worse" are those who rely on the intercst study in planning their retirements.

These are not the only worst case scenarios that intercst failed to account for, but they are two of the most important ones.

Raddr expressed some surprise at the fact that changing the sequence of returns a small amount causes only slight changes upward in the SWR in some cases but relatively large changes downward in others. I believe that the cause of this disparity is intercst's use of the datamining technique. This is the exact criticism that is usually made of the technique, that it produces results that appear on the surface to be statistically valid but which are biased because of the desire on the part of the "researcher" to come to one particular conclusion rather than to just permit the data to speak for itself.

What makes it confusing is that in this case the usual signs of datamining are being shown to us in a mirror. Everything is reversed because the goal of the exercise was not to come up with the best possible answer (a high stock return), but the worst possible answer (the worst case scenario). But I believe that the bias with which the exercise was conducted produces the same sort of flaws as the more common form of datamining. Here the bias is not that intercst looked for good results that are not statistically likely to repeat (as in the Foolish Four strategy), but that he ignored bad results that would have pulled the safe withdrawal rate downward.

Does anyone see what I am getting at in saying that the root problem with the intercst study is its reliance on datamining (and a failure to do what would be required to make datamining appropriate in this case)? Am I right that the reason for the large drops in SWR from small sequence changes pointed out by Raddr is the use of a form of "datamining" in determining the SWR? I am not a statistics expert, or anything remotely close to it. So I do not want to make this claim on the other board if there are not others more knowledgable on this stuff agreeing with the essence of what I am saying here.
raddr
*** Veteran
Posts: 265
Joined: Mon Nov 25, 2002 3:25 am
Contact:

Post by raddr »

One of the criticisms that has been raised re the intercst SWR study (but not by me) is that it is produced through DataMining, and therefore the results are not reliable. Intercst acknowledges that the study is an exercise in datamining, but defends his use of this statistical technique in this particular case on grounds that what he is doing is really an example of reverse datamining.

Oh my. I don't remember him saying this when I was there but it doesn't surprise me. I would argue that this is datamining of the worse kind. First of all, just the fact that his study came out when it did at the height of the bubble and market euphoria is an exercise in hindsight bias in and of itself. I mean why didn't he choose real estate or Japanese stocks? Because he wanted to look at what had been hottest lately, i.e. he picked the winner in retrospect. This would be like running the study on fidelity magellan when Lynch was there and extrapolating forward to the future.

Second, far from being "reverse" datamining he actually optimized the allocations to fit the efficient frontier in retrospect. For example, IIRC he says that something like 75:25 mix was optimal for 30 years. Well, yes, it was but who knows what the efficient frontier will be going forward? In order to take advantage of his hindsight bias you have to know in advance what allocation will perform best the next few decades. Maybe it will be 75:25 but it may be 60:40 or 90:10. Guess wrong and you lower your SWR.

Next, there is absolutely no statistical significance to the amount of data he's got. There is so much overlap from year to year that it useless to perform any statistical analysis since that would require independent data points of which there are pitifully few, maybe 3 or 4 depending on the period of time studied.

Since there is no way to get any statistically usable info from such a few non-overlapping periods the next best thing you can do is run some simulations like I've previously done, much like the same thing that was done to analyze mean reversion. These cast severe doubts on the future chances that the SWR will stay at or near 4%.

There are all sorts of worst case scenarios that intercst failed to take into account in his study. One is the possibility that someday stocks might reach a level of overvaluation higher than they ever had in the 130 years he examined. That in fact happened in the late 1990s, and the failure to take this reality into account means that the study does not in fact consider the worst case scenario. Bernstein's analysis does factor in the overvaluation levels of the late 1990s, and that is why he came up with a SWR of 2 percent.

This is something that should not be ignored. If you look at the ratio of past peak-to-peak PE ratios you get this:



You can see the two peaks at 1929 and in the late 60's - not coincidentally the the two times that a 4% WR failed for periods longer than 30 years. Right now we sit at about 18 on the chart which is still pretty close to prior peaks. Obviously the mother of all bubbles occured right before this and it will be years before we know it plays out. So, there is nothing in the last 130 years even remotely close to the current bubble and it is absurd to assume that nothing worse can happen than has already happened.


Does anyone see what I am getting at in saying that the root problem with the intercst study is its reliance on datamining (and a failure to do what would be required to make datamining appropriate in this case)? Am I right that the reason for the large drops in SWR from small sequence changes pointed out by Raddr is the use of a form of "datamining" in determining the SWR?

Yes
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Post by hocus »

Thanks, raddr.

My common sense and understanding of the SWR concept have told me for a long time that the fundamental problem with the intercst study is that it is a case of datamining gone awry. But I do not have the skill with numbers that you possess, and the stuff you have put forward here has been most helpful to me in my efforts to show that the claims he makes about the study misrepresent a whole bunch of the historical data.

I'd like to try to get one more piece of feedback from you for use in a future post, if I may.

DataSnooper was quoted at one point in the Great Debate as saying what you say in your post above, that the intercst study does not examine nearly enough independent data points to claim any significant statistical validity. I found intercst's response to that claim most interesting, and was disappointed that not much was made of it in the debate.

Here is a link to his response.

http://boards.fool.com/Message.asp?mid=17775982

Intercst says here that "I claim no 'statistical confidence levels to the withdrawal rates' in the study."

OK, that sounds right. But how then can he offer a post like this one from last week--

http://boards.fool.com/Message.asp?mid=18246186

In this post, he is making a claim as to the "statistical confidence" that an investor may have in a portfolio surviving as he ages, which is something that is going to happen in the future. How can he make this claim, knowing that the study offers no basis on which to claim statistical confidence levels as to future withdrawal rates?

The question I am asking is so basic that it is probably a bit insulting to you. But I have to ask because I want to make this point in a future post and I want to be certain that I am correct before venturing forth.

Is it fair to say that the statement made in the first link above and the claim made in the second link above are in complete contradiction with each other? If the first statement is accurate (as I think it is), then the second claim is nonsense, right?

And intercst must understand what a "statistical confidence level" is or he could not have made the statement in the first link. Which means that the claim made in the second link is not the result of a misunderstanding or a mistake, but is a deliberate deception.

Is there any flaw in my thought process here? I need to know before I push this thing forward any further.
raddr
*** Veteran
Posts: 265
Joined: Mon Nov 25, 2002 3:25 am
Contact:

Post by raddr »

hocus,

You owe me one, man. :wink:I stooped low enough to sign up for a 30 day free trial at TMF so I could read the links. :oops:

The first bit of misinformation:

Using the UP-1984 Mortality Table we get the following:

A 59-year-old has a 1% chance (99th percentile) of living another 40 years to age 99.

A 54-year-old has a 5% chance (95th percentile) of living another 40 years to age 94.

A 51-year old has a 10% chance (90th percentile) of living another 40 years to age 91.

The numbers I get for mean life expectancy for a healthy 51, 54, and 59 year old are: 100, 100, and 101. IOW all have at a greater than 50% chance of living past the ages he mentions above.

Then he compunds the error further:

Combining the probabilities of the portfolio running dry and the odds that our retiree will actually live long enough to see his portfolio run dry yields the following table:

Odds of 40-YrLife Expectancy -------------40-Yr withdrawal rate---------.
4.10%(99% safe)....4.23%(95% safe)...4.57%(90% safe).

59yr old (1% odds) 99.9% 99.7% 97.9.
54yr old (5% odds) 99.8% 99.1% 96.4%.
51yr old (10% odds) 99.5% 98.6% 95.3%

Pretty amazing, huh? A 5% chance that your portfolio runs dry in 40 years, combined with a 5% chance you'll die in less than 40 years, still keeps your overall financial picture solvent more than 99% of the time.

See this thread: http://nofeeboards.com/boards/viewtopic.php?t=129

Clearly Chris (Dagrims) must have read the intercst post and didn't realize the flaw in the analysis which I point out in the following post. Intercst wrongly assumes two things. One is that the two probabilities can simply be multiplied together to get the result. This would be true if the portfolios go broke at the end of each period and not before. John's execllent work has shown that in fact there are many early failures when the retiree would have a very high probability of still being alive.

Secondly, of course, is the problem that the has the life expectancies wrong (at least for healthy people with good family histories) and he uses optimized historical SWR's for the equations. Multiplying two flawed numbers together isn't a good idea. :wink:

His findings are not only wrong but carry no statistical significance. Other than that his analysis is good. :lol:
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Post by hocus »

His findings are not only wrong but carry no statistical significance.

I understand, raddr.

But I really need an answer to a somewhat different question here, if I can press your patience just a tad farther.

Let's forget for a moment whether what intercst says is statistically valid or not. Just leave that to the side.

What I am trying to nail down here is whether, given the two links I provided in my earlier post, is there any possible way to read this in which intercst is not engaging in deliberate deception?

The things you are pointing out are errors, and all sorts of errors are made on discussion boards all the time, and it's just part of the risk you take getting information on the internet. But the Motley Fool posting rules do not permit deliberate deception, in my reading of them. They require that posting be done in "good faith."

So what I am trying to nail down is, is there any possible way that the same person could have made both statements and not be guilty of deliberate deception? To me, it seems that his acknowledgement in the first link that the study lacks statistical validity for assessing what may happen in the future rules out any legitimate purpose for making the claim in the second link, which assumes that the study does provide statistically valid information as to what may happen in the future.

I am 99 percent sure of what I am saying here. My only uncertainty is over this phrase "statistical confidence levels." With no background in statistics, I don't know the precise meaning of this phrase. I think I see what it is getting at, but I wanted to run this by an expert before going forward on the other board. Is it as clear to you as it is to me that intercst is not only wrong in what he is saying about the study, but that he is engaging in deliberate deception in at least this one claim he makes for it? The key point is not that what he is saying is wrong, but that he knows what he is saying is wrong and continues saying it regardless.

I'm sorry to press you on this, but in my mind it is an important point.
JWR1945
***** Legend
Posts: 1697
Joined: Tue Nov 26, 2002 3:59 am
Location: Crestview, Florida

Post by JWR1945 »

hocus asks:

Let's forget for a moment whether what intercst says is statistically valid or not. Just leave that to the side.

What I am trying to nail down here is whether, given the two links I provided in my earlier post, is there any possible way to read this in which intercst is not engaging in deliberate deception?

The things you are pointing out are errors, and all sorts of errors are made on discussion boards all the time, and it's just part of the risk you take getting information on the internet. But the Motley Fool posting rules do not permit deliberate deception, in my reading of them. They require that posting be done in "good faith."

My answer is Yes. In isolation his comments can be attributed to poor semantics. They do not necessarily indicate a deliberate deception.

First, internal to the Retire Early Safe Withdrawal Rate Study, he avoided making any claims about statistical confidence levels. If fact, that is a strength of the study.

External to the study he has repeated suppressed any critical thinking involving his study's applicability. He has repeatedly made the same claims even after he has been shown with clear-cut, easily understood evidence that he was wrong. Those kinds of claims cannot possibly be in good faith. He has repeatedly suppressed investigations to determine the reliability of projections, especially those related to withdrawal rates.

One could advance a plausibility argument that he is afraid of learning something new instead of deliberately lying. It is similar to staying away from your doctor because he might find something seriously wrong with you. Both approaches end up with similar results.

The second post is actually a probability estimate and not a statistical finding. You can usually make a best estimate even when you do not know its accuracy.

Do not be overly concerned about the term data mining. It is an important consideration when estimating the study's applicability and the reliability of its projections. But all such considerations are being actively suppressed. Data mining is just one item among many.

Have fun.

John R.
raddr
*** Veteran
Posts: 265
Joined: Mon Nov 25, 2002 3:25 am
Contact:

Post by raddr »

hocus,

What I am trying to nail down here is whether, given the two links I provided in my earlier post, is there any possible way to read this in which intercst is not engaging in deliberate deception?

As little respect as I have for the guy and as sure as I am that he does deliberately deceive at times he has a Clintonesque ability to get away with it more or less on technicalities. His second post, while clearly far off base, makes no claim of statistical significance although IMO he obviously ascribes way to much precision to his findings. Maybe you can call him out on this though:

With all the furor over "2% is the safe withdrawal rate", most folks have forgotten that there are many things that will actually IMPROVE your safe withdrawal rate -- and some of them are easily to calculate with a tremendous degree of accuracy.

Besides being patently wrong, by using the terminology I highlight above he clearly implies that there is great validity to his conclusions but he doesn't come right out and say something like it is "significant to p < .05" or other statistical terminology.

The fact that the second post has gotten 31 recs and has not been seriously challenged by anyone over there just shows how inbred that board has become. :lol:

One last thought, though. Scroll down the thread to this intercst post:

Let's say you want to move from 99.9% safe to 99.9999% safe (we all agree that in terms of theoretical probabilities (as opposed to historical withdrawal rates) there is no 100%.

If that meant reducing your withdrawal rate from 4% to 2% and required you to amass twice as much money, would most people even be able to retire before they reached age 86 and started "limping along in a walker in Boca Raton?"

Here he does IMO ascribe a high degree of statistically probability to his numbers, equivalent to p < .001 for his analysis at a WR of 4% and p < .00001 for a WR of 2%. No one that I'm aware has come out with anything like those precise probabilities for the future.
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Post by hocus »

In isolation his comments can be attributed to poor semantics. They do not necessarily indicate a deliberate deception.

JWR1945:

As you know, I greatly respect your viewpoint and you clearly have a far firmer grasp of the statistical concepts at issue here than I do. The reason that I am asking this question is that my next post on this matter is titled "What DataSnooper Says" and will focus on the particular deception I have laid out here. I will not be offering the post for some time, partly because I want to be 100 percent sure of its accuracy before going forward (and for other reasons too). The question of whether the contradiction in the two statements noted above can be attributed by a reasonable person to "poor semantics" or not is an important one.

I don't want to be argumentative on this. I asked for your opinion, and I need to hear it out regardless of whether it is what I expected to hear or not. But I personally do not see how the second statement could be a case of poor semantics, given what he says in the first statement. In the first statement, he reveals himself to be something of a statistics expert. He clearly understood DataSnooper's criticism and responds to it in a way that only someone familier with the statistical realities could.

Then in the second statement he uses numbers pulled from his study in a new calculation that would make sense to perform only if those numbers provide a statistically valid insight into what may happen in the future. If you think it over some more, and what I am saying here seems to make any more sense, please let me know. Otherwise, I may have to drop this idea for my next post and try something else. If you look at raddr's last post on this thread, he seems to be getting at what I am driving at in the comments he makes near the end of his post.

Here's what raddr says: "Here he does IMO ascribe a high degree of statistically probability to his numbers, equivalent to p < .001 for his analysis at a WR of 4% and p < .00001 for a WR of 2%. No one that I'm aware has come out with anything like those precise probabilities for the future." I think this is a big deal, especially given the context in which these comments were offered for the board's consideration.

These comments came after I requested revocation of posting privileges on grounds of deception and after I offered a number of references to the wording of the Motley Fool posting rules. He was on notice that deception was a problem for the board, and he did what raddr said above anyway. In my mind, this single post offers more grounds for revocation of posting privileges than would 30 posts in which he calls me dirty names. The board has been trying to come to terms with this matter, and he is engaging in deliberate deception (while possessing a good understanding of the statistical concepts at issue) on a critical point.

If you have any further observations on this one particular point of the contradiction between these two statements, I would very much like to hear them.

internal to the Retire Early Safe Withdrawal Rate Study, he avoided making any claims about statistical confidence levels. If fact, that is a strength of the study.

I agree with this statement. I do not believe that the study itself is deceptive. It is intercst's interpretations of the study that are deceptive. Intercst understands statistics well enough to appreciate the limitations of the study, and he is respected on the board as the author of a valuable study, and he abuses that trust by deliberately deceiving the board as to the study's import. This is my testimony to the REHP board community. I see this as a serious violation of the Motley Fool posting rules, serious enough to justify revocation of posting privileges when the possible consequence is that board members who fall for the deception could see their retirements go bust.

One could advance a plausibility argument that he is afraid of learning something new instead of deliberately lying.

I asked you to come up with anything you possibly could imagine, and you did so. I thank you for that. However, I don't think this explanation works when considering the contradiction brought out by examination of the statements in the two links I provided above. The statement in the first link shows that he possesses a clear appreciation of the limitations of the study. It is not me or DataSnooper or raddr saying that the study lacks statistical validity. Intercst says that. So why use numbers from the study as part of a calculation in which you purport to tell people how likely as a statistical (or probability, if you prefer) matter they are to have their retirement plans succeed as they age, which is something that happens i the future?

The second post is actually a probability estimate and not a statistical finding.

This is exactly the sort of thing I was worried someone was going to say if I went forward with the "What DataSnooper Says" post. I do not understand the distinction you are making here. It's not your fault, it is mine, but I truly want to be able to grasp it if my numbers-challenged mind is capable.

The way I see it, he is using a number from his study to make a probability claim. We know from Link One that he understands that the number he is using lacks statistical validity. Nowhere in this post does he offer the caveat that "of course this number has no statistical validity, I'm just engaging in an academic exercise here."

He is offering this claim on a board where people come to find information on how to construct retirement plans, so he knows that they may make use of the claim he offers for that purpose. He knows the danger of offering such a claim and he offers it anyway.

The question in my mind is not so much whether the claim made is in the final analysis a statistical one or a probability one. It is whether someone with the knowledge of statistics and probability that intercst has been demonstrated to possess could possibly make this claim without bad intent. You suggest that he might be trying to hide the truth even from himself, and that comment is directly relevant to what I am trying to explore here. But I do not think he can be trying to fool himself on the key question as he revealed in his response to Datasnooper that he knows the truth about the limitations of the study.

If intercst knows the truth about the study, and he stills tries to fool board members who do not possess the background needed to know the truth, I see that as a serious violation of the Motley Fool posting rules.

all such considerations are being actively suppressed. Data mining is just one item among many.

I understand. But I hope in future posts to move away from general claims that intercst has violated the TMF rules and focus on specific cases in which the case can be made with certainty to any fair-minded person. The view of many board members is that the intercst study is far from perfect, but that it is more or less right and that is good enough. My claim is that the factors intercst failed to consider cause the results to be off not by a little but by a lot, and that that is why an examination of the effects of these factors is critical.

Many people do not grasp why a few ignored factors should cause the withdrawal rate to drop from 4 percent to 2 percent or lower. The results of Raddr's analysis at the beginning of the sequencing of returns thread are counter-intuitive. You would think that a reveral of one or two years would yield a small change, not a big one.

The explanation of the mystery, I believe, is that the study relies on a form of datamining as its methodology. I believe that one of the problems of constucting a study pursuant to a datamining strategy is that, when you are wrong about something, you are not just a little wrong, but a whole bunch wrong. I acknowledge that I am a little over my head re the statistical stuff, but raddr seems to be backing me up on this point. So I am left still not 100 percent sure that I am right on the datamining issue but still not persuaded that I am on the wrong track either.
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Post by hocus »

As little respect as I have for the guy and as sure as I am that he does deliberately deceive at times he has a Clintonesque ability to get away with it more or less on technicalities.

I understand. I stopped following the Clinton matter too closely after awhile, but my recollection is that his ability to practice law was taken from him for a time. I'm not entirely sure, but that is what I seem to remember. So it seems to me that the deceptions that he engaged in had consequences, even if those consequences ended up being something less than removal from office. It seems possible to me that something similar could happen in the intercst matter. Perhaps the board community will ultimately elect not to ask Motley Fool to revoke his posting privileges, but will elect to appoint board referees to block dirty posting practices in the future. I think that something like that would be a big step forward for the board.

On this point, I feel a need to observe here that <i<it is in the interests of this board community that the REHP board be returned to health. Even if you never intend to pay the fee, you benefit from that board exploring these issues. The way this should work is that one community works through an issue, and then someone who participates in both communities offers some thoughts at Community B that result from what he learned through participation in Community A. The TMF board can serve as a spur to learning here, and it can serve as a farm system generating new posters at this board. Motley Fool spends a lot of money on advertising pulling people to that site, and the natural way for this to work is for some of those people to find their way to this place from links at the REHP board after they have used that board to get clear on the basics.

There is no reason why people interested in this subject matter should be satisfied with having a single board in existence at which the realities of safe withdrawal rates may be discussed. So long as we are willing to follow the TMF posting rules, there is no good reason why we should not benefit from discussions on this subject held at the TMF board. For that to happen, however, we need to ask for it to happen. It won't happen unless we are willing to request that intercst follow the TMF posting rules. My point here is that I do not believe that this discussion is off-topic for this board.

His second post, while clearly far off base, makes no claim of statistical significance although IMO he obviously ascribes way to much precision to his findings.

As in my response to JWR, I appreciate you saying this, as it is a direct response to the question I asked. I still don't see it this way myself, however. Perhaps he does not make a direct claim of statistical significance. But he uses a number from his study in making the probability claim he offers here. Isn't the use of a number from his study a showing that either he believes the study has statistical validity or that he is acting in bad faith? And can't we rule out the possibility that he believes the study has statistical validity from what he says in the DataSnooper post?

If you went to a doctor, and the doctor gave you a drug that the FDA has banned because it kills 50 percent of those who use it, would it be a defense to say "I never claimed that there are any statistically valid showings that the drug works, I only said that there was a 99 percent chance it would cure you." I think that's crazy. The act of citing the study for purposes of a probability calculation is an implicit endorsement of its statistical validity. If intercst did not know that the study was statistically invalid, this could be chalked up to a misunderstanding. But his comments in the DataSnooper post show that he knows the drug being offered may result in retirement death, and he goes ahead and prescribes it anyway.

The fact that the second post has gotten 31 recs and has not been seriously challenged by anyone over there just shows how inbred that board has become.

Yes and no. It shows that the regular posters have been compromised. To understand this, you have to recall how this thing played out. When I first raised questions re the study, I was not aware of the Bernstein quote. I made a good case on logic, but I did not possess data proving my point. When a lot of board members (over 80) expressed a desire to learn more about this issue, the group that likes the study panicked and went into attack mode.

A good number of regular posters adopted positions at that time that were stated in such strong terms that it became hard for them to back down when the Bernstein quote (based on data) was brought forward. They now see that their earlier statements were too strong, but it is hard for them to reverse positions. The regular posters who are supporting intercst do not possess the same bad intent that he does, in my view. They are just sort of "stuck" on this issue.

It's helpful to recall here that I knew about the limitations of the intercst study a long time before I said anything about them on the board. So I was "compromised" too on this issue for a long time. I played a part in The Great Deception during that time when I was offering posts that were pulling people to a board that I knew was offering fraudulent investment advice. My intent was not bad. I just didn't see any way that I would be permitted to participate on the board if I told the truth on this issue, and I really wanted to participate. I'm not in a position to offer much judgment of posters other than intercst who have taken the pro-intercst line on this.

Intercst is the only one acting with truly bad intent, in my assessment. Intercst has the abiility to bring the whole thing to an end with one short statement that he has exagerrated the findings of the study in his comments on it. And he understands the limitations of the study far better than most others because he wrote the study and he possesses at least a working knowledge of statistical concepts. He is the only one offering deceptions in the hope of achieving personal gain, in my view. The others are doing it out of loyalty for a friend, which is something very different, and they probably have not thought through the possible negative consequences for people who will see their retirements go bust.

Also, it is entirely unfair to judge the board as a whole by making reference to posts that appear on the board. There are far more lurkers than there are posters. The lurkers have been intimidated from commenting on this issue by the brutal personal attacks that have been made against those questioning the study. But they have shown through their use of the rec tool that they would like for intercst to permit me to have my say.

One of the key points in my "Community Rules!" post from last Monday morning is that 47 community members recently endorsed a plea that I be allowed to speak. It is only a segment of the board population that is compromised. There are a good number of honest people who congregate there in the hopes of finding information that will provide a real help with their plans for early retirement. It's easy to forget this when you view the posts going up, but it is my desire to learn from interactions with this group of 47 (or whatever the number) that makes it worth my while to pursue this.

Here he does IMO ascribe a high degree of statistically probability to his numbers, equivalent to p < .001 for his analysis at a WR of 4% and p < .00001 for a WR of 2%. No one that I'm aware has come out with anything like those precise probabilities for the future.

This is the crux of my point. I don't think that Bernstein's analysis showing a 2 percent safe withdrawal rate is the last word on SWRs any more than the intercst study is the last word. It's going to be at least 20 years before we know the exact number in all circumstances, in my assessment. But Bernstein does not claim to have delivered the last word. He just performs the analysis that it makes sense for him to perform, and presents the results to interested parties.

Intercst engages in deliberate deception as to the results of his analysis, putting board members who place trust in his statements at great financial risk. It is this practice of his that is the basis of my concern. When you put forward figures like the ones you note above, you must have a basis for doing so. And intercst's response to the DataSnooper criticism shows that he understands perfectly well that there is no basis for the claims he is making.
wanderer
*** Veteran
Posts: 363
Joined: Tue Nov 26, 2002 9:33 am
Location: anytown, usa

Post by wanderer »

here's a source on what data mining is:

http://www.investorhome.com/mining.htm

the precision to 3 significant digits in the SWR analysis and the exact allocation 74/26 are pretty indicative of "curve-fitting". raddr has gone into some of the more salient the details.

one of the statistical types can also give a brief thumbnail on hypotheses, null hypotheses, normal distributions, requirements for rejection of the null hypothesis, confidence intervals, and the like.

to get an idea of what is required to reject a null hypopthesis ata statistically significant level, here's an example: when i was an auditor and i wanted to reject the hypothesis that controls were inadequate on a population of, say, 10,000 checks, i would have to sample between 38 and 76 checks for all sorts of attributes (proper authorization, payee, amount, extensions, etc.). those items selected for the sample had to be randomly chosen and independent. more than 1 exception meant that i would usually have to expand the scope of my substantive testing significantly(rather than relying on controls over disbursments). think cost of audit going up big time.

iow, it takes a lot to overcome the null hypothesis to any significant level of confidence (one sufficient to opine on financial statements). the results of the rehp "study" aren't remotely statistically significant.

(btw, i think i mentioned elsewhere that we once tested lease controls at a property management company and found 41 exceptions in 152 items (not good).)

wanderer
regards,

wanderer

The field has eyes / the wood has ears / I will see / be silent and hear
raddr
*** Veteran
Posts: 265
Joined: Mon Nov 25, 2002 3:25 am
Contact:

Post by raddr »

one of the statistical types can also give a brief thumbnail on hypotheses, null hypotheses, normal distributions, requirements for rejection of the null hypothesis, confidence intervals, and the like.

Unfortunately, the nature of the data overlap doesn't allow for classic statistical analysis to take place. I think datasnooper commented on this once and I agree. The data points have to independent of each other in order to apply meaningful statistical techniques. With each 30 year period overlapping the next by 29 years each point is heavily dependent on the next.

That said, however, there are simulation techniques you can perform such as the ones datasnooper and I did when debating m**n r*v*rs**n. If we can define the problem(s) at hand I think I could come with something along those lines.
User avatar
ataloss
**** Heavy Hitter
Posts: 559
Joined: Mon Nov 25, 2002 3:00 am

Post by ataloss »

here's a source on what data mining is: www.investorhome.com/mining.htm


I was left with the impression that data mining was useful for hypothesis generation but out of sample testing is required for confirmation. The TMF crew data mined and curve fit and then applied statistics as if they were analyzing observational data.

Is there any meaningful way to statistically validate a data mined study (or reverse data mined studie) like intercst's swr?
Have fun.

Ataloss
JWR1945
***** Legend
Posts: 1697
Joined: Tue Nov 26, 2002 3:59 am
Location: Crestview, Florida

Post by JWR1945 »

I intend to look into this issue further. However, when hocus says:

I don't want to be argumentative on this. I asked for your opinion, and I need to hear it out regardless of whether it is what I expected to hear or not. But I personally do not see how the second statement could be a case of poor semantics, given what he says in the first statement. In the first statement, he reveals himself to be something of a statistics expert. He clearly understood DataSnooper's criticism and responds to it in a way that only someone familiar with the statistical realities could.

I don't think that intercst has anything more than a superficial understanding of statistics. He does say that he has taken some courses. But I have seen nothing in his posts to make me believes that he understands the subject.

Can anybody provide me with an example that shows that intercst understands statistics?

This may be the critical insight.

Have fun.

John R.
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Post by hocus »

The precision to 3 significant digits in the SWR analysis and the exact allocation 74/26 are pretty indicative of "curve-fitting".

Fantastic link, wanderer. Lots of food for thought there.

The responses from you, JWR, raddr, and ataloss have caused an idea for a new way to approach this to spring into my head. I will probably need some help from this board to pull this off, so I hope that at least a few here will be agreeable.

What if we were to work together to develop a "Frequently Asked Questions (FAQs) on the Proper Application of the Intercst SWR Study?" We could come up with a clear statement both of what the study does indeed show, and what it does not, with short explanations of the flawed interpretations that intercst and some others have put forward on the REHP board.

Then, each time the subject of SWRs came up over there, there would be no need to enter into a long, nasty debate going over all the same stale topics once again. Someone could just post a link to the FAQ and say, "you need to take a look at Item Six to see why it is deceptive to make this claim."

I think this would be a huge help to those who have not been able to follow the entire debate because of its length and complexity. If we could have the FAQ on the intercst study incorporated into the general REHP board FAQ, I also think it is a constructive way of dealing with the fraud problem. So long as board participants are put on notice that serious-minded people have concerns about some of the claims that intercst and others make in regard to the study, they are being given both sides of the story and then it is up to them whether to pursue the intercst recommendations or not.

I don't have a whole bunch of time to move forward on this right now. But I could participate in discussions here in an effort to develop a FAQ beginning in early January. Does this idea have appeal to any others here?

I would be particularly interested in getting the involvement of someone like Dagrims, who has taken what I would describe as a "moderate" position on the question on the other board. We're not going to persuade the intercst troops, and we don't need to persuade outselves. So I think we should be focusing on responding to any concerns or questions held by those who at this point see some merit in the arguments being made by both sides.
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Post by hocus »

I don't think that intercst has anything more than a superficial understanding of statistics. He does say that he has taken some courses. But I have seen nothing in his posts to make me believes that he understands the subject.

I can't say in a general sense whether his understanding of statistics is superficial or not. But his response to the DataSnooper criticism shows that he understands things well enough to know that the claim he put forward last week was deceptive. Doesn't it?

If he had argued that what DataSnooper said about the lack of independent data-points was no big deal, then I might go along with what you are suggesting here. But he didn't. He acknowledged that he didn't use enough independent data-points to make any statistically valid claims about the returns that stocks may provide in the future.

Given that, I believe it is fair to charge him with deliberate deception when he indeed makes such claims. I don't know exactly how much he knows, but I know that he knows enough to realize that the claims he makes are not justified by the research he did. Whether he knows anything beyond that or not is not any of my business, so far as I can see.

My concern is protecting readers of the REHP board from the consequences of placing any confidence in his invalid claims. Once that job is done, I would just as soon turn my attention to Retire Early topics of greater intrinsic appeal to me.
Dagrims
* Rookie
Posts: 8
Joined: Mon Dec 02, 2002 1:10 am
Location: Orlando Florida

Post by Dagrims »

I don't have a whole bunch of time to move forward on this right now. But I could participate in discussions here in an effort to develop a FAQ beginning in early January. Does this idea have appeal to any others here?

I would be particularly interested in getting the involvement of someone like Dagrims, who has taken what I would describe as a "moderate" position on the question on the other board. We're not going to persuade the intercst troops, and we don't need to persuade outselves. So I think we should be focusing on responding to any concerns or questions held by those who at this point see some merit in the arguments being made by both sides.


I'm not against this idea, but I will have very limited time over the next three to six months. As a CPA and MBA with an undergraduate degree in Economics, I understand how numbers and statistics can be used to present and 'prove' nearly any conclusion. That's what really intrigues me about the SWR study. Trying to get a grasp on what it really purports to say, and as importantly, what it doesn't say, is an interesting challenge. Educated and well-meaning people can take a look at the data and come up with completely different ideas of what value can be gleaned from it.

I don't believe that intercst is intentionally deceiving people with his comments. I believe he attributes certain findings to the study, and that investors can use such results to plan their future activities. He is single-minded in his beliefs of the 'best' way to retire early. Sometimes when something has worked for a person in the past, and data has been found to support the activity, it's very difficult to get that person to be as open-minded towards both criticism of the activity and towards alternative methods. That's as far as I think it goes in this case, but that's simply my opinion.

I appreciate the suggestion to involve me in helping to better understand the SWR study and provide information to prospective users of the study. I'm interested in doing so, but I honestly don't have the time right now.

Chris

p.s. Raddr, thanks for your response to my post about multiplying the chance of portfolio failure by life expectancy. Your explanations illuminated the issue for me.
User avatar
ElSupremo
Admin Board Member
Posts: 343
Joined: Thu Nov 21, 2002 12:53 pm
Location: Cincinnati, Ohio

Post by ElSupremo »

Greetings Hocus :)
I don't have a whole bunch of time to move forward on this right now. But I could participate in discussions here in an effort to develop a FAQ beginning in early January. Does this idea have appeal to any others here?


If anyone wants to put a "FIRE FAQ" together I'll be happy to add a page to the site for it. I did this for the index funds board(The FAQ can be viewed at http://home.cinci.rr.com/indexfunds/ for the time being.) and it made a world of difference. All I ask is that it is done in HTML and not PDF.
"The best things in life are FREE!"

www.nofeeboards.com
raddr
*** Veteran
Posts: 265
Joined: Mon Nov 25, 2002 3:25 am
Contact:

Post by raddr »

Hi Chris,

p.s. Raddr, thanks for your response to my post about multiplying the chance of portfolio failure by life expectancy. Your explanations illuminated the issue for me.

Hey, no problem. You'll undoubtedly have the opportunity at some point soon to educate me on something. :wink:
hocus
Moderator
Posts: 435
Joined: Mon Dec 02, 2002 12:56 am

Post by hocus »

I don't believe that intercst is intentionally deceiving people with his comments. I believe he attributes certain findings to the study, and that investors can use such results to plan their future activities. He is single-minded in his beliefs of the 'best' way to retire early. Sometimes when something has worked for a person in the past, and data has been found to support the activity, it's very difficult to get that person to be as open-minded towards both criticism of the activity and towards alternative methods. That's as far as I think it goes in this case, but that's simply my opinion.

This is a well-crafted expression of the middle-ground viewpoint. I believe that this middle-ground group is the one that will determine which way the REHP board goes on this thing.

This group has not been terribly vocal of late, for understandable reasons. But this group will appreciate a FAQ that makes a clear and concise statement of the realities of SWRs. We have far more information available to us today for use in development of such a statement than we had back on May 13. For starters, we have the research that raddr and JWR1945 have contributed here. We have lots of posts in the transcript showing the sorts of logical fallacies that people can fall into, and what they need to watch out for. We have a good sense of the concerns of the REHP community, as there have been some genuine posts mixed in with the larger number of disruptive ones. We have links to outside sources of information, such as the one that wanderer pointed us to a few posts up in this thread.

I do think we need to get some middle-ground REHP posters involved in the effort. I of course understand the limits on Dagrim's ability to participate. My thought is that, if this group is willing to proceed on something like this, I could put up a post at the REHP board letting the community there know what we plan to do and asking for any middle-ground posters with an interest to join us.

At the end of the process, we could get "sign-offs" on the FAQ. Say that we came up with something with the following endorsements: BenSolar; Ataloss; raddr; wanderer; JWR1945; Dagrims; hocus; StubbleJumper; Galagan; RJMcDonald; and perhaps a few others. It would be hard to ignore a balanced statement endorsed by 10 or 12 current or former community members.

I think that something like that would fly, and incorporation of our statement into the board FAQ would solve the problem holding up on-topic discussion at the board. We might even be able to avoid the need to deal with the deception issue if we went about this the right away. The important thing to focus on is not what has happened in the past, but what is going to happen in the future. Open the board to discussion of alternative investment options, and the core problem is solved.

I do think we would need to use the intercst statement that he made in response to DataSnooper and note some examples of things he has said as illustrations of the mistakes it is possible to make with the study if one is not careful what one is saying. But I am entirely open to using the most diplomatic possible language if doing so helps us increase the number of endorsements we can get for a statement of this kind.

I see this as a constructive way to proceed. But it doesn't make too much sense unless people here have some concern about fixing the REHP board. I made my pitch above for why I think that getting that board back to functioning status is a plus for the people who congregate here. There won't necessarily be any need for people who congregate here to post there. But we would need to provide links to some of the research done here, and I would like to be able to list the screen-names who endorse the statement.

If no one has any major objection, I'll start thinking about agenda items, and about the wording of the statement I'll need to place back at the other board. Then I'll put up a post here shortly after the turn of the year letting people know how I propose that we proceed. As far as I'm concerned, we don't need to go about this thing in any particular rush. We just do what we can when we can, and when it's done, it's done. So I don't see that there would be too much distraction from any other discussions that people were hoping to hold here.

If anyone has questions or concerns, I'll be checking the board occasionally over the course of today and tomorrow to try to respond to them. To me, it sounds like a promising approach. And, of course, the likelihood is that those of us participating would end up discovering a few things about how SWRs work that we didn't know starting out.
JWR1945
***** Legend
Posts: 1697
Joined: Tue Nov 26, 2002 3:59 am
Location: Crestview, Florida

Post by JWR1945 »

hocus suggests and I heartily endorse:

What if we were to work together to develop a "Frequently Asked Questions (FAQs) on the Proper Application of the Intercst SWR Study?" We could come up with a clear statement both of what the study does indeed show, and what it does not, with short explanations of the flawed interpretations that intercst and some others have put forward on the REHP board.

Then, each time the subject of SWRs came up over there, there would be no need to enter into a long, nasty debate going over all the same stale topics once again. Someone could just post a link to the FAQ and say, "you need to take a look at Item Six to see why it is deceptive to make this claim."

I recommend that we include other models as well. There are some questions that are best answered using exact historical sequences and others that are best answered using alternative methods. It is usually a good idea to look at the same issue from a variety of viewpoints.

Underneath any approach there are many (unstated) assumptions. It is a huge advantage to know which assumptions are critical to reaching any particular conclusion. Then you know what to look for. It helps alert you before something goes seriously wrong.

Have fun.

John R.
Post Reply