Below is a simulation of the consequence of controlling for fixed effects. In some cases this is desirable, in others not.
In the wage equation below, the wage depends on gender (b = 2), effort (b = 10), and ability (b = 3). There is no unobserved heterogeneity. A straightforward OLS estimation will return the correct b's.
However, as gender and ability are panel-invariant, the fixed effect regression does not yield any effect -- although the effect of effort is unbiased.
On the other hand, say that ability and effort are correlated, and ability is not observed, the estimated beta if effort in OLS would be biased, but not in a fixed effects estimation. The latter, however, would not allow estimating the beta for gender.
Good riddance.
clear
gene byte gender = .
gene float effort = .
gene float wage = .
gene float ability = .
gene long id = .
forvalues i = 1/1000 {
local a = runiform()
local g = runiform() > .5
set obs `=`i'*5'
replace ability = `a' if missing(id)
replace gender = `g' if missing(id)
replace id = `i' if missing(id)
}
replace effort = runiform()
replace wage = 2*gender + 10*effort + 3*ability + .5*runiform()
regress wage gender effort ability
areg wage gender effort ability, abs(id)
/*
. regress wage gender effort ability
Source | SS df MS Number of obs = 5,000
-------------+---------------------------------- F(3, 4996) > 99999.00
Model | 50282.7806 3 16760.9269 Prob > F = 0.0000
Residual | 104.339671 4,996 .020884642 R-squared = 0.9979
-------------+---------------------------------- Adj R-squared = 0.9979
Total | 50387.1202 4,999 10.0794399 Root MSE = .14452
------------------------------------------------------------------------------
wage | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
gender | 1.999089 .0040916 488.58 0.000 1.991067 2.00711
effort | 10.00101 .0070513 1418.33 0.000 9.987187 10.01483
ability | 3.004343 .0071066 422.75 0.000 2.990411 3.018275
_cons | .2498592 .0058902 42.42 0.000 .2383118 .2614065
------------------------------------------------------------------------------
.
end of do-file
. do "/var/folders/p2/2v2ckxtd2794655ypfbmfg9w0000gn/T//SD31177.000000"
. areg wage gender effort ability, abs(id)
note: gender omitted because of collinearity
note: ability omitted because of collinearity
Linear regression, absorbing indicators Number of obs = 5,000
F( 1, 3999) = 1602779.17
Prob > F = 0.0000
R-squared = 0.9983
Adj R-squared = 0.9979
Root MSE = 0.1450
------------------------------------------------------------------------------
wage | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
gender | 0 (omitted)
effort | 10.00054 .0078993 1266.01 0.000 9.985056 10.01603
ability | 0 (omitted)
_cons | 2.810143 .0044419 632.65 0.000 2.801435 2.818852
-------------+----------------------------------------------------------------
id | F(999, 3999) = 401.808 0.000 (1000 categories)
*/
Posts tonen met het label statistics. Alle posts tonen
Posts tonen met het label statistics. Alle posts tonen
vrijdag 12 mei 2017
maandag 6 februari 2017
Making sense of measuring effect size (eta-squared)
Introduction: it's impossible
We like to know whether effects are significant, or more significant than others, but generally avoid the issue of whether effects are substantial. On rare occasions, when an OLS regression suffices, a statistician may want to report R-squared values. However, do not expect much enthusiasm, as R-squared can only increase when adding additional variables even if they do not make sense. Adjusted R-squared, AIC and BIC measures correct for this. Also, R-squared has to be computed differently depending on whether or not there is a constant in the model (see Hayashi, 2000, p. 20). Worse, if your model is not OLS, you will need to define some pseudo-R-squared.Needless to say, if you want an R-squared value for any subset of explanatory factors, statisticians will hate you. This is not possible, because of collinearity, they will say. You can stepwise add or delete variables and check the change in R-squared, but this will include or remove variance that is due to other variables. Sadly!
Objection: it may be possible
The people at UCLA, in fact Philip B. Ender, thought differently. They have a Stata package called -regeffectsize- for grabs which partitions the variance explained by the separate explanatory factors, giving a good indication of how substantial effects are. The help-file is not very instructive, so I deciphered the .ado myself, and here's the report. I'm sorry this is not (yet) in LaTeX, but speed over quality for now.The structure of the program is as follows:
- From the regression, they take:
- TSS, total sum of squares in Y, which is RSS+MSS
- RSS: residual sum of squares
- MSS: the model sum of squares
- DFR: residual degrees of freedom, which is n-k (n: sample size, k: number of variables in the model)
- If you divide RSS by DFR you get the mean summed error MSE
- From the F-test (simply -test-), they take:
- The F-value
So how do they get to the (semi)partial eta-squared formula? There are three formulations, which are more or less the same and build up. The basis is the effect sum of squares, which is ESS = F*MSE.
Semipartial eta-squared
ETA2 = ESS/TSS = F*MSE/TSS
A glimpse of genius, if you ask me, to use a distribution unit (F) into the equation for the partitioning of variance. You might think that if F is really key, then our partitioning will be little more than a comparison of significance levels. The higher ESS, the more important the explanatory variable – for this it needs to be significant. In Hayashi, 2000, p. 53, I found some intuition that may be helpful. The F-test is a monotone transformation of the likelihood ratio.
Recall that MSE = RSS/DFR = RSS/(n-k) and add that F = (n-k)*(RSSr/RSSu-1) when there is only one test variable (#r = 1) to compare with the null. It is easy to see that n-k drops from the RHS and we are left with:
ETA2 = [ RSS*(RSSr-RSSu)/RSSu ] / TSS
RSSu are the unrestricted errors when beta is as calculated. Hence RSSu = RSS (the model residual sum of squares). RSSr are the restricted errors under the null (i.e. beta = 0). We can rewrite the equation to:
ETA2 = (RSSr-RSS)/TSS
Say there is only one variable in the model, then RSSr = TSS, and ETA2 = R2 = MSS/TSS. In general, the better the prediction, the lower RSS will be and the higher ETA2. Note that by definition RSS > RSSr, unless there is nothing explained, and then ETA2 = 0. What matters is the difference if we 'exclude' a variable: then RSSr rises proportionally to the importance of that variable.
Is this a good metric? I would tend to think so. The advantage of this approach in my view is that the beta's of other variables are unbiased by the omission of one variable, so MSS is unchanged and RSSr will be correct. There is also no difference in DFR in the models to compare, as it would be with stepwise deletion. To me, this looks useful as an approach to explain how much of total variance is explained by one explanatory variable, although I may overlook something. In that case, let me know.
Oh, I don't know why it would be semi-partial – the .ado uses the macro `spe2', so it's just a guess.
Percentage eta-squared
%ETA2 = 100*ETA2/R2
I think that is obvious enough. The output says 'change eta-squared', but there is no change involved.
Partial eta-squared
Here we have a little change, to not include the variance explained by other variables in the denominator. This will always be larger than the semipartial eta-squared, and will be exponentially higher when ESS approaches MSS. It could be a solution to avoid the issue of having much shared variance in MSS which is not in ESS, I'm not sure.
Part.ETA2 = ESS / (ESS+RSS)
References
Philip B. Ender – UCLA, 2015, http://www.ats.ucla.edu/stat/stata/faq/regeffectsize.htm
dinsdag 3 februari 2015
Maximum Likelihood - with examples of regressions of continuous and discrete dependent variables
The idea of Maximum Likelihood is very simple, yet powerful. A regression returns a residual for every case, telling us how far the estimation is off. In Ordinary Least Squares, we minimize the residual. In Maximum Likelihood, we ... maximize the likelihood of the prediction. It sounds like it is the same thing, and in some cases it is the same thing. But whereas OLS mathematically computes the parameters of the model, ML uses a maximization algorithm (generally Newton-Raphson) that checks multiple possibilities. These are called iterations, and when the model works well, they will 'converge'. If there is no maximum (i.e. the maximizand is non concave), or several maxima, you are out of luck.
Now let f(e) = (2.pi.sigma^2)^-(1/2) . exp(- (e^2)/(2.sigma^2)
In this case f is a normal density function or normal distribution. If the error is 0, the density is highest (around .44), so a good estimation will need to have small errors.
The likelihood function we want to maximize is the joint probability distribution of the sample. This means we compute the density of the error for every case and multiply all of them. We may just as well take the log and sum the logs of the error density. When this sum of logs is maximizes, we have the same set of coefficients b as in OLS. Note that this will be the true b only if the error term is indeed normally distributed.
y = xb
Yet in this case we cannot simply compute a residual. In the probit case, we take the cumulative probability of xb. It is obvious that this depends on a scaling parameter sigma.
The likelihood function is then P(xb) if y_obs = 1, and 1-P(xb) if y_obs = 1. Another way to express this is:
L_i = P(xb)^y_obs * (1-P(xb))^(1-y_obs).
Maximizing the product of all L_i or the sum of the logs yields good estimates of b.
Note: an alternative to probit is the logit or logistic regression. Then the function P is not the normal cumulative distribution function, but instead it is the inverse logit:
P(xb) = exp(xb) / (1+ exp(xb) )
The advantage here is that no scaling sigma has to be estimated. It used to be popular in the early days of sociology, before econometrics became dominant in the social sciences.
Continuous dependent variables
Take a simple regression like y = Xb, and suppose - as in simplified OLS - that the error term is normally distributed. Then e = y - Xb.Now let f(e) = (2.pi.sigma^2)^-(1/2) . exp(- (e^2)/(2.sigma^2)
In this case f is a normal density function or normal distribution. If the error is 0, the density is highest (around .44), so a good estimation will need to have small errors.
The likelihood function we want to maximize is the joint probability distribution of the sample. This means we compute the density of the error for every case and multiply all of them. We may just as well take the log and sum the logs of the error density. When this sum of logs is maximizes, we have the same set of coefficients b as in OLS. Note that this will be the true b only if the error term is indeed normally distributed.
Discrete dependent variables
When the outcome variable y_obs is binary (0,1), such a linear model will not work. Instead we estimate a function that lies behind the observed outcome. If that function results in a number greater than 0, we will observe a success (1) and if not, a failure (0). Hence again, we have:y = xb
Yet in this case we cannot simply compute a residual. In the probit case, we take the cumulative probability of xb. It is obvious that this depends on a scaling parameter sigma.
The likelihood function is then P(xb) if y_obs = 1, and 1-P(xb) if y_obs = 1. Another way to express this is:
L_i = P(xb)^y_obs * (1-P(xb))^(1-y_obs).
Maximizing the product of all L_i or the sum of the logs yields good estimates of b.
Note: an alternative to probit is the logit or logistic regression. Then the function P is not the normal cumulative distribution function, but instead it is the inverse logit:
P(xb) = exp(xb) / (1+ exp(xb) )
The advantage here is that no scaling sigma has to be estimated. It used to be popular in the early days of sociology, before econometrics became dominant in the social sciences.
Links
The last link has a nice picture comparing the cumulative distribution function and the inverse logit (or logistic) function. The latter has 'fatter' tails, even though the difference is negligible in practice.
- http://en.wikipedia.org/wiki/Logit
- http://en.wikipedia.org/wiki/Probit
- http://en.wikipedia.org/wiki/Logistic_regression#Bayesian_logistic_regression
donderdag 20 maart 2014
Type I and Type II errors
Type I: Ho true -- accept Ha / reject Ho -- try to limit this through minimizing the standard errors on your estimates, estimates should be efficient
Type II: Ha true -- reject Ha / accept Ho -- harder to know, estimates should be more consistent
http://en.wikipedia.org/wiki/Type_I_and_type_II_errors
Type II: Ha true -- reject Ha / accept Ho -- harder to know, estimates should be more consistent
http://en.wikipedia.org/wiki/Type_I_and_type_II_errors
donderdag 29 juli 2010
What stats package to use?
Introduction
Boys like their toys, and this is not different with statistical packages. It's a perpetual and heated debate and when you've landed at some point and think your workflow is good, technology passes you and sets you back.
Here's an old discussion that I first consulted, but below I make my own considerations. In short: I'd use Python for big data, Stata for analysis, and R if I have to (e.g. for some graphs). Everything else I would ditch.
Stata
Stata is my program of choice. It is quite expensive, but mind you that for a couple of hundred euros, depending on the flavour, you'll not only get an easy and robust statistical software package, but also fast support, a good community, useful user commands, and a great documentation source. Fun fact: all documentation is read by the wife of the founder, who's not a statistician but perhaps even smarter. If she doesn't understand what the statisticians are saying, it goes back to the drawing board.
The bad things: forget about ever copy-pasting anything. You'll also need to have a lot of memory on your computer, as Stata loads the whole file and just one at a time (although you can 'preserve' a file temporarily to work on something else in between).
Python
Python is the next language I will learn. I have used chunks without understanding what I was doing, but I like the sound of the language, and it's the logical step-up after Stata, it seems. Many people are using it and so will I.
R
I don't like R. There is a thorough discussion here, circling around leaving Stata for R, but ending up in concluding what I conclude about: R is a mixture of a coding language like Python and a statistical language like Stata, but because it is open source the support is unsure, the community tends to be geeky and unfriendly, the documentation is poor, and the language consistency - even if the structure is good because it is a programming language - is bad. Some commands have their own inner programming language and that is plain bad.
The good things: it is free and R Studio is a great user interface. It has good graphic capabilities, and
Mplus
I don't know Mplus. Colleagues use it when there are issues with missing values, and the programmers are said to be the best statisticians in the world. So it must be good, but I don't use it.
SAS
This is old software. It is too complicated, and while it can do a lot through obscure options, it is not flexible enough to do what you want.
SPSS
This is bad software. It is a scandal that some universities still teach this.
Some R resources
Apparently the single best manual for R: https://r4ds.had.co.nz.- http://www.burns-stat.com/ - een man met een missie
- http://www.burns-stat.com/pages/Present/Excel_to_R_annotated.pdf - bewijst eigenlijk dat R graphics niet deugen
- http://www.burns-stat.com/pages/Tutor/R_relative_statpack.pdf
- http://www.burns-stat.com/pages/Tutor/spreadsheet_addiction.html
Abonneren op:
Posts (Atom)