Composition bias in small samples with skewed distributions

If regressing on wages that are not normally distributed (right-skewed, in general), and comparing groups with very different relative frequencies (like men and women in the STEM-field), added a censoring (e.g. wage floors), then a positive bias for the larger group is likely for small sample sizes.

I demonstrate this below using a simple Monte Carlo simulation.

In a tobit-like fashion, one would like to control for the selection chance, but this is given by gender which is already in the model.

It would be interesting and easy to make this simulation based on actual means, sd, skewness in the (male) population.


clear
cap erase mc.dta
foreach n of numlist 60 80 100 120 140 200 300 500 800 1000 2000 {
forvalues r = 1/500 {
clear
*local n = 1000
set obs `n'
local m = 2000 // mean
local s = 200 // standard deviation (spread)
local f = 0.1 // feminisation
local a = 3 // positive: right skewed (it is possible to compute the skewness metric based on a)
gene g = 1-(runiform()<`f')
gene rn = runiform()
gene d = 2*invnormal(rn)*normal(rn*`a')
gene w = max(1400,`m'+d*`s')
*twoway kdensity w
collapse w, by(g)
gene nsize = `n'
gene c = 1
reshape wide w, i(c) j(g)
list 
cap gene gpg = w1/w0
cap gene gpg = .
cap append using mc.dta
cap save mc.dta, replace
}
}
replace gpg = round(100*gpg-100,.01)
tabstat gpg, by(nsize)
*scatter gpg nsize

exit

Stata user command -norm-

Normalizing a variable is not so hard to do. The formula is:

normvar = (oldvar - minimum) / (maximum - minimum)

Hence you get a long way with the -summarize- command and r(min) and r(max).

However, in some cases you want to normalize to a wider range than appears in the data, e.g. rescale a variable that was coded 1-5 to 0-100, despite there not being units with either value 1 or 5. Maybe such cases exist in another pool, so you want to transform the variable in the same way. Again this is easy to do, but it may get tedious and make your syntax look ugly and confusing.

This is where the -norm- command may be of use.

Syntax

norm var [, replace generate(newvarname) scale(integer) maximum(integer) minimum(integer) [no]label]

The default scale is 1, hence range: 0-1.
By default value 0 gets the label 'Minimum' and the maximum gets a label too. You can wipe the label using the option nolabel.

Examples

norm q89, scale(100) replace
norm q93, scale(20) max(5)

Installation

Download the files and put them in your ado-folder (plus/personal) or refer to the containing folder by adopath + folderpath.

Coding sectors based on NACE rev 1.1 and NACE rev. 2

Below is a broad categorisation of the NACE activities in four groups:

1. Primary sector and construction
2. Manufacturing and energy (including water, waste and recycling)
3. Services and entertainment
4. Public sector, education and health (including libraries)

The syntax below can be used in Stata straight away:

Based on NACE rev. 1.1

gene s_industryR1 = .
replace s_industryR1 = 1 if y10_nace_rev1 >= 1 & y10_nace_rev1 <= 14 // primary sector + construction
replace s_industryR1 = 1 if y10_nace_rev1 == 45 // primary sector + construction
replace s_industryR1 = 2 if y10_nace_rev1 >= 15 & y10_nace_rev1 <= 41 // manufacturing + energy ressources
replace s_industryR1 = 2 if y10_nace_rev1 == 90 // manufacturing + energy ressources
replace s_industryR1 = 3 if y10_nace_rev1 >= 50 & y10_nace_rev1 <= 74 // services + entertainment
replace s_industryR1 = 3 if y10_nace_rev1 >= 91 & y10_nace_rev1 <= 97 // services + entertainment
replace s_industryR1 = 4 if y10_nace_rev1 >= 75 & y10_nace_rev1 <= 85 // PEH
replace s_industryR1 = 4 if y10_nace_rev1 == 99 // PEH

Based on NACE rev. 2

gene s_industryR2 = .
replace s_industryR2 = 1 if y10_nace_rev2 >= 1 & y10_nace_rev2 <= 9 // primary sector + construction
replace s_industryR2 = 1 if y10_nace_rev2 >= 41 & y10_nace_rev2 <= 43 // primary sector + construction
replace s_industryR2 = 2 if y10_nace_rev2 >= 10 & y10_nace_rev2 <= 39 // manufacturing + energy ressources
replace s_industryR2 = 3 if y10_nace_rev2 >= 45 & y10_nace_rev2 <= 82 // services + entertainment
replace s_industryR2 = 3 if y10_nace_rev2 >= 92 & y10_nace_rev2 <= 98 // services + entertainment
replace s_industryR2 = 3 if y10_nace_rev2 == 90 // services + entertainment
replace s_industryR2 = 4 if y10_nace_rev2 >= 84 & y10_nace_rev2 <= 88 // PEH
replace s_industryR2 = 4 if y10_nace_rev2 == 91 // PEH
replace s_industryR2 = 4 if y10_nace_rev2 == 99 // PEH

Info on the box: http://www.quackit.com/html/codes/horizontal_scroll.cfm

Stata regression prefixes

Lagged values
l.var = l1.var = variable at t-1
l2.var = variable at t-2
l(1/3).var = l1.var l2.var l3.var
etc.

Leading values
f.var = f1.var = variable at t+1
f2.var = variable at t+2

Differences
d.var = var[t] - var[t-1]
d2.var = ( var[t] - var[t-1] ) - (var[t-1] - var[t-2])

Seasonal differences
s.var = var[t] - var[t-1]
s4.var = var[t] - var[t-4]

Macro's in Stata

Ik heb nog steeds geen idee wat het verschil is tussen global en local macro's in de praktijk. Mocht ik tijd hebben, dan vind ik het wellicht hier:

http://data.princeton.edu/stata/programming.aspx

Dummy codering: macro voor SPSS

Om het gesukkel met REGRESSION wat te verlichten, hier een macro voor simple dummy coding van de variabele 'orgvar' met 3 levels. Je zal nu drie variabelen krijgen:
orgvar1 = 1 als orgvar = 1, anders 0
orgvar2 = 1 als orgvar = 2, anders 0
orgvar3 = 1 als orgvar = 3, anders 0

DEFINE macrodummycoding
(orig=!CHAREND('/')
/values=!CHAREND('/')).
!DO !x !IN (!values)
DO IF !orig = !x.
COMPUTE !CONCAT(!orig,!x) = 1.
ELSE IF missing(!orig) = 1.
COMPUTE !CONCAT(!orig,!x) = $SYSMIS.
ELSE.
COMPUTE !CONCAT(!orig,!x) = 0.
END IF.
!DOEND
EXECUTE.
!ENDDEFINE.


macrodummycoding
orig = orgvar /
values = 1 2 3/.

Voor wie het wil geloven. Het kan sneller en gemakkelijker met Stata.