401(k) dataset again

Nuisance parameters

$\psi_a$ and $\psi_b$ of each scores

PL
ATE
ATT
PL-IV
LATE

A generic function for Double ML with linear score

Results

ATT vs. ATE

LATE vs. ATE

How large is the difference between ATT and ATE in %?

Goals:

Hand-code partially linear (IV) and AIPW for ATE/ATT/LATE with the generic Double ML recipe
Test $H_0: ATT = ATE$ , $H_0: LATE = ATE$ and $H_0: (ATT - ATE) / ATE \times 100 = 0$ using the chain rule of influence functions

401(k) dataset again

We again use the data of the hdm package. The data was used in Chernozhukov and Hansen (2004). Their paper investigates the effect of participation in the employer-sponsored 401(k) retirement savings plan (p401) on net assets (net_tfa). Since then, the data was used to showcase many new methods. It is not the most comprehensive dataset with basically ten covariates/regressors/predictors:

age: age
db: defined benefit pension
educ: education (in years)
fsize: family size
hown: home owner
inc: income (in US $)
male: male
marr: married
pira: participation in individual retirement account (IRA)
twoearn: two earners

However, it is publicly available and the few controls ensure that the programs run not as long as with datasets that you hope to have for your applications.

library(hdm)
library(causalDML)
library(grf)
library(tidyverse)

set.seed(1234) # for replicability

# Get data
data(pension)
# Outcome
Y = pension$net_tfa
# Treatment
W = pension$p401
# Treatment
Z = pension$e401
# Create main effects matrix
X = model.matrix(~ 0 + age + db + educ + fsize + hown + inc + male + marr + pira + twoearn, data = pension)

We want to estimate the effect of 401(k) participation on net assets. We leverage two identification strategies:

Based on unconfoundedness as before
using eligibility for 401(k) as instrument for actual participation

and use estimators that assume constant effects (partially linear) and allow for heterogeneous effects (ATE, ATT, LATE).

In total we want to implement five estimators:

partially linear Double ML (PL)
AIPW-ATE Double ML (ATE)
AIPW-ATE Double ML (ATT)
partially linear IV Double ML (PL-IV)
AIPW-LATE Double ML (LATE)

Nuisance parameters

We need a bunch of nuisance parameters to do so. The ones we know already are

$e(X)=E[W|X]$ coded as exhat
$m(X)=E[Y|X]$ coded as mxhat
$m(0,X)=E[Y|W=0,X]$ coded as mwhat0
$m(1,X)=E[Y|W=1,X]$ coded as mwhat1

Additionally, we need

$h(X)=E[Z|X]$ to be coded as hhat
$m_z(0,X)=E[Y|Z=0,X]$ to be coded as mzhat0
$m_z(1,X)=E[Y|Z=1,X]$ to be coded as mzhat1
$e(0,X)=E[W|Z=0,X]$ to be coded as ezhat0
$e(1,X)=E[W|Z=1,X]$ to be coded as ezhat1

n = length(Y)
nfolds = 5
fold = sample(1:nfolds,n,replace=T)

exhat = mxhat = mwhat0 = mwhat1 = hhat = mzhat0 = mzhat1 = ezhat0 = ezhat1 = rep(NA,n)
  
for (i in 1:nfolds){
  rfe = regression_forest(X[fold != i,],W[fold != i])
  exhat[fold == i] = predict(rfe, X[fold == i,])$predictions
  
  rfm = regression_forest(X[fold != i,],Y[fold != i])
  mxhat[fold == i] = predict(rfm, X[fold == i,])$predictions
  
  rfm0 = regression_forest(X[fold != i & W==0,],Y[fold != i & W==0])
  mwhat0[fold == i] = predict(rfm0, X[fold == i,])$predictions

  rfm1 = regression_forest(X[fold != i & W==1,],Y[fold != i & W==1])
  mwhat1[fold == i] = predict(rfm1, X[fold == i,])$predictions

  rfh = regression_forest(X[fold != i,],Z[fold != i])
  hhat[fold == i] = predict(rfh, X[fold == i,])$predictions

  rfmz0 = regression_forest(X[fold != i & Z==0,],Y[fold != i & Z==0])
  mzhat0[fold == i] = predict(rfmz0, X[fold == i,])$predictions

  rfmz1 = regression_forest(X[fold != i & Z==1,],Y[fold != i & Z==1])
  mzhat1[fold == i] = predict(rfmz1, X[fold == i,])$predictions

  rfez0 = regression_forest(X[fold != i & Z==0,],W[fold != i & Z==0])
  ezhat0[fold == i] = predict(rfez0, X[fold == i,])$predictions

  rfez1 = regression_forest(X[fold != i & Z==1,],W[fold != i & Z==1])
  ezhat1[fold == i] = predict(rfez1, X[fold == i,])$predictions
}

and finally the easy $e=E[W]$ coded as ehat

ehat = mean(W)

$\psi_a$ and $\psi_b$ of each scores

Now we code up the two components of the score, that will help us later to get the point estimate and the standard error using the same function.

PL

The empirical version of the score looks like this: $\psi^{PL}(O;\hat{\tau},\hat{\eta}) = \hat{\tau} \underbrace{(-1)(W - \hat{e}(X))^2}_{\psi_a^{PL}} + \underbrace{ (Y - \hat{m}(X)) (W - \hat{e}(X))}_{\psi_b^{PL}}$ To follow the recipe, we need two components:

$\psi_a^{PL}$ to be coded as pa_pl
$\psi_b^{PL}$ to be coded as pb_pl

pa_pl = -(W - exhat)^2
pb_pl = (Y - mxhat) * (W - exhat)
-sum(pb_pl) / sum(pa_pl)

[1] 13972.06

ATE

The empirical version of the score looks like this: $\psi^{ATE}(O;\hat{\tau}_{ATE},\hat{\eta})= \hat{\tau}_{ATE} \underbrace{(-1)}_{\psi_a^{ATE}} + \underbrace{\hat{m}(1,X) -\hat{ m}(0,X) + \frac{W (Y - \hat{m}(1,X) )}{\hat{e}(X)} - \frac{(1-W) (Y - \hat{m}(0,X) )}{1-\hat{e}(X)} }_{\psi_b^{ATE}}$ To follow the recipe, we need two components:

$\psi_a^{ATE}$ to be coded as pa_ate
$\psi_b^{ATE}$ to be coded as pb_ate

pa_ate = rep(-1,length(Y))
pb_ate = mwhat1 - mwhat0 + W * (Y - mwhat1) / ehat - (1 - W) * (Y - mwhat0) / (1-ehat)
-sum(pb_ate) / sum(pa_ate)

[1] 12356.5

ATT

The empirical version of the score looks like this: $\psi^{ATT}(O;\hat{\tau}_{ATT},\hat{\eta})= \hat{\tau}_{ATT} \underbrace{(-1)\frac{W}{\hat{e}}}_{\psi_a^{ATT}} + \underbrace{\dfrac{W}{\hat{e}} (Y - \hat{m}(0,X)) - \dfrac{ (1-W) \hat{e}(X) }{\hat{e} (1-\hat{e}(X))} (Y - \hat{m}(0,X)) }_{\psi_b^{ATT}}$ To follow the recipe, we need two components:

$\psi_a^{ATT}$ to be coded as pa_att
$\psi_b^{ATT}$ to be coded as pb_att

pa_att = -W / ehat
pb_att = W * (Y - mwhat0) / ehat - ( (1 - W) * exhat ) * (Y - mwhat0) / (ehat * (1 - ehat))
-sum(pb_att) / sum(pa_att)

[1] 14796

PL-IV

The empirical version of the score looks like this: $\psi^{IV}(O;\hat{\tau}_{IV},\hat{\eta})= \hat{\tau}_{IV} \underbrace{(-1) (W - \hat{e}(X)) (Z - \hat{h}(X))}_{\psi_a^{IV}} + \underbrace{(Y - \hat{m}(X)) (Z - \hat{h}(X))}_{\psi_b^{IV}}$ To follow the recipe, we need two components:

$\psi_a^{IV}$ to be coded as pa_iv
$\psi_b^{IV}$ to be coded as pb_iv

pa_iv = -(W - exhat) * (Z - hhat)
pb_iv = (Y - mxhat) * (Z - hhat) 
-sum(pb_iv) / sum(pa_iv)

[1] 12892.28

LATE

The empirical version of the score looks like this: $\begin{align} \psi^{LATE}(O;\hat{\tau}_{LATE},\hat{\eta}) & = \hat{\tau}_{LATE} \underbrace{ (-1) \Bigg[ \hat{e}(1,X) - \hat{e}(0,X) + \dfrac{Z (W - \hat{e}(1,X))}{\hat{h}(X)} - \dfrac{(1-Z) (W - \hat{h}(0,X))}{1-\hat{h}(X)} \Bigg] }_{\psi_a^{LATE}} \\ & \quad + \underbrace{\hat{m}_z(1,X) - \hat{m}_z(0,X) + \dfrac{Z (Y - \hat{m}_z(1,X))}{\hat{h}(X)} - \dfrac{(1-Z) (Y - \hat{m}_z(0,X))}{1-\hat{h}(X)}}_{\psi_b^{LATE}} \end{align}$ To follow the recipe, we need two components:

$\psi_a^{LATE}$ to be coded as pa_late
$\psi_b^{LATE}$ to be coded as pb_late

pa_late = -( ezhat1 - ezhat0 + Z * (W - ezhat1) / hhat - (1 - Z) * (W - ezhat0) / (1-hhat) )
pb_late = mzhat1 - mzhat0 + Z * (Y - mzhat1) / hhat - (1 - Z) * (Y - mzhat0) / (1-hhat)
-sum(pb_late) / sum(pa_late)

[1] 11194.11

A generic function for Double ML with linear score

Now we write a function that takes any $\psi_a$ and $\psi_b$ as inputs and outputs

point estimate
standard error
t-value
p-value

The function implements the following steps:

Calculate point estimate as $ = - $
$\hat{\theta}$ can then be used to complete the empirical score as $\psi(O;\hat{\theta},\hat{\eta}) = \hat{\theta} \psi_a(O_i;\hat{\eta}_i) + \psi_b(O_i;\hat{\eta}_i)$
Create influence function $\Psi(O;\hat{\theta},\hat{\eta}) = - \frac{\psi(O;\hat{\theta},\hat{\eta})}{N^{-1} \sum_i \psi_a(O_i;\hat{\eta}_i)}$
Estimate the variance as $\hat{\sigma}^2 = Var(\Psi(O;\hat{\theta},\hat{\eta}))$ or $\hat{\sigma}^2 = \frac{N^{-1} \sum_i \psi(O_i;\hat{\theta},\hat{\eta}_i)^2}{[N^{-1} \sum_i \psi_a(O_i;\hat{\eta}_i)]^2}$
Get the standard error as $se(\hat{\theta}) = \sqrt{\frac{\hat{\sigma}^2}{N}}$

DML_inference = function(psi_a,psi_b) {
  N = length(psi_a)
  theta = -sum(psi_b) / sum(psi_a)
  psi = theta * psi_a + psi_b
  Psi = - psi / mean(psi_a)
  sigma2 = var(Psi)
  # sigma2 = mean(psi^2) / mean(psi_a)^2
  se = sqrt(sigma2 / N)
  t = theta / se
  p = 2 * pt(abs(t),N,lower = FALSE)
  result = c(theta,se,t,p)
  return(result)
}

Results

results = matrix(NA,5,4)
rownames(results) = c("PL","ATE","ATT","PL-IV","LATE")
colnames(results) = c("Effect","S.E.","t","p")
results[1,] = DML_inference(pa_pl,pb_pl)
results[2,] = DML_inference(pa_ate,pb_ate)
results[3,] = DML_inference(pa_att,pb_att)
results[4,] = DML_inference(pa_iv,pb_iv)
results[5,] = DML_inference(pa_late,pb_late)
printCoefmat(results,has.Pvalue = TRUE)

       Effect    S.E.      t         p    
PL    13972.1  1538.8 9.0800 < 2.2e-16 ***
ATE   12356.5  1469.2 8.4102 < 2.2e-16 ***
ATT   14796.0  1647.5 8.9808 < 2.2e-16 ***
PL-IV 12892.3  1923.4 6.7029 2.154e-11 ***
LATE  11194.1  1699.1 6.5881 4.681e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

data.frame(thetas = results[,1],ses = results[,2],
                Estimator = rownames(results),
                cil = results[,1] - 1.96*results[,2],
                ciu = results[,1] + 1.96*results[,2])  %>% 
  ggplot(aes(x=Estimator,y=thetas,ymin=cil,ymax=ciu)) + geom_point(size=2.5) + geom_errorbar(width=0.15)  +
  geom_hline(yintercept=0)

ATT vs. ATE

Assume that we want to test whether $ATT=ATE$ .

For this purpose, we create the new parameter $\Delta_{ATT} = \tau_{ATT} - \tau_{ATE}$ and want to test $H_0:\Delta(\tau_{ATT},\tau_{ATE}) = 0$

The influence function of this new parameter can derived like this: $\begin{align*} \Psi_\Delta & = \overbrace{\frac{\delta \Delta}{\delta \tau_{ATT}}}^{=1} \Psi_{\tau_{ATT}} + \overbrace{\frac{\delta \Delta}{\delta \tau_{ATE}}}^{=-1} \Psi_{\tau_{ATE}} \\ & = \Psi_{\tau_{ATT}} - \Psi_{\tau_{ATE}} \end{align*}$

Define for convenience and later use a Psi_maker() function that creates the influence function for generic $\psi_a$ and $\psi_b$ :

Psi_maker = function(psi_a,psi_b) {
  theta = -sum(psi_b) / sum(psi_a)
  psi = theta * psi_a + psi_b
  Psi = - psi / mean(psi_a)
  return(Psi)
}

Now this can be used to test $H_0:\Delta(\tau_{ATT},\tau_{ATE}) = 0$ :

# Calculate parameters
att = - sum(pb_att) / sum(pa_att)
ate = - sum(pb_ate) / sum(pa_ate)
Delta_att = att - ate
# Create influence function for new parameter
Psi_Delta_att = Psi_maker(pa_att,pb_att) - Psi_maker(pa_ate,pb_ate)
# Calculate standard errors, t and pvalues
se_Delta_att = sqrt(var(Psi_Delta_att)/length(Psi_Delta_att))
t_Delta_att = Delta_att / se_Delta_att
p_Delta_att = 2 * pt(abs(t_Delta_att),length(Psi_Delta_att),lower = FALSE)
# Print results
result = matrix(c(Delta_att,se_Delta_att,t_Delta_att,p_Delta_att),nrow = 1)
rownames(result) = c("ATT-ATE")
colnames(result) = c("Delta","S.E.","t","p")
printCoefmat(result,has.Pvalue = TRUE)

          Delta    S.E.      t         p    
ATT-ATE 2439.50  516.31 4.7249 2.334e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

We observe that the different between ATT and ATE is highly significant. This may come as a surprise given that the confidence intervals overlap in the graph above. However, such eyeballing does not work for correlated estimators and the ATT and ATE estimator are clearly correlated.

LATE vs. ATE

Similarly, we can investigate $H_0:\Delta_{LATE} = \tau_{LATE} - \tau_{ATE} = 0$ . (0.5P)

The derivation works like before:

$\begin{align} \Psi_{\Delta_{LATE}} &= \underbrace{\frac{\partial \Delta_{LATE}}{\partial \tau_{LATE}}}_{=1} \Psi_{LATE} + \underbrace{\frac{\partial \Delta_{LATE}}{\partial \tau_{ATE}}}_{=-1} \Psi_{ATE} \\ \Psi_{\Delta_{LATE}} &=\Psi_{LATE} - \Psi_{ATE} \end{align}$

and also the inference follows the same recipe:

# CalcuLATE
late = -sum(pb_late)/sum(pa_late) 
# New target parameter
Delta_late = late - ate 
# Create influence function for new parameter
Psi_Delta_late = Psi_maker(pa_late,pb_late) - Psi_maker(pa_ate,pb_ate)
# Print results
se_Delta_late = sqrt(var(Psi_Delta_late)/length(Psi_Delta_late))
t_Delta_late = Delta_late/se_Delta_late
p_Delta_late = 2 * pt(abs(t_Delta_late),length(Psi_Delta_late),lower = FALSE) # get a p-value (at what level would be not reject?)

results = matrix(c(Delta_late,se_Delta_late,t_Delta_late,p_Delta_late),nrow = 1)
rownames(results) = c("LATE-ATE")
colnames(results) = c("Delta","S.E.","t","p")
printCoefmat(results,has.Pvalue = TRUE)

           Delta    S.E.      t      p
LATE-ATE -1162.4  1247.2 -0.932 0.3514

However, the difference between LATE and ATE is not statistically significant.

How large is the difference between ATT and ATE in %?

Finally, let’s look at a more complicated new parameter $\Delta\% = \frac{\tau_{ATT} - \tau_{ATE}}{\tau_{ATE}}\times 100$ .

Following the recipe, we derive the new influence function:

$\begin{align} \Psi_{\Delta\%} &= \frac{\partial \Delta\%}{\partial \tau_{ATT}} \Psi_{ATT} + \frac{\partial \Delta\%}{\partial \tau_{ATE}} \Psi_{ATE} \\ \Psi_{\Delta\%} &=100 \cdot \frac{1}{\tau_{ATE}} \Psi_{ATT} -100 \cdot \frac{\tau_{ATT}}{\tau_{ATE}^2} \Psi_{ATE} \end{align}$

$\Psi_{ATT}$ and $\Psi_{ATE}$ can be obtained using Psi_maker. $\tau_{ATE}$ and $\tau_{ATT}$ can be replaced by their estimates:

# New parameter
Delta_pc = (att-ate)/ate*100 
# New IF
Psi_Delta_pc = 100 / ate * Psi_maker(pa_att, pb_att) - 100 * att / (ate^2) * Psi_maker(pa_ate, pb_ate)
# Results
se_Delta_pc = sqrt(var(Psi_Delta_pc)/length(Psi_Delta_pc))
t_Delta_pc = Delta_pc/se_Delta_pc
p_Delta_pc = 2 * pt(abs(t_Delta_pc),length(Psi_Delta_pc),lower = FALSE)
results = matrix(c(Delta_pc,se_Delta_pc,t_Delta_pc,p_Delta_pc),nrow = 1)
rownames(results) = c("Delta%")
colnames(results) = c("Delta","S.E.","t","p")
printCoefmat(results,has.Pvalue = TRUE)

         Delta    S.E.      t         p    
Delta% 19.7426  4.3854 4.5019 6.812e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Not surprisingly the percentage change is significant, like the plain level difference between ATT and ATE. However, the t-values are not identical, which illustrates that we explicitly calculate the inference for such new parameters. And influence functions provide a neat tool to do so.

LS0tDQp0aXRsZTogIkNhdXNhbCBNTDogRG91YmxlIE1MIGFzIGdlbmVyaWMgcmVjaXBlIg0Kc3VidGl0bGU6ICJBcHBsaWNhdGlvbiBub3RlYm9vayINCmF1dGhvcjogIk1pY2hhZWwgS25hdXMiDQpkYXRlOiAiYHIgZm9ybWF0KFN5cy50aW1lKCksICclbS8leScpYCINCm91dHB1dDogDQogIGh0bWxfbm90ZWJvb2s6DQogICAgdG9jOiB0cnVlDQogICAgdG9jX2Zsb2F0OiB0cnVlDQogICAgY29kZV9mb2xkaW5nOiBzaG93DQotLS0NCg0KDQo8YnI+DQoNCkdvYWxzOg0KDQotIEhhbmQtY29kZSBwYXJ0aWFsbHkgbGluZWFyIChJVikgYW5kIEFJUFcgZm9yIEFURS9BVFQvTEFURSB3aXRoIHRoZSBnZW5lcmljIERvdWJsZSBNTCByZWNpcGUNCg0KLSBUZXN0ICRIXzA6IEFUVCA9IEFURSQsICRIXzA6IExBVEUgPSBBVEUkIGFuZCAkSF8wOiAoQVRUIC0gQVRFKSAvIEFURSBcdGltZXMgMTAwID0gMCQgdXNpbmcgdGhlIGNoYWluIHJ1bGUgb2YgaW5mbHVlbmNlIGZ1bmN0aW9ucw0KDQo8YnI+DQoNCiMgNDAxKGspIGRhdGFzZXQgYWdhaW4NCg0KV2UgYWdhaW4gdXNlIHRoZSBkYXRhIG9mIHRoZSBgaGRtYCBwYWNrYWdlLiBUaGUgZGF0YSB3YXMgdXNlZCBpbiBbQ2hlcm5vemh1a292IGFuZCBIYW5zZW4gKDIwMDQpXShodHRwczovL2RpcmVjdC5taXQuZWR1L3Jlc3QvYXJ0aWNsZS84Ni8zLzczNS81NzU4Ni9UaGUtRWZmZWN0cy1vZi00MDEtSy1QYXJ0aWNpcGF0aW9uLW9uLXRoZS1XZWFsdGgpLiBUaGVpciBwYXBlciBpbnZlc3RpZ2F0ZXMgdGhlIGVmZmVjdCBvZiBwYXJ0aWNpcGF0aW9uIGluIHRoZSBlbXBsb3llci1zcG9uc29yZWQgNDAxKGspIHJldGlyZW1lbnQgc2F2aW5ncyBwbGFuICgqcDQwMSopIG9uIG5ldCBhc3NldHMgKCpuZXRfdGZhKikuIFNpbmNlIHRoZW4sIHRoZSBkYXRhIHdhcyB1c2VkIHRvIHNob3djYXNlIG1hbnkgbmV3IG1ldGhvZHMuIEl0IGlzIG5vdCB0aGUgbW9zdCBjb21wcmVoZW5zaXZlIGRhdGFzZXQgd2l0aCBiYXNpY2FsbHkgdGVuIGNvdmFyaWF0ZXMvcmVncmVzc29ycy9wcmVkaWN0b3JzOg0KDQotICphZ2UqOiBhZ2UNCg0KLSAqZGIqOiBkZWZpbmVkIGJlbmVmaXQgcGVuc2lvbg0KDQotICplZHVjKjogZWR1Y2F0aW9uIChpbiB5ZWFycykNCg0KLSAqZnNpemUqOiBmYW1pbHkgc2l6ZQ0KDQotICpob3duKjogaG9tZSBvd25lcg0KDQotICppbmMqOiBpbmNvbWUgKGluIFVTICQpDQoNCi0gKm1hbGUqOiBtYWxlDQoNCi0gKm1hcnIqOiBtYXJyaWVkDQoNCi0gKnBpcmEqOiBwYXJ0aWNpcGF0aW9uIGluIGluZGl2aWR1YWwgcmV0aXJlbWVudCBhY2NvdW50IChJUkEpDQoNCi0gKnR3b2Vhcm4qOiB0d28gZWFybmVycw0KDQpIb3dldmVyLCBpdCBpcyBwdWJsaWNseSBhdmFpbGFibGUgYW5kIHRoZSBmZXcgY29udHJvbHMgZW5zdXJlIHRoYXQgdGhlIHByb2dyYW1zIHJ1biBub3QgYXMgbG9uZyBhcyB3aXRoIGRhdGFzZXRzIHRoYXQgeW91IGhvcGUgdG8gaGF2ZSBmb3IgeW91ciBhcHBsaWNhdGlvbnMuDQoNCmBgYHtyLCB3YXJuaW5nPUYsbWVzc2FnZT1GfQ0KbGlicmFyeShoZG0pDQpsaWJyYXJ5KGNhdXNhbERNTCkNCmxpYnJhcnkoZ3JmKQ0KbGlicmFyeSh0aWR5dmVyc2UpDQoNCnNldC5zZWVkKDEyMzQpICMgZm9yIHJlcGxpY2FiaWxpdHkNCg0KIyBHZXQgZGF0YQ0KZGF0YShwZW5zaW9uKQ0KIyBPdXRjb21lDQpZID0gcGVuc2lvbiRuZXRfdGZhDQojIFRyZWF0bWVudA0KVyA9IHBlbnNpb24kcDQwMQ0KIyBUcmVhdG1lbnQNClogPSBwZW5zaW9uJGU0MDENCiMgQ3JlYXRlIG1haW4gZWZmZWN0cyBtYXRyaXgNClggPSBtb2RlbC5tYXRyaXgofiAwICsgYWdlICsgZGIgKyBlZHVjICsgZnNpemUgKyBob3duICsgaW5jICsgbWFsZSArIG1hcnIgKyBwaXJhICsgdHdvZWFybiwgZGF0YSA9IHBlbnNpb24pDQpgYGANCg0KV2Ugd2FudCB0byBlc3RpbWF0ZSB0aGUgZWZmZWN0IG9mIDQwMShrKSBwYXJ0aWNpcGF0aW9uIG9uIG5ldCBhc3NldHMuIFdlIGxldmVyYWdlIHR3byBpZGVudGlmaWNhdGlvbiBzdHJhdGVnaWVzOg0KDQotIEJhc2VkIG9uIHVuY29uZm91bmRlZG5lc3MgYXMgYmVmb3JlDQoNCi0gdXNpbmcgZWxpZ2liaWxpdHkgZm9yIDQwMShrKSBhcyBpbnN0cnVtZW50IGZvciBhY3R1YWwgcGFydGljaXBhdGlvbg0KDQphbmQgdXNlIGVzdGltYXRvcnMgdGhhdCBhc3N1bWUgY29uc3RhbnQgZWZmZWN0cyAocGFydGlhbGx5IGxpbmVhcikgYW5kIGFsbG93IGZvciBoZXRlcm9nZW5lb3VzIGVmZmVjdHMgKEFURSwgQVRULCBMQVRFKS4gDQoNCkluIHRvdGFsIHdlIHdhbnQgdG8gaW1wbGVtZW50IGZpdmUgZXN0aW1hdG9yczoNCg0KLSBwYXJ0aWFsbHkgbGluZWFyIERvdWJsZSBNTCAoUEwpDQoNCi0gQUlQVy1BVEUgRG91YmxlIE1MIChBVEUpDQoNCi0gQUlQVy1BVEUgRG91YmxlIE1MIChBVFQpDQoNCi0gcGFydGlhbGx5IGxpbmVhciBJViBEb3VibGUgTUwgKFBMLUlWKQ0KDQotIEFJUFctTEFURSBEb3VibGUgTUwgKExBVEUpDQoNCjxicj4NCjxicj4NCg0KIyBOdWlzYW5jZSBwYXJhbWV0ZXJzDQoNCldlIG5lZWQgYSBidW5jaCBvZiBudWlzYW5jZSBwYXJhbWV0ZXJzIHRvIGRvIHNvLiBUaGUgb25lcyB3ZSBrbm93IGFscmVhZHkgYXJlIA0KDQotICRlKFgpPUVbV3xYXSQgY29kZWQgYXMgYGV4aGF0YA0KDQotICRtKFgpPUVbWXxYXSQgY29kZWQgYXMgYG14aGF0YA0KDQotICRtKDAsWCk9RVtZfFc9MCxYXSQgY29kZWQgYXMgYG13aGF0MGANCg0KLSAkbSgxLFgpPUVbWXxXPTEsWF0kIGNvZGVkIGFzIGBtd2hhdDFgDQoNCkFkZGl0aW9uYWxseSwgd2UgbmVlZA0KDQotICRoKFgpPUVbWnxYXSQgdG8gYmUgY29kZWQgYXMgYGhoYXRgDQoNCi0gJG1feigwLFgpPUVbWXxaPTAsWF0kIHRvIGJlIGNvZGVkIGFzIGBtemhhdDBgDQoNCi0gJG1feigxLFgpPUVbWXxaPTEsWF0kIHRvIGJlIGNvZGVkIGFzIGBtemhhdDFgDQoNCi0gJGUoMCxYKT1FW1d8Wj0wLFhdJCB0byBiZSBjb2RlZCBhcyBgZXpoYXQwYA0KDQotICRlKDEsWCk9RVtXfFo9MSxYXSQgdG8gYmUgY29kZWQgYXMgYGV6aGF0MWANCg0KDQpgYGB7cn0NCm4gPSBsZW5ndGgoWSkNCm5mb2xkcyA9IDUNCmZvbGQgPSBzYW1wbGUoMTpuZm9sZHMsbixyZXBsYWNlPVQpDQoNCmV4aGF0ID0gbXhoYXQgPSBtd2hhdDAgPSBtd2hhdDEgPSBoaGF0ID0gbXpoYXQwID0gbXpoYXQxID0gZXpoYXQwID0gZXpoYXQxID0gcmVwKE5BLG4pDQogIA0KZm9yIChpIGluIDE6bmZvbGRzKXsNCiAgcmZlID0gcmVncmVzc2lvbl9mb3Jlc3QoWFtmb2xkICE9IGksXSxXW2ZvbGQgIT0gaV0pDQogIGV4aGF0W2ZvbGQgPT0gaV0gPSBwcmVkaWN0KHJmZSwgWFtmb2xkID09IGksXSkkcHJlZGljdGlvbnMNCiAgDQogIHJmbSA9IHJlZ3Jlc3Npb25fZm9yZXN0KFhbZm9sZCAhPSBpLF0sWVtmb2xkICE9IGldKQ0KICBteGhhdFtmb2xkID09IGldID0gcHJlZGljdChyZm0sIFhbZm9sZCA9PSBpLF0pJHByZWRpY3Rpb25zDQogIA0KICByZm0wID0gcmVncmVzc2lvbl9mb3Jlc3QoWFtmb2xkICE9IGkgJiBXPT0wLF0sWVtmb2xkICE9IGkgJiBXPT0wXSkNCiAgbXdoYXQwW2ZvbGQgPT0gaV0gPSBwcmVkaWN0KHJmbTAsIFhbZm9sZCA9PSBpLF0pJHByZWRpY3Rpb25zDQoNCiAgcmZtMSA9IHJlZ3Jlc3Npb25fZm9yZXN0KFhbZm9sZCAhPSBpICYgVz09MSxdLFlbZm9sZCAhPSBpICYgVz09MV0pDQogIG13aGF0MVtmb2xkID09IGldID0gcHJlZGljdChyZm0xLCBYW2ZvbGQgPT0gaSxdKSRwcmVkaWN0aW9ucw0KDQogIHJmaCA9IHJlZ3Jlc3Npb25fZm9yZXN0KFhbZm9sZCAhPSBpLF0sWltmb2xkICE9IGldKQ0KICBoaGF0W2ZvbGQgPT0gaV0gPSBwcmVkaWN0KHJmaCwgWFtmb2xkID09IGksXSkkcHJlZGljdGlvbnMNCg0KICByZm16MCA9IHJlZ3Jlc3Npb25fZm9yZXN0KFhbZm9sZCAhPSBpICYgWj09MCxdLFlbZm9sZCAhPSBpICYgWj09MF0pDQogIG16aGF0MFtmb2xkID09IGldID0gcHJlZGljdChyZm16MCwgWFtmb2xkID09IGksXSkkcHJlZGljdGlvbnMNCg0KICByZm16MSA9IHJlZ3Jlc3Npb25fZm9yZXN0KFhbZm9sZCAhPSBpICYgWj09MSxdLFlbZm9sZCAhPSBpICYgWj09MV0pDQogIG16aGF0MVtmb2xkID09IGldID0gcHJlZGljdChyZm16MSwgWFtmb2xkID09IGksXSkkcHJlZGljdGlvbnMNCg0KICByZmV6MCA9IHJlZ3Jlc3Npb25fZm9yZXN0KFhbZm9sZCAhPSBpICYgWj09MCxdLFdbZm9sZCAhPSBpICYgWj09MF0pDQogIGV6aGF0MFtmb2xkID09IGldID0gcHJlZGljdChyZmV6MCwgWFtmb2xkID09IGksXSkkcHJlZGljdGlvbnMNCg0KICByZmV6MSA9IHJlZ3Jlc3Npb25fZm9yZXN0KFhbZm9sZCAhPSBpICYgWj09MSxdLFdbZm9sZCAhPSBpICYgWj09MV0pDQogIGV6aGF0MVtmb2xkID09IGldID0gcHJlZGljdChyZmV6MSwgWFtmb2xkID09IGksXSkkcHJlZGljdGlvbnMNCn0NCmBgYA0KDQphbmQgZmluYWxseSB0aGUgZWFzeSAkZT1FW1ddJCBjb2RlZCBhcyBgZWhhdGANCg0KYGBge3J9DQplaGF0ID0gbWVhbihXKQ0KYGBgDQoNCjxicj4NCjxicj4NCg0KIyAkXHBzaV9hJCBhbmQgJFxwc2lfYiQgb2YgZWFjaCBzY29yZXMNCg0KTm93IHdlIGNvZGUgdXAgdGhlIHR3byBjb21wb25lbnRzIG9mIHRoZSBzY29yZSwgdGhhdCB3aWxsIGhlbHAgdXMgbGF0ZXIgdG8gZ2V0IHRoZSBwb2ludCBlc3RpbWF0ZSBhbmQgdGhlIHN0YW5kYXJkIGVycm9yIHVzaW5nIHRoZSBzYW1lIGZ1bmN0aW9uLg0KDQojIyBQTA0KDQpUaGUgZW1waXJpY2FsIHZlcnNpb24gb2YgdGhlIHNjb3JlIGxvb2tzIGxpa2UgdGhpczoNCiQkXHBzaV57UEx9KE87XGhhdHtcdGF1fSxcaGF0e1xldGF9KSA9IFxoYXR7XHRhdX0gXHVuZGVyYnJhY2V7KC0xKShXIC0gXGhhdHtlfShYKSleMn1fe1xwc2lfYV57UEx9fSArICBcdW5kZXJicmFjZXsgKFkgLSBcaGF0e219KFgpKSAoVyAtIFxoYXR7ZX0oWCkpfV97XHBzaV9iXntQTH19JCQNClRvIGZvbGxvdyB0aGUgcmVjaXBlLCB3ZSBuZWVkIHR3byBjb21wb25lbnRzOg0KDQoxLiAkXHBzaV9hXntQTH0kIHRvIGJlIGNvZGVkIGFzIGBwYV9wbGANCg0KMi4gJFxwc2lfYl57UEx9JCB0byBiZSBjb2RlZCBhcyBgcGJfcGxgDQoNCmBgYHtyfQ0KcGFfcGwgPSAtKFcgLSBleGhhdCleMg0KcGJfcGwgPSAoWSAtIG14aGF0KSAqIChXIC0gZXhoYXQpDQotc3VtKHBiX3BsKSAvIHN1bShwYV9wbCkNCmBgYA0KDQo8YnI+DQoNCiMjIEFURQ0KDQpUaGUgZW1waXJpY2FsIHZlcnNpb24gb2YgdGhlIHNjb3JlIGxvb2tzIGxpa2UgdGhpczoNCiQkXHBzaV57QVRFfShPO1xoYXR7XHRhdX1fe0FURX0sXGhhdHtcZXRhfSk9IFxoYXR7XHRhdX1fe0FURX0gXHVuZGVyYnJhY2V7KC0xKX1fe1xwc2lfYV57QVRFfX0gKyBcdW5kZXJicmFjZXtcaGF0e219KDEsWCkgIC1caGF0eyBtfSgwLFgpICArIFxmcmFje1cgKFkgLSBcaGF0e219KDEsWCkgKX17XGhhdHtlfShYKX0gLSBcZnJhY3soMS1XKSAoWSAtIFxoYXR7bX0oMCxYKSApfXsxLVxoYXR7ZX0oWCl9IH1fe1xwc2lfYl57QVRFfX0kJA0KVG8gZm9sbG93IHRoZSByZWNpcGUsIHdlIG5lZWQgdHdvIGNvbXBvbmVudHM6DQoNCjEuICRccHNpX2Fee0FURX0kIHRvIGJlIGNvZGVkIGFzIGBwYV9hdGVgDQoNCjIuICRccHNpX2Jee0FURX0kIHRvIGJlIGNvZGVkIGFzIGBwYl9hdGVgDQoNCg0KYGBge3J9DQpwYV9hdGUgPSByZXAoLTEsbGVuZ3RoKFkpKQ0KcGJfYXRlID0gbXdoYXQxIC0gbXdoYXQwICsgVyAqIChZIC0gbXdoYXQxKSAvIGVoYXQgLSAoMSAtIFcpICogKFkgLSBtd2hhdDApIC8gKDEtZWhhdCkNCi1zdW0ocGJfYXRlKSAvIHN1bShwYV9hdGUpDQpgYGANCg0KPGJyPg0KDQojIyBBVFQNCg0KVGhlIGVtcGlyaWNhbCB2ZXJzaW9uIG9mIHRoZSBzY29yZSBsb29rcyBsaWtlIHRoaXM6DQokJFxwc2lee0FUVH0oTztcaGF0e1x0YXV9X3tBVFR9LFxoYXR7XGV0YX0pPSBcaGF0e1x0YXV9X3tBVFR9IFx1bmRlcmJyYWNleygtMSlcZnJhY3tXfXtcaGF0e2V9fX1fe1xwc2lfYV57QVRUfX0gKyBcdW5kZXJicmFjZXtcZGZyYWN7V317XGhhdHtlfX0gKFkgLSBcaGF0e219KDAsWCkpIC0gIFxkZnJhY3sgKDEtVykgXGhhdHtlfShYKSAgfXtcaGF0e2V9ICgxLVxoYXR7ZX0oWCkpfSAoWSAtIFxoYXR7bX0oMCxYKSkgfV97XHBzaV9iXntBVFR9fSQkDQpUbyBmb2xsb3cgdGhlIHJlY2lwZSwgd2UgbmVlZCB0d28gY29tcG9uZW50czoNCg0KMS4gJFxwc2lfYV57QVRUfSQgdG8gYmUgY29kZWQgYXMgYHBhX2F0dGANCg0KMi4gJFxwc2lfYl57QVRUfSQgdG8gYmUgY29kZWQgYXMgYHBiX2F0dGANCg0KDQpgYGB7cn0NCnBhX2F0dCA9IC1XIC8gZWhhdA0KcGJfYXR0ID0gVyAqIChZIC0gbXdoYXQwKSAvIGVoYXQgLSAoICgxIC0gVykgKiBleGhhdCApICogKFkgLSBtd2hhdDApIC8gKGVoYXQgKiAoMSAtIGVoYXQpKQ0KLXN1bShwYl9hdHQpIC8gc3VtKHBhX2F0dCkNCmBgYA0KDQo8YnI+DQoNCg0KIyMgUEwtSVYNCg0KVGhlIGVtcGlyaWNhbCB2ZXJzaW9uIG9mIHRoZSBzY29yZSBsb29rcyBsaWtlIHRoaXM6DQokJFxwc2lee0lWfShPO1xoYXR7XHRhdX1fe0lWfSxcaGF0e1xldGF9KT0gXGhhdHtcdGF1fV97SVZ9IFx1bmRlcmJyYWNleygtMSkgKFcgLSBcaGF0e2V9KFgpKSAoWiAtIFxoYXR7aH0oWCkpfV97XHBzaV9hXntJVn19ICsgXHVuZGVyYnJhY2V7KFkgLSBcaGF0e219KFgpKSAoWiAtIFxoYXR7aH0oWCkpfV97XHBzaV9iXntJVn19JCQNClRvIGZvbGxvdyB0aGUgcmVjaXBlLCB3ZSBuZWVkIHR3byBjb21wb25lbnRzOg0KDQoxLiAkXHBzaV9hXntJVn0kIHRvIGJlIGNvZGVkIGFzIGBwYV9pdmANCg0KMi4gJFxwc2lfYl57SVZ9JCB0byBiZSBjb2RlZCBhcyBgcGJfaXZgDQoNCg0KYGBge3J9DQpwYV9pdiA9IC0oVyAtIGV4aGF0KSAqIChaIC0gaGhhdCkNCnBiX2l2ID0gKFkgLSBteGhhdCkgKiAoWiAtIGhoYXQpIA0KLXN1bShwYl9pdikgLyBzdW0ocGFfaXYpDQpgYGANCg0KPGJyPg0KDQoNCiMjIExBVEUNCg0KVGhlIGVtcGlyaWNhbCB2ZXJzaW9uIG9mIHRoZSBzY29yZSBsb29rcyBsaWtlIHRoaXM6DQokJA0KXGJlZ2lue2FsaWdufQ0KXHBzaV57TEFURX0oTztcaGF0e1x0YXV9X3tMQVRFfSxcaGF0e1xldGF9KSAmID0gXGhhdHtcdGF1fV97TEFURX0gXHVuZGVyYnJhY2V7ICgtMSkgXEJpZ2dbIFxoYXR7ZX0oMSxYKSAtIFxoYXR7ZX0oMCxYKSArIFxkZnJhY3taIChXIC0gXGhhdHtlfSgxLFgpKX17XGhhdHtofShYKX0gLSBcZGZyYWN7KDEtWikgKFcgLSBcaGF0e2h9KDAsWCkpfXsxLVxoYXR7aH0oWCl9IFxCaWdnXSB9X3tccHNpX2Fee0xBVEV9fSBcXA0KJiBccXVhZCArIFx1bmRlcmJyYWNle1xoYXR7bX1feigxLFgpIC0gXGhhdHttfV96KDAsWCkgKyBcZGZyYWN7WiAoWSAtIFxoYXR7bX1feigxLFgpKX17XGhhdHtofShYKX0gLSBcZGZyYWN7KDEtWikgKFkgLSBcaGF0e219X3ooMCxYKSl9ezEtXGhhdHtofShYKX19X3tccHNpX2Jee0xBVEV9fQ0KXGVuZHthbGlnbn0NCiQkDQpUbyBmb2xsb3cgdGhlIHJlY2lwZSwgd2UgbmVlZCB0d28gY29tcG9uZW50czoNCg0KMS4gJFxwc2lfYV57TEFURX0kIHRvIGJlIGNvZGVkIGFzIGBwYV9sYXRlYA0KDQoyLiAkXHBzaV9iXntMQVRFfSQgdG8gYmUgY29kZWQgYXMgYHBiX2xhdGVgDQoNCg0KYGBge3J9DQpwYV9sYXRlID0gLSggZXpoYXQxIC0gZXpoYXQwICsgWiAqIChXIC0gZXpoYXQxKSAvIGhoYXQgLSAoMSAtIFopICogKFcgLSBlemhhdDApIC8gKDEtaGhhdCkgKQ0KcGJfbGF0ZSA9IG16aGF0MSAtIG16aGF0MCArIFogKiAoWSAtIG16aGF0MSkgLyBoaGF0IC0gKDEgLSBaKSAqIChZIC0gbXpoYXQwKSAvICgxLWhoYXQpDQotc3VtKHBiX2xhdGUpIC8gc3VtKHBhX2xhdGUpDQpgYGANCg0KPGJyPg0KPGJyPg0KDQojIEEgZ2VuZXJpYyBmdW5jdGlvbiBmb3IgRG91YmxlIE1MIHdpdGggbGluZWFyIHNjb3JlDQoNCk5vdyB3ZSB3cml0ZSBhIGZ1bmN0aW9uIHRoYXQgdGFrZXMgYW55ICRccHNpX2EkIGFuZCAkXHBzaV9iJCBhcyBpbnB1dHMgYW5kIG91dHB1dHMNCg0KLSBwb2ludCBlc3RpbWF0ZSANCg0KLSBzdGFuZGFyZCBlcnJvcg0KDQotIHQtdmFsdWUNCg0KLSBwLXZhbHVlDQoNCg0KVGhlIGZ1bmN0aW9uIGltcGxlbWVudHMgdGhlIGZvbGxvd2luZyBzdGVwczoNCg0KMS4gQ2FsY3VsYXRlIHBvaW50IGVzdGltYXRlIGFzICRcaGF0e1x0aGV0YX0gPSAtXGZyYWN7XHN1bV9pXHBzaV9iKE9faTtcaGF0e1xldGF9X2kpfXtcc3VtX2kgXHBzaV9hKE9faTtcaGF0e1xldGF9X2kpfSAkDQoNCjIuICRcaGF0e1x0aGV0YX0kIGNhbiB0aGVuIGJlIHVzZWQgdG8gY29tcGxldGUgdGhlIGVtcGlyaWNhbCBzY29yZSBhcyAkXHBzaShPO1xoYXR7XHRoZXRhfSxcaGF0e1xldGF9KSA9IFxoYXR7XHRoZXRhfSBccHNpX2EoT19pO1xoYXR7XGV0YX1faSkgKyBccHNpX2IoT19pO1xoYXR7XGV0YX1faSkkDQoNCjMuIENyZWF0ZSBpbmZsdWVuY2UgZnVuY3Rpb24gJFxQc2koTztcaGF0e1x0aGV0YX0sXGhhdHtcZXRhfSkgPSAtIFxmcmFje1xwc2koTztcaGF0e1x0aGV0YX0sXGhhdHtcZXRhfSl9e05eey0xfSBcc3VtX2kgXHBzaV9hKE9faTtcaGF0e1xldGF9X2kpfSQNCg0KNC4gRXN0aW1hdGUgdGhlIHZhcmlhbmNlIGFzICRcaGF0e1xzaWdtYX1eMiA9IFZhcihcUHNpKE87XGhhdHtcdGhldGF9LFxoYXR7XGV0YX0pKSQgb3IgJFxoYXR7XHNpZ21hfV4yID0gXGZyYWN7Tl57LTF9IFxzdW1faSAgXHBzaShPX2k7XGhhdHtcdGhldGF9LFxoYXR7XGV0YX1faSleMn17W05eey0xfSBcc3VtX2kgIFxwc2lfYShPX2k7XGhhdHtcZXRhfV9pKV1eMn0kDQoNCjUuIEdldCB0aGUgc3RhbmRhcmQgZXJyb3IgYXMgJHNlKFxoYXR7XHRoZXRhfSkgPSBcc3FydHtcZnJhY3tcaGF0e1xzaWdtYX1eMn17Tn19JA0KDQpgYGB7cn0NCkRNTF9pbmZlcmVuY2UgPSBmdW5jdGlvbihwc2lfYSxwc2lfYikgew0KICBOID0gbGVuZ3RoKHBzaV9hKQ0KICB0aGV0YSA9IC1zdW0ocHNpX2IpIC8gc3VtKHBzaV9hKQ0KICBwc2kgPSB0aGV0YSAqIHBzaV9hICsgcHNpX2INCiAgUHNpID0gLSBwc2kgLyBtZWFuKHBzaV9hKQ0KICBzaWdtYTIgPSB2YXIoUHNpKQ0KICAjIHNpZ21hMiA9IG1lYW4ocHNpXjIpIC8gbWVhbihwc2lfYSleMg0KICBzZSA9IHNxcnQoc2lnbWEyIC8gTikNCiAgdCA9IHRoZXRhIC8gc2UNCiAgcCA9IDIgKiBwdChhYnModCksTixsb3dlciA9IEZBTFNFKQ0KICByZXN1bHQgPSBjKHRoZXRhLHNlLHQscCkNCiAgcmV0dXJuKHJlc3VsdCkNCn0NCmBgYA0KDQo8YnI+DQo8YnI+DQoNCg0KIyBSZXN1bHRzDQoNCmBgYHtyfQ0KcmVzdWx0cyA9IG1hdHJpeChOQSw1LDQpDQpyb3duYW1lcyhyZXN1bHRzKSA9IGMoIlBMIiwiQVRFIiwiQVRUIiwiUEwtSVYiLCJMQVRFIikNCmNvbG5hbWVzKHJlc3VsdHMpID0gYygiRWZmZWN0IiwiUy5FLiIsInQiLCJwIikNCnJlc3VsdHNbMSxdID0gRE1MX2luZmVyZW5jZShwYV9wbCxwYl9wbCkNCnJlc3VsdHNbMixdID0gRE1MX2luZmVyZW5jZShwYV9hdGUscGJfYXRlKQ0KcmVzdWx0c1szLF0gPSBETUxfaW5mZXJlbmNlKHBhX2F0dCxwYl9hdHQpDQpyZXN1bHRzWzQsXSA9IERNTF9pbmZlcmVuY2UocGFfaXYscGJfaXYpDQpyZXN1bHRzWzUsXSA9IERNTF9pbmZlcmVuY2UocGFfbGF0ZSxwYl9sYXRlKQ0KcHJpbnRDb2VmbWF0KHJlc3VsdHMsaGFzLlB2YWx1ZSA9IFRSVUUpDQpgYGANCg0KDQpgYGB7cn0NCmRhdGEuZnJhbWUodGhldGFzID0gcmVzdWx0c1ssMV0sc2VzID0gcmVzdWx0c1ssMl0sDQogICAgICAgICAgICAgICAgRXN0aW1hdG9yID0gcm93bmFtZXMocmVzdWx0cyksDQogICAgICAgICAgICAgICAgY2lsID0gcmVzdWx0c1ssMV0gLSAxLjk2KnJlc3VsdHNbLDJdLA0KICAgICAgICAgICAgICAgIGNpdSA9IHJlc3VsdHNbLDFdICsgMS45NipyZXN1bHRzWywyXSkgICU+JSANCiAgZ2dwbG90KGFlcyh4PUVzdGltYXRvcix5PXRoZXRhcyx5bWluPWNpbCx5bWF4PWNpdSkpICsgZ2VvbV9wb2ludChzaXplPTIuNSkgKyBnZW9tX2Vycm9yYmFyKHdpZHRoPTAuMTUpICArDQogIGdlb21faGxpbmUoeWludGVyY2VwdD0wKQ0KYGBgDQoNCg0KPGJyPg0KPGJyPg0KDQoNCiMgQVRUIHZzLiBBVEUNCg0KQXNzdW1lIHRoYXQgd2Ugd2FudCB0byB0ZXN0IHdoZXRoZXIgJEFUVD1BVEUkLg0KDQpGb3IgdGhpcyBwdXJwb3NlLCB3ZSBjcmVhdGUgdGhlIG5ldyBwYXJhbWV0ZXIgJFxEZWx0YV97QVRUfSA9IFx0YXVfe0FUVH0gLSBcdGF1X3tBVEV9JCBhbmQgd2FudCB0byB0ZXN0ICRIXzA6XERlbHRhKFx0YXVfe0FUVH0sXHRhdV97QVRFfSkgPSAwJA0KDQpUaGUgaW5mbHVlbmNlIGZ1bmN0aW9uIG9mIHRoaXMgbmV3IHBhcmFtZXRlciBjYW4gZGVyaXZlZCBsaWtlIHRoaXM6DQokJA0KXGJlZ2lue2FsaWduKn0NCiAgXFBzaV9cRGVsdGEgJiA9IFxvdmVyYnJhY2V7XGZyYWN7XGRlbHRhIFxEZWx0YX17XGRlbHRhIFx0YXVfe0FUVH19fV57PTF9IFxQc2lfe1x0YXVfe0FUVH19ICsgXG92ZXJicmFjZXtcZnJhY3tcZGVsdGEgXERlbHRhfXtcZGVsdGEgXHRhdV97QVRFfX19Xns9LTF9IFxQc2lfe1x0YXVfe0FURX19IFxcDQogICYgPSBcUHNpX3tcdGF1X3tBVFR9fSAtIFxQc2lfe1x0YXVfe0FURX19DQpcZW5ke2FsaWduKn0NCiQkDQoNCkRlZmluZSBmb3IgY29udmVuaWVuY2UgYW5kIGxhdGVyIHVzZSBhIGBQc2lfbWFrZXIoKWAgZnVuY3Rpb24gdGhhdCBjcmVhdGVzIHRoZSBpbmZsdWVuY2UgZnVuY3Rpb24gZm9yIGdlbmVyaWMgJFxwc2lfYSQgYW5kICRccHNpX2IkOg0KDQpgYGB7cn0NClBzaV9tYWtlciA9IGZ1bmN0aW9uKHBzaV9hLHBzaV9iKSB7DQogIHRoZXRhID0gLXN1bShwc2lfYikgLyBzdW0ocHNpX2EpDQogIHBzaSA9IHRoZXRhICogcHNpX2EgKyBwc2lfYg0KICBQc2kgPSAtIHBzaSAvIG1lYW4ocHNpX2EpDQogIHJldHVybihQc2kpDQp9DQpgYGANCg0KTm93IHRoaXMgY2FuIGJlIHVzZWQgdG8gdGVzdCAkSF8wOlxEZWx0YShcdGF1X3tBVFR9LFx0YXVfe0FURX0pID0gMCQ6DQoNCmBgYHtyfQ0KIyBDYWxjdWxhdGUgcGFyYW1ldGVycw0KYXR0ID0gLSBzdW0ocGJfYXR0KSAvIHN1bShwYV9hdHQpDQphdGUgPSAtIHN1bShwYl9hdGUpIC8gc3VtKHBhX2F0ZSkNCkRlbHRhX2F0dCA9IGF0dCAtIGF0ZQ0KIyBDcmVhdGUgaW5mbHVlbmNlIGZ1bmN0aW9uIGZvciBuZXcgcGFyYW1ldGVyDQpQc2lfRGVsdGFfYXR0ID0gUHNpX21ha2VyKHBhX2F0dCxwYl9hdHQpIC0gUHNpX21ha2VyKHBhX2F0ZSxwYl9hdGUpDQojIENhbGN1bGF0ZSBzdGFuZGFyZCBlcnJvcnMsIHQgYW5kIHB2YWx1ZXMNCnNlX0RlbHRhX2F0dCA9IHNxcnQodmFyKFBzaV9EZWx0YV9hdHQpL2xlbmd0aChQc2lfRGVsdGFfYXR0KSkNCnRfRGVsdGFfYXR0ID0gRGVsdGFfYXR0IC8gc2VfRGVsdGFfYXR0DQpwX0RlbHRhX2F0dCA9IDIgKiBwdChhYnModF9EZWx0YV9hdHQpLGxlbmd0aChQc2lfRGVsdGFfYXR0KSxsb3dlciA9IEZBTFNFKQ0KIyBQcmludCByZXN1bHRzDQpyZXN1bHQgPSBtYXRyaXgoYyhEZWx0YV9hdHQsc2VfRGVsdGFfYXR0LHRfRGVsdGFfYXR0LHBfRGVsdGFfYXR0KSxucm93ID0gMSkNCnJvd25hbWVzKHJlc3VsdCkgPSBjKCJBVFQtQVRFIikNCmNvbG5hbWVzKHJlc3VsdCkgPSBjKCJEZWx0YSIsIlMuRS4iLCJ0IiwicCIpDQpwcmludENvZWZtYXQocmVzdWx0LGhhcy5QdmFsdWUgPSBUUlVFKQ0KYGBgDQoNCldlIG9ic2VydmUgdGhhdCB0aGUgZGlmZmVyZW50IGJldHdlZW4gQVRUIGFuZCBBVEUgaXMgaGlnaGx5IHNpZ25pZmljYW50LiBUaGlzIG1heSBjb21lIGFzIGEgc3VycHJpc2UgZ2l2ZW4gdGhhdCB0aGUgY29uZmlkZW5jZSBpbnRlcnZhbHMgb3ZlcmxhcCBpbiB0aGUgZ3JhcGggYWJvdmUuIEhvd2V2ZXIsIHN1Y2ggZXllYmFsbGluZyBkb2VzIG5vdCB3b3JrIGZvciBjb3JyZWxhdGVkIGVzdGltYXRvcnMgYW5kIHRoZSBBVFQgYW5kIEFURSBlc3RpbWF0b3IgYXJlIGNsZWFybHkgY29ycmVsYXRlZC4NCg0KPGJyPg0KPGJyPg0KDQoNCiMgTEFURSB2cy4gQVRFDQoNCg0KU2ltaWxhcmx5LCB3ZSBjYW4gaW52ZXN0aWdhdGUgJEhfMDpcRGVsdGFfe0xBVEV9ID0gXHRhdV97TEFURX0gLSBcdGF1X3tBVEV9ID0gMCQuICgwLjVQKQ0KDQpUaGUgZGVyaXZhdGlvbiB3b3JrcyBsaWtlIGJlZm9yZToNCg0KJCRcYmVnaW57YWxpZ259DQpcUHNpX3tcRGVsdGFfe0xBVEV9fSAmPSBcdW5kZXJicmFjZXtcZnJhY3tccGFydGlhbCBcRGVsdGFfe0xBVEV9fXtccGFydGlhbCBcdGF1X3tMQVRFfX19X3s9MX0gXFBzaV97TEFURX0gKyBcdW5kZXJicmFjZXtcZnJhY3tccGFydGlhbCBcRGVsdGFfe0xBVEV9fXtccGFydGlhbCBcdGF1X3tBVEV9fX1fez0tMX0gXFBzaV97QVRFfSBcXA0KXFBzaV97XERlbHRhX3tMQVRFfX0gJj1cUHNpX3tMQVRFfSAtIFxQc2lfe0FURX0NClxlbmR7YWxpZ259JCQNCg0KYW5kIGFsc28gdGhlIGluZmVyZW5jZSBmb2xsb3dzIHRoZSBzYW1lIHJlY2lwZToNCg0KYGBge3J9DQojIENhbGN1TEFURQ0KbGF0ZSA9IC1zdW0ocGJfbGF0ZSkvc3VtKHBhX2xhdGUpIA0KIyBOZXcgdGFyZ2V0IHBhcmFtZXRlcg0KRGVsdGFfbGF0ZSA9IGxhdGUgLSBhdGUgDQojIENyZWF0ZSBpbmZsdWVuY2UgZnVuY3Rpb24gZm9yIG5ldyBwYXJhbWV0ZXINClBzaV9EZWx0YV9sYXRlID0gUHNpX21ha2VyKHBhX2xhdGUscGJfbGF0ZSkgLSBQc2lfbWFrZXIocGFfYXRlLHBiX2F0ZSkNCiMgUHJpbnQgcmVzdWx0cw0Kc2VfRGVsdGFfbGF0ZSA9IHNxcnQodmFyKFBzaV9EZWx0YV9sYXRlKS9sZW5ndGgoUHNpX0RlbHRhX2xhdGUpKQ0KdF9EZWx0YV9sYXRlID0gRGVsdGFfbGF0ZS9zZV9EZWx0YV9sYXRlDQpwX0RlbHRhX2xhdGUgPSAyICogcHQoYWJzKHRfRGVsdGFfbGF0ZSksbGVuZ3RoKFBzaV9EZWx0YV9sYXRlKSxsb3dlciA9IEZBTFNFKSAjIGdldCBhIHAtdmFsdWUgKGF0IHdoYXQgbGV2ZWwgd291bGQgYmUgbm90IHJlamVjdD8pDQoNCnJlc3VsdHMgPSBtYXRyaXgoYyhEZWx0YV9sYXRlLHNlX0RlbHRhX2xhdGUsdF9EZWx0YV9sYXRlLHBfRGVsdGFfbGF0ZSksbnJvdyA9IDEpDQpyb3duYW1lcyhyZXN1bHRzKSA9IGMoIkxBVEUtQVRFIikNCmNvbG5hbWVzKHJlc3VsdHMpID0gYygiRGVsdGEiLCJTLkUuIiwidCIsInAiKQ0KcHJpbnRDb2VmbWF0KHJlc3VsdHMsaGFzLlB2YWx1ZSA9IFRSVUUpDQpgYGANCg0KSG93ZXZlciwgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiBMQVRFIGFuZCBBVEUgaXMgbm90IHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQuDQoNCjxicj4NCjxicj4NCg0KIyBIb3cgbGFyZ2UgaXMgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiBBVFQgYW5kIEFURSBpbiAlPw0KDQpGaW5hbGx5LCBsZXQncyBsb29rIGF0IGEgbW9yZSBjb21wbGljYXRlZCBuZXcgcGFyYW1ldGVyICRcRGVsdGFcJSA9IFxmcmFje1x0YXVfe0FUVH0gLSBcdGF1X3tBVEV9fXtcdGF1X3tBVEV9fVx0aW1lcyAxMDAkLiANCg0KRm9sbG93aW5nIHRoZSByZWNpcGUsIHdlIGRlcml2ZSB0aGUgbmV3IGluZmx1ZW5jZSBmdW5jdGlvbjoNCg0KJCRcYmVnaW57YWxpZ259DQpcUHNpX3tcRGVsdGFcJX0gJj0gXGZyYWN7XHBhcnRpYWwgXERlbHRhXCV9e1xwYXJ0aWFsIFx0YXVfe0FUVH19IFxQc2lfe0FUVH0gKyBcZnJhY3tccGFydGlhbCBcRGVsdGFcJX17XHBhcnRpYWwgXHRhdV97QVRFfX0gXFBzaV97QVRFfSBcXA0KXFBzaV97XERlbHRhXCV9ICY9MTAwIFxjZG90IFxmcmFjezF9e1x0YXVfe0FURX19IFxQc2lfe0FUVH0gLTEwMCBcY2RvdCBcZnJhY3tcdGF1X3tBVFR9fXtcdGF1X3tBVEV9XjJ9IFxQc2lfe0FURX0NClxlbmR7YWxpZ259JCQNCg0KJFxQc2lfe0FUVH0kIGFuZCAkXFBzaV97QVRFfSQgY2FuIGJlIG9idGFpbmVkIHVzaW5nIGBQc2lfbWFrZXJgLiAkXHRhdV97QVRFfSQgYW5kICRcdGF1X3tBVFR9JCBjYW4gYmUgcmVwbGFjZWQgYnkgdGhlaXIgZXN0aW1hdGVzOg0KDQpgYGB7cn0NCiMgTmV3IHBhcmFtZXRlcg0KRGVsdGFfcGMgPSAoYXR0LWF0ZSkvYXRlKjEwMCANCiMgTmV3IElGDQpQc2lfRGVsdGFfcGMgPSAxMDAgLyBhdGUgKiBQc2lfbWFrZXIocGFfYXR0LCBwYl9hdHQpIC0gMTAwICogYXR0IC8gKGF0ZV4yKSAqIFBzaV9tYWtlcihwYV9hdGUsIHBiX2F0ZSkNCiMgUmVzdWx0cw0Kc2VfRGVsdGFfcGMgPSBzcXJ0KHZhcihQc2lfRGVsdGFfcGMpL2xlbmd0aChQc2lfRGVsdGFfcGMpKQ0KdF9EZWx0YV9wYyA9IERlbHRhX3BjL3NlX0RlbHRhX3BjDQpwX0RlbHRhX3BjID0gMiAqIHB0KGFicyh0X0RlbHRhX3BjKSxsZW5ndGgoUHNpX0RlbHRhX3BjKSxsb3dlciA9IEZBTFNFKQ0KcmVzdWx0cyA9IG1hdHJpeChjKERlbHRhX3BjLHNlX0RlbHRhX3BjLHRfRGVsdGFfcGMscF9EZWx0YV9wYyksbnJvdyA9IDEpDQpyb3duYW1lcyhyZXN1bHRzKSA9IGMoIkRlbHRhJSIpDQpjb2xuYW1lcyhyZXN1bHRzKSA9IGMoIkRlbHRhIiwiUy5FLiIsInQiLCJwIikNCnByaW50Q29lZm1hdChyZXN1bHRzLGhhcy5QdmFsdWUgPSBUUlVFKQ0KYGBgDQoNCk5vdCBzdXJwcmlzaW5nbHkgdGhlIHBlcmNlbnRhZ2UgY2hhbmdlIGlzIHNpZ25pZmljYW50LCBsaWtlIHRoZSBwbGFpbiBsZXZlbCBkaWZmZXJlbmNlIGJldHdlZW4gQVRUIGFuZCBBVEUuIEhvd2V2ZXIsIHRoZSB0LXZhbHVlcyBhcmUgbm90IGlkZW50aWNhbCwgd2hpY2ggaWxsdXN0cmF0ZXMgdGhhdCB3ZSBleHBsaWNpdGx5IGNhbGN1bGF0ZSB0aGUgaW5mZXJlbmNlIGZvciBzdWNoIG5ldyBwYXJhbWV0ZXJzLiBBbmQgaW5mbHVlbmNlIGZ1bmN0aW9ucyBwcm92aWRlIGEgbmVhdCB0b29sIHRvIGRvIHNvLg0KDQo8YnI+DQo=

Causal ML: Double ML as generic recipe

Application notebook

Michael Knaus

02/23

401(k) dataset again

Nuisance parameters

$\psi_a$ and $\psi_b$ of each scores

PL

ATE

ATT

PL-IV

LATE

A generic function for Double ML with linear score

Results

ATT vs. ATE

LATE vs. ATE

How large is the difference between ATT and ATE in %?

Causal ML: Double ML as generic recipe

Application notebook

Michael Knaus

02/23

401(k) dataset again

Nuisance parameters

ψa\psi_a and ψb\psi_b of each scores

PL

ATE

ATT

PL-IV

LATE

A generic function for Double ML with linear score

Results

ATT vs. ATE

LATE vs. ATE

How large is the difference between ATT and ATE in %?

$\psi_a$ and $\psi_b$ of each scores