Update: we have not received details for treatment delay. An author reports that treatment initiation time was not recorded: [osf.io]
. Conflicting estimates are provided in a comment of the article and independent analysis, with reports indicating missing data in the dataset. Also see [medrxiv.org]
(companion PEP trial), and Pullen et al. [ncbi.nlm.nih.gov]
, which shows shipping delay for these trials of 19 - 68 hours. Only one third of participants completed enrollment weekdays between 8:00am and 4:00pm, with 44% outside of these hours during the week, and 22% during the weekend. With enrollment up to 4 days after symptom onset, this implies delivery 19 - 164 hours after onset (19 hours would require instantaneous enrollment).
~70 to 140 hour (inc. shipping) delayed outpatient treatment with HCQ showing lower hospitalization/death and faster recovery, but not reaching statistical significance. There was one hospitalized control death and one non-hospitalized HCQ death. It is unclear why there was a non-hospitalized death, external factors such as lack of standard care may be involved. Excluding that case results in one control death and zero HCQ deaths. Details for the hospitalizations and deaths such as medication adherence and treatment delay may be informative but are not provided.
The paper states the end point was changed to symptom severity because they would have required 6,000 participants. However, if the same event rates continued, they would hit 95% significance on the reduction in hospitalization after adding less than 500 patients per arm.
Treatment is relatively late, ~70 to 140 hours after symptoms, including the shipping delay. The paper does not mention the shipping delay but partial details are provided in the study protocol. They are not clear but suggest no shipping on the weekends and a possible 12pm cutoff for same day dispensing and mailing. Assuming that enrollments were evenly distributed between 6am and 12am each day, we get an average of approximately 46 hours shipping delay. We have asked for shipping details and will update with more accurate values when available. In any case the treatment delay is relatively long and there is likely little overlap with the more typical delays used such as 0 - 36 hours for oseltamivir.
The paper compares 0 - 36 hour delayed treatment with oseltamivir (influenza) and ~70 to 140 hour delayed treatment with HCQ (COVID-19), noting that oseltamivir seemed more effective. However, a more comparable study is McLean (2015) who showed that 48 - 119 hour delayed treatment with oseltamivir has no effect. This suggests that HCQ is more effective than oseltamivir, and that HCQ may still have significant effect for some amount of delay beyond the delay where oseltamivir is effective.
6 people were included that enrolled with >4d symptoms, although they do not match the study inclusion criteria. This reduces observed effectiveness. The paper says 56% (236) were enrolled within 1 day of symptoms, but results show only 40% for "<1d", 56% is possibly for <48hrs, we have asked for clarification.
Patients in this study are relatively young and most of them recover without assistance. This reduces the room for a treatment to make improvements. The maximum improvement of an effective treatment would be expected before all patients approach recovery. Authors focus on the end result where most have recovered, but it is more informative to examine the curve and the point of maximum effectiveness. Authors did not collect data for every day but they do have interim results for days 3, 5, 10. The results are consistent with an effective treatment and show a statistically significant improvement, p
= 0.05, at day 10 (other unreported days might show increased effectiveness).
Results also show a larger treatment effect for those >50, not statistically significant due to the small sample, but noted as COVID-19 risk dramatically increases with age. The effect may be more visible here because younger patients may on average have more mild cases with less room for improvement. In general patients in this study have relatively mild symptoms on average, limiting the chance to observe improvement.
The study relies on Internet surveys. Known fake surveys were submitted to the similar PEP trial and there could be an unknown number of undetected fake surveys in both trials. The study shows a high incidence of side effects in the placebo arm, which could be in part due to fake entries 
The granularity change in the histograms of Figure S4 has raised questions 
. Data on increasing severity, less affected by the lower bound where everyone has recovered, also supports effectiveness 
Research shows the placebo used in the US may be protective for COVID-19 [frontiersin.org]
so the true effectiveness of HCQ could be higher than observed. Also see [acpjournals.org]
Treatment delay reporting has changed from the companion PEP trial which reported results for enrollment delays 1, 2, 3, and 4 separately (and from which we can confirm a statistically significant delay-response relationship), while this trial combines 1-2 and 3-4, and adds <1. Since the two trials share reporting (some patients were moved between trials) the reason for the differences are not clear.
RCT of 423 patients with Internet surveys. Medication adherence was only 77% so the true effect of treatment is likely higher. Analysis of primarily low risk patients, authors note the results are not generalizable to the COVID high-risk population. We will update when hearing back on questions asked.
Also see [drive.google.com]
regarding flaws in this study. NCT04308668