Archive for the ‘Stats’ Category
Cross-posted from Design Research for Global Health.
Giving talks is not one of my strong suits, but it seems to be a part of the job requirement. Earlier this month, I had the opportunity (even though I’m no good, I do consider it an opportunity), to give a couple talks, one to the Interdisciplinary MPH Program at Berkeley and one to a group of undergraduate design students, also at Berkeley. Despite the difference in focus, age, and experience of the two groups, the topic was roughly the same: How do we effectively use design thinking as an approach in public health?
The first session was so-so, and I suspect that the few people who were excited about it were probably excited in spite of the talk. It started well, but about halfway through, something began to feel very wrong and that feeling didn’t go away until some time later that evening. Afterwards, I received direct feedback from the instructor and from the students in the form of an evaluation. I recommend this if it is ever presented as an option. Like any “accident”, this one was a “confluence of factors”: lack of clarity and specificity, allowing the discussion to get sidetracked, poor posture, and a tone that conveyed a lack of excitement for the topic.
It’s one thing to get feedback like this, another to act on it.
The second session went much better, gauging by the student feedback, the comments from the instructor, and my own observations. This in spite of a larger group (60 vs. 20) that would be harder to motivate (undergraduates with midterms vs. professionals working on applied problems in public health). I chalk it all up to preparation and planning. Certainly there are people that are capable of doing a great job without preparation – I just don’t think I’m one of those people.
Most of that preparation by the way was not on slides. I did use slides, but only had five for an hour session and that still proved to be too many. Most of the time that I spent on slides, I spent developing a single custom visual to convey precisely the information that was relevant to the students during this session (see image). The rest of the preparation was spent understanding the audience needs by speaking to those running the class; developing a detailed plan for the hour, focusing on how to make the session a highly interactive learning experience; designing quality handouts to support the interactive exercise; and doing my necessary homework. For this last one, I spent 20 minutes on the phone with a surgeon friend, since the session was built around a case study discussing surgical complications and design.
Three resources I found really useful:
- Why Bad Presentations Happen to Good Causes, Andy Goodman, 2006. This commissioned report was developed to help NGOs with their presentations, but I think there is value here for anyone whose work involves presentations. It is evidence-based and provides practical guidance on session design, delivery, slides (PowerPoint), and logistics. Most importantly, it is available as a free download. I was fortunate enough to pick up a used copy of the print edition for US$9 at my local bookstore, which was worth the investment for me because of the design of the physical book. It’s out-of-print now and it looks like the online used copies are quite expensive – at least 3x what I paid – so I recommend the PDF.
- Envisioning Information, Edward Tufte, 1990. I read this when I was writing my dissertation. Folks in design all know about Tufte, but I still recommend a periodic refresher. This is the sort of book that will stay on my shelf. Also potentially useful is The Visual Display of Quantitative Information. For those working in global health, don’t forget how important the display of information can be: (a) Bill Gates and the NYTimes, (b) Hans Rosling at TED.
- Software for creating quality graphics. The drawing tools built into typical office applications, though they have improved in recent years, are still limited in their capability and flexibility, especially if you’re looking at #2 above. In the past 10 days, three people in my socio-professional network have solicited advice on such standalone tools, OmniGraffle (for Mac) and Visio (Windows): a graphic designer in New York, an energy research scientist in California, and a healthcare researcher in DC. Both are great options. I use OmniGraffle these days, though I used to use Visio a few years back. If cost is an issue, there are open-source alternatives available, though I’m not at all familiar with them (e.g., the Pencil plug-in for Firefox).
A few days back Aman wrote a post about Google Flu Trends. Thought I’d add a few thoughts here after reading the draft manuscript that the Google-CDC team posted in advance of its publication in Nature.
By the way, here’s what Nature says: Because of the immediate public-health implications of this paper, Nature supports the Google and the CDC decision to release this information to the public in advance of a formal publication date for the research. The paper has been subjected to the usual rigor of peer review and is accepted in principle. Nature feels the public-health consideration here makes it appropriate to relax our embargo rule
Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L. Detecting influenza epidemics using search engine query data. Draft manuscript for Nature. Retrieved 14 Nov 2008.
Assuming that few folks will read the manuscript or the article, here’s some highlights. I should say I appreciated that the article was clearly written. If you need more context, check out Google Flu Trends How does this work?…
- Targets health-seeking behavior of Internet users, particularly Google users [not sure those are different anymore], in the United States for ILI (influenza-like illness)
- Compared to previous work attempting to link online activity to disease prevalence, benefits from volume: hundreds of billions of searches over 5 years
- Key result – reduced reporting lag to one day compared to CDC’s surveillance system of 1-2 weeks
- Spatial resolution based on IP address goes to nearest big city [for example my current IP maps to Oakland, California right now], but the system is right now only looking to the level of states – this is more detailed CDC’s reporting, which is based on 9 U.S. regions
- CDC data was used for model-building (linear logistic regression) as well as comparison [for stats nerds - the comparison was made with held-out data]
- Not all states publish ILI data, but they were still able to achieve a correlation of 0.85 in Utah without training the model on that state’s data
- There have attempted to look at disease outbreaks of enterics and arboviruses, but without success.
- For those familiar with GPHIN and Healthmap, two other online , the major difference is in the data being examined – Flu Trends looks at search terms while the other systems rely on news sources, website, official alerts, and the such
- There is a possibility that this will not model a flu pandemic well since the search behavior used for modeling is based on non-pandemic variety of flu
- The modeling effort was immense – “450 million different models to test each of the candidate queries”
So what does this mean for developing world applications?
Here’s what the authors say: “Though it may be possible for this approach to be applied to any country with a large population of web search users, we cannot currently provide accurate estimates for large parts of the developing world. Even within the developed world, small countries and less common languages may be challenging to accurately survey.”
The key is whether there are detectable changes in search in response to disease outbreaks. This is dependent on Internet volume, health-seeking search behavior, and language. And if there is no baseline data, like with CDC surveillance data, then what is the best strategy for model-building? How valid will models be from one country to another? That probably depends on the countries. Is it perhaps possible to have a less refined output, something like a multi-level warning system for decision makers to followup with on-the-ground resources? Or should we be focusing on news+ like GPHIN and Healthmap?
Another thought is that we could mine SMS traffic for detecting disease outbreaks. The problem becomes more complicated, since we’re now looking at data that is much more complex than search queries. And there is often segmentation due to the presence of multiple phone providers in one area. Even if the data were anonymized, this raises huge privacy concerns. Still it could be a way to tap in to areas with low Internet penetration and to provide detection based on very real-time data.
This piece is cross-posted from the Uganda output-based aid (OBA) site which just got a major under-the-hood overhaul in its move to a blog format. The Uganda OBA project contracts private clinics to see qualified patients for complaints of suspected sexually transmitted infections (STIs). Patients who buy a subsidized voucher from local drug shops and pharmacies are entitled to seek care for themselves and their partner at any of the contracted clinics. Clinics are reimbursed on a negotiated fee-for-service schedule.
The following report (“VSHD, 2007, Assessment of OBA Clinic Utilization”) is an evaluation of the OBA program’s first year impact on utilization at participating clinics (July 2006 to June 2007). The study, led by Berkeley graduate students Richard Lowe and Ben Bellows, was undertaken June to August 2007 and required an extensive review of thousands of handwritten lab and outpatient entries at OBA facilities. Records were kept differently at many of the clinics and,at several clinics, data were simply not available. However, we have information from 7 of the 16 clinics and they indicate a strong patient uptake and program improvement in the first year of OBA. One of the more dramatic findings is that the total number of patient visits at contracted clinics increased 226% in the first year of OBA compared to the year before OBA.
It does not appear that patients who have attended OBA clinics simply substituted the OBA voucher for their own out-of-pocket spending. Taking all seven clinics together, the number of non-OBA patients seeking STI treatment actually increased in the first year of OBA. One likely reason is that social marketing stimulated greater demand for STI treatment beyond the voucher-using population.
Program adherence also appears to be improving over the first year of OBA as the number of fully paid claims increased from 30% of all submitted claims in July/August 2006 to 70% of all claims in June 2007. Although it should be stressed that claims quality varied significantly between providers.
There is some concern about the quality of lab testing at participating clinics. Lab technicians could benefit from better on-site follow-up and incentives for high quality diagnoses. However, the percent of positive gonorrhea tests more than doubled, indicating increased awareness of this infection in the community and at provider clinics.
The report paints a detailed picture of the participating clinics in their first year of OBA and it is hoped that findings can be used for program improvement as the expansion is planned.
Our many thanks go to both Microcare and MSI who graciously assisted with our many requests for supplemental data and assistance reaching clinic providers. Many thanks as well to the KfW Development Bank and the Bixby Program at UC Berkeley for funding the research.