Kilkari is a mobile health (mHealth) program operated by ARMMAN, a non-profit organization based in India, which uses IVR technology to deliver time-sensitive audio information to pregnant women and mothers to reduce maternal and child mortality rates. To improve beneficiary retention, we present a preliminary study aimed at targeting interventions for beneficiaries with low listenership. We model this problem as a time series prediction task and assess the efficacy of different machine learning (ML) models. Our experiments reveal that ML models can improve the prediction of low listenership from 5% (as obtained through random selection) to 25%. However, more sophisticated ML algorithms do not perform any better than logistic regression, at least based on the inputs and context as discussed in this paper,. These results highlight the need for novel machine learning research to help better target ARMMAN’s limited intervention resources.