In this paper we study the working conditions of Community Health Workers in performing house calls within low-income, violent communities in order to understand the challenges in delivering primary care in developing countries. We conducted field studies in two primary healthcare clinics and telephone surveys for 5 months in 2017 within systematic sample of 1690 community health workers based on clinics distributed along the health regions of the city of Rio de Janeiro, Brazil. A number of 759 interviews were completed, approximately 50% of the sample, 86% men and 14% women. Most participants are 30-39 years old (35%), followed by 27% of 40-49 years old participants. Results show that exposure to territorial violence and environmental or health-related diseases significantly affects CHWs. Moreover, CHWs have to develop a significant set of skills to cope with aspects of the territory, and those skills are not present in their training. As community health workers work on the sharp end of the healthcare system, responsible for outreaching, community education, counseling, and social support, our study presents contributions to government and management levels on working conditions inside communities, constraints in assistance, and difficulties in implementing primary care policies.