Automated Data Cleaning for Sleep Data from Wearables

Written by: Gabriel Mongefranco, mongefrg@umich.edu, Sr. Mobile Data Architect, University of Michigan.

The full article is available in the Eisenberg Family Depression Center’s Knowledge Base.


Sometimes, the best solution to a complex problem can be found in your typical everyday tools. That was the case for U-M’s Circadian Rhythm Research Laboratory, which was struggling to clean detailed sleep data from Fitbit devices and text messages used in a clinical study, necessitating two staff members to do tedious manual work for 4-5 hours per week… with just two participants! In collaboration with the Mobile Technologies Core at the Eisenberg Family Depression Center, the lab used a tool that everyone is familiar with to automate the parsing, cleaning, and analysis of data: Excel. Further automation to parse text (SMS) messages was done in Power Automate, another Microsoft 365 tool.

The Mobile Technologies Core developed a way to automatically apply an algorithm used by the Sleep & Circadian lab to determine the correct sleep and wakeup times, calculate many sleep and HRV metrics, and complete other tedious processes for cleaning Fitbit data. The Core also created an automation to parse electronic sleep diary entries that are sent from a text message to an Outlook mailbox. Given that most researchers would have access to Excel, the core used Excel and its Power Query component to ensure reproducibility and to make the code reusable. The sleep diary automation was done in Power Automate, a cloud product from Microsoft for low-code workflow automation.

This innovative yet simple solution tackles a problem that sleep researchers have tried to solve for many years. It has saved the Circadian Lab countless hours, allowing them to expand their studies and enroll many more participants concurrently, thus accelerating the pace of mental health and sleep research. This project has been selected for a presentation at the SLEEP 2024 National Scientific Conference, showcasing the latest and most impactful research in the field of sleep medicine.

Learn more and download the code at: michmed.org/sleepdata.

Leave a Reply

Your email address will not be published. Required fields are marked *