Data Wrangling

I have spent my whole week at the office data wrangling, and yes you are allowed to add as much southern twang to that phrase as you'd like. We [scientists] often use data wrangling to describe searching for, collecting, cleaning up, and working with data that we need for our own research, and it can be a pretty exhausting process. For our work on the Mississippi Sound model, we are working with four major data sets from four different agencies. Each data set and agency has unique codes, formatting, and even syntax in some cases. However, when we use coding software to extract information from these data sets, it is easier on us and the computer if we have simplified data and if we are using the bare minimum of the data. Here you can see the size of the data sets I have been working with, and the data set that is 22.6 million observations is the combination of the four agencies' data prior to my cleaning. You can see that my computer is not enjoying the size of the data and the complexity of the extraction processes, as the memory usage in the top of the picture is extremely high.

I have been enjoying the wrangling process, though, as tiring as it has been. This process allows me to do a lot of problem solving and refine my coding skills, and I have the authority at this stage to make decisions regarding the data. Since I am making decisions, though, I am keeping detailed notes on how I made the decisions and any evidence to support those decisions, since scientific research should be as transparent as possible. Additionally, once I submit this work, someone else on the team might disagree with a decision, so I will need to explain why I made a choice and, if necessary, edit the notes to reflect an alternative choice.

I am hoping to finish data wrangling tomorrow, but the final step is one I've never completed before, so I may need to work into next week to finish this process. Here's hoping, but taking more time to learn new coding skills and create reproducible results is worth it. Stay tuned for next week's blog! 

---