Driftly — Let’s Find Your Next Home!
--
How Our Team Overcame Losing Members to Achieve Results
Project Background
My team’s project was to create a web application for nomads looking to move to a new city. The user can search and compare multiple cities through multiple features: job market, housing, and weather. Going into the project, I was a bit nervous at the shear amount of data I would need to collect, clean, and model as a data scientist. It was my first time working on an application which would take over a month to complete, and I knew time management and smart planning would be crucial to our success. Little did I know those challenges would not be the most difficult hurdle our team would face.
My Adapting Role
Initially, my role on the team was to query and clean the weather data. Another’s role was to get housing data, yet another to get job market data, and our last data scientist would set up our API. For querying the data, I chose to use the World Weather Organization (WWO) API, due to their free trial allowing up to 500 queries a day, which although a lot, proved to still be quite an obstacle. Below is my code for querying the WWO API:
frequency = 24 # time between each data row (hours) - max of 24
start_date = '01-JAN-2015'
end_date = '31-DEC-2019'
api_key = 'api_key_goes_here'
hist_weather_data = retrieve_hist_data(api_key, location_list, start_date, end_date, frequency,
location_label = False, export_csv = True, store_df = True)Cleaning the weather data into JSON proved to be quite an obstacle as well. I and a teammate tackled this by first splitting the data in two, one for summer and one for winter, then creating 2-D arrays and looping over them to check if each city-state was in our allowable dictionary, taking the averages of each feature, and then embedding the dictionaries inside each other so our WEB team had an easier time querying the data.
After we had queried and cleaned all the data and were ready to move it to the API, we lost our team member who was supposed to set up the API, and he had pushed none of his code to our GitHub repository. Seeing how crucial it was to the team’s success, I jumped right in to tackle the DS API deployment.
Feeling a bit behind, I attempted deploying to Heroku, which I’ve worked with previously. However, after running into multiple “pre-receive hook declined” errors when deploying, I reached out to the owner of the GitHub repository, who then instructed me I should use AWS and the Elastic Beanstalk CLI to deploy, which made me a bit nervous as it was entirely new territory.
It all began with the Docker installation, which installing on Windows Home was a nightmare. After spending an entire day trying to get Docker set up in Windows Home, I bit the bullet and purchased Windows Pro for $99, activating the “Containers” and “Hyper-V” features on my machine, allowing me to effortlessly install Docker.
Once docker was set up and I installed Elastic Beanstalk CLI through a GitHub Repository which would launch EB in a virtual environment, it was smooth sailing towards deployment. Deploying to AWS was a huge hurdle our team had to overcome, and I’m certainly glad I took on the challenge.
Project Features and Future
Once the API was set up with our endpoints containing job market, housing, and weather data from around 700 U.S. cities for , we could focus our efforts on making visualizations. Utilizing the Plotly-express library, I created multiple visualizations for these data, an example shown below.
Once our basic time-series visualizations were complete, I focused my attention on gathering data for new features. Our data team set our sights high for new features, and I decided to search for general health data, COVID-19 data, and restaurant data. Going forward, the team plans on gathering data on our discussed new features, and then moving on to modeling predictions for our user’s to make more informed decisions.
Closing Remarks
With a vision, determination, and a plan, anything is possible. Losing team members is certainly a challenge, but with the right mind-set and a plan of attack, a team can accomplish anything. By being willing to adapt to unforeseen circumstances and tackle unknown technologies, you only make yourself, and your team, stronger as a result. So jump right in! What’s stopping you?
