Building a City Metrics Comparison Tool as a Coding Bootcamp Project — My Story
A Brief Introduction
My Journey as a Full Stack Web Developer began roughly 8 months ago, just when the lockdown was taking effect across the globe, so it actually was the perfect timing for me since I had just enrolled at Lambda School; a 9-month Remote Coding Bootcamp.
This is the story of my final project at Lambda School, which involves creating a product as a group for an actual stakeholder. In our case, the stakeholder is a Lambda Staff Member and one of the people in charge of what we call ‘Lambda Labs’.
Lambda Labs is the name of the section dedicated to our final project, which I’m completing today. This product will be in essence the culmination of 8 months of hard work from the first line of code I ever wrote to the realization of a fully fleshed out project that meets the stakeholder’s vision to the best extent possible.
Have you ever wondered what the average rent for single or double bedroom apartments are in a given city? There is probably a website that will give you that information. How about the weather? There will probably be another website that will tell you about that too, and also perhaps another website that will list the job landscape in that city. The sheer amount of data one can find online nowadays can be sparse and become overwhelming when wanting to know several metrics of a specific city, and even more when trying to compare that data to another city’s data.
This is where Driftly comes in. This tool is an accessible platform to compare various city data, such as seasonal weather patterns, job and unemployment history, and housing and rent prices. The best feature is that it allows the user to choose up to 3 cities to compare in real time and render graphs that represent all this data in a very visually pleasing manner that permits fast and easy comparisons between each city at a quick glance. If the city you are comparing or gathering data from meets your expectations, you might be looking at your next destination.
This project has been constantly testing my abilities as a full stack software developer since day one. We had to face challenges such as scraping all the data from reliable sources and APIs, and figuring out ways to render and display this data into graphs. We had to learn how to apply several advanced search filters at once and how we would connect and store all the data in the database.
Even more concerning is the fact that we were 9 people in the team and now we are down to 4. The initial plans and work distribution we had has been totally shifting week by week. We would all be focusing on an area of the project but we have had to adapt quickly and thankfully all of us remaining are able to fill in the actual ‘full stack’ of the project and are equally comfortable working in the Frontend or the Backend. Having to readjust to this situation has been hard at times, because this has meant wondering if we will be able to meet our stakeholder’s expectations in the time we have been given.
All in all, I do believe that losing people along the way and asking ourselves if we will meet the deadlines and client expectations are certainly things we will be encountering constantly once we get out to the ‘real’ world and outside this ‘safe’ zone that is Lambda School.
Software Development 101: Facing the Technical Challenges
Whether you are just beginning in the world of software development or you are a hardened veteran, one thing that is common across the board is that you will constantly encounter challenges that require sharp problem-solving skills to overcome. The true value of any software developer is to quickly adapt to these challenges and to be able to overcome them in an efficient way.
This project has been no different. Coming out of the first product roadmap and stakeholder meeting, I asked myself how we were going to implement several of the features that were talked about. One of the constraints we had coming into this project is that we had to use the scaffolding given to us, just like in the real world it is most likely that you will inherit and contribute to an already established code base.
Implementing Okta
One of the requirements that was in the initial scaffolding was using Okta as a user validation and authentication tool. I was initially in charge of the Backend portion of the project (more details on why that ended up not being the case in just a second), so the first challenge was to figure out how to use a third party service and implement it to the project.
I had used packages such as Yup for input validation and had created user registration / login forms before, generating and passing a token using JWT and hashing user passwords to store those hashed values in a database. The problem with this flow is that it is very prone to information being leaked and hackers could potentially extract sensitive data, especially as a developer who has used this technology only a few times before.
This is why user authentication should most times be handled by a third party, such as Okta, as it is harder for something to go wrong and if it does, they take full responsibility. I recommend this approach to any software developer that has to deal with user authentication. Thankfully it was very straightforward to set up an Okta user login feature using Node.js and Express, as I found a very helpful video on YouTube. A link to the video can be found here.
Parsing the Data
Another problem we encountered relates to how our initial workload was distributed. A few of the members of our team were going to focus on scraping all the data we needed to render for our project, and then I would be in charge of parsing their data (in the form of huge JSON objects) and making it into easily accessible JavaScript objects to be stored in the backend database using PostgreSQL / pgAdmin and Knex. Then these objects containing all the necessary data would be safely stored there and I would provide all the endpoints for the frontend team to interact with.
I Made a few parsing functions to achieve this goal, parsing in jobs and housing data directly from the Data Science API built for this project. These functions would return formatted JavaScript objects that could easily be added to the Knex seed data and stored in the PostgreSQL database.
After hitting the DS API, I parsed the data. City names were coming as ‘City, State’ and I removed the state from the city to store them in separate arrays to be used later.
Then I proceeded to transform the strings into integers, as the housing data we use are numbers. Once I got all the data separated into three individual arrays, I simply made a for loop to cycle through each element in the array and map it out to the housingData object below.
This is the end result and what I eventually used to seed the data in the Knex table and storing it in our database:
I ended up doing a very similar parsing function to grab job data and parse it accordingly to our idea of a JavaScript object as well. I will not show any code snippets from that because it is almost the same as the code above.
Adapting to a Changing Group Environment
Even after the hard work I invested building these functions, we had to totally adapt when 4 of our team members dropped out of the team for one reason or the other (we were 9 initially and 5 currently). I had to move my efforts to the frontend of the project, and we ended up deciding that the best thing to do moving forward would be to hit the Data Science API directly instead of having our own API and database, as this process took too much time and I needed to focus on other aspects of the project now.
The issue moving forward is that we have to make some complex axios calls in the frontend to exactly grab the data we need for any given metric we want to display. We also had to figure out how to grab object data that is in JSON format and has the keys as strings with spacing between each word. We found a way eventually to make calls in functions such as the one below, using square bracket notation:
Even though there will have to be many functions accommodating each and every different metric we will want to implement in the project, in the long run it is a better alternative than having me parsing and storing all the data in the database, since it is a lengthy process and I need to be focusing on the frontend aspect of the project now.
Displaying the Graphs
The final challenge we faced as a team was to translate the graph plots from JSON format to a component that could actually render those visualizations in React. We ended up using React-Plotly. This took several days (even a week maybe?) to do well, since none of us in the team had ever used this package.
For now, we are just rendering dummy data but we will update it soon with all the real data for each city stored in the DS API.
These have been all the technical challenges for now, and an insight on how we have overcome these challenges. Keep in mind, however, that this project is still a work in progress. I will update soon.
setCurrentState();
The current state of the Product
Given that we had a lot of restructuring and readjusting going on within the team for the past few weeks, there are still several bugs that we have to figure out in the front end. This is an example:
Currently, state is not storing the correct cities in the comparison tool, it constantly replaces the second and third cities with the first city added to it in both images and location name.
Having said that, there are several features that are working fine right now.
- All the cities we are grabbing from the DS API are correctly being displayed on the application.
- Dynamic button with the ability to ‘’add’ the city into another component, where the data comparison is made, and when in the comparison tool, the button can ‘remove’ it from the component.
- Main data being shown in the cards, for now it is housing, total manufacturing, and weather in F.
- The ‘more info’ button takes the user to another route where they are able to see more city data in detail, including graphs (even though it is dummy data for now).
- Search bar is working, cities can also be stored from the search bar.
As I mentioned earlier, this project has been through a lot of problems, however, there is still one month left to go and the ones of us who remain in the group want to put all our efforts into this application.
Update
As of October 23, the project has been completed. Features include:
- State has been fixed. City cards correctly add to the comparison container with their respective images.
- Over 650 US cities have been added to the application.
- Search bar has been updated.
- Added modals when city cards are clicked, showing different weather, housing, jobs and COVID-19 data.
- When the user adds the desired cities, click on ‘Compare Cities’ and this will take the user to a route full of interactive graph visualizations for the data mentioned above. Up to 3 cities can be compared at a time.
- Alternatively, the user can click on ‘Advanced Search’ to filter out cities by population, temperature, job markets, housing prices, and COVID-19 positive cases.
- Link to the deployed site can be found here.
Future Challenges
The biggest challenges I foresee moving forward are working with several graphs at the same time, how to display them correctly, and how to overlap the data. So the actual city metrics comparison layout in one big graph for users to extrapolate data and information at a quick glance. How will it be rendered on the page? What is the best possible way to do so? How do we get all the data from the API and parse it into good-looking graphs? These are some of the questions I keep asking myself.
Other challenges may include how to properly introduce advanced search filters to our application. What metrics will the user be able to filter and search for? Will I use checkboxes or scrollbars? What types of inputs are best to achieve this goal? The idea is to have it be the most efficient without being too convoluted, as well as very user friendly, so anyone can use the application with ease.
Update
Happy to say we have solved all the challenges above!
- Graphs compare up to 3 cities at once :
- Advanced Search Filters:
Preparing for the Real World
Despite facing many setbacks and losses since the start of the project, I think this situation has resemblance to what can and will occur in the real world. Adapting constantly to a changing team environment and having the ability to take on many roles, be it more design oriented or working with React components or making a database to store all the application’s data, this is a key feature of any good Full Stack Software Developer.
Another good skill I’m learning is to adapt and meet the stakeholder’s vision, deadlines, and first and foremost, their expectations towards the product. In the real world there are all these factors coming into place and it is very important to begin having a feel for them as soon as possible.
Finally, another way this project is helping me polish my skills and advance my career is the ability to use technology I hadn’t used before. From Okta to React-Plotly, and parsing all sorts of data from a huge API, all this has helped me become a better researcher on unknown technologies and has given me the ability and confidence to apply them to my own projects.
