Tag: Data Science

Coding a function, an app, or a website is an act of creation no different from writing a short story or painting a picture. From very simple tools we create something where there never was something. Painters have pigments. Writers have words. Coders have data types.

Data types govern just about every aspect of coding. Each type represents a specific kind of thing stored in a computer’s memory, and has different ways of being used in writing code. They also range in complexity from the humble integer to the sophisticated dictionary. This article will lay out the basic data types, using Python as the base language, and will also discuss some more advanced data types that are relevant to data analysts and data scientists.

Computers, even with the wondrous innovations of generative AI in the last few years, are still— just as their name suggests—calculators. The earliest computers were people who performed simple and complex calculations faster than the average person. (All the women who aided Alan Turing in cracking codes at Bletchley Park officially held the title of computers.)

Simple Numbers

It’s appropriate, then, that the first data type we discuss is the humble integer. These are the whole numbers 0 to 9, in their millions of combinations from 0 to 999,999,999,999 and beyond, including negative numbers. Different programming languages handle integers differently.

Python supports integers, usually denoted as int, as “arbitrary precision.” This means it can hold as many places as the computer has memory for. Java, on the other hand, recognizes int as the set of 32-bit integers ranging from -2,147,483,648 to 2,147,483,647. (A bit is the smallest unit of computer information, representing the logical state of on or off, 1 or 0. A 32-bit system can store 2^32 different values.)

The next step up the complexity ladder brings us to floating point numbers, or more commonly, floats. Floating point numbers approximate real numbers that include decimalized fractions from 0.0 to 99.999 and so on, again including negative numbers and repeating to the limits of a computer’s memory. The level of precision (number of decimal places) is constrained by a computer’s memory, but Python implements floats as 64-bit numbers.

With these two numeric data types, we can perform calculations. All the arithmetic operations are available with Python, along with exponentiation, rounding, and modulo operation. (Denoted %, modulo returns the remainder of division. For example, 3 % 2 returns 1.) It is also possible to convert a float to an int, and vice versa.

From Numbers to Letters and Words

Although “data” and “numbers” are practically synonymous in popular understanding, data types also exist for letters and words. These are called strings or string literals. 

“Abcdefg” is a string.

So is:

“When in the course of human events it becomes necessary for one people to dissolve the political bands which have connected them with another.”

For that matter so is:

“Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur vitae lobortis enim.”

Computer languages don’t care whether the string in question is a letter, a word, a sentence, or complete gobbledygook. They only care, really, whether the thing being stored is a number or not a number, and specifically whether it is a number the computer can perform arithmetic on. 

An aside: to humans, “1234” and 1234 are roughly the same. We humans can infer from context whether something is a calculable number, like on a receipt, or a non-calculable number, like a ZIP code. “1234” is a string. 1234 is an integer. We call those quotation marks “delimiters” and they serve the same role for the computer that context serves for us humans.  

Strings come with their own methods that allow us to do any number of manipulations. We can capitalize words, reverse them, and slice them up in a multitude of ways. Just as with floats and ints, it’s also possible to tell the computer to treat a number as if it were a string and do those same manipulations on it. This is handy when, for example, you encounter a street address in data analysis, or if you need to clean up financial numbers that someone left a currency mark on.

Encoding numbers and letters is a valuable tool, but we also need ways to check that things are what they claim to be. This is the role of the boolean data type, usually shortened to bool. This data type takes on one of two values: True or False, although it is also often represented as 1 or 0, respectively. The bool also gives rise to the boolean operators (<,>, !=, ==) which evaluate the truth value of some expression, like 2 < 3 (True) or “Thomas” == “Tom” (False).

Ints, floats, and strings are the most basic data types that you will find across computing languages. And they are very useful for storing singular things: a number, a word. (Strictly speaking, an entire library could be encoded as a single string.)

These individual data types are great if we only want to encode simple, individual things, or make one set of calculations. But the real value of (digital) computers is their ability to do lots of calculations very very quickly. So we need some additional data types to collect the simpler ones into groups.

From Individuals to Collections

The first of these collection data types is the list. This is simply an unordered collection of objects of any data type. In Python, lists are set off by brackets ([]). (“Ordered” means that the elements are arranged any old way, and not sorted high-to-low or low-to-high.)

For example, the below is a list:

[1, 3.7, 2, 3.4, 4, 6.74, 5.0]

 This is also a list:

[“John”, “Mary”, “Sue”, “Alphonse”]

Even this a list:

[1, “John”, 2.2, “Mary”, 3] 

It’s important to note that within a list, each element (or item) of the list still operates according to the rules of its data type. But know that the list as a whole also has its own methods. So it’s possible to remove elements from a list, add things to a list, sort the list, as well as do any of the manipulations the individual data types support on the appropriate elements (e.g., calculations on a float or an integer, capitalize a string).

The last and probably most complicated of the basic data types is the dictionary, usually abbreviated dict. Some languages refer to this as a map. Dictionaries are unordered collections of key-value pairs, set off by braces ({}). Individual key-value pairs inside a dictionary are set off with a comma. They enable a program to use one value to get another value. So, for example, a dictionary for a financial application might contain a stock ticker and that ticker’s last closing price,like so:

{“AAPL”: 167.83,

“GOOG”: 152.62,

“META”: 485.58}

In this example, “AAPL” is the key, 167.83 is the value. A program that needed the price of Apple’s stock could then get that value by calling the dictionary key “AAPL.” And as with lists, the individual items of the dictionary, both keys and values, retain the attributes of their own data types.

These pieces form the basics of data types in just about every major scripting language in use today. With these (relatively) simple tools you can code up anything from the simplest function to a full spreadsheet program or a Generative Pre-trained Transformer (GPT). 

Extending Data Types into Data Analysis

If we want to extend the data types that we have into even more complicated forms, we can get out of basic Python and into libraries like Numpy and Pandas. These bring additional data types that expand Python’s basic capabilities into more robust data analysis and linear algebra, two essential functions for machine learning and AI.

First we can look at the Numpy array. This handy data type allows us to set up matrices, though they really just look like lists, or lists of lists. However, they are less memory intensive than lists, and allow more advanced calculation. They are therefore much better when working with large datasets.

If we combine a bunch of arrays, we wind up with a Pandas DataFrame. For a data analyst or machine learning engineer working in Python this is probably your most common tool. It can hold and handle all the other data types we have discussed. The Pandas DataFrame is, in effect, a more powerful and efficient version of the Excel spreadsheet. It can handle all the calculations that you need for exploratory data analysis and data visualization.

Data types in Python, or any programming language, form the basic manipulable unit. Everything a language uses to store information for future use is a data type of some kind, and each type has specific things it can store, and rules governing what it can do.

Learn Data Types and Structures in Flatiron’s Data Science Bootcamp

Ready to gain a deeper understanding of data types and structures to develop real-world data science skills? Our Data Science Bootcamp can help take you from basic concepts to in-demand applications. Learn how to transform data into actionable insights—all in a focused and immersive program. Apply today and launch your data science career!

Predictive modeling is a process in data science that forecasts future outcomes based on historical data and statistical algorithms. It involves building mathematical models that learn patterns from past data to make predictions about unknown or future events. These models analyze various variables or features to identify relationships and correlations, which are then used to generate predictions. Well over half of the Flatiron School’s Data Science Bootcamp program involves learning about various predictive models.

Applications of Predictive Modeling

One common application of predictive modeling is in finance, where it helps forecast stock prices, predict market trends, and assess credit risk. In marketing, predictive modeling helps companies target potential customers more effectively by predicting consumer behavior and preferences. For example, companies can use customer data to predict which products a customer is likely to purchase next or which marketing campaigns will yield the highest return on investment.
Healthcare is another field that uses predictive modeling. Predictive modeling plays a vital role in identifying patients at risk of developing certain diseases. It also helps improve treatment outcomes and optimize resource allocation. By analyzing patient data, such as demographics, medical history, and lifestyle factors, healthcare providers can predict potential health issues and intervene early to prevent or manage them effectively. 

Manufacturing and logistics widely use predictive modeling to optimize production processes, predict equipment failures, and minimize downtime. By analyzing data from sensors and machinery, manufacturers can anticipate maintenance needs and schedule repairs before breakdowns occur, reducing costs and improving efficiency.

Overall, predictive modeling has diverse applications across various industries, helping businesses and organizations make more informed decisions, anticipate future events, and gain a competitive advantage in the marketplace. Its ability to harness the power of data to forecast outcomes makes it a valuable tool for driving innovation and driving success in today’s data-driven world.

The Steps for Building a Predictive Model

Below is a step-by-step guide to building a simple predictive machine learning model using Python pseudocode. Python is a versatile, high-level programming language known for its simplicity and readability, making it an excellent choice for beginners and experts alike. Its extensive range of libraries and frameworks, particularly in fields such as data science, machine learning, artificial intelligence, and scientific computing, has solidified its place as a cornerstone in the data science community. While Flatiron School Data Science students learn other technical skills and languages, such as dashboards and SQL, the primary language that students learn and use is Python.

Step 1

In Step 1 below (in the gray box), Python libraries are imported. A Python library is a collection of functions and methods that allows you to perform many actions without writing your code. It is a reusable chunk of code that you can use by importing it into your program, saving time and effort in coding from scratch. Libraries in Python cover a vast range of programming needs, including data manipulation, visualization, machine learning, network automation, web development, and much more. 

The two most widely used Python libraries are NumPy and pandas. The former adds support for large, multi-dimensional arrays and matrices, along with a large collection of high-level functions to operate on these arrays. The latter is a high-level data manipulation tool built on top of the Python programming language. It is most well-suited for structured data operations and manipulations, akin to SQL but in Python. 

The third imported Python library is scikit-learn, which is an open-source machine learning library that proves a wide range of supervised and unsupervised learning algorithms. It is built on NumPy, SciPy, and Matplotlib, offering tools for statistical modeling, including classification, regression, clustering, and dimensionality reduction. In data science, scikit-learn is extensively used for developing predictive models and conducting data analysis.  Its simplicity, efficiency, and easy for integration with other Python libraries makes it an essential tool for machine learning practitioners and researchers.

Step 2

Now that the libraries have been imported in Step 1, the data needs to be brought in—as can be seen in Step 2. Since we’re considering predictive modeling, we’ll use the feature variables to predict the target variable. 

In a dataset for a predictive model, feature variables (also known as predictors or independent variables) are the input variables that are used to predict the outcome. They represent the attributes or characteristics that help the model learn patterns to make predictions. For example: In a dataset for predicting house prices, feature variables might include:

• Square_Feet: The size of the house in square feet
• Number_of_Bedrooms: The number of bedrooms in the house
• Age_of_House: The age of the house in years
• Location_Rating: A rating representing the desirability of the house’s location

The target variable (also known as the dependent variable) is the output variable that the model is trying to predict. Continuing with our housing example, the target variable would be:

• House_Price: The price of the house

Thus, In this scenario, the model learns from the feature variables (Square_Feet, Number_of_Bedrooms, Age_of_House, Location_Rating) to accurately predict the target variable (House_Price).

Step 3

Note, We split the dataset into training and test sets in Step 2. We did this to evaluate the predictive model’s performance on unseen data, ensuring it can generalize well beyond the data it was trained on. This split helps identify and mitigate overfitting, where a model performs well on its training data but poorly on new, unseen data, by providing a realistic assessment of how the model is likely to perform in real-world scenarios.

Now comes the key moment in Step 3, where we use our statistical learning model. In this case, we’re using multiple linear regression, which is an extension of simple linear regression. It is designed to predict an outcome based on multiple independent variables, and fits a linear equation to the observed data where the target variable is modeled as a linear combination of two or more feature variables, incorporating a separate coefficient (slope) for each independent variable plus an intercept. This approach allows for the examination of how various feature variables simultaneously affect the outcome. It provides a more comprehensive analysis of the factors influencing the dependent variable.

Step 4

In Step 4, we evaluate the model to find out how well it fits the data. There are a myriad of metrics that one can use to evaluate predictive learning models. In the pseudocode below, we use the MSE, or the mean squared error.

The MSE is a commonly used metric to evaluate the performance of a regression model. It measures the average squared difference between the actual values (observed values) and the predicted values generated by the model. Mathematically, it is calculated by taking the average of the squared differences between each predicted value and its corresponding actual value. The formula for MSE is:

In this formula, 

• n is the number of observations
• y_i represents the actual value of the dependent variable for the i^th observation
• ŷ_i represents the predicted value of the dependent variable for the i^th observation

A lower MSE value indicates that the model’s predictions are closer to the actual values, suggesting a better fit of the model to the data. Conversely, a higher MSE value indicates that the model’s predictions are further away from the actual values, indicating poorer performance.

Step 5

At this point, one usually would want to tune (i.e., improve on the model). But for this introductory explanation, Step 5 will be to use our model to make predictions.

Summary of Predictive Modeling

The pseudocode in Steps 1 through 5 shows the basic steps involved in building a simple predictive machine learning model using Python. You can replace placeholders like `’your_dataset.csv’`, `’feature1’`, `’feature2’`, etc., with actual data and feature names in your dataset. Similarly, you can replace `’target_variable’` with the name of the target variable you are trying to predict. Additionally, you can experiment with different models, preprocessing techniques, and evaluation metrics to improve the model’s performance.

 Predictive modeling in data science involves using statistical algorithms and machine learning techniques to build models that predict future outcomes or behaviors based on historical data. It encompasses various steps, including data preprocessing, feature selection, model training, evaluation, and deployment. Predictive modeling is widely applied across industries for tasks such as forecasting, classification, regression, anomaly detection, and recommendation systems. Its goal is to extract valuable insights from data to make informed decisions, optimize processes, and drive business outcomes. 

Effective predictive modeling requires a combination of domain knowledge, data understanding, feature engineering, model selection, and continuous iteration to refine and improve model performance over time that lead to actionable insights.

Learn About Predictive Modeling (and More) in Flatiron’s Data Science Bootcamp

Forge a career path in data science in as little as 15 weeks by attending Flatiron’s Data Science Bootcamp. Full-time and part-time opportunities await, and potential career paths the field holds include ones in data analysis, AI engineering, and business intelligence analysis. Apply today or schedule a call with our Admissions team to learn more!

The data visualizations in this blog post—which are tied to best camping locales for viewing the 2024 total solar eclipse—are not optimized for a mobile screen. For the best viewing experience, please read on a desktop or tablet.

I once read about a married couple who annually plan vacations to travel the world in pursuit of solar eclipses. They spoke about how, regardless of their location, food preferences, or language abilities, they always managed to share a moment of awe with whoever stood near them as they gazed up at the hidden sun.

While I can’t speak to the experience of viewing an eclipse abroad, I did travel to the path of totality for a solar eclipse in 2017, and I can confirm the feeling of awe and the sense of shared experience with strangers. Chasing eclipses around the world isn’t something I can easily squeeze into my life, but when an eclipse is nearby, I make an effort to go see it. 

On April 8, 2024, a solar eclipse will pass over the United States. It will cast the moon’s shadow over the states of Texas, Oklahoma, Arkansas, a sliver of Tennessee, Missouri, Kentucky, Illinois, Indiana, Ohio, Michigan, Pennsylvania, New York, Vermont, New Hampshire, and Maine.

As I’ve been making plans for this eclipse, I’ve been using NASA’s data tool for exploring where in the path of totality I might view the eclipse from. This tool is exceptional and includes the time the eclipse will begin by location, a simulation of the eclipse’s journey, and a weather forecast. I have several friends who have traveled to see an eclipse only to be met with a gray cloudy sky, so keeping an eye on the weather is very important. 

But, as I was using this tool, I found myself wanting to know what camping options fall within the path of totality for the 2024 total solar eclipse. I, like many, prefer to make a small camping vacation out of the experience, and the location of parks is information not provided by NASA’s data tool. So I found a dataset detailing the location of 57,000 parks in the United States, isolated the parks that fall within the path of totality, and plotted them.

Below is the final visualization, with an added tooltip for viewing the details of the park. For those interested in the code for this project, check out the links to the data and my data manipulation and visualization code.

Happy eclipsing, everyone!

Gain Data Science Skills at Flatiron School

Learn the data science skills that employers are after in as little as 15 weeks at Flatiron School. Our Data Science Bootcamp offers part-time and full-time enrollment opportunities for both onsite and online learning. Apply today or download the syllabus for more information on what you can learn. Interested in seeing the types of projects our students complete upon graduation? Attend our Final Project Showcase.

Natural Language Processing (NLP) is a subfield of artificial intelligence (AI) and linguistics that focuses on the interaction between computers and human languages. It encompasses a range of techniques aimed at enabling computers to understand, interpret, and generate human language in a manner that is both meaningful and contextually relevant. 

In data science, NLP plays a pivotal role in extracting insights from vast amounts of textual data. Through techniques such as text classification, sentiment analysis, named entity recognition, and language translation, NLP empowers data scientists to analyze and derive actionable insights from unstructured text data sources such as social media, customer reviews, emails, and news articles. By harnessing the power of NLP, data scientists can uncover patterns, trends, and sentiments within textual data. This enables organizations to make data-driven decisions and enhance various aspects of their operations, from customer service to product development and market analysis.

NLP is fundamental to generative AI models like ChatGPT. Natural language processing techniques enable these models to understand and generate human-like text, making them capable of engaging in meaningful conversations with users. NLP provides the framework for tasks such as language understanding, sentiment analysis, summarization, and language generation. All are essential components of generative AI systems.

Applications of NLP

NLP techniques are extensively utilized in text classification and sentiment analysis, offering a wide array of applications across various industries.

Text Classification

NLP enables automatic categorization of textual data into predefined classes or categories. Applications include:

• Spam detection: NLP algorithms can classify emails or messages as spam or non-spam, helping users manage their inbox efficiently.
• Topic classification: NLP models categorize news articles, research papers, or social media posts into relevant topics, aiding in content organization and information retrieval.
• Language identification: NLP models can identify the language of a given text, which is useful for multilingual platforms and content analysis.

Sentiment Analysis

NLP techniques are employed to analyze the sentiment or emotion expressed in textual data, providing valuable insights for decision-making. Applications include:

• Brand monitoring: Sentiment analysis helps businesses monitor online conversations about their brand, products, or services. This enable them to gauge public perception and address potential issues promptly.
• Customer feedback analysis: NLP algorithms analyze customer reviews, surveys, and social media comments. Then, they work to understand customer sentiment towards specific products or services, facilitating product improvement and customer satisfaction.
• Market research: Sentiment analysis aids in analyzing public opinion and sentiment towards specific topics or events, providing valuable insights for market research, trend analysis, and forecasting.
• Social media analysis: NLP techniques are utilized to analyze sentiment in social media posts, tweets, and comments, enabling businesses to track customer sentiment, identify influencers, and engage with their audience effectively.

NLP Techniques

NLP encompasses a variety of techniques designed to enable computers to understand and process human languages. Two fundamental techniques in NLP are tokenization and stemming, which play crucial roles in text preprocessing and analysis.

Tokenization

Tokenization is the process of breaking down a piece of text into smaller units (called tokens). These tokens can be words, phrases, or other meaningful elements. The primary goal of tokenization is to divide the text into individual units for further analysis. There are different tokenization strategies, including:

• Word tokenization divides the text into words or word-like units. For example, the sentence “The quick brown fox jumps over the lazy dog” would be tokenized into [“The”, “quick”, “brown”, “fox”, “jumps”, “over”, “the”, “lazy”, “dog”].
• Sentence tokenization splits the text into sentences. For instance, the paragraph “Natural Language Processing (NLP) is a fascinating field. It involves analyzing and understanding human language” would be tokenized into [“Natural Language Processing (NLP) is a fascinating field.”, “It involves analyzing and understanding human language.”].

Stemming 

Stemming is the process of reducing words to their root or base form, known as the stem. The goal of stemming is to normalize words so that different forms of the same word are treated as identical. Stemming algorithms apply heuristic rules to remove suffixes and prefixes from words. For example:

• Original Word: “Running”
• Stemmed Word: “Run”
• Original Word: “Jumped”
• Stemmed Word: “Jump”

Stemming is particularly useful in tasks such as text mining, information retrieval, and search engines. Why? Because stemming reduces words to their base forms can improve indexing and retrieval accuracy. Both tokenization and stemming are essential preprocessing steps in many NLP applications, including text classification, sentiment analysis, machine translation, and information retrieval. These techniques help transform raw textual data into a format suitable for further analysis and modeling, facilitating the extraction of meaningful insights from large volumes of text data.

Natural Language Processing (NLP) Resources

Given the comprehensive overview of NLP’s applications and techniques, several resources can significantly aid in deepening your understanding and skills in this field. Books such as Natural Language Processing in Action by Lane, Howard, and Hapke, and Speech and Language Processing by Jurafsky and Martin provide foundational knowledge and practical examples. These texts are excellent for understanding the underlying principles of NLP. They’re also great for reference on specific topics like tokenization, stemming, and machine learning models used in NLP. Regardless of which NLP resource is used, the key is to practice coding the models.

Learn More about NLP in Flatiron’s Data Science Bootcamp

Flatiron School’s Data Science Bootcamp teaches natural language processing, data analysis and engineering, machine learning fundamentals and much more. Full-time and part-time enrollment opportunities await! Apply today or schedule a call with Admissions to learn more about what Flatiron can do for you and your career.