Free AI web copilot to create summaries, insights and extended knowledge, download it at here

3926

Abstract

egorical variables. If the number of categories are few compared to the total number values, it is better to use the category data type instead of object. It saves a great amount of memory depending on the data size.The following code will go over columns with object data type. If the number of categories are less than 5 percent of the total number of values, the data type of the column will be changed to category.<div id="0b82"><pre>cols = marketing.select_dtypes(include='object').columns for col in cols: ratio = len(marketing[col].value_counts()) / len(marketing) if ratio < 0.05: marketing[col] = marketing[col].astype('category')</pre></div>We have done three steps of data cleaning and manipulation. Depending on the task, the number of steps might be more.Let’s create a pipe that accomplish all these tasks.The pipe function takes functions as inputs. These functions need to take a dataframe as input and return a dataframe. Thus, we need to define functions for each task.<div id="4e5c"><pre>def drop_missing(df): thresh = len(df) * 0.6 df.dropna(axis=1, thresh=thresh, inplace=True) return df </pre></div><div id="3a6e"><pre>def remove_outliers(df, column_name): low = np.quantile(df[column_name], 0.05) high = np.quantile(df[column_name], 0.95) return df[df[column_name].between(low, high, inclusive=True)]</pre></div><div id="dcd6"><pre>def to_category(df): cols = df.select_dtypes(include='object').columns for col in cols: ratio = len(df[col].value_counts()) / len(df) if ratio < 0.05: df[col] = df[col].astype('category') return df</pre></div>You may argue that what the point is if we need to define functions. It does not seem like simplifying the workflow. You are right for one particular task but we need to think more generally. Consider you are doing the same operations many times. In such case, creating a pipe makes the process easier and also provides cleaner code.We have mentioned that the pipe function takes a function as input. If the function we pass to the pipe function has any arguments, we can pass it to the pipe function along with the function. It makes the pipe function even more efficient.For instance, the remove_outliers function takes a column name as argument. The function removes the outliers in that column.We can now create our pipe.</

Options

p><div id="df66"><pre>marketing_cleaned = (marketing. pipe(drop_missing). pipe(remove_outliers, 'Salary'). pipe(to_category))</pre></div>It looks neat and clean. We can add as many steps as needed. The only criterion is that the functions in the pipe should take a dataframe as argument and return a dataframe. Just like with the remove_outliers function, we can pass the arguments of the functions to the pipe function as an argument. This flexibility makes the pipes more useful.One important thing to mention is that the pipe function modifies the original dataframe. We should avoid changing the original dataset if possible.To overcome this issue, we can use a copy of the original dataframe in the pipe. Furthermore, we can add a step that makes a copy of the dataframe in the beginning of the pipe.<div id="95d3"><pre>def copy_df(df): return df.copy()</pre></div><div id="f711"><pre>marketing_cleaned = (marketing. pipe(copy_df). pipe(drop_missing). pipe(remove_outliers, 'Salary'). pipe(to_category))</pre></div>Our pipeline is complete now. Let’s compare the original dataframe with the cleaned to confirm it is working.<div id="b937"><pre>marketing.shape (1000,10)</pre></div><div id="e53b"><pre>marketing.dtypes Age object Gender object OwnHome object Married object Location object Salary int64 Children int64 History object Catalogs int64 AmountSpent int64 </pre></div><div id="3641"><pre>marketing_cleaned.dtypes (900,10)</pre></div><div id="e065"><pre>marketing_cleaned.dtypes Age category Gender category OwnHome category Married category Location category Salary int64 Children int64 History category Catalogs int64 AmountSpent int64</pre></div>The pipeline is working as expected.<h2 id="8b85">Conclusion</h2>The pipes provide cleaner and more maintainable syntax for data analysis. Another advantage is that they automatize the steps of data cleaning and manipulation.If you are doing the same operations over and over, you should definitely consider creating a pipeline.Thank you for reading. Please let me know if you have any feedback.</article></body>

My Secret Leveling-Up Weapon: Coffee Meetings

As I look back on the last 3 years of my Product career compared to the first few years, I see acceleration. While some of my growth has been due to skill development and working for a few great DC tech companies (shout out to Vistaprint Digital + Upside Travel!), there was something else behind this leveling-up.

Beginning about 2 years ago, I made a conscious decision that I wanted to get a coffee or a drink with someone (old or new) every week.

While this sounds simple, at first, it was very difficult for me. My mornings are my most productive time, how could I possibly sacrifice 30–45 minutes of that time to simply talk with someone!?

However, I quickly recognized the power of this simple habit and over the past 2 years, it’s been my secret weapon.

As I look back at my calendar, I’ve met 112 times for coffee or drinks (!) over the last 2 years with people old and new.

5 Things I’ve Learned From My Coffees

1. You never know who will help you and you never know when

As organized as you try to be, you simply can’t plan for everything. You don’t know all of the opportunities that exist out there in the world and you won’t without talking to people!

Coffees will allow you to express your interests and goals for your personal life and your professional career. Often, others can help you achieve those faster.

2. It’s fun and rewarding helping others

For the past 9 months, I’ve been meeting monthly with the founders of Stratus, a financial planning platform for Millennials, by Millennials.

Their energy and excitement is infectious and it’s been immensely rewarding helping them with at least a few (hopefully decent) ideas on how they can continue to level-up themselves and their company.

3. Challenging your brain is healthy

As you set up your own coffees, make sure not to restrict yourself to people in your direct field or sub-industry.

I’ve found some of my most interesting conversations have been with people at companies that have no connection whatsoever with mine.

These conversations activate parts of your brain that your day-to-day conversations and job may not.

4. DC Tech/Tech is a small world

As you meet with more and more people for coffee, you’ll learn how connected the world is. I’m constantly surprised when new people I meet have worked or met with people I’ve built relationships with (although LinkedIn sometimes spoils the surprise…)

Because of this “graph” of connections you’ll build with coffees, each subsequent connection is likely to strengthen your network by more than the previous one did.

5. Your productivity will actually get a boost from other’s ideas

I saved this one for last because it completely refuted my initial worry of these coffees being time poorly spent. Rather, the truth was completely the opposite: These coffees have increased my productivity.

Through the course of conversation, I’d share some of the hard problems I was working on and, often, I would get a completely new perspective from the person I was talking with.

They see my company, my challenges, with different eyes and I’ve been amazed (and very thankful) for the ideas they’ve given me to attack these problems.

What’s Your Leveling-Up Secret Weapon?

Share it in the comments! Or share it with me on Twitter: @amitch5903

My Secret Leveling-Up Weapon: Coffee Meetings

5 Things I’ve Learned From My Coffees

1. You never know who will help you and you never know when

2. It’s fun and rewarding helping others

3. Challenging your brain is healthy

4. DC Tech/Tech is a small world

5. Your productivity will actually get a boost from other’s ideas

What’s Your Leveling-Up Secret Weapon?

More From Alex Mitchell

Top 10 Chinese Startups to Watch in 2018

The focus of startup power is shifting west (or east, depending where you live) and it has been for quite some time…

10 Bold Technology Predictions for the Next 10 Years

NOTE: These predictions are over 2 years old! Author: Alex Mitchell

Top 10 Books for Product Managers of All Levels

Top 10 Books for Product Managers — Broken down by PM experience level. Author: Alex Mitchell

The Death of Thinking

When's the last time you let your brain rest? Author: Alex Mitchell

An Open (Financial) Letter to Millennials

A Generation of Financials Wins and Financial Fails. Author: Alex Mitchell