What I Learned from Surviving a Coaching Plan in Silicon Valley

Failures made me who I am today.

So… Where do I begin?

A couple of years ago, I was this young and passionate data engineer working at a Fortune 500 retail company in the Midwest US. My manager trusted me with the most innovative projects that other senior engineers couldn’t deliver. I got promoted to Senior only after one year. I was proud of my work and I felt I could conquer anything.

But still, I believed I wasn’t challenged enough at work. I wanted more.

“You should come to the Bay Area! This is the center of the universe!”

After a friend of mine repeatedly tried to sell me a brighter future in Silicon Valley, I decided to make a move. I got a job offer that was two times my salary at the time. Oh man, I was so excited about what was ahead of me.

Six months after I started at my shiny new job in Silicon Valley, I got put on a coaching plan.

Womp womp.

You must be wondering… “What went wrong, Xinran?”

How did I fall from “awesome” to “awful” so quickly?

What Went Wrong

Expectation Management

“When we hired you, we expected you to be taking care of everything in our data warehouse. But you didn’t deliver that.”

My manager said this during our first conversation about coaching. That was also the first time we ever explicitly talked about expectations.

I never even heard of the term “expectation management” back then, but now I have learned its significance in the most painful way possible.

Until this day, I still wonder if things would have been different if my manager and I had aligned on expectations earlier:

• What if we talked about what success should look like at 90-day or 6-month marks?
• What if we had discussed the challenges I might face moving from a large structured data organization to being the first data engineer on a nano-size startup data team?

I will never know how the other Xinran’s story would have gone in the parallel universe. But I know I will never make the same mistake again.

Onboarding

What went wrong during onboarding?

There was not much onboarding. I had to work on delivery on Week 2.

I took many things at my previous team for granted, and one of their most valuable gifts is how everyone generously offered their time to help me learn. Unfortunately, at the new company, people were too swamped with their own delivery to help me out.

To be honest, I felt quite discouraged and lost after the first few 1-on-1 meetings with my new colleagues. One of them sent a couple of links to me on Slack and then left the meeting. Another told me that I wouldn’t understand how Kafka internals work even if it was explained to me.

Looking back, perhaps I could have tried harder to obtain the information I need for my job. But nobody can change the past.

More importantly, what have I learned? I came to the belief that an onboarding roadmap is crucial for any role or seniority. And successful onboarding requires commitments not only from the manager and the new hire but also from their teammates and partners.

The Bigger Picture

If I can pick one and only one reason for why I ended up on a coaching plan, it would be that:

I was missing the bigger picture.

All of my failed projects were in the sales domain. Why? I didn’t fully comprehend how business-to-business (B2B) sales work or the business problems behind my projects. For example, how a deal is landed from end to end, what different stages there are for a deal pipeline, and how to forecast deals given existing pipelines (and vice versa).

In short, I failed to deliver the right data because I didn’t entirely understand how the data would be used.

On the contrary, I was an awesome data engineer on the marketing side. I got to directly work with the growth marketing team, allowing me to learn our marketing strategy on social media and the business problems we aimed to solve with the marketing data.

Feedback

At the new job, the feedback from data scientists is the sole deciding factor in my performance. Unfortunately, the data scientists and I never felt comfortable directly exchanging feedback with each other. Why? Because there was very little trust in our transactional relationship.

As a result, all the feedback I received always came through my manager. You can probably imagine how this went. Whenever I got feedback from my manager, I asked for clarification and examples, but most of the time my manager didn’t know enough context to explain further. Having a middleman in the feedback cycle only led to more and more frustrations and misunderstandings among all of us.

The Not-So-Happy Ever After

In order to survive, I did everything I could to right the wrongs (including the wrongs I didn’t agree with). Fortunately, about one month later, I managed to make it out of the coaching plan.

Time to celebrate and enjoy the happy ever after, hooray? Nope.

Although I survived, it’s no longer a job I want to keep, because I was never happy there, neither before nor after the coaching.

A few months later, I left.

Although the coaching plan was the darkest period of my life, I’m grateful that it pushed me to reflect deeply on work (data engineering) and life.

What I Learned About Data Engineering

Business Context Matters

As I confessed in the previous section, the main reason behind my failed projects is that I was missing the bigger picture. It is so critical for data engineers to be close to business because business context provides the bigger picture of data engineering problems:

• How does the business operate? How does the business map to data?
• What are the short-term and long-term business goals?
• What and how are different systems and teams involved?
• How will the source data be produced?
• How will the output data be consumed?

Knowing the bigger picture is essential for data engineers to pin down requirements and design optimized data solutions accordingly.

(I explained more about the importance of business context and how to build the bridge between data engineering and business in this article.)

Data Engineering Culture Matters

“Data engineering is customer service to data science. So all it matters is we making our data scientists happy.”

The moment my manager said this to me, I realized my manager wasn’t really my boss, the data scientists were: they decide what we work on and their words determine our performance. The power dynamic made me feel like a code monkey all the time.

What’s worse? Most of the company didn’t even know data engineers existed. At the end of the day, all you heard was:

“Kudos to the data science team!”

“Big thank you to our data scientist for making it happen!”

If you are a data engineer too, you probably know this feeling all too well.

The lack of data engineering culture is the problem here, but it shouldn’t be this way. Data engineering teams deserve visibility, autonomy, and healthy partnerships with stakeholders:

• Visibility: Data engineers should be recognized and celebrated along with other data roles.
• Autonomy: Data engineers should have the autonomy to prioritize long-term investments (such as tooling or infrastructure innovation) to solve common data engineering problems that are not stakeholder driven.
• Partnerships: Data engineers should feel empowered to say NO to stakeholders to pursue better overall outcomes and avoid burnout.

(I talked more about my opinions on data engineering excellency in this article.)

What I learned about life

Here is the most important lesson I learned after everything:

I cared way too much about work.

My mental health took a toll. I was depressed for a long time because all of my self-worth depended on my career success and nothing else. Even after surviving the coaching plan and getting a new job, the unbearable amount of shame and guilt still tortured me for months.

My mental struggles become too painful that I eventually had to see a therapist for help. Don’t I wish someone had told me this sooner:

“You deserve to be happy, respected, and loved, regardless of your professional success.”

Growing up, love and happiness were never unconditional to me, and now I had to train myself to believe in the opposite. With the help of my therapist, I also started to explore personal values and goals outside of work, such as pottery, dog rescue, and mentorship.

Failures will always be part of life and I might have difficult times at work again. But I know if it happens again, I will be able to cope better. I will look for help. I will be kind to myself. And I will move on.

Why I wrote this down

Having been on a coaching plan isn’t a badge I wore around every day. In fact, I have only spoken about it with very few people. I was too cowardly to share my story because I was afraid of repercussions:

• What if people think I’m a loser?
• What if they don’t trust me anymore?
• What if I get fired after they find out I’m far from perfect?
• What if I never get hired again because of my past?

While I am publicly criticizing the fact that perfection is overvalued in the tech industry, I hide away my own imperfection.

I realized what a hypocrite I am.

That’s why, after much hesitation, I decided to write my story down. If there are going to be repercussions for sunshining the mistakes I have made in the past, so let them be.

Because the truth is: failures made all of us who we are today.

Even though we are not perfect, we all deserve to pursue our careers.

Selfishly, by writing this down, I can finally accept that part of me. So, thank you for hearing me out.

Imperfection is in all of us. May you also find the strength and courage to embrace yours.