The State of Big Data Recruiting

Big data -- i.e. data technology and data science -- has seen an incredible rise over the past few years. Enterprise in every industry is investing to be more data-driven in how they do business. This movement has enabled the job market for data tech and data science expertise to grow at an exponential pace. Some of the most sought-after talent include the following:

• Data Scientists – referred to by the Harvard Business Review as the sexiest job of the 21st century
• Data Technology Engineers / DBAs – especially expertise in distributed computing (Hadoop, Spark, etc)
• Quantitative Analysts and Managers – team players who can drive forward a data-driven culture

Demand for talent far outpaces the supply of talent

Data science and data technology roles demand a high degree of specialized expertise, held by a limited supply of highly-sought professionals. The supply of talent is growing, though demand is growing at an even faster pace. The McKinsey Global Institute released a study stating that:

Put another way, this estimate suggests that demand for analytic talent will exceed supply by 50-60%.

What this means for data science and data technology professionals

If you have the skills to be a key player in enabling a company's 'big data' initiative, it is a very good time for your career. Recruiters will be knocking at your door constantly and you'll always have job options to consider. Enjoy having the upper hand in being selective where you want to work.

What this means for employers/recruiters?

In recruiting, some positions are tougher to fill than others – and, roles relating to data science and data technology will be among the toughest. Do whatever you can to maximize your chances. Cast a wide net, network extensively, and offer attractive compensation. Most expert data science and data infrastructure compensation pushes well into the six-figures, easily over 100k and possibly even going up into the 200k range for some roles.

---

Recruiting data scientists

Data scientists possess a potent combination of skills in quant, engineering, and business acumen. They are highly-demanded, which means you generally don't see an unemployed data scientist. But you might find one who would rather work for your company than the one they are at now.

We'd like to clear up some common misconceptions around how to look for data scientists:

• Misconception: Must have a Ph.D (in math or some other research area). This is a blunt approach that limits your pool and may not align well with true needs. Data scientists are multidisciplinary, not just experts in one field only. Also, strong data scientists are often motivated autodidacts who constantly learn through experience and investigation, not necessarily requiring a degree program to have the needed skills.
• Misconception: Must be a Hadoop expert. A data scientist definitely needs technical skills, but many recruiters confuse this with being an infrastructure engineer. A data scientist must be comfortable interacting with various types of systems including possibly Hadoop, but it shouldn't be a filter when trying to find leads. However, data scientists should be proficient with SQL and Python, R, or SAS.
• Beware: 'Imposter' data scientists. Many people label themselves data scientists but may have rudimentary experience, e.g. may have run a regression in Excel at some point, but don't yet have extended technical and quantitative depth. When interviewing, make sure skills are vetted thoroughly. How is their statistics knowledge? Can they write code?

A key trait of data scientists is intellectual curiosity. This natural enthusiasm for discovery tends to lead them to investigate and build things on their own. Two interview questions to help identify rockstar potential:

"Tell me about a technical project that you started on your own, outside of school and work"
"Tell me about a topic that you taught yourself because you wanted to learn it"

If a candidate can list many self-taught skills and name projects they initiated on their own because they were curious, it is a good signal.

How much should data scientists get paid?
Data scientist compensation extends well into the 6-figure range. Learn more about data scientist salary.

---

Recruiting data technology engineers / DBAs

Data engineers are key players in building up the core infrastructure of big data systems. This may involve working with constantly-shifting, relatively nascent technologies. Finding engineers who are already domain experts in the exact technology used by your company is always a nice win; however, the specificity of that requirement often makes for an overly narrow search. Alternatively, if an engineer has not worked with the exact technology, but has impressive technical problem-solving and architectural intuition with servers/data systems in general, those skills are still invaluable. The idea is that smart engineers with a good foundation can figure anything out.

While there is a wide variety of technology associated with big data infrastructure, a common element tends to be MPP (massively parallel processing) architecture. Technologies include Hadoop, Spark, Cassandra, MongoDB, and countless others. These are very different from traditional database systems. Thus, someone who spent years as DBA for a non-MPP system (e.g. Microsoft SQL Server) may have a longer jump to managing an MPP computing cluster (e.g. Hadoop/HBase). If you are a recruiter, it is a good idea to develop general background knowledge around the different types of big data technologies, in order to fully grasp candidate skill sets amidst the alphabet soup of tech talk out there.

How much should big data engineers and DBAs get paid?
These technical roles often demand premium conpensation. Learn more about data engineer and DBA salary.

---

The Bottom Line

Demand for big data talent is booming. Here are some broad takeaway points for both sides of the market.

Recruiters – understand the space very well
The nuance around each specialty is commonly misunderstood, which can lead to poor targeting and misalignment with candidates. Don't be a recruiter who doesn't know what exactly they're trying to hire.

Recruiters – be hyper aggressive
Go above and beyond in your search for these roles, relative to other roles. The level of competition in the market is sky-high and you need to do everything you can.

Big data professionals – seize the opportunity to write your own ticket
It is an incredibly advantageous time for you. This is your market, where companies are competing for your talent. You can be selective and find the company that suits you best.