MyPage is a personalized page based on your interests.The page is customized to help you to find content that matters you the most.

I'm not curious

Senior Data Engineer - New Jersey

Location Edison, United States
Posted 11-November-2019

Senior Data Engineer - New Jersey

Location: New Jersey

Technology: Azure Big Data

Job Type: Full Time Permanent

Date Posted: October 14, 2019

Data is at the core of our business. The data engineer is a technical job that requires substantial expertise in a broad range of software development and programming fields. The data engineer should especially have sufficient knowledge of big data solutions to be able to implement those on premises or in the cloud.

A data engineer generally works on implementing complex big data projects with a focus on collecting, parsing, managing, analyzing and visualizing large sets of data to turn information into insights using multiple platforms. He or she should be able to decide on the needed hardware and software design needs and act according to the decisions. The big data engineer should be able to develop prototypes and proof of concepts for the selected solutions.

Ideal candidate for this role is someone with a strong background in computer programming, statistics, and data science who is eager to tackle problems with large, complex datasets using the latest Python, R, and/or PySpark. You are a self-starter who will take ownership of your projects and deliver high-quality data-driven analytics solutions.

Specific responsibilities are as follows:

Work with data science team to deploy Machine Learning Models
You will be using Data wrangling techniques converting one raw form into another including data visualization, data aggregation, training a statistical model etc.
Work with various relational and non-relational data sources with the target being Azure based SQL Data Warehouse & Cosmos DB repositories
Clean, unify and organize messy and complex data sets for easy access and analysis
Hands on data preparation activities using the Azure technology stack especially Azure Databricks is strongly preferred
Implement discovery solutions for high speed data ingestion
Work with the Sr. Data Engineers on the team to develop APIs
Sourcing data from multiple applications, profiling, cleansing and conforming to create master data sets for analytics use
Utilize state of the art methods for data manning especially unstructured data
Experience with Complex Data Parsing (Big Data Parser) and Natural Language Processing (NLP) Transforms on Azure a plus
Design solutions for managing highly complex business rules within the Azure ecosystem
Performance tune data loads

Skills Required:

Mid to advanced level knowledge of Python and Pyspark is an absolute must.
Knowledge of Azure, Hadooop 2.0 ecosystems, HDFS, MapReduce, Hive, Pig, Sqoop, Spark etc. a must
Experience with Web Scraping frameworks (Scrapy or Beautiful Soup or similar)
Extensive experience working with Data APIs (Working with RESTful endpoints and/or SOAP)
Significant programming experience (with above technologies as well as Java, R and Python on Linux) a must
Knowledge of any commercial distribution like HortonWorks, Cloudera, MapR etc. a must
Excellent working knowledge of relational databases, MySQL, Oracle etc.
Natural Language Processing (NLP) skills with experience in Apache Solr, Python a plus
Knowledge of High-Speed Data Ingestion, Real-Time Data Collection and Streaming is a plus

5+ years of solid experience in Big Data technologies a must
Microsoft Azure certifications a huge plus
Data visualization tool experience a plus


Unlimited PTO
2 days Remote a week
15% Bonus
Gym Membership
Uber Membership
Regular in-office social events, happy hours, etc.
401K + 5% Match
Health/Medical/Dental benefits [full family coverage]
Open vacation policy
Free Breakfast & Lunch in newly renovated kitchen

If you or someone you know is interested in this position, please send your resume directly to or call (Brittany Toth) 212-731-8282. My client is looking to start the interview process as soon as possible.

Nigel Frank International is the global leader for Microsoft recruitment, advertising more Azure roles than any other agency. We deal with both Microsoft Partners & End Users throughout North America. By specializing solely in placing candidates in the Azure market I have built relationships with most of the key employers in The Greater New York area and have a complete understanding of where the best Azure opportunities are.

Backed by private equity firm TPG Growth, we have a proven track record servicing the Microsoft permanent and contract recruitment market and, to date, have worked with over 30,000 organizations globally from our offices in North America, Europe, and Asia-Pacific.

Interested in this Job? Do you want to Re-Apply?

Location *
LinkedIn Profile
Tap to Attach your LinkedIn Profile

Awards & Accolades for MyTechLogy
Winner of
Top 100 Asia
Finalist at SiTF Awards 2014 under the category Best Social & Community Product
Finalist at HR Vendor of the Year 2015 Awards under the category Best Learning Management System
Finalist at HR Vendor of the Year 2015 Awards under the category Best Talent Management Software
Hidden Image Url