Skip to main content

Developershome

This is Developer's Home. I am passionate about Technology and Data. Here I share my learning with various data technologies.

Search This Blog

Home
VLog
Big Data Engineering
Docker
Data Engineering Problems
Spark

Showing posts from February, 2023

Posts

Data Engineering Problem 1 (Get Max Salary of each employee)

on February 05, 2023

dataengineering level_medium postgresql pyspark sparksql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Data Engineering Problem 0 (Employees with salary more than 100K)

on February 04, 2023

dataengineering hive postgresql pyspark sql

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Data Engineering Tool Suite

on February 04, 2023

container dataengineering docker-compose image mongodb mysql postgresql pyspark spark

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Deploy Spark Using Docker

on February 04, 2023

container docker image pyspark spark

+

Get link
Facebook
X
Pinterest
Email
Other Apps

Developershome

Archive

February 20239

Labels

app-deployment
application
bigdata
container
dataengineering
deployment
docker
docker-compose
fastapi
hive

image
level_medium
mongodb
mysql
nosql
postgresql
pyspark
spark
sparksql
sql
vm
webserver

Show more Show less

Total Pageviews

Popular Posts

Deploy Spark Using Docker

In this blog, we will learn how to deploy spark using Docker. We will use this environment in future for learning spark and solving data engineering problems. With Spark, we also want to install a few of the packages (connectors) so that we don't need to install them separately. When we will learn Spark, we will mostly require packages below. Delta Lake AWS S3 ( s3a:// ) Google Cloud Storage ( gs:// ) Azure Blob Storage ( wasbs:// ) Azure Datalake generation 1 ( adls:// ) Azure Datalake generation 2 ( abfss:// ) Snowflake Hadoop cloud magic committer for AWS PostgreSQL MySQL MongoDB If we check Docker hub, we see that there are multiple options available for Spark We will select below image, as it by default provides datamechanics/spark All the required connectors which we need Jupyter notebook Built-in with Python & PySpark support ...

Data Engineering Problem 1 (Get Max Salary of each employee)

Introduction: As we discussed earlier, we will start solving Data Engineering problems using SQL (PostgreSQL and MySQL), NoSQL (MongoDB or Cassandra) and Apache Spark (PySpark and Spark SQL) We will start from very easy SQL problems to difficult SQL Problems, we will also solve problems regarding data loads (Batch, replication and Streaming). Problem Statement We have a table with employees and their salaries, however, some of the records are old and contain outdated salary information. Find the current salary of each employee assuming that salaries increase each year. Output their id, first name, last name, department ID, and current salary. Order your list by employee ID in ascending order. Problem Difficulty Level: Medium Data Structure Data for this problem In CSV Format Please also clone below GitHub repo for the SQL queries and Jupyter notebook. https://github.com/developershomes/DataEngineeringProblems/tree/main/Problem%201 P...

Big Data Engineer Skills

Data engineers need to know a combination of programming languages, database skills, and data processing tools in order to be successful in their careers. Also need to have understanding of below technology Big Data Processing Frameworks: Apache Hadoop, Apache Spark, Apache Storm, and Apache Flink are some of the popular big data processing frameworks used in the industry. Distributed Storage Systems: Experience with distributed storage systems such as HDFS, Apache Cassandra, and Apache Kafka is essential for a data engineer. Data Warehousing: Knowledge of data warehousing technologies such as Apache Hive, Apache Impala, and Apache Pig is also important. Data Integration: Experience with data integration technologies such as Apache NiFi, Apache Nifi Registry, Apache Airflow, and Apache Beam is important. Cloud Computing: Familiarity with cloud computing platforms such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform ...

Report Abuse

Powered by Blogger

Theme images by mammamaart

https://developershome.blog/