Developershome

Introduction: As we discussed earlier, we will start solving Data Engineering problems using SQL (PostgreSQL and MySQL), NoSQL (MongoDB or Cassandra) and Apache Spark (PySpark and Spark SQL) We will start from very easy SQL problems to difficult SQL Problems, we will also solve problems regarding data loads (Batch, replication and Streaming). Problem Statement We have a table with employees and their salaries, however, some of the records are old and contain outdated salary information. Find the current salary of each employee assuming that salaries increase each year. Output their id, first name, last name, department ID, and current salary. Order your list by employee ID in ascending order. Problem Difficulty Level: Medium Data Structure Data for this problem In CSV Format Please also clone below GitHub repo for the SQL queries and Jupyter notebook. https://github.com/developershomes/DataEngineeringProblems/tree/main/Problem%201 P...

Deploy Spark Using Docker

In this blog, we will learn how to deploy spark using Docker. We will use this environment in future for learning spark and solving data engineering problems. With Spark, we also want to install a few of the packages (connectors) so that we don't need to install them separately. When we will learn Spark, we will mostly require packages below. Delta Lake AWS S3 ( s3a:// ) Google Cloud Storage ( gs:// ) Azure Blob Storage ( wasbs:// ) Azure Datalake generation 1 ( adls:// ) Azure Datalake generation 2 ( abfss:// ) Snowflake Hadoop cloud magic committer for AWS PostgreSQL MySQL MongoDB If we check Docker hub, we see that there are multiple options available for Spark We will select below image, as it by default provides datamechanics/spark All the required connectors which we need Jupyter notebook Built-in with Python & PySpark support ...

Big Data Engineer Skills

Data engineers need to know a combination of programming languages, database skills, and data processing tools in order to be successful in their careers. Also need to have understanding of below technology Big Data Processing Frameworks: Apache Hadoop, Apache Spark, Apache Storm, and Apache Flink are some of the popular big data processing frameworks used in the industry. Distributed Storage Systems: Experience with distributed storage systems such as HDFS, Apache Cassandra, and Apache Kafka is essential for a data engineer. Data Warehousing: Knowledge of data warehousing technologies such as Apache Hive, Apache Impala, and Apache Pig is also important. Data Integration: Experience with data integration technologies such as Apache NiFi, Apache Nifi Registry, Apache Airflow, and Apache Beam is important. Cloud Computing: Familiarity with cloud computing platforms such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform ...

Report Abuse

Powered by Blogger

Theme images by mammamaart

https://developershome.blog/