![TechWithViresh](/img/default-banner.jpg)
- Видео 113
- Просмотров 770 281
TechWithViresh
Индия
Добавлен 25 дек 2018
TechWithViresh is committed and specializes in the technology areas like : Machine Learning,AI,Spark,Big Data,Nosql, graph DB,Cassandra and Hadoop ecosystem.
Contact us at : techwithviresh@gmail.com
facebook : Tech-Greens
Contact us at : techwithviresh@gmail.com
facebook : Tech-Greens
Intro | Big Data Career Switch
#Bigdata #career #Hadoop #Spark #Scala
In this particular video, we have discussed to start a new initiative / video series on how to do the self preparation for the career switch in the BigData world
Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more
Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg
About us:
We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big Data,Nosql, graph DB,Cassandra and Hadoop ecosystem.
Mastering Spark : ruclips.net/video/bU57q5R5eTc/видео.html
Mastering Hive : ruclips.net/vide...
In this particular video, we have discussed to start a new initiative / video series on how to do the self preparation for the career switch in the BigData world
Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more
Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg
About us:
We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big Data,Nosql, graph DB,Cassandra and Hadoop ecosystem.
Mastering Spark : ruclips.net/video/bU57q5R5eTc/видео.html
Mastering Hive : ruclips.net/vide...
Просмотров: 2 453
Видео
Azure DataBricks Cluster Deployment | Spark Cluster | Spark Job
Просмотров 3,6 тыс.3 года назад
#Azure #Databricks #Spark # Scala In this particular video, we have discussed in detail about how to get the spark job created and deployed on to the Azure Databricks cluster Code github Link : github.com/vireshku/SparkAzureDataLake Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to su...
Spark Scala | Connection with Azure Data Lake | Read Data | Write Data | Azure Activity Directory
Просмотров 5 тыс.4 года назад
#Azure #Spark #DataLake In this particular video, we have discussed how to establish the connectivity between the spark and the Azure data lake Code github Link : github.com/vireshku/SparkAzureDataLake.git Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/chann...
Apache Spark | Delta Lake | New Features | Part-2
Просмотров 2,8 тыс.4 года назад
#Apache #Execution #Model #SparkUI #BigData #Spark #Partitions #Shuffle #Stage #Internals #Performance #optimisation #DeepDive #Join #Shuffle,#Azure #Cloud #Event #Streaming ,#hadoop #Hdfs #Mapreduce #Tutorial #Apache #Spark3 #DeltaLake #ACID In this particular video, we have discussed in detail about the New Features available as part of Apache Spark 3. Some of the major features are 1. ACID T...
Delta Lake | Spark 3 | Apache Spark New Features
Просмотров 6 тыс.4 года назад
#Apache #Spark3 #DeltaLake #ACID In this particular video, we have discussed in detail about the New Features available as part of Apache Spark 3. Some of the major features are 1. ACID Transactions in Spark 2. Schema Enforcement 3. DML Support - Delete , Update , Upsert 4. Time Travel Code github Link : gist.github.com/vireshku/1c1c34fc2d342077285c3368a0936205 Please join as a member in my cha...
Apache Spark Scala development project setup with Eclipse
Просмотров 12 тыс.4 года назад
#Apache #Spark #Eclipse #Beginner In this series, we are starting a new step by step tutorial to understand the developement and deployment of the spark scala application developement , Here in this particular video we have setup the spark scala local developement environment with Eclipse as the IDE Please join as a member in my channel to get additional benefits like materials in BigData , Dat...
Hadoop Tutorial | HDFS Blocks | Step by Step
Просмотров 1,9 тыс.4 года назад
#Apache #Hadoop #Introduction #Hadoop #HDFS Blocks In this series, we are starting a new step by step Hadoop tutorial for beginner to experts.In this particular video, we have discussed the design and architecture of the Hadoop HDFS Blocks. Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click he...
Apache Spark 3 | New Feature | Performance Optimization | Dynamic Partition Pruning
Просмотров 8 тыс.4 года назад
#Apache #Spark3 #Performance #Dynamic Partition Pruning In this particular video, we have discussed New Features available as Dynamic Partition Pruning for the query optimisation in Spark 3 Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC...
Spark Performance Optimization | Join | UNION vs OR
Просмотров 8 тыс.4 года назад
#Apache #Spark #Performance #Optimization In this particular video, we have discussed spark join performance Optimization in the scenario where 'OR' operator is used within the Joins. Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHE...
Apache Spark 3 | Design | Architecture | New Features | Interview Question
Просмотров 8 тыс.4 года назад
#Apache #Spark3 #Design #Architecture In this particular video, we have discussed New Features available , Design and Architecture in Spark 3 Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg About us: We are a technology consul...
Spark Performance Tuning | Memory Architecture | Interview Question
Просмотров 9 тыс.4 года назад
#Apache #Spark #Performance #Memory In this particular video, we have discussed Spark performance optimisation for the efficient memory management Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg About us: We are a technology c...
What is Hadoop | Introduction | Hadoop Tutorial | Architecture
Просмотров 1,1 тыс.4 года назад
#Apache #Hadoop #Introduction #Hadoop1 vs #Hadoop2 In this series, we are starting a new step by step Hadoop tutorial for beginner to experts Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg About us: We are a technology consul...
Spark Interview Question | Bucketing | Spark SQL
Просмотров 14 тыс.4 года назад
#Apache #Spark #SparkSQL #Bucketing Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg About us: We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big Data...
Spark Interview Questions | PySpark and Apache Arrow | What is Apache Arrow
Просмотров 4,5 тыс.4 года назад
#Apache #spark #PYSpark #Apache #Arrow Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg About us: We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big D...
Spark Interview Question | fold vs reduce
Просмотров 4,3 тыс.4 года назад
#Apache #spark #fold #reduce #Analytics Please join as a member in my channel to get additional benefits like materials in BigData , Data Science, live streaming for Members and many more Click here to subscribe : ruclips.net/channel/UCZqHmLZxX0KC6PiJHETflOg About us: We are a technology consulting and training providers, specializes in the technology areas like : Machine Learning,AI,Spark,Big ...
Spark Interview Question | Clickstream Aanalytics
Просмотров 2,8 тыс.4 года назад
Spark Interview Question | Clickstream Aanalytics
Spark Scenario Based Question | ClickStream Analytics
Просмотров 7 тыс.4 года назад
Spark Scenario Based Question | ClickStream Analytics
Spark Interview Question | Cost Based Optimizer
Просмотров 7 тыс.4 года назад
Spark Interview Question | Cost Based Optimizer
Hadoop Interview Question | Split Brain Problem
Просмотров 3,9 тыс.4 года назад
Hadoop Interview Question | Split Brain Problem
Spark Interview Question | Map vs MapPartition vs MapPartitionWithIndex
Просмотров 9 тыс.4 года назад
Spark Interview Question | Map vs MapPartition vs MapPartitionWithIndex
Spark Interview Questions | Spark Context Vs Spark Session
Просмотров 19 тыс.4 года назад
Spark Interview Questions | Spark Context Vs Spark Session
Spark Interview Question | Partition Pruning | Predicate Pushdown
Просмотров 13 тыс.4 года назад
Spark Interview Question | Partition Pruning | Predicate Pushdown
Apache Spark Basics | How Spark Works | Interview Question
Просмотров 6 тыс.4 года назад
Apache Spark Basics | How Spark Works | Interview Question
Spark Scenario Interview Question | Persistence Vs Broadcast
Просмотров 13 тыс.4 года назад
Spark Scenario Interview Question | Persistence Vs Broadcast
RDD Size Programmatically | Hands-on Code | Interview Question
Просмотров 5414 года назад
RDD Size Programmatically | Hands-on Code | Interview Question
Cache vs Persist | Spark Tutorial | Deep Dive
Просмотров 32 тыс.4 года назад
Cache vs Persist | Spark Tutorial | Deep Dive
Spark Execution Model | Spark Tutorial | Interview Questions
Просмотров 2,5 тыс.4 года назад
Spark Execution Model | Spark Tutorial | Interview Questions
Managing Spark Partitions | Spark Tutorial | Spark Interview Question
Просмотров 23 тыс.4 года назад
Managing Spark Partitions | Spark Tutorial | Spark Interview Question
Apache Spark Tutorial | NoSql Database
Просмотров 7054 года назад
Apache Spark Tutorial | NoSql Database
Spark Performance Tuning | Avoid GroupBy | Interview Question
Просмотров 11 тыс.4 года назад
Spark Performance Tuning | Avoid GroupBy | Interview Question
What about aggregateByKey function in RDD
To whom may be concerned when to use GroupByKey over ReduceByKey: groupByKey() can be used for non-associative operations, where the order of application of the operation matters. For example, if we want to calculate the median of a set of values for each key, we cannot use reduceByKey(), since median is not an associative operation.
Hindu again 🤢
Hi, I like your Spark videos. Please create a dedicated video for top 100 most frequently used Spark Commands. - Pankaj C
Video recommendatin at the end are blocking the content...
i see, for RDD its memory and for Dataframe it is mem + disk
why are you talking like sleppy mode ??
Please provide aws questions and answers. Thank you 🙏
what is MSCK ?
What if I have multiple spark jobs in parallel in on spark session
For ORC format, schema evolution is not just limited to adding new columns. Backward Compatibility: Adding Columns: New columns can be added to the schema without affecting existing data files. When reading old ORC files with a new schema that includes additional columns, the new columns will be treated as optional and filled with default values. Removing Columns: Similar to Parquet, existing columns can be removed without breaking compatibility. When reading old ORC files with a new schema that excludes certain columns, those columns will be ignored. Changing Data Types: Data types of existing columns can be changed, and ORC will attempt to convert the data to the new type. However, similar to Parquet, this conversion might result in data loss if the types are not compatible. Forward Compatibility: Adding Columns: New columns can be added, and existing files can still be read without errors. The new columns will be filled with default values when data from the old files is read. Removing Columns: Files written with a schema that has fewer columns can still be read with a newer schema containing additional columns. The additional columns will be treated as optional. Changing Data Types: Forward compatibility is generally maintained for changing data types, but careful consideration is needed to avoid potential data loss or conversion issues. above points are what I found supplementing with your content. thanks for your videos and dedication in making them, it is really helpful for my preparation.
Hi! Why you say Avro is row oriented, isn't also columnar storage?
Thank you
Super content thank you
Good sir
Could you please tell what is the difference between partition pruning and predicate pushdown
Both same
Very Nice and clear explanation before this video i was very confused regarding executor tuning part now after this video it is now crystal clear.
Hi, 10 nodes means including the master node? i have a configuration like this: "Instances": { "InstanceGroups": [ { "Name": "Master nodes", "Market": "SPOT", "InstanceRole": "MASTER", "InstanceType": "m5.4xlarge", "InstanceCount": 1 }, { "Name": "Worker nodes", "Market": "SPOT", "InstanceRole": "CORE", "InstanceType": "m5.4xlarge", "InstanceCount": 9 } ], "KeepJobFlowAliveWhenNoSteps": false, "TerminationProtected": false },
@TechWithViresh: no recent videos. Can you please add . your videos are very useful brother. thanks
Thanks, for sure videos coming soon :)
Thanks! A great and concise explanation!
The 2nd map will not executed as no action performed on result data set after collect.
hello, i find the content very interesting especially on when the hash join is better than the sort merge join. could you please tell me where you found the documentation on that?
Many thanks to you sir. 😊 i learnt spark from you
very good. please make videos as interview questions on spark as a group of videos
nice
Audio quality is not good content is good
Limit comes after order by in query execution order, how using limit will reduce the number of records to be sorted? Am I missing anything here?
Why are you converting dataframe to rdd ?? It is very bad practice in terms of performance
video from 11:30, we are adding random key to exiting towerid key for Example. tower id: 101 and salt key : 67 then 101+67= 168 hash value of the 168 would be a final value right. what in case of partition column is string datatype. ??
Incase of strings, we can add surrogate keys, based on string column values and then do the salting.
bhai ye hindi m bta dega toh tera kuch chla jaa rha h kya??
Perfect 👌 explanation
Very good and descriptive comparison. Thank you!
You gave the all information about Hive.. is this enough for interview?
How the last map operation will run on driver see till collect a job will be completed and whenever we call another action it will create new job with new Dag which will again distributed and run on executors??
Good explanation.. Thank you 👍
can we get ppt that you show in the videos?
What If each node has only 8cores?? How does spark allocate 5cores per jvm ?
Awesome✨
bro if you have 6 blocks in Hadoop 3 then it consumes 15 blocks. Suppose we have a file which consists of 2 Blocks (B1 and B2). 1) With current HDFS setup, we will have total (2×3 = 6 blocks in total). For Block B1 -> B1.1, B1.2, B1.3 For Block B2 -> B2.1, B2.2, B2.3 2) With EC setup, we will have total (2×2 + 2/2 = 5 blocks in total). For Block B1 -> B1.1, B1.2 For Block B2 -> B2.1, B2.2 The 3rd Copy of each Block will be Xor’ed together and stored as a single Parity Block as (B1.1 xor B2.1) -> Bp In this setup: If B1.1 is corrupted, we can recompute B1.1 = Bp xor B2.1 If B2.1 is corrupted, we can recompute B2.1 = Bp xor B1.1 If both B1.1 and B2.1 are corrupted, then we have another copy of both the blocks (B1.2 and B2.2) If parity Block Bp is corrupted, then it is again recomputed as B1.1 xor B2.1
@Ankit Bansal can you please solve this using SQL please
This isn't instagram where you can tag channels lol
Is there any differences with performance issues?
Crisp , concise and to the point explanation in great detail. Anyone can understand through this video. Extremely well done. Kudos...
Glad it was helpful!
Thank you
Welcome!
Thank you
Welcome!
Good content
Voice and explanation not clear!
Sir will you please make a video that explains the rand() function?
how can we do percentile() avoiding groupBy ...can you explain it ?
Good one
Thank you! Cheers!