Join Newsletter

Unveiling much simplified Functional Programming in Scala for Data Engineering

YOW! Lambda Jam 2020

I will talk about a much simplified version of functional programming in Scala, in building an abstraction for Feature Generation in Data Engineering space.
The program made using this abstraction will get interpreted to the popular data source languages of our choice - such as Spark or Flink. However, before it gets interpreted to any of these engines, we will explain how these programs could be optimised by introspecting its nodes, and help run these interpretations faster. The core idea is similar to that of Free Applicative, however, implementing it in Scala hasn't been straight forward. Here, we provide a similar capability but without mentioning much about FreeAp, and without the usual Scala boilerplates of implicits, macros and a proliferated usage of type classes.

The purpose of the talk is not just to demonstrate a set of code, but to showcase the fact that sticking on to fundamentals of Functional Programming, and finding the right abstraction enables writing solutions quickly and relatively easily.

It proves we don't need to learn a bulk of libraries to apply these concepts in real world applications. The learning curve and a massive set of libraries was often termed as the functional programming in Scala industry, resulting in lesser adoption and developers moving away from it. With this talk my intention is to motivate developers to come back and start writing FP even if they are in the world of JVM.

Afsal Thaj

Principal Consultant

Simple Machines

Australia

Simple Machines, Consultant

I am a Software Developer/Consultant, in Simple Machines, a technology consultancy in Australia that always strive to develop high quality purely functional software.

I have been working in Sydney for the last 5+ years. I started in CBA as an ETL developer, and most of the time, I worked as Feature Engineering (Data Engineering) Squad Lead.

Later on, I moved to Simple Machines as a consultant, and enjoying here for the last 3 years. I have got the rest of my experience as a developer in India.

Interests:

I believe in writing robust software that has a high positive impact on business. During recent years, I strongly felt my passion as to build frameworks and libraries with the help of functional programming constructs, and enjoy releasing it to production with zero bugs ! With my personal time learning Haskell, I do enjoy the power of FP in JVM world through Scala.

Blogging and sharing knowledge:

I am active in blogging programming concepts in general, principles that we should follow, and you can find my latest blogs here: https://medium.com/@afsal.taj06 and gists in https://gist.github.com/afsalthaj

Open source contributions:

I have got commits in various projects across languages.
The latest project that I author and maintain is zio-config (https://github.com/zio/zio-config), which unveils simplified version of advanced functional programming with Scala. I have also got a few other pet projects as you will see in my GitHub (such as a safe-string-interpolation to disallow meaningless logging in applications, which is not quite exciting but always my favourite)

What next?

It makes me excited to share my experiences deploying applications that sticked on to principles of Functional Programming. This year and coming years, my focus will be on sharing this wonderful experience that had tangible business outcomes that resulted in less number of re-iterations and bug fixes. By doing this, I intend to improve adoption of Functional Programming in the industry surrounding me!