Pies, Lies and AIs

Exploring the world of data and organisational intransigence

What makes a data engineer?

Note: this post is mainly about Azure but it can apply to any cloud. You get to a stage when you hear enough definitions of what people...

Documentation the easy way

So a slight departure from Spark (sort of) for this post, but I wanted to look at one of the most commonly overlooked aspects of building...

Why leave bad data to chance?

Something that we often see as Spark jobs are moved into production is that handling of bad data is either ignored, or a lot of effort...

Pivot, Step, Pivot, Twist, Un-pivot

Getting data into a good shape is a key part to Data Engineering, and we often get data in all sorts of shape and quality

When in doubt, shell out

The command line is a powerful environment that lets you do a lot of work quickly, easily, and in a repeatable way

Making development easier in Databricks

It's now easier to see the documentation directly in the notebook, rather than having to call "help" or go searching the Internet

Microsoft's new approach to data

Over the past few months we've been hearing more and more about Azure Synapse. For those of you that heard my talk and saw my talk at the...

Sustainability is coming of age

Over the past year I've been engaged working with Finextra on topics of sustainable finance as part of an expert committee on the...

Kusto Lightning Fact 5: make-series

A key capability of Kusto Query Language and Azure Data Explorer is the ability to make time series. This is the start of a 3-series set...

Kusto Lightning Fact 4: .show

A key facet of using Azure Data Explorer is the ability to easily use command control. For those that have a good SQL background some of...


Subscribe Form

Stay up to date



Thanks for submitting!