Depending on things make them harder to change
The more you start to use things, depend on them, the harder they become to change.
This story starts with a data lake that all the scientist use. The data lake is created somewhere upstream (bubble) and it just happily exists somewhere and it has all the data we need. The data scientists can query this data lake and people live happily ever after.
Years went by and as the team grew the data scientists started creating production services using this data lake. Most of these systems just pull some data from there, do things with it, and store that information somewhere else. These systems are Zap, Bar, and Foo respectively.
Now, there’s a lone and sad data engineer that needed to make an important change to the data lake. But after working on this for weeks he has come to the realisation that it is simply too much work to make this change. Why? Because now that three production services depend on it, any change in the data lake must be propagated in the services using it, but the people maintaining Zap, Bar, and Foo are long gone, and no one knows how to do it.
A small change in the data lake is now a big change in the whole system because we depend on the data lake.
The moral of the story is: if you start depending on something, it becomes harder to change.
Comments