[ad_1]
Lead story – The modern stack has a messy data problem
Judging by the platitudes of data management vendors, I thought we had this data quality thing figured out. Not so fast, says Neil Raden, after attending a surprisingly vigorous panel:
Usually, these panels are dull, especially when the panelists are all founders or executives of software companies. This was different.
At issue? The so-called “Modern Data Stack,” (MDS) and what it takes to achieve it:
One particularly memorable moment came when George Fraser clarified Fivetran’s role in the MDS. As a data replication company, Fivetran focuses on replicating data into the desired destination without getting involved in any of the workflows intended by the user… However, as the MDS evolves, the “messy problem of data integration” will undoubtedly remain a challenge for all players in the industry.
Data management vendors are increasingly sophisticated, but: the messy data problem hasn’t gone away. Neil quotes Fraser:
People need to realize that the sources produce very unclean data. And if you need to send the data to a relational database that supports updates and things like that, the data you will be looking at will be very ugly.
AI raises this problem to an fever pitch:
Timely and virtually frictionless access to data is a critical requirement for the expanding need for data science and AI/ML. The precious time of skilled practitioners is often spent managing data instead of building models.
What did this panel of luminaries propose to do about it?
Some panelists believed that federated data (e.g. distributed cloud data), is alleviating the messy data problem; others weren’t so sure. Phrases like “stupid idea” definitely make for a memorable panel. Neil says the panel did agree on one thing: data lakes are not the answer. Bottom line: solutions to the dirty data problem are emerging, but it’s not a one-size-fits-all situation.
Ergo, organizations in pursuit of an MDS can’t ignore the data…
..
[ad_2]
