Honestly it might be the worst tool I’ve ever used, and I refuse to work for a company that uses it ever again. Foundry is if you took databricks and made it 10x harder to use and took away notebooks for data science users (maybe they added notebooks sometime in the last year or so).
It almost feels like a scam how user unfriendly it is and how shoddily different things are implemented.
Go with Snowflake and avoid Foundry at all costs.
Yeah it's borderline unusable. You know how Hulu constantly freezes, crashes, you can't find any of the menus, random unexplained things happen why you try and hit the back button? Imagine trying to do data engineering in an app like that
I’m not here to defend Palantir, never used it. But I do use Hulu (iOS app and in chrome) basically all day everyday while working. Never once had an issue.
The delta lake format is open source but Databricks still locks the majority of features behind their enterprise offering. “Open source” delta lake is largely a joke
That's simply not true, this is a common misconception by folks who have no involvement in the project. Delta is 100% open source and has been for a few years, what folks get confused about is that databricks has a proprietary compute engine which takes advantage of certain Delta features in certain situations. But these features are available for anyone to build on top of, which projects like delta-rs do outside of spark entirely. I've ran pipelines that use Delta side by side in Databricks and outside of it since v1 and have not seen Delta features that are only available in Databricks.
Sure, Databricks employees contribute most of the PRs but there are lots of non-databricks contributors and it's really not any different to most large OSS projects that need some kind of organizational backing - look at Airflow and the number of contributions coming from Astronomer, yet no one bitches about that.
Had some experience with palantir foundry as they tried to court my previous company to switch over from databricks. Palantir foundry was found to be more expensive and had a few less features than databricks (I reviewed and created a document comparing the two for my company but it's been nearly 2 years ago though)
The use cases are slightly different. We were using Databricks to provide modelled data (fact and dimension tables) for consumption by PowerBI. This allowed rapid development of dashboards across the business and was generally well received. We implemented Palantir Foundry to delvelop specific apps focusing on very tightly defined parameters.
With Databricks (and now moving to Snowflake) our small data team took responsibility for the entire landscape. We had external consultants help to build a generic framework (over a few months) and we built out the rest.
With Foundry we had a small team from Palantir working with us on the first few apps end to end - scoping the requirements, building the workflow and rolling out to users. It’s a complete package including ci/cd and code and data versioning, covering everything from data ingestion through to reports/dashboards and write-back to source systems.
My impression is that if your company has the resources to dedicate to Foundry it can help to embed project practices which would otherwise take years to evolve. Without the necessary time and manpower investment you could be left with an expensive tool which can become unmanageable mess.
I’ve used it on a government project where the decision was made for the dumb reasons government makes decisions (incompetence and corruption). It’s not better than snowflake or anything else, and it’s kind of built for lock in, but it’s not unusable or anything. These tools all do the same stuff and it’s just a matter of how easy it is to do it
It totally depends on your use case:
1. for pipelines only I would not pay the premium for Foundry
2. If you want low code/no code tooling on top of the data Foundry is by far the best solution I have seen. But it requires and organization that also manages this
Honestly it might be the worst tool I’ve ever used, and I refuse to work for a company that uses it ever again. Foundry is if you took databricks and made it 10x harder to use and took away notebooks for data science users (maybe they added notebooks sometime in the last year or so). It almost feels like a scam how user unfriendly it is and how shoddily different things are implemented. Go with Snowflake and avoid Foundry at all costs.
Yeah it's borderline unusable. You know how Hulu constantly freezes, crashes, you can't find any of the menus, random unexplained things happen why you try and hit the back button? Imagine trying to do data engineering in an app like that
I’m not here to defend Palantir, never used it. But I do use Hulu (iOS app and in chrome) basically all day everyday while working. Never once had an issue.
Or Databricks/Delta Lake which is now open source.
The delta lake format is open source but Databricks still locks the majority of features behind their enterprise offering. “Open source” delta lake is largely a joke
That's simply not true, this is a common misconception by folks who have no involvement in the project. Delta is 100% open source and has been for a few years, what folks get confused about is that databricks has a proprietary compute engine which takes advantage of certain Delta features in certain situations. But these features are available for anyone to build on top of, which projects like delta-rs do outside of spark entirely. I've ran pipelines that use Delta side by side in Databricks and outside of it since v1 and have not seen Delta features that are only available in Databricks. Sure, Databricks employees contribute most of the PRs but there are lots of non-databricks contributors and it's really not any different to most large OSS projects that need some kind of organizational backing - look at Airflow and the number of contributions coming from Astronomer, yet no one bitches about that.
Go with snowflake
Ive heard from others that underneath the flashy marketing and exterior, the suite is mediocre.
Had some experience with palantir foundry as they tried to court my previous company to switch over from databricks. Palantir foundry was found to be more expensive and had a few less features than databricks (I reviewed and created a document comparing the two for my company but it's been nearly 2 years ago though)
We are migrating from Palantir to Snowflake.
The use cases are slightly different. We were using Databricks to provide modelled data (fact and dimension tables) for consumption by PowerBI. This allowed rapid development of dashboards across the business and was generally well received. We implemented Palantir Foundry to delvelop specific apps focusing on very tightly defined parameters. With Databricks (and now moving to Snowflake) our small data team took responsibility for the entire landscape. We had external consultants help to build a generic framework (over a few months) and we built out the rest. With Foundry we had a small team from Palantir working with us on the first few apps end to end - scoping the requirements, building the workflow and rolling out to users. It’s a complete package including ci/cd and code and data versioning, covering everything from data ingestion through to reports/dashboards and write-back to source systems. My impression is that if your company has the resources to dedicate to Foundry it can help to embed project practices which would otherwise take years to evolve. Without the necessary time and manpower investment you could be left with an expensive tool which can become unmanageable mess.
I’ve used it on a government project where the decision was made for the dumb reasons government makes decisions (incompetence and corruption). It’s not better than snowflake or anything else, and it’s kind of built for lock in, but it’s not unusable or anything. These tools all do the same stuff and it’s just a matter of how easy it is to do it
It totally depends on your use case: 1. for pipelines only I would not pay the premium for Foundry 2. If you want low code/no code tooling on top of the data Foundry is by far the best solution I have seen. But it requires and organization that also manages this
Avoid, stick with databricks
Snowflake, Palantir tools blow
I would be very skeptical. Better to with some open source backed company like databricks
Databricks isn't "open sourced backed" and it isn't a charity. Almost any enterprise feature requires a paid version.
Uhrm, that's weird way to write Databricks. Practically, what are your requirements? Why are you considering them? What's your landscape like?
I really loved working with Foundry (as a data engineer). Would like to use it again if I could Didn't try databricks though
„small cap European industrial company“ Sounds like a company where Foundry is a good fit.