Frontside the Podcast g565t

In this episode, Austin Parker, Principal Developer Advocate at Lightstep talks about the OpenTelemetry Framework, which is an observability framework for cloud-native software and a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior. TRANSCRIPT: CHARLES: Hello everybody, and welcome to The Frontside podcast. My name is Charles. I’m a developer here at The Frontside and with me today also from The Frontside is Taras. Hello, Taras. TARAS: Hello, Charles. And today we are here with Austin Parker, a Principal Developer Advocate at LightStep. Welcome, Austin. AUSTIN: Hello. Thanks for having me. It's great to be here. CHARLES: Yeah. And we are going to talk about a project called OpenTelemetry, and I guess we should start by talking about Telemetry in general. And then we'll talk about how you come to be working on OpenTelemetry at LightStep and what kind of relationship is there. But maybe we could just start by filling people in on what Telemetry with the capital T is, oh sorry, with the little t, and what the OpenTelemetry project is. AUSTIN: So it is Telemetry with a capital T. I think the easiest way to explain it is something we're all familiar with, logging. How many times is the first thing you do when you sit down and it's like, I'm going to write a program, or I'm going to write something? You console.log("Hello, World!"). That's really the basis of how we understand what our programs are doing. And those logs are a type of Telemetry signal. They're a signal from our process, from our application that says what it's doing. So obviously when you get a more complex application and you add more services, the scale of your application increases, and you have more components and more things you need to be aware of both the quantity and the type of Telemetry that you use will increase. So a lot of times you're going to sit there and say, “Hmm, yeah. I have all these logs and that's great, but I need to know a number. I want to know how many requests per second do I have, or what's the latency of a request.” So that's a time-series metric, so you can express that. You write some instrumentation code or you use a library that does it for you, and it emits a number that is p50 or average latency. And then you keep adding services. You keep adding all this stuff. And you're like, ah, now I have 20 or 30 or 2000 different services running and a single request is going to hop through 15 or 30 or 500 of them, and gosh, that's really hard to understand by just metrics and logs. And so you can go in and add a distributed tracing, and that's a way to look at the entire request as it flows from all of these services. And those three concepts, logs, metrics, and traces are sometimes referred to as the three pillars of observability, and that's what OpenTelemetry is about; it's an observability framework for cloud-native applications that really seeks to make all of those things I just talked about: metrics, logs, and tracing a built-in integrated feature of cloud-native software. So, how does it do this? The basic idea behind the project is -- let's actually go back to logs because it's the simplest one. You probably have a logging framework you use. And there are a lot of options, and it depends on your language. It depends on if you're using Java, then you probably have a log for J or if you're using the spring framework, then there's logging things there. If you're using Go, then there might be Zap or other types of structured loggers. If you're using Python, there are loggers there if you’re using C#, there are loggers there. And the cool thing about cloud-native is that it lets us ignore, you know, I don’t have to have a system comprised just of Java or just of Go. I can have multiple different types of applications and application frameworks and languages and all this stuff playing together abstracted from me. But it can be really challenging to integrate all those different log sources together. The same thing happens with metrics, and the same thing happens with traces. You need some neutral piece in the middle. You need some sort of API that can be used by not only people developing applications but also library authors also people writing infrastructure components, things like queue libraries or SQL clients or whatever. If they want to provide Telemetry to their s, to people that are integrating their software, you want a standardized way to express these logs and traces and metrics. And to get value out of those, it's not enough to just have them all go to a file or go to standard out. You want to put them somewhere. You want to put them in a tool that can help you analyze that data and give you some insight about what's going on. You also need a way to have a common exposition format for them so that you're not writing endless levels of translators. So OpenTelemetry provides all of these things. It's an API right now creating distributed traces and metrics. It's also an SDK that you can integrate that will produce those traces and metrics. And then it's also a set of tools that can integrate traces, metrics, and logs that are created by all sorts of different existing libraries, tie them together, and put them into a common format that your analysis tools can understand. So if I have a bunch of Apache logs and I've got Prometheus metrics and maybe I've got Jaeger traces, I can throw all those together in an OpenTelemetry collector. And then it can translate them to a bunch of different things. It can send them to proprietary vendors. It can send them to other open-source tools. It can transform them into a common format called OTLP and write them to object storage so that I can go back later and do queries across them using SQL dialect. It’s a Swiss Army Knife almost for all this different Telemetry data. And the eventual goal is that this will all be integrated into the rest of the cloud-native ecosystem so that you, as an end-, you're not having to go implement a bunch of this stuff. It’s there; it's a toggle you turn on. CHARLES: It’s just there. So how does this compare and contrast to more traditional approaches? I mean, this is certainly outside my personal domain of expertise, but how would this compare to tools like Splunk and Logstash? And are they in a similar domain and if so, what would be the differentiator? AUSTIN: That's a good question. There are a lot, I mean, there's obviously Splunk, like you said. And what I think you see is that there's -- maybe if I say metrics you think Prometheus or you think Nagios or you think Datadog. CHARLES: Nagios, now there is a name I haven’t heard in years. [laughter] Of all the podcasts in the world, Nagios just comes walking into mine. AUSTIN: Yeah, how many are name-dropping Nagios in 2021? CHARLES: [Laughs] AUSTIN: But if you think about it, traditionally, we've thought of this very much like a one-to-one mapping where you have a tool for logs. Maybe you're using Splunk for your logs and you've got a bunch of Splunk log collectors deployed everywhere. Technology changes, new things come along and you say, oh, well, now I want to add traces. I want to add an APM. So what am I going to do? Either I'm stuck with my existing vendor, or I want to get a new one. But if I want to get a new one, now I have to integrate a whole new thing. I have to integrate a whole new library, a whole new agent. I have to send data off to this other place, and now I've got two different tabs to click through. And you can be in this situation where you have a bunch of different really disconnected experiences and tools because you're using all these different proprietary solutions. The advantage of OpenTelemetry is that it takes that middle layer, the layer between your application code and your infrastructure, and at the top of this, you have your analysis platform, your observability tools, that's your Splunks, that's your Grafanas, that’s your LightStep or your Datadogs. Open Telemetry sits between those, and it collects the data from your application code, translates it, and then sends it up to whatever you're going to use to analyze it. And because it's open-source, it's all out there in the open. If something doesn't work right, you can see what's going on. You don't have to be limited or restricted by what your vendor offers you. Maybe you're using a version of a library that isn't instrumented by your vendor, by their proprietary agent with OpenTelemetry, you can go see did someone in the community already contribute this plugin so that my particular version of Express has instrumentation for metrics and traces? TARAS: We had this experience recently on a project where we had to integrate logging and tracing into a GraphQL application. And it's tricky because if you've never done it before, it's a bit of like what am I doing here? [laughs] How are we actually going to go about doing this? AUSTIN: Yeah, it can be very challenging. TARAS: Yeah. It's definitely not trivial. One of the solutions that we ended up going with in addition to -- so in a stack, it's a cloud-native platform that's being created internally and one of the challenges that we've seen is that the data ends up in a data lake. The logging gets collected and sent over, but we've found that a tool like Sentry provided better information about the logging that we have or the way that we could consume that information was much more detailed in Sentry, so we ended up actually integrating Sentry directly. But now we have a situation, we have a platform that is designed to do this that has Prometheus and has a bunch of things. But the vendor, an external vendor, actually offers a better developer experience. Is the granularity or the kind of data -- like, is anything getting lost in the process? So if you use something like OpenTelemetry between, for example, Sentry so if we could write to OpenTelemetry and then have that collected and sent over to Sentry, would you expect the quality of the data to be lost in the process? How transparent do you think that would be? AUSTIN: So I'm not 100% familiar with the instrumentation Sentry does. I know broadly what it does, but I don't know the details, so I don't want to speak poorly of it. There's nothing technically that would stop it from being just as good because at the end of the day, what we're hoping to encourage with this is that -- there are two parts like you said; there's the quality of the data that's coming out of the application process and then there's the analysis piece of that data that's being done by the proprietary tool. So since it's all open source, since the part where you're instrumenting the code to collect that data and you're building all these hooks and stuff like that that's all open source, there's no reason that the data quality can't be the same. The advantage of OpenTelemetry is that as the community improves these plugins and improves these integrations, then the benefits of that improvement are applied equally. So now it's not just like oh, you have to use Sentry because Sentry gives you a plugin that gives you the best data quality. You can use OpenTelemetry, and OpenTelemetry will naturally get that improved data quality through contributions and through people and vendor partners contributing their code back to it. So the real question then becomes which tool is going to work best for me? Maybe Sentry is going to do since it has some whiz-bang collection or sampling algorithm or a way to do analysis that sets them apart from other vendors and then they can compete on that. What I see happening right now a lot is people get forced into using things they don't want to use because of this problem, because it's like, well, I have to use this because this is what -- this might not be the best tool for me, but it's what IT approved, or it's what we already have a contract for. And we can't use this new open source thing that might work better for us, or we can't use this other proprietary thing as a point solution to fix or to give us a really good analysis about this one set of our services. Open Telemetry makes it very easy to take the same data and then type it off to different places. So you can actually do stuff like, hey, I have stuff from the web. I have a web app and then I have a back-end app and all that data is going to OpenTelemetry collectors. And then maybe I take all the web stuff and I forward it to Sentry and then all the back end stuff and I forward it to Jaeger. So there are a lot of options. It’s like Kubernetes in that way. Right now OpenTelemetry works at a pretty low level, which is good and bad. It's good in the sense that it's very easy to make it do what you want it to do because we're giving you access to a lot of stuff really down in the weeds. It's maybe bad because it's a little hard to get started with because of that. You have to know what you're doing. So that's one of the things that I think the project is working on. What we’re really focused on over the next few months is trying to make that getting started experience a lot easier. CHARLES: Could I ask a little bit? Because this is the question that was brewing in my mind as I was listening and then you started talking about okay, we've got a front end, we've got a back end, and maybe we've got an app. What does the developer experience look like now if I want to start collecting metrics with OpenTelemetry? Do I install an SDK? Do I install some middleware into my app? What does it look like if my back end is, you know, I've got Node services on the back end, and I've got an iPhone app and a web app, how would I work in each of those different environments? AUSTIN: That's a good question. The overall process is pretty similar in each language. You are going to install an API, and you're going to install an SDK, and then you're going to install the appropriate integrations. Now, in some languages, this is a lot easier than others, probably the easiest is Java. So let's say I have a spring boot application. I can go a repository; it’s OpenTelemetry-Java auto-instrumentation. I go there, they publish a JAR file. I that JAR, I add the Java agent flag to my command line and I give it some config properties. And then that code when the application starts up, it'll look at all the libraries. It'll see what has instrumentation available? What has plugins available? It'll hook into all those and then it'll start collecting data and sending it off wherever I told it. So that's the easiest. And honestly, for languages that that integration, that's what we would love to see everything get to. For other stuff, it's a little trickier right now because one of the good things and bad things about a project like this is that it's designed to be very broad. We're not trying to make a really good thing for Java and just Java. We're trying to make a really good thing for pretty much every language, every modern language, at least which includes stuff like Swift and PHP, even though I just said modern language. But we use PHP, right? CHARLES: [laughs] PHP is a postmodern language. AUSTIN: Exactly. But a lot of people use it. [laughter] It's not abandoned, and it's not dead. It's actually like there's a ton of PHP in production. The problem is that you have different levels of -- there are just different amounts of in the community for those languages. There are a lot of people that know Java and Go really well and JavaScript and Node and things like that that are working on it, and so that's where a lot of heat and light is. Maybe the Swift repo is a little behind. It's not quite there. One way we've helped with this as a project or it will help you as an end- understand the maturity of things is that when you go in, you go and look at one, there's a maturity matrix. This is all being done through -- there's a specification and our big recent thing is that specification hit 1.0 for the distributed tracing part of this. And then the next thing will be metrics, and then the next thing will be logs. So those metrics and logs are both technically in an alpha/experimental stage. It really just means they can change, but we're not throwing out the entire thing and starting over. But you can see oh, Swift is maybe a little bit behind. Swift doesn't everything. It’s not completely up-to-date to the spec and maybe s spec version 0.5 rather than spec 1.0. But you would go and you would install the API; you would install the SDK and then if integrations existed, you would import those based on however it works. Like in Node, if you have a Node app, there are instrumentation plugins for Express. And so you can import that and then that as a middleware and then that will start tracing your incoming requests. There's an instrumentation plugin for HTTP and HTTPS, so you can import that and you can have that hook into your outgoing HTTP requests. There are plugins for gRPC. So if you're using gRPC for RPCs, you can bring that in, configure your gRPC server and client with those as middleware and go from there. It's all designed to be somewhat conventional to the language it's in. Each language SIG is supposed to hue to whatever the conventions of their language are for lack of a better term. So we want the Java stuff to look and feel like Java stuff. We want the C# stuff to look and feel like C# stuff. We want the Node stuff to look and feel like Node stuff. The advantage though is because all of these are running off the same specification, the default state, everything should work together. So if I have my front end single-page app in React and I go and I Install the React instrumentation and I install the XMLHTTPRequest instrumentation to my web app and then I have my Express server that's running the Express plugin, when I give a request from the single-page app back to my API server running in Express, that should emit a single trace. Then it'll handle all the tricky things like injecting context, extracting context. All the things that used to be really hard in of inter-op between multiple systems, out of the box, most of that stuff should just work. And as the project matures, it'll get from it should just work to it will just work. [laughter] TARAS: And it does just work. AUSTIN: The spec is 1.0 but the implementations aren't quite there yet, but in Java, Go, Node, Python probably by the time this is out most of those will have their first release candidate for 1.0. And that'll be for tracing and context and stuff. The metrics stuff should be 1.0 by this fall. But one other part of it also is we're not trying to sit here and say like, oh, you have to change everything. Because a big part of this is playing nicely with stuff that already exists in the cloud-native ecosystem so Prometheus and OpenMetrics, for example. We have our own metrics API that exists for a lot of reasons and I think solves a lot of problems that people writing metrics have identified in the past. But we also want to make sure that, hey, you're using Prometheus? Cool, that will still work with OpenTelemetry, and you can integrate that and you can have your Prometheus stuff play nicely in this ecosystem. We're not writing our own logging API. There are already a million logging APIs, and people are generally pretty happy with whatever they use. So we don't want to come in and reinvent the wheel. What we want to say is that okay, cool, your logging API can integrate with this idea of OpenTelemetry like the context layer of it so that you can correlate your existing logs with the traces or the metrics that OpenTelemetry is generating. We want to be a good -- I don't want to say we want to be invisible, but we don't want people to have to change the way they do everything. We want to work with what you're already doing. A huge part of this project was making sure that it's composable. You can reimplement parts of it if they don't work for you. Or if you already have a legacy tracing system, because a lot of people do, they might have a more basic one. They might have just simple correlation IDs that are sent through logs. But we want to make sure that hey, you've got some system that you wrote five years ago and it works well for you and you're thinking oh, I want to do something more advanced? Well, cool, we designed this in a way that you should be able to without a ton of extra work stitch that old stuff and this new stuff together and have everything work in harmony. TARAS: I can see this being really powerful especially in the context of now with cloud-native platforms and companies are creating their own internal cloud-native platforms. And most quite often building a cloud-native platform from scratch no matter how many people you have, you are probably going to be understaffed. So having something like OpenTelemetry as an API, like a document, that API that you could use right off-the-shelf so you don't have to document your own solution, which I think is often the biggest challenge with integrating with cloud-native platforms is that the infrastructure that a developer is supposed to use it's usually not documented enough. And so having something like OpenTelemetry where you could just take it right off-the-shelf and then have the platform essentially say just use OpenTelemetry in whatever language you're using to write your service, I think that's a really big value. So, developers don't need to learn or reverse engineer their particular platforms set up for logging and maybe how that thing works with Jaeger and all that stuff. I think it would be really helpful there. AUSTIN: I think one other thing as well by offering an open-source standard for this, something that is very strictly being done under the auspices of the CNCF -- and so there's a governing structure that does prevent this quite deliberately from ever being taken over by one vendor, which was something that I think a lot of people were concerned about. And I'm concerned about it. I work for a pretty small company compared to a lot of people in this space. Nobody wants this to just become the fang show. Nobody really wants there to be a situation where the big five or the big three cloud vendors of the world are calling the shots and everyone else just has to follow along without at least having the ability to influence things. So the governing structure of this project is set up to ensure that there is representation from multiple companies on the governance and technical committees. You can't just have one thing take it and run. But I think what that helps with is that since that does make it truly agnostic, it allows these big cloud players to say like, okay, well, here's something we haven't done before. Like if you think about Amazon, let's assume you're running everything on Amazon, there are tools in Amazon to give you -- there are things like X-Ray and then CloudTrail and CloudWatch and stuff where it's like, okay, you can get visibility into metrics and traces that maybe go into managed services. You can see a trace about S3 or about Aurora or whatever. OpenTelemetry by existing outside of this a little bit lets them say like, “Okay, we'll also make this data available in OpenTelemetry format,” and that way, you can take it. Stuff that previously you could only use within the confines of a proprietary vendor in their tools, now you can take that and export it and send it somewhere else. So you can send it to your own open-source, maybe you built your own observability stack, cool. You can take that data that's coming to you in OTLP, and you can put it into your stuff. You can take it and put it into another proprietary vendor. That gives you I think a lot of freedom and flexibility as an end- to build what works for you. I don't think everyone needs the same thing is maybe the best way to put it. Some people don't have either the operational complexity or the organizational maturity or whatever for a big hotshot observability solution. Not everyone needs all the features of every single platform out there either. Some people are going to be fine using completely off-the-shelf open-source thing, or they're going to want to do their own weird custom stuff. Maybe they're providing Telemetry back to their end-s in some way so they need something that lets them split-brain this and send some stuff back out to the internet and to who knows. So the advantage of OpenTelemetry now is because it acts as this neutral agnostic standard in the middle, suddenly vendors that before wouldn't provide Telemetry data to you or couldn't provide Telemetry data to you now can say in the future, “Oh yeah, we offer OTLP. We offer this Telemetry, these traces, these metrics, these logs, or whatever, and you just give us an endpoint to write to, and we'll send you this stuff in OpenTelemetry format. And then it'll integrate with whatever else you got as long as it's all talking OpenTelemetry as well.” CHARLES: So who else is involved in the Telemetry project/consortium? You mentioned it was part of the cloud-native...was it Cloud Native Foundation? AUSTIN: Yeah. So it's like a Cloud Native Computing Foundation sandbox project right now. We're hoping to move into incubating this year. We're actually one of the most popular CNCF projects by contributor so second or third only to Kubernetes. It depends on when you look at the stats. But we have contributors and governing from everyone from Google to Amazon, Splunk, Datadog, New Relic, Honeycomb, LightStep, Dynatrace, AppDynamics is in there a little bit. Any sort of observability company you can think of anyone that's selling dashboards and traces and metrics and all that stuff they're involved, Grafana, Elastic is involved. So there's an extremely broad base of . It's been really exciting to see all of these different vendors that have their own agendas come together and work on this. I think, for the most part, it's all with very pure motives. TARAS: Yeah. The cloud-native community to me is very inspiring because it's really great to see how this vendor-neutral approach they've taken has worked out. It seems to be working. I'm sure it's not perfect. I'm sure it has some rough edges, but it's like what you're saying it's different companies that are operating in the same space that are coming together to have something that does bring value and does provide flexibility. That's really great to see especially for something like cloud-native space where we are talking about building huge platforms that require huge investments over a long time. So being able to have flexibility and be able to have evolution is really important. AUSTIN: Well, a lot of it also especially, in this case, let's go back to earlier talking about Sentry and the data quality issue. The question of instrumenting all these different existing pieces of code, was actually really silly when you think about it because maybe you were using New Relic. Oh, I'm sorry. I forgot to shout-out New Relic. I don't if I shouted-out New Relic, but they're also very involved in OpenTelemetry. Before, you would say like okay, maybe I'm using New Relic, and I have this proprietary agent that's going to do this instrumentation and send data to their APM endpoints and then Datadog says, “Well, we want to do an APM tool.” So Datadog has to go in and they have to reimplement all this stuff and signal effects before they [inaudible 30:12] by Splunk. It's like oh, we have an APM tool, so we have to reimplement all this stuff and so on and so forth. And it's silly because there's only really a layer of software there, that layer of instrumentation for Express or whatever is completely undifferentiated. It's just like I have a hook on something starting, and there’s something finishing. CHARLES: Right. The value is in what you do with the data not in actually saying here's the data. AUSTIN: Right. These were incredibly undifferentiated, incredibly commodified instrumentation libraries. But all these different vendors had to provide and maintain roughly the same code base duplicated across 5 or 6 or 10 or however many different companies. And so I think a lot of the inspiration behind OpenTelemetry is simply that one, wouldn't it be nice if we didn't have to do that? There are a bunch of individual vendors -- I'm not going to put anything on anyone's motive here, but there's certainly a story in open source about socializing maintenance costs. But I think the bigger thing is probably it's not so much about the today; it's about the tomorrow. And the tomorrow looks a lot more like well, if there is an actual open standard that everyone agrees on, there's an API that everyone accepts and there's an exposition format for this data that is neutral and that we can all work with, then why not just that in the hopes that upstream maintainers will integrate it natively into their libraries? So instead of having to install a plugin to Express to get it to do tracing, why doesn't Express just have OpenTelemetry ? And so I think that's what you're going to see over the next couple of years is this API hits 1.0 because we have stability guarantees on this for three years. So once we get an actual 1.0, I think there's going to be a bit more of like oh well, we'll just take this code that was previously this third-party plugin that you installed. And instead of having this external middleware package, what if there was just a built-in tracing middleware for Express that when you create your Express server or you just say, “Express that use tracing.” And then it'll just look for OpenTelemetry, an OpenTelemetry SDK, and ta-da! Now, this is not an OpenTelemetry problem; this is an upstream framework maintainer problem. And that also helps everyone too because that means that the people that actually write the software, the people that actually write the frameworks and libraries can start to have better control over the instrumentation and can say, “This is what we think is really important. We can provide a recommended dashboard or a recommended troubleshooting guide that says, hey, this is what you should watch for. These are the attributes that are important for you to be aware of. This is maybe a good metrics query that you can run or an alert or whatever that we can tell you about because now there's this common language for expressing those metrics, there’s a common language for expressing those traces for expressing the semantic attributes of a request.” And I think that could be I think a pretty big revolution in the way that we, as an industry, think about performance and talk about performance and how to optimize performance and understand incidents and things like that just because now we have this lingua franca of OpenTelemetry to discuss these sort of problems in. TARAS: The topic of Telemetry, so is there overhead to -- I don't really know how to ask this question, but I’m curious what is the standard that you measure for performance of any of these tools? Is there an acceptable impact that you would see? In theory, I would imagine there'd be like no impact at all, but is there overhead that's introduced when you start to do tracing that is measurable? AUSTIN: I mean, it depends on the perspective. So what I would say is this: we have standards and specifications. We've specified benchmarking suites and things like that. That part of this journey to GA includes these full benchmarking suites and testing and being able to give a quantified answer to this question. What I would say is from an end- perspective, generally, no, there's no real end- impact or at least not a ton. So if you think about how a span is created, it's an allocation, and you have a span for every single request, if you're doing 100% sampling, you're creating spans for every single request that moves through your server. So there is some overhead because every single request is now going to have the additional overhead of oh, I had to create a span and that span needs all these attributes and so on and so on and so forth. But the actual act of exporting that data and doing that stuff is done off the main threads, so to speak. So that part is pretty snappy, and we've done, I think, the best we can, and we can always do better. But certainly, when we developed these SDKs, they were done with an eye towards performance. So where you can, avoid unnecessary allocations, where you can, avoid expensive operations. The best case where you're most likely going to see it is on the export side of things because that requires you have to write this data somewhere. And so you're either writing over a network link or you’re writing into a file. If you're writing into a file, then that's pretty snappy. But over a network link, then you obviously have the additional overhead of creating requests out to this other service through gRPC or through HTTP or whatever, and then compression and TLS negotiation and all that. That said, in my experience, most of the overhead people incur with distributed tracing aren't necessarily -facing impact. It's not like oh, I've added 10 milliseconds round trip. It's mostly memory pressure or U overhead on my process goes up because I'm just doing a little extra work for every single request and a little extra work for every single request will add up. So that just means you need to horizontally scale more. In my opinion, the benefit of tracing far outweighs the trade-off in headroom just because having that ability to zoom in on one individual request and see what happened at every single step is invaluable when you're dealing with distributed system problems and having 100% -- this is one thing that we do at LightStep. We encourage you to use 100% on sampled data and send that to us and then we use fancy dynamic sampling stuff to make it manageable, but the ability to just take this entire firehose of data and look and drill all the way down to one individual request and one individual service out of potentially millions is it's really unimaginable. And this is a tangent, and this is not a shill, I promise, but Spotify uses us. And so I've talked to some of their engineers, and you can imagine the request load of Spotify at every second of every day the amount of traffic moving through that system, and the ability that they have to look into those billions and billions of requests and narrow it down and just say oh, I want to find this one error or this error that's localized to this zip code is just incredibly powerful, and that's something that's really hard to do I think with traditional -- certainly, it's hard to do with just metrics. It's annoying to do with metrics and logs, but when you have metrics and logs and traces and you’ve integrated all of these things, it becomes easier to do. And it also lets you fix problems a lot faster and also understand how problems are impacting actual s rather than aggregate groups of s. CHARLES: Yeah, that actually sounds amazing. TARAS: Do you encourage to capture all the traces and then provide tooling to find a needle in a haystack? AUSTIN: To be a little more clear here, OpenTelemetry lets you pick how you sample, and we offer tools to help you do things like tail-based sampling where you send all of your traces to a pool of OpenTelemetry collectors and then you ensure that all the spans for one trace wind up on the same collector, and then you can make a tail-based decision about hey, does this have an error, or does this meet some criteria? Now, LightStep also s OpenTelemetry. We actually natively ingesting the OpenTelemetry protocol and our unique architecture allows us to take 100% of that data with no upfront sampling and then do dynamic sampling on it where we can create trace aggregates, and we can do a lot of really cool stuff to not only give you that high-level picture but also let you see we can pick the important stuff out to keep for longer for that long-term analysis. I didn't necessarily come here wanting to talk about that too much, but if you're interested, lightstep.com, we have a bunch of very fun marketing things that can tell you about how great we are. TARAS: Well, it's interesting. Part of the reason why I wanted to talk to you about this is because one of the challenges that Frontside has been dealing with is we’re building this brand new testing platform. It's essentially a really complex distributed system that you run locally. I mean, from the developer perspective or from the perspective, a lot of the complexity of the implementation is to make it really easy for developers and make it really fast. So we have all these perks, but one of the trade-offs you make when you have a distributed system is that you have different processes that are communicating over HTTP and in some cases over web sockets. And then when something breaks down, you're like, I have no idea what the hell is going on. [laughs] And that's the situation that we're dealing with. And one of the things I was thinking about is actually integrating OpenTelemetry into the library that we use to create the testing tool so we can have a Jaeger trace that shows us a structured log of the particular operation. So you run a test, and you can actually see the entire structured trace of the test. And then the coolest thing about this is that when you have this as a starting point, then you can actually extend this with the test results that reach all the way into the system. So the root is the initiation of the execution of the test, but your trace can include the data that is coming from essentially one long trace going all the way into the infrastructure. So you can actually get at a place where you see the output. You can actually see information about what's deeply inside of the microservice that is providing the data for the test result. That seems like a really appealing thing to be able to do. AUSTIN: Totally. I think that’s a really good example of when you would want to do no upfront sampling because -- and this is a challenge with the way -- I think people have this idea they’re like oh, sampling is...there's maybe two camps here. There's you can't do this without sampling, and then there's you sample everything. You can't lose anything. And the truth is in the middle because, in any production system, 90%, 99% of the data is honestly not that useful. The reason why is because a system at rest tends to stay at rest, or you tend to get very similar stuff. And a system where everything is broken, everything is going to be broken in a similar way. With OpenTelemetry, we give you the ability, I think, to separate the sampling decision and move them to a different layer. So instead of saying, “I have to make these sample decisions in my process,” it’s more okay, you can exfiltrate your data to a collector or to a pool of collectors and then you can do the sampling there, which I think will unlock in the future more advanced sound sampling algorithms and the ability to make better decisions about your data and what gets sampled and what gets sampled out. For something like you were describing, actually, you'd probably want to err on the side of yeah, keep everything because you're not running tests every single minute of every single day. And you do want to have that full visibility into everything that's going on because tests can tend to be, from run to run, there can be a high level of variance in what actually happens because the code is changing all the time, right? You're not generally running the exact same tests every time. I would definitely say OpenTelemetry would be a perfect thing to integrate there to get that visibility into what's going on. TARAS: Well, we'll give it a shot, and we'll let you know how it goes. [laughs] AUSTIN: Hey, poke me on Twitter if you have any questions. TARAS: Yeah, awesome. And where can people find you on Twitter? AUSTIN: Yeah, I'm @austinlparker. I tweet regularly. It's like the only social media I pay attention to. CHARLES: All right. Well, thank you so much for coming by, Austin L. Parker. We already mentioned you work for LightStep, lightstep.com. Where can people go to find out more about the OpenTelemetry project? Is there a URL there? AUSTIN: Check out our website opentelemetry.io and that has links to our GitHub, to our chat on the CNCF Slack. You can find meetings for all the SIGs, and you can start there, best place. CHARLES: Fantastic. I love the longer that I'm a developer, I love just how in many ways it becomes a more and more comfortable experience because all of these things that were hard to do or that you didn't even know about you should be doing all of a sudden you just wait a few years and they just happen for free, and you're like, oh, wow. AUSTIN: Yeah. CHARLES: It's like you're Neo in The Matrix where you just keep learning all these different kungfus, and all you have to do is just flutter your eyes for a little bit. So it definitely sounds like this is one of those things that is in the works and that people are going to get to benefit from with very little effort going forward. AUSTIN: Yeah, definitely. Like I said, check it out. By the time this goes up, you should see a lot more 1.0s starting to pop up out there. So we definitely want . CHARLES: Well, fantastic. Thank you again and thank you, everybody, for listening, and we will see everybody next time. Thank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain, old email at @frontside.io. Thanks and see you next time. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript\r\n\nCHARLES: Hello everybody, and welcome to The Frontside podcast. My name is Charles. I’m a developer here at The Frontside and with me today also from The Frontside is Taras. Hello, Taras.\r\n\nTARAS: Hello, Charles. And today we are here with Austin Parker, a Principal Developer Advocate at LightStep. Welcome, Austin.\r\n\nAUSTIN: Hello. Thanks for having me. It\'s great to be here.\r\n\nCHARLES: Yeah. And we are going to talk about a project called OpenTelemetry, and I guess we should start by talking about Telemetry in general. And then we\'ll talk about how you come to be working on OpenTelemetry at LightStep and what kind of relationship is there. But maybe we could just start by filling people in on what Telemetry with the capital T is, oh sorry, with the little t, and what the OpenTelemetry project is.\r\n\nAUSTIN: So it is Telemetry with a capital T. I think the easiest way to explain it is something we\'re all familiar with, logging. How many times is the first thing you do when you sit down and it\'s like, I\'m going to write a program, or I\'m going to write something? You console.log("Hello, World!"). That\'s really the basis of how we understand what our programs are doing. And those logs are a type of Telemetry signal. They\'re a signal from our process, from our application that says what it\'s doing. So obviously when you get a more complex application and you add more services, the scale of your application increases, and you have more components and more things you need to be aware of both the quantity and the type of Telemetry that you use will increase. So a lot of times you\'re going to sit there and say, “Hmm, yeah. I have all these logs and that\'s great, but I need to know a number. I want to know how many requests per second do I have, or what\'s the latency of a request.” So that\'s a time-series metric, so you can express that. You write some instrumentation code or you use a library that does it for you, and it emits a number that is p50 or average latency. And then you keep adding services. You keep adding all this stuff. And you\'re like, ah, now I have 20 or 30 or 2000 different services running and a single request is going to hop through 15 or 30 or 500 of them, and gosh, that\'s really hard to understand by just metrics and logs. And so you can go in and add a distributed tracing, and that\'s a way to look at the entire request as it flows from all of these services.\r\n\nAnd those three concepts, logs, metrics, and traces are sometimes referred to as the three pillars of observability, and that\'s what OpenTelemetry is about; it\'s an observability framework for cloud-native applications that really seeks to make all of those things I just talked about: metrics, logs, and tracing a built-in integrated feature of cloud-native software. So, how does it do this? The basic idea behind the project is -- let\'s actually go back to logs because it\'s the simplest one. You probably have a logging framework you use. And there are a lot of options, and it depends on your language. It depends on if you\'re using Java, then you probably have a log for J or if you\'re using the spring framework, then there\'s logging things there. If you\'re using Go, then there might be Zap or other types of structured loggers. If you\'re using Python, there are loggers there if you’re using C#, there are loggers there. And the cool thing about cloud-native is that it lets us ignore, you know, I don’t have to have a system comprised just of Java or just of Go. I can have multiple different types of applications and application frameworks and languages and all this stuff playing together abstracted from me.\r\n\nBut it can be really challenging to integrate all those different log sources together. The same thing happens with metrics, and the same thing happens with traces. You need some neutral piece in the middle. You need some sort of API that can be used by not only people developing applications but also library authors also people writing infrastructure components, things like queue libraries or SQL clients or whatever. If they want to provide Telemetry to their s, to people that are integrating their software, you want a standardized way to express these logs and traces and metrics. And to get value out of those, it\'s not enough to just have them all go to a file or go to standard out. You want to put them somewhere. You want to put them in a tool that can help you analyze that data and give you some insight about what\'s going on. You also need a way to have a common exposition format for them so that you\'re not writing endless levels of translators. So OpenTelemetry provides all of these things. It\'s an API right now creating distributed traces and metrics. It\'s also an SDK that you can integrate that will produce those traces and metrics. And then it\'s also a set of tools that can integrate traces, metrics, and logs that are created by all sorts of different existing libraries, tie them together, and put them into a common format that your analysis tools can understand. So if I have a bunch of Apache logs and I\'ve got Prometheus metrics and maybe I\'ve got Jaeger traces, I can throw all those together in an OpenTelemetry collector. And then it can translate them to a bunch of different things. It can send them to proprietary vendors. It can send them to other open-source tools. It can transform them into a common format called OTLP and write them to object storage so that I can go back later and do queries across them using SQL dialect. It’s a Swiss Army Knife almost for all this different Telemetry data. And the eventual goal is that this will all be integrated into the rest of the cloud-native ecosystem so that you, as an end-, you\'re not having to go implement a bunch of this stuff. It’s there; it\'s a toggle you turn on.\r\n\nCHARLES: It’s just there. So how does this compare and contrast to more traditional approaches? I mean, this is certainly outside my personal domain of expertise, but how would this compare to tools like Splunk and Logstash? And are they in a similar domain and if so, what would be the differentiator?\r\n\nAUSTIN: That\'s a good question. There are a lot, I mean, there\'s obviously Splunk, like you said. And what I think you see is that there\'s -- maybe if I say metrics you think Prometheus or you think Nagios or you think Datadog.\r\n\nCHARLES: Nagios, now there is a name I haven’t heard in years. [laughter] Of all the podcasts in the world, Nagios just comes walking into mine.\r\n\nAUSTIN: Yeah, how many are name-dropping Nagios in 2021?\r\n\nCHARLES: [Laughs]\r\n\nAUSTIN: But if you think about it, traditionally, we\'ve thought of this very much like a one-to-one mapping where you have a tool for logs. Maybe you\'re using Splunk for your logs and you\'ve got a bunch of Splunk log collectors deployed everywhere. Technology changes, new things come along and you say, oh, well, now I want to add traces. I want to add an APM. So what am I going to do? Either I\'m stuck with my existing vendor, or I want to get a new one. But if I want to get a new one, now I have to integrate a whole new thing. I have to integrate a whole new library, a whole new agent. I have to send data off to this other place, and now I\'ve got two different tabs to click through. And you can be in this situation where you have a bunch of different really disconnected experiences and tools because you\'re using all these different proprietary solutions.\r\n\nThe advantage of OpenTelemetry is that it takes that middle layer, the layer between your application code and your infrastructure, and at the top of this, you have your analysis platform, your observability tools, that\'s your Splunks, that\'s your Grafanas, that’s your LightStep or your Datadogs. Open Telemetry sits between those, and it collects the data from your application code, translates it, and then sends it up to whatever you\'re going to use to analyze it. And because it\'s open-source, it\'s all out there in the open. If something doesn\'t work right, you can see what\'s going on. You don\'t have to be limited or restricted by what your vendor offers you. Maybe you\'re using a version of a library that isn\'t instrumented by your vendor, by their proprietary agent with OpenTelemetry, you can go see did someone in the community already contribute this plugin so that my particular version of Express has instrumentation for metrics and traces?\r\n\nTARAS: We had this experience recently on a project where we had to integrate logging and tracing into a GraphQL application. And it\'s tricky because if you\'ve never done it before, it\'s a bit of like what am I doing here? [laughs] How are we actually going to go about doing this?\r\n\nAUSTIN: Yeah, it can be very challenging.\r\n\nTARAS: Yeah. It\'s definitely not trivial. One of the solutions that we ended up going with in addition to -- so in a stack, it\'s a cloud-native platform that\'s being created internally and one of the challenges that we\'ve seen is that the data ends up in a data lake. The logging gets collected and sent over, but we\'ve found that a tool like Sentry provided better information about the logging that we have or the way that we could consume that information was much more detailed in Sentry, so we ended up actually integrating Sentry directly. But now we have a situation, we have a platform that is designed to do this that has Prometheus and has a bunch of things. But the vendor, an external vendor, actually offers a better developer experience. Is the granularity or the kind of data -- like, is anything getting lost in the process? So if you use something like OpenTelemetry between, for example, Sentry so if we could write to OpenTelemetry and then have that collected and sent over to Sentry, would you expect the quality of the data to be lost in the process? How transparent do you think that would be?\r\n\nAUSTIN: So I\'m not 100% familiar with the instrumentation Sentry does. I know broadly what it does, but I don\'t know the details, so I don\'t want to speak poorly of it. There\'s nothing technically that would stop it from being just as good because at the end of the day, what we\'re hoping to encourage with this is that -- there are two parts like you said; there\'s the quality of the data that\'s coming out of the application process and then there\'s the analysis piece of that data that\'s being done by the proprietary tool. So since it\'s all open source, since the part where you\'re instrumenting the code to collect that data and you\'re building all these hooks and stuff like that that\'s all open source, there\'s no reason that the data quality can\'t be the same. The advantage of OpenTelemetry is that as the community improves these plugins and improves these integrations, then the benefits of that improvement are applied equally. So now it\'s not just like oh, you have to use Sentry because Sentry gives you a plugin that gives you the best data quality. You can use OpenTelemetry, and OpenTelemetry will naturally get that improved data quality through contributions and through people and vendor partners contributing their code back to it. So the real question then becomes which tool is going to work best for me? Maybe Sentry is going to do since it has some whiz-bang collection or sampling algorithm or a way to do analysis that sets them apart from other vendors and then they can compete on that.\r\n\nWhat I see happening right now a lot is people get forced into using things they don\'t want to use because of this problem, because it\'s like, well, I have to use this because this is what -- this might not be the best tool for me, but it\'s what IT approved, or it\'s what we already have a contract for. And we can\'t use this new open source thing that might work better for us, or we can\'t use this other proprietary thing as a point solution to fix or to give us a really good analysis about this one set of our services. Open Telemetry makes it very easy to take the same data and then type it off to different places. So you can actually do stuff like, hey, I have stuff from the web. I have a web app and then I have a back-end app and all that data is going to OpenTelemetry collectors. And then maybe I take all the web stuff and I forward it to Sentry and then all the back end stuff and I forward it to Jaeger. So there are a lot of options. It’s like Kubernetes in that way. Right now OpenTelemetry works at a pretty low level, which is good and bad. It\'s good in the sense that it\'s very easy to make it do what you want it to do because we\'re giving you access to a lot of stuff really down in the weeds. It\'s maybe bad because it\'s a little hard to get started with because of that. You have to know what you\'re doing. So that\'s one of the things that I think the project is working on. What we’re really focused on over the next few months is trying to make that getting started experience a lot easier.\r\n\nCHARLES: Could I ask a little bit? Because this is the question that was brewing in my mind as I was listening and then you started talking about okay, we\'ve got a front end, we\'ve got a back end, and maybe we\'ve got an app. What does the developer experience look like now if I want to start collecting metrics with OpenTelemetry? Do I install an SDK? Do I install some middleware into my app? What does it look like if my back end is, you know, I\'ve got Node services on the back end, and I\'ve got an iPhone app and a web app, how would I work in each of those different environments?\r\n\nAUSTIN: That\'s a good question. The overall process is pretty similar in each language. You are going to install an API, and you\'re going to install an SDK, and then you\'re going to install the appropriate integrations. Now, in some languages, this is a lot easier than others, probably the easiest is Java. So let\'s say I have a spring boot application. I can go a repository; it’s OpenTelemetry-Java auto-instrumentation. I go there, they publish a JAR file. I that JAR, I add the Java agent flag to my command line and I give it some config properties. And then that code when the application starts up, it\'ll look at all the libraries. It\'ll see what has instrumentation available? What has plugins available? It\'ll hook into all those and then it\'ll start collecting data and sending it off wherever I told it. So that\'s the easiest. And honestly, for languages that that integration, that\'s what we would love to see everything get to. For other stuff, it\'s a little trickier right now because one of the good things and bad things about a project like this is that it\'s designed to be very broad. We\'re not trying to make a really good thing for Java and just Java. We\'re trying to make a really good thing for pretty much every language, every modern language, at least which includes stuff like Swift and PHP, even though I just said modern language. But we use PHP, right?\r\n\nCHARLES: [laughs] PHP is a postmodern language.\r\n\nAUSTIN: Exactly. But a lot of people use it. [laughter] It\'s not abandoned, and it\'s not dead. It\'s actually like there\'s a ton of PHP in production. The problem is that you have different levels of -- there are just different amounts of in the community for those languages. There are a lot of people that know Java and Go really well and JavaScript and Node and things like that that are working on it, and so that\'s where a lot of heat and light is. Maybe the Swift repo is a little behind. It\'s not quite there. One way we\'ve helped with this as a project or it will help you as an end- understand the maturity of things is that when you go in, you go and look at one, there\'s a maturity matrix. This is all being done through -- there\'s a specification and our big recent thing is that specification hit 1.0 for the distributed tracing part of this. And then the next thing will be metrics, and then the next thing will be logs. So those metrics and logs are both technically in an alpha/experimental stage. It really just means they can change, but we\'re not throwing out the entire thing and starting over. But you can see oh, Swift is maybe a little bit behind. Swift doesn\'t everything. It’s not completely up-to-date to the spec and maybe s spec version 0.5 rather than spec 1.0. But you would go and you would install the API; you would install the SDK and then if integrations existed, you would import those based on however it works. Like in Node, if you have a Node app, there are instrumentation plugins for Express. And so you can import that and then that as a middleware and then that will start tracing your incoming requests. There\'s an instrumentation plugin for HTTP and HTTPS, so you can import that and you can have that hook into your outgoing HTTP requests. There are plugins for gRPC. So if you\'re using gRPC for RPCs, you can bring that in, configure your gRPC server and client with those as middleware and go from there.\r\n\nIt\'s all designed to be somewhat conventional to the language it\'s in. Each language SIG is supposed to hue to whatever the conventions of their language are for lack of a better term. So we want the Java stuff to look and feel like Java stuff. We want the C# stuff to look and feel like C# stuff. We want the Node stuff to look and feel like Node stuff. The advantage though is because all of these are running off the same specification, the default state, everything should work together. So if I have my front end single-page app in React and I go and I Install the React instrumentation and I install the XMLHTTPRequest instrumentation to my web app and then I have my Express server that\'s running the Express plugin, when I give a request from the single-page app back to my API server running in Express, that should emit a single trace. Then it\'ll handle all the tricky things like injecting context, extracting context. All the things that used to be really hard in of inter-op between multiple systems, out of the box, most of that stuff should just work. And as the project matures, it\'ll get from it should just work to it will just work. [laughter]\r\n\nTARAS: And it does just work.\r\n\nAUSTIN: The spec is 1.0 but the implementations aren\'t quite there yet, but in Java, Go, Node, Python probably by the time this is out most of those will have their first release candidate for 1.0. And that\'ll be for tracing and context and stuff. The metrics stuff should be 1.0 by this fall. But one other part of it also is we\'re not trying to sit here and say like, oh, you have to change everything. Because a big part of this is playing nicely with stuff that already exists in the cloud-native ecosystem so Prometheus and OpenMetrics, for example. We have our own metrics API that exists for a lot of reasons and I think solves a lot of problems that people writing metrics have identified in the past. But we also want to make sure that, hey, you\'re using Prometheus? Cool, that will still work with OpenTelemetry, and you can integrate that and you can have your Prometheus stuff play nicely in this ecosystem. We\'re not writing our own logging API. There are already a million logging APIs, and people are generally pretty happy with whatever they use. So we don\'t want to come in and reinvent the wheel. What we want to say is that okay, cool, your logging API can integrate with this idea of OpenTelemetry like the context layer of it so that you can correlate your existing logs with the traces or the metrics that OpenTelemetry is generating.\r\n\nWe want to be a good -- I don\'t want to say we want to be invisible, but we don\'t want people to have to change the way they do everything. We want to work with what you\'re already doing. A huge part of this project was making sure that it\'s composable. You can reimplement parts of it if they don\'t work for you. Or if you already have a legacy tracing system, because a lot of people do, they might have a more basic one. They might have just simple correlation IDs that are sent through logs. But we want to make sure that hey, you\'ve got some system that you wrote five years ago and it works well for you and you\'re thinking oh, I want to do something more advanced? Well, cool, we designed this in a way that you should be able to without a ton of extra work stitch that old stuff and this new stuff together and have everything work in harmony.\r\n\nTARAS: I can see this being really powerful especially in the context of now with cloud-native platforms and companies are creating their own internal cloud-native platforms. And most quite often building a cloud-native platform from scratch no matter how many people you have, you are probably going to be understaffed. So having something like OpenTelemetry as an API, like a document, that API that you could use right off-the-shelf so you don\'t have to document your own solution, which I think is often the biggest challenge with integrating with cloud-native platforms is that the infrastructure that a developer is supposed to use it\'s usually not documented enough. And so having something like OpenTelemetry where you could just take it right off-the-shelf and then have the platform essentially say just use OpenTelemetry in whatever language you\'re using to write your service, I think that\'s a really big value. So, developers don\'t need to learn or reverse engineer their particular platforms set up for logging and maybe how that thing works with Jaeger and all that stuff. I think it would be really helpful there.\r\n\nAUSTIN: I think one other thing as well by offering an open-source standard for this, something that is very strictly being done under the auspices of the CNCF -- and so there\'s a governing structure that does prevent this quite deliberately from ever being taken over by one vendor, which was something that I think a lot of people were concerned about. And I\'m concerned about it. I work for a pretty small company compared to a lot of people in this space. Nobody wants this to just become the fang show. Nobody really wants there to be a situation where the big five or the big three cloud vendors of the world are calling the shots and everyone else just has to follow along without at least having the ability to influence things. So the governing structure of this project is set up to ensure that there is representation from multiple companies on the governance and technical committees. You can\'t just have one thing take it and run. But I think what that helps with is that since that does make it truly agnostic, it allows these big cloud players to say like, okay, well, here\'s something we haven\'t done before. Like if you think about Amazon, let\'s assume you\'re running everything on Amazon, there are tools in Amazon to give you -- there are things like X-Ray and then CloudTrail and CloudWatch and stuff where it\'s like, okay, you can get visibility into metrics and traces that maybe go into managed services. You can see a trace about S3 or about Aurora or whatever. OpenTelemetry by existing outside of this a little bit lets them say like, “Okay, we\'ll also make this data available in OpenTelemetry format,” and that way, you can take it. Stuff that previously you could only use within the confines of a proprietary vendor in their tools, now you can take that and export it and send it somewhere else. So you can send it to your own open-source, maybe you built your own observability stack, cool. You can take that data that\'s coming to you in OTLP, and you can put it into your stuff. You can take it and put it into another proprietary vendor. That gives you I think a lot of freedom and flexibility as an end- to build what works for you.\r\n\nI don\'t think everyone needs the same thing is maybe the best way to put it. Some people don\'t have either the operational complexity or the organizational maturity or whatever for a big hotshot observability solution. Not everyone needs all the features of every single platform out there either. Some people are going to be fine using completely off-the-shelf open-source thing, or they\'re going to want to do their own weird custom stuff. Maybe they\'re providing Telemetry back to their end-s in some way so they need something that lets them split-brain this and send some stuff back out to the internet and to who knows. So the advantage of OpenTelemetry now is because it acts as this neutral agnostic standard in the middle, suddenly vendors that before wouldn\'t provide Telemetry data to you or couldn\'t provide Telemetry data to you now can say in the future, “Oh yeah, we offer OTLP. We offer this Telemetry, these traces, these metrics, these logs, or whatever, and you just give us an endpoint to write to, and we\'ll send you this stuff in OpenTelemetry format. And then it\'ll integrate with whatever else you got as long as it\'s all talking OpenTelemetry as well.”\r\n\nCHARLES: So who else is involved in the Telemetry project/consortium? You mentioned it was part of the cloud-native...was it Cloud Native Foundation?\r\n\nAUSTIN: Yeah. So it\'s like a Cloud Native Computing Foundation sandbox project right now. We\'re hoping to move into incubating this year. We\'re actually one of the most popular CNCF projects by contributor so second or third only to Kubernetes. It depends on when you look at the stats. But we have contributors and governing from everyone from Google to Amazon, Splunk, Datadog, New Relic, Honeycomb, LightStep, Dynatrace, AppDynamics is in there a little bit. Any sort of observability company you can think of anyone that\'s selling dashboards and traces and metrics and all that stuff they\'re involved, Grafana, Elastic is involved. So there\'s an extremely broad base of . It\'s been really exciting to see all of these different vendors that have their own agendas come together and work on this. I think, for the most part, it\'s all with very pure motives.\r\n\nTARAS: Yeah. The cloud-native community to me is very inspiring because it\'s really great to see how this vendor-neutral approach they\'ve taken has worked out. It seems to be working. I\'m sure it\'s not perfect. I\'m sure it has some rough edges, but it\'s like what you\'re saying it\'s different companies that are operating in the same space that are coming together to have something that does bring value and does provide flexibility. That\'s really great to see especially for something like cloud-native space where we are talking about building huge platforms that require huge investments over a long time. So being able to have flexibility and be able to have evolution is really important.\r\n\nAUSTIN: Well, a lot of it also especially, in this case, let\'s go back to earlier talking about Sentry and the data quality issue. The question of instrumenting all these different existing pieces of code, was actually really silly when you think about it because maybe you were using New Relic. Oh, I\'m sorry. I forgot to shout-out New Relic. I don\'t if I shouted-out New Relic, but they\'re also very involved in OpenTelemetry. Before, you would say like okay, maybe I\'m using New Relic, and I have this proprietary agent that\'s going to do this instrumentation and send data to their APM endpoints and then Datadog says, “Well, we want to do an APM tool.” So Datadog has to go in and they have to reimplement all this stuff and signal effects before they [inaudible 30:12] by Splunk. It\'s like oh, we have an APM tool, so we have to reimplement all this stuff and so on and so forth. And it\'s silly because there\'s only really a layer of software there, that layer of instrumentation for Express or whatever is completely undifferentiated. It\'s just like I have a hook on something starting, and there’s something finishing.\r\n\nCHARLES: Right. The value is in what you do with the data not in actually saying here\'s the data.\r\n\nAUSTIN: Right. These were incredibly undifferentiated, incredibly commodified instrumentation libraries. But all these different vendors had to provide and maintain roughly the same code base duplicated across 5 or 6 or 10 or however many different companies. And so I think a lot of the inspiration behind OpenTelemetry is simply that one, wouldn\'t it be nice if we didn\'t have to do that? There are a bunch of individual vendors -- I\'m not going to put anything on anyone\'s motive here, but there\'s certainly a story in open source about socializing maintenance costs. But I think the bigger thing is probably it\'s not so much about the today; it\'s about the tomorrow. And the tomorrow looks a lot more like well, if there is an actual open standard that everyone agrees on, there\'s an API that everyone accepts and there\'s an exposition format for this data that is neutral and that we can all work with, then why not just that in the hopes that upstream maintainers will integrate it natively into their libraries? So instead of having to install a plugin to Express to get it to do tracing, why doesn\'t Express just have OpenTelemetry ?\r\n\nAnd so I think that\'s what you\'re going to see over the next couple of years is this API hits 1.0 because we have stability guarantees on this for three years. So once we get an actual 1.0, I think there\'s going to be a bit more of like oh well, we\'ll just take this code that was previously this third-party plugin that you installed. And instead of having this external middleware package, what if there was just a built-in tracing middleware for Express that when you create your Express server or you just say, “Express that use tracing.” And then it\'ll just look for OpenTelemetry, an OpenTelemetry SDK, and ta-da! Now, this is not an OpenTelemetry problem; this is an upstream framework maintainer problem. And that also helps everyone too because that means that the people that actually write the software, the people that actually write the frameworks and libraries can start to have better control over the instrumentation and can say, “This is what we think is really important. We can provide a recommended dashboard or a recommended troubleshooting guide that says, hey, this is what you should watch for. These are the attributes that are important for you to be aware of. This is maybe a good metrics query that you can run or an alert or whatever that we can tell you about because now there\'s this common language for expressing those metrics, there’s a common language for expressing those traces for expressing the semantic attributes of a request.” And I think that could be I think a pretty big revolution in the way that we, as an industry, think about performance and talk about performance and how to optimize performance and understand incidents and things like that just because now we have this lingua franca of OpenTelemetry to discuss these sort of problems in.\r\n\nTARAS: The topic of Telemetry, so is there overhead to -- I don\'t really know how to ask this question, but I’m curious what is the standard that you measure for performance of any of these tools? Is there an acceptable impact that you would see? In theory, I would imagine there\'d be like no impact at all, but is there overhead that\'s introduced when you start to do tracing that is measurable?\r\n\nAUSTIN: I mean, it depends on the perspective. So what I would say is this: we have standards and specifications. We\'ve specified benchmarking suites and things like that. That part of this journey to GA includes these full benchmarking suites and testing and being able to give a quantified answer to this question. What I would say is from an end- perspective, generally, no, there\'s no real end- impact or at least not a ton. So if you think about how a span is created, it\'s an allocation, and you have a span for every single request, if you\'re doing 100% sampling, you\'re creating spans for every single request that moves through your server. So there is some overhead because every single request is now going to have the additional overhead of oh, I had to create a span and that span needs all these attributes and so on and so on and so forth. But the actual act of exporting that data and doing that stuff is done off the main threads, so to speak. So that part is pretty snappy, and we\'ve done, I think, the best we can, and we can always do better. But certainly, when we developed these SDKs, they were done with an eye towards performance. So where you can, avoid unnecessary allocations, where you can, avoid expensive operations.\r\n\nThe best case where you\'re most likely going to see it is on the export side of things because that requires you have to write this data somewhere. And so you\'re either writing over a network link or you’re writing into a file. If you\'re writing into a file, then that\'s pretty snappy. But over a network link, then you obviously have the additional overhead of creating requests out to this other service through gRPC or through HTTP or whatever, and then compression and TLS negotiation and all that. That said, in my experience, most of the overhead people incur with distributed tracing aren\'t necessarily -facing impact. It\'s not like oh, I\'ve added 10 milliseconds round trip. It\'s mostly memory pressure or U overhead on my process goes up because I\'m just doing a little extra work for every single request and a little extra work for every single request will add up. So that just means you need to horizontally scale more.\r\n\nIn my opinion, the benefit of tracing far outweighs the trade-off in headroom just because having that ability to zoom in on one individual request and see what happened at every single step is invaluable when you\'re dealing with distributed system problems and having 100% -- this is one thing that we do at LightStep. We encourage you to use 100% on sampled data and send that to us and then we use fancy dynamic sampling stuff to make it manageable, but the ability to just take this entire firehose of data and look and drill all the way down to one individual request and one individual service out of potentially millions is it\'s really unimaginable. And this is a tangent, and this is not a shill, I promise, but Spotify uses us. And so I\'ve talked to some of their engineers, and you can imagine the request load of Spotify at every second of every day the amount of traffic moving through that system, and the ability that they have to look into those billions and billions of requests and narrow it down and just say oh, I want to find this one error or this error that\'s localized to this zip code is just incredibly powerful, and that\'s something that\'s really hard to do I think with traditional -- certainly, it\'s hard to do with just metrics. It\'s annoying to do with metrics and logs, but when you have metrics and logs and traces and you’ve integrated all of these things, it becomes easier to do. And it also lets you fix problems a lot faster and also understand how problems are impacting actual s rather than aggregate groups of s.\r\n\nCHARLES: Yeah, that actually sounds amazing.\r\n\nTARAS: Do you encourage to capture all the traces and then provide tooling to find a needle in a haystack?\r\n\nAUSTIN: To be a little more clear here, OpenTelemetry lets you pick how you sample, and we offer tools to help you do things like tail-based sampling where you send all of your traces to a pool of OpenTelemetry collectors and then you ensure that all the spans for one trace wind up on the same collector, and then you can make a tail-based decision about hey, does this have an error, or does this meet some criteria? Now, LightStep also s OpenTelemetry. We actually natively ingesting the OpenTelemetry protocol and our unique architecture allows us to take 100% of that data with no upfront sampling and then do dynamic sampling on it where we can create trace aggregates, and we can do a lot of really cool stuff to not only give you that high-level picture but also let you see we can pick the important stuff out to keep for longer for that long-term analysis. I didn\'t necessarily come here wanting to talk about that too much, but if you\'re interested, lightstep.com, we have a bunch of very fun marketing things that can tell you about how great we are.\r\n\nTARAS: Well, it\'s interesting. Part of the reason why I wanted to talk to you about this is because one of the challenges that Frontside has been dealing with is we’re building this brand new testing platform. It\'s essentially a really complex distributed system that you run locally. I mean, from the developer perspective or from the perspective, a lot of the complexity of the implementation is to make it really easy for developers and make it really fast. So we have all these perks, but one of the trade-offs you make when you have a distributed system is that you have different processes that are communicating over HTTP and in some cases over web sockets. And then when something breaks down, you\'re like, I have no idea what the hell is going on. [laughs] And that\'s the situation that we\'re dealing with. And one of the things I was thinking about is actually integrating OpenTelemetry into the library that we use to create the testing tool so we can have a Jaeger trace that shows us a structured log of the particular operation. So you run a test, and you can actually see the entire structured trace of the test. And then the coolest thing about this is that when you have this as a starting point, then you can actually extend this with the test results that reach all the way into the system. So the root is the initiation of the execution of the test, but your trace can include the data that is coming from essentially one long trace going all the way into the infrastructure. So you can actually get at a place where you see the output. You can actually see information about what\'s deeply inside of the microservice that is providing the data for the test result. That seems like a really appealing thing to be able to do.\r\n\nAUSTIN: Totally. I think that’s a really good example of when you would want to do no upfront sampling because -- and this is a challenge with the way -- I think people have this idea they’re like oh, sampling is...there\'s maybe two camps here. There\'s you can\'t do this without sampling, and then there\'s you sample everything. You can\'t lose anything. And the truth is in the middle because, in any production system, 90%, 99% of the data is honestly not that useful. The reason why is because a system at rest tends to stay at rest, or you tend to get very similar stuff. And a system where everything is broken, everything is going to be broken in a similar way. With OpenTelemetry, we give you the ability, I think, to separate the sampling decision and move them to a different layer. So instead of saying, “I have to make these sample decisions in my process,” it’s more okay, you can exfiltrate your data to a collector or to a pool of collectors and then you can do the sampling there, which I think will unlock in the future more advanced sound sampling algorithms and the ability to make better decisions about your data and what gets sampled and what gets sampled out.\r\n\nFor something like you were describing, actually, you\'d probably want to err on the side of yeah, keep everything because you\'re not running tests every single minute of every single day. And you do want to have that full visibility into everything that\'s going on because tests can tend to be, from run to run, there can be a high level of variance in what actually happens because the code is changing all the time, right? You\'re not generally running the exact same tests every time. I would definitely say OpenTelemetry would be a perfect thing to integrate there to get that visibility into what\'s going on.\r\n\nTARAS: Well, we\'ll give it a shot, and we\'ll let you know how it goes. [laughs]\r\n\nAUSTIN: Hey, poke me on Twitter if you have any questions.\r\n\nTARAS: Yeah, awesome. And where can people find you on Twitter?\r\n\nAUSTIN: Yeah, I\'m @austinlparker. I tweet regularly. It\'s like the only social media I pay attention to.\r\n\nCHARLES: All right. Well, thank you so much for coming by, Austin L. Parker. We already mentioned you work for LightStep, lightstep.com. Where can people go to find out more about the OpenTelemetry project? Is there a URL there?\r\n\nAUSTIN: Check out our website opentelemetry.io and that has links to our GitHub, to our chat on the CNCF Slack. You can find meetings for all the SIGs, and you can start there, best place.\r\n\nCHARLES: Fantastic. I love the longer that I\'m a developer, I love just how in many ways it becomes a more and more comfortable experience because all of these things that were hard to do or that you didn\'t even know about you should be doing all of a sudden you just wait a few years and they just happen for free, and you\'re like, oh, wow.\r\n\nAUSTIN: Yeah.\r\n\nCHARLES: It\'s like you\'re Neo in The Matrix where you just keep learning all these different kungfus, and all you have to do is just flutter your eyes for a little bit. So it definitely sounds like this is one of those things that is in the works and that people are going to get to benefit from with very little effort going forward.\r\n\nAUSTIN: Yeah, definitely. Like I said, check it out. By the time this goes up, you should see a lot more 1.0s starting to pop up out there. So we definitely want .\r\n\nCHARLES: Well, fantastic. Thank you again and thank you, everybody, for listening, and we will see everybody next time.\r\n\nThank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain, old email at @frontside.io. Thanks and see you next time.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n

In this episode, Puneet Lath, Director of Research and Development at Expensify, talks about the unique way Expensify is using open source with their products by not just open-sourcing software tools but also open-sourcing the front end of the product itself. They are rebuilding their products on React Native to be fully cross-platform and doing so in an open-source manner. All code is public, and anyone can see it and contribute to it. TRANSCRIPT: CHARLES: Hello, everybody, and welcome to the Frontside podcast. My name is Charles Lowell. With me today also from the Frontside is Taras. Hello, Taras. TARAS: Hey, Charles. CHARLES: And we're going to be talking with Puneet Lath, the director of R&D at Expensify today about a lot of interesting topics. Hey, Puneet. PUNEET: Hey, thanks for having me. CHARLES: One of the reasons that we wanted to have you here on the podcast was to talk about the unique way that you're using open source with your products. Maybe you could tell us a bit about the strategy and how you guys came to start implementing it. PUNEET: Yeah, absolutely. So we've always been big fans of open source. We've had some parts of our technology stack that have been open-sourced in the past for example our database layer called Bedrock is something that we open-sourced quite a while back. But what we're doing now is a little bit different, a little bit more unique I think basically not just open-sourcing software tools that we use but actually open-sourcing the front end of our product itself. So we're basically rebuilding our products, the front end at least, on React Native to be fully cross-platform, and we're doing that in an open-source manner. so all the code is public, anyone can see it and contribute to it. We're fully committed to open source. CHARLES: That's crazy. There are so many implications there. It's definitely the conventional wisdom that you maybe open-source some libraries that you use. Of course, you let everyone use your front end, and you keep your back end close source. But what's the advantage of having an open-source front end? There is only one Expensify. What's the advantage for me as a developer of being able to actually modify this code? Like, I can't deploy it on my own systems. PUNEET: Sure. The benefit that we see is we get the advantage of the open-source community. We get people who can see our code. If they use our software and they care about our software, they have a chance to be a part of building that software. So we see ourselves as more than just a company but a community of people. And there's so much engineering talent out there in the world and so many good ideas and so many different perspectives. So we just see it as an opportunity to try to tap into some of that. TARAS: It's mind-blowing because there are so many times that you use something and you really can't do anything with the software if you find there's something wrong with it. And so much of our stacks are now converging in similar technologies the idea that you're using an app and you're like -- and something like Expensify too because it's part of your core business. I could imagine a consulting company like Frontside using Expensify for our internal workflow, and we have a few people on the team who can do React, [Chuckles] I mean every single one. So you encounter something you're like, “Look, I think this could be different.” And the idea that we have a SaaS that we use and we can just drop into their code and like, “Hey, you know what? I don't like this. [Chuckles] Can we change this? This doesn't really work for how we're using it.” CHARLES: How many people would actually change it so that when you merge a pull request the number of pull requests actually decrements by one? [Laughter] So is it mostly that kind of thing where you get people who just see ways that they can improve the product that they themselves are using? And they’re like, “Hey, I want to contribute to it.” PUNEET: That's the dream, certainly. We think it can create a totally different relationship with customers where up until this point, if you want us to fix a bug, or if you want us to implement a feature, you had to convince us to do it. Now you can go fix it yourself, and we'll happily merge it. Or if there's a feature that you think we should build and we're not totally convinced, you can build it. Send us a PR, we'll look at it, experience it, and it just creates a more collaborative way for us to build this software. We hope that it empowers our s and gives them more of an opportunity to see the things that they want. In theory, if we don't merge it, they can run that local version of Expensify that has their changes if that's what they want to do. That's certainly an option that's available to them as well. But our hope is more that it just allows people to be a part of building our products and not just in their ideas but in actually seeing those ideas implemented. CHARLES: How far along are you on this experiment? Have you started to see some significant traction or contributions to your codebase from the outside community? PUNEET: We’re pretty early on in the experiment. We’re a few months into it so most of the contributions from people to the codebase that aren't employees at Expensify are contributors that we're paying to implement changes. So one of the big benefits is that we can squash bugs at a faster rate. When a bug comes up, that gets filed in the repo and then we can put it on a platform like Upwork and basically say, “Hey, we'll pay $500 or $1,000 or whatever it is depending on the bug to anyone who wants to come and fix it. And it just makes the speed of bug fixing so much faster, the same thing with implementing features or polish. Historically, we always had to prioritize, do you build new big features, or do you polish the existing ones? Now we don't really have to make that trade-off. We can focus on building new features, and we can have external people pick up and polish items if they want to earn extra cash, whether they're s or people out there in the world that do this kind of freelancing work. TARAS: It's really interesting. I like how you guys are trying a few different things at the same time because this idea that you can engage people in contributing to open source while having a system for getting paid for doing that it's kind of the best of both worlds in some ways because many times some people would like to break in and have some credibility on GitHub where they could show that I've done this work, and they really don't have a lot of choices. And especially if you don't have a lot of bandwidth to just sit and write code, if you don't have that luxury to just sit down and not earn any money, then this could be a way for people to break through that barrier and get paid while also building up their GitHub portfolio. So I think it's a really positive thing, and I really hope that it gets traction. I hope more people start to use it. PUNEET: Yeah, that's something that we're already seeing where we're having people who have contributed who are coming back over and over again, and they're getting better. They're building skills through doing things in our codebase, maybe they're early on in their software engineering career, maybe they just graduated from school or they're maybe making a career transition. They can get some experience in a way that doesn't require them to get hired for a full-time job but like you said, still get paid for it and then not have to do it for free. CHARLES: I think it's amazing. From a purely practical sense, it's just one way to completely and totally circumvent the whole barrier to contribution of hiring somebody, getting them on either a contract or paying benefits and doing all this stuff. So you just do these micro chunks of work without having to go through all of that. PUNEET: Yeah, and it kind of feeds each other. So we're hiring. We're happy to hire great engineers who want to work here but working at Expensify comes with everything with Expensify. You have to be bought into the vision. You have to care about the business. You have to care about the product roap. You have to care about our values. You have to want to work full-time in a job that takes a lot from you. And that's great. We want those people. That's what we want to do, and that's why we work there. But at the same time, there's tons of expertise out there. There are people who don't want to do that who just want to be freelance, have more freedom, pick up chunks of work here and there. And we love the ability to lean on their experience and their specific skills. There are a lot of people with very specific skills that we can tap into even if they don't work at Expensify because we've done this. TARAS: I can see this being also a way to pre-qualify people in a way because you're always -- if you want to hire, if you want to grow your team, it always helps to find people who already know something about you. You're not starting completely cold. They're attracted to you for whatever reason and so having them be familiar with your codebase, having seen them in action -- because you could tell so much about someone with their pull request hygiene, their commit hygiene, their attention to detail. There are all these little things that you can read from how people describe their work in a pull request that I think it would be really telling about the stage that they are in in their career or the way that they think about technology and that doesn't necessarily mean -- because there's so many different kinds of engineers but being able to read between the lines and what the person is saying I could see that being really helpful if you ever consider hiring somebody from the community. PUNEET: Absolutely. One of the things that we see when we do hire engineers and they don't succeed at Expensify, it's rarely a skills issue or a technical ability issue; it's almost always a communication and collaboration issue. How do they communicate? How do they collaborate? And one of the nice things about this is we're getting to see that. They get to see how we communicate and collaborate, and we get to see how they do. And so yeah, absolutely, it gives you a better sense for both sides of would this make sense in the future for them to work here? And I think that's a really cool aspect of it. CHARLES: One of the things that I'm curious to know about is about the existing Expensify engineers and if there's a mental mode-switch that happens when the product is open source. Because I know I certainly feel the pull to make my best absolute product when I know that it's going to be open source and that it's not going to be in front of an audience of 10 people or 1,000 people. It could be potentially in front of an audience of 10 million. And so I really like -- Even though I try to always be the best version of my development self that I can be when I'm writing code no matter the context, I have to say the incentive is different when it's open source. And I think that's why we see, in my experience, we see basically a higher quality of code just absolutely when you're developing an open-source codebase. And I'm wondering did you see that in your internal staff in the way that people treat the open-source? Not only is it going to have a bigger audience but you just have to have, because everybody's not in your Slack, you have to have better documentation so that people can discover things for themselves. Your pull requests have to be more descriptive because there's not as much shared context, things like that. PUNEET: Yeah. It's exactly what you said. These people don't work here. They're not in Slack, they don't necessarily know everything. And so yeah, our REEs have gotten better as a result of this, how we set up our tools and make it easy to get set up so that you can develop on our product has gotten better as a result of external people who don't have access to our internal Stack Overflow on these things. They need to be able to set up quickly for this small job that they're doing in order to be able to send a pull request. Our communication and PRs all of that has gotten better as a result of this and ultimately, that's making us all better engineers as well just being forced to have this higher level of thought in our communication. TARAS: What kind of tools do you have around the project to help developers who are not part of the team, like someone who is contributing open source to make sure that the work that they're doing is fruitful? Because I can imagine it being really discouraging to put a bunch of work in and then you create a pull request and then you find out that oh, you know what? We actually can't really merge this. PUNEET: So we experienced some of that in the beginning. We hired people to do work and they'd send a pull request, and it was totally different than what we were expecting, whether it was in style or in approach or that sort of thing. And so we've done some experimentation. We experimented with hiring two people for one job and then paying them both but maybe merging the code that was better. That created some kind of weird incentive, so we've gone away from that. What we do now, is we try to – basically, what we're doing is more of our internal processes are becoming public as a result of this. So internally, we have a process where you have to explain, you have to propose “Here's how I'm going to approach this problem.” You get on that before you actually go and write the code. We're having our contributors do the same thing, and we're having them break down “Okay, what's going to be your approach to this problem? Let's have a discussion about that and agree on that first before you actually go and make the changes and submit the pull request.” So we expect that the we're going to have for you on the pull request is going to be minimal and more tactical rather than “No, this is all wrong.” CHARLES: And along that same line, how do you keep people productive just in of making sure that they have the pieces of the stack that they need to do their job? When I think about the first day on the job, you sit down, they're like okay, this is how you set up your front end. And now we've got a little vagrant box that has the back end or a Docker container whatever. That's not really an option especially when you have the closed source back end and you have the core business logic that you may not want to give out. How do you resolve that tension? PUNEET: So it's been super interesting because they're not setting up a dev environment. They don't have a local server, so they're basically coding against our production servers. So they're running a local version of the front end, but they're developing against the production servers, which works because Expensify s are free to create. So you can create as many s as you want with as many email addresses as you want. You can play around, so it works fine to be using our production servers. So really they just need to have locally, the set up to run cross-platform front end. So they need to either have an Android or an iPhone, or they need to have local simulators set up so that they can run those. Anyone that has web is easy, but they just need to be able to run those things. So there are some small limitations there. And for that reason, we expect that they've done some either mobile work or React Native work before, so they already have those things set up. They already have the ability to run local simulators and those types of things. But other than that, you just got to fork the repo and pull it down and you're good to go. TARAS: What kind of automated testing do you guys use? I’m really curious about how you have it set up in of providing on things that are working or might not be working. PUNEET: So right now it's kind of the Wild West. I would say the automated testing is pretty minimal. We are not massive proponents of doing tons of automated testing. We found that doing manual testing, regression testing is more valuable. So we have some specific requirements of the people that are contributing that they test in all platforms and post screenshots of what they've seen. We have a QA company that we use that does manual testing for us across devices and those sorts of things. But our automated testing is actually pretty minimal. TARAS: That's interesting. It's a little bit different than how I think about organizing that kind of work, but it's really cool to see that you guys are putting it into practice, so it's fascinating to see it in motion. PUNEET: Absolutely. And I think our automated testing will get more robust. And what's cool is we can also have contributors create those automated tests as well. That's part of the work that we can have them do also. But I think we certainly believe that there's always going to be a large manual testing component especially when it comes to the myriad of devices that we're thinking about people running on; we definitely think mobile-first. And so we always think manual testing is going to be a big part of it. TARAS: One thing that comes to mind one of the reasons why I was really interested in this topic right off the bat was that it really feels like in today's world, the way we consume software feels almost in some ways inadequate because, from a company perspective, you have to invest into software to be able to get value from it. But you also have to invest into it to find out if the software is not going to work for you, and that's the part that I find really interesting about what you guys are doing is that in some ways, there seems to be the possibility of being able to pick a piece of software but know that you don't have to make a switch in case it doesn't work for you exactly as you might imagine on day one. So I wonder if you have any kind of experiences like this or if you have any thoughts about how this might evolve beyond this first experiment. PUNEET: That's super interesting. I never really thought about it from that perspective. I suppose that's true. You could get a full experience of using Expensify without necessarily needing to commit to it. That's built into the product as well. We have free trials; anyone can create an for free and play around. So I'm not sure that I see a huge benefit from that perspective specifically. I think I could see people building interesting versions of Expensify on top of what we have and creating their own stuff from it; I think that's interesting. And then what's cool is we can pick and choose what they've created. We actually want to fold into the main application, that's the direction I see it more from. TARAS: That's also what I'm thinking as well. Because feature-wise, that you can determine. But as your needs evolve, so if you find you want to integrate with this other piece of software or you find that you've got a few contractors and you'd like to be able to give them a specific way of reporting time as opposed to open-ended, that applies to everybody and then if you make it available to -- I would imagine from a perspective of someone who is making software for a variety of people, you always have this conflict between making something that is customizable which means you're adding additional options that you have to manage or making something that is generic in which case it might be too generic for some groups of people. So finding a balance between those two things I would imagine would be really difficult. So this might not fit for everyone because not everyone is a consulting company where every single person is familiar with React but increasingly, these skills are becoming a basis; they’re becoming part of software literacy. I can imagine many kids today who are going to be 10 years from now they're going to be 15 or 20 with 10 years of experience doing React. [Chuckles] So in that world, I could see them being able to pick apart the app and be like, “You know what? I work at this company, but the way we use this app doesn't really work for what we want. What do you guys think about this build of that particular app?” PUNEET: I think I can totally see that happening. I hope that happens. I think that'd be incredible. You're right, from a product perspective, we're always thinking about, okay, what's the most simple version of this? What's the widest version of this? We're not trying to build for very specific use cases. We're trying to build for the masses. And so it's all about simplicity, always, in of how we're thinking about the product. And yeah, I think people will build more specific features for more specific use cases or more specific versions. We don't have an explicitly open API but as a result of having an open front end, we have an open API. And so people can do whatever they want in of creating whatever they want that interacts with our server and database infrastructure. CHARLES: So what we've been talking about and what you've been describing, it's a pretty radical departure, as I said at the beginning, from the conventional way of doing things. And when I'm listening to it now, it feels like oh, well this is just a no-brainer. That's not how most apps are and so obviously people see that there is risk to this. So from the perspective of someone who's over on the other side of this decision and this move, how do you perceive that risk? When you were evaluating this decision, what were you seeing as the trade-offs and the risks of doing this? PUNEET: Totally. And I think the risk that most people would consider is what happens if someone steals our stuff? And I can totally understand that. For us, I mean, first of all, it's just the front end. So everything under the hood is still private. Maybe at some point in the future if we're thinking really crazy, maybe it's all open. CHARLES: [Laughs] PUNEET: That's not in the plans right now. So there's still a lot that you would have to be able to build on your own if you really wanted to try to steal everything that we have. The other thing is we're established; we've been around for 10 years. We spent 10 years perfecting this market. We have over 10 million s. There's a lot that has to be learned beyond just the front-end code in order to be able to replicate what we do. It's basically impossible to replicate. And so there's a lot of confidence there that yeah, sure, maybe there's a risk and maybe there's something hiding around the corner that we don't see. But we have a hard time picturing what that might be. The benefits seem to far outweigh the risks. TARAS: What are you excited about now? Because it's obvious that you guys are doing some really interesting stuff. You have your database open-source project; you have your front-end open source. What do you see that's on your horizon that you think is going to be really exciting from a technology perspective? PUNEET: So I think it's going to be cool to see this community of developers start to grow. We certainly believe in React Native that's why we adopted it. We're using not just React Native for mobile, but we’re also using React Native for web. So our goal for the longest time has been to not have to write three different sets of front-end code being able to write code once and have it run cross-platform that's certainly the dream. I think historically, it's been very difficult to do that and have it be good quality and feel good quality and feel Native. But we believe now that with React Native and where it's at that it’s possible to have a Native feeling experience while only having to write code once. So I'm excited about that future, and I think so far so good. CHARLES: How does that work? And I know that's a very general question. So specifically, the things that are in my mind, definitely when I tried out the React Native web, it was bedeviled by performance problems ironically, the Native platform on the web. But also it's just the underlying components are just different, the way that the layout containment and the way events are handled. And this question comes out of left. Does it make it easier to have a design system? Do you guys have a design system, and does that help? And how does that play in making that compatibility dream real? PUNEET: We do have a design system, but the reality is even when you're developing for web, you're developing for all screen sizes. So in of layout and those types of things, we've been thinking mobile-first even when doing web development anyways, so we're always thinking small screens. And in of actual writing the code perspective, writing React Native code is super similar to writing React code. The components have different names, but instead of a div you have view, but ultimately, the code is so similar. So we haven't found that it's really that hard to make the transition to using it. It's just believing that you can, and once you believe that you can, you realize that it's actually not as hard as you might think. CHARLES: Okay. That is exciting. If that is actually working out, that's a pretty awesome development. Do you find that you're able to do things like making sure that you're able to meet the same accessibility requirements on both platforms with the same set of code? PUNEET: So that's one area where we need more research done is the accessibility aspect. So we've started hiring some external contributors to focus on accessibility for us and figure that part out. My answer to your question is I'm not sure yet. I think that should be pretty solvable. When I say that we're using React Native for web, that doesn't mean there aren't areas or ways in which we write web-specific code but the vast majority of the code is written in a cross-platform way and sometimes you have to fill it in with Native elements. But all of the business logic and most of the main structure is written in a cross-platform way. CHARLES: So, how do you handle things? Do you have things like routing? Do you just use React Router on both platforms in the sense that one platform does not have the concept of a URL bar? PUNEET: Yeah, that's a great question. Right now we use React Router on both platforms. We're actually in the process of transitioning to React Navigation because it just handles Native Navigation much better. And it actually works quite well for web as well. So React Navigation works with React Native for web, and it works quite well for web. And you do have to think about navigation in a cross-platform way. So you have to think about Native stacks and how do those relate to URL bars and a back button? And so you have to think about them, and you have to come up with ways where things that exist on Native what are those the equivalent of? So a right-click on web, what is that the equivalent of on Native? Is that the equivalent of a long press? Okay, so we're going to code with that in mind and right-click and our long press are the same or a back button and a back swipe are the same or those types of things are how we have to think as a result of doing it in a cross-platform way. But the libraries like React Navigation it just fine as long as you know what you want to do. CHARLES: I guess if you're thinking in of those abstract operations, it's the same way that you can on an event handler you're thinking of is the Meta key down? Not is the option key or the ALT key because it's different on Unix, Mac, and Windows but that abstract operation is the same. PUNEET: Exactly. CHARLES: Okay. Man, I'm excited to go give it a try. TARAS: What kind of features have you had your open source contributors implement? What's the biggest thing that someone has contributed? PUNEET: The biggest thing probably is -- So the main functionality is the chat application. So it allows people to chat with each other and facilitate payments within this chat structure. And so one of the things that we in this chat app is using Markdown and so we have this engine that basically converts between HTML and Markdown and plain text because depending on what device you're on, it's getting sent to the server in HTML or plain text. And how do we convert Markdown into HTML and vice versa? And so the biggest contribution that we had is something super specific which we have no expertise of which is around basically that HTML conversion and how browsers navigate the HTML tree and style it, and it's actually far beyond me. But those kinds of really specific deep things where you need this deep, deep knowledge of HTML and CSS that most web developers don't have, we have been able to use contributors to come in, drop-in. They understand that well. They can write this engine for us that does that conversion stuff like that. TARAS: That's a really great example of something that we've observed as well. We’ve actually recently changed how we structure engagements with our clients because we observed this ourselves as well that there are certain problems that solutions to those problems exist in specific groups and most likely you just don't have that. A perfect example of this is that we collaborated with another consulting company or actually two consulting companies to create a Bluetooth simulator. And we could have taken a crack at it; we probably could have figured it out. But they have years of experience doing Bluetooth. It made so much more sense for us to provide access to our client to that experience as opposed to trying to do that ourselves. And so facilitating the creation of that was actually really impactful because now they're using this tool, and they did not have this experience internally. We did not have all that experience internally, but we were still able to create the technology that we wanted to. What you're describing is a perfect example of what we've seen ourselves as well that there is certain expertise that you can get if you frame the relationship in a specific way. CHARLES: How is it that you match this? When you say, “Oh, we have this specific need,” how do you go about finding it? Do you just issue a bounty and they come to you, or how do you write it up and talk about it? PUNEET: There are two ways that we do it. So one is issuing a bounty and seeing if someone comes to us and proposes something that makes sense to us. That seems to work more with smaller things and less specific things. The other is we go seek people out. So for example, I told you guys we are making this transition from React Router to React Navigation. So we've written that all ourselves. One of our engineers has been working on that project, but no one on our team really has a ton of experience with React Navigation. So we're thinking okay, now that we've basically written this, let's go have someone who has a bunch of experience or expertise with React Navigation review it and reverse the roles where they're reviewing our code versus us reviewing contributor code. And so we went and looked in the React Navigation repo, and we're seeing okay, who are some of the big contributors? Maybe we can reach out to them and pay them for their time if they do freelance work and have them review this and give us . So we seek people out, and we let people come to us. We're trying both approaches, and they seem to have different times in which they work. TARAS: And you can do that because you're open. Because it's hard to engage when you're like, “Well, let's sign a contract first,” and then “Are you available then?” Blah, blah, blah. Instead of just like, “Hey, there's a link here, and we have this problem. Can you help us with this?” [Laughter] PUNEET: Exactly. It's so much easier to just be like, “Here's the link to the branch, check it out. If you have some , we'd love to pay you for it.” It's so much easier to have that conversation because of people's just natural curiosity they want to go look at it. TARAS: Especially if someone's using your tool. You're going to be like, oh, let me see how they actually put that in practice. PUNEET: Yeah, absolutely. TARAS: Again, going back to the idea that-- I think it's a really healthy thing because we have so many conversations around changing how people get compensated in open source and how we compensate people for open source. But at the same time, in some ways, there can be inherent barriers that exist for no good reason other than the fact that they just simply exist. We see this with especially deg tools like when integrating with other vendors like we have a project to create Azure simulation, and I would not want to build Azure simulation without having Azure folks at the table. It makes no point to do that. It's a sophisticated piece of technology and being able to do that in the open is really helpful. If you don't think that that's possible, you would never even attempt it. PUNEET: Totally. Yeah. I think it takes a certain level of confidence that hey, we can engage in public and nothing bad is going to happen. It's actually going to be better for all of us. It's a win-win; we get something out of it, and it helps our business and our product grow but so do the people that we're interacting with. TARAS: What triggered this? Where did this come from? Why did you guys think we got to do this? This is something that we should organize our backlog, and we got to do all this work. We got to figure out the legal stuff. What was it that gave you the idea that this is something that you should do? PUNEET: So we for a long time for years and years and years, we've been talking about how we can get to cross-platform. We've always had been frustrated by the need to do all this duplicate effort in of writing front-end code. Way back in the day, we wrote our own cross-platform but back when it wasn't just iPhone and Android but back when Blackberry was still a big thing, Windows phone, it seemed like it was going to become a big thing. So we had this cross-platform layer that we called Yet Another Platform Layer (YAPL) which was to write cross-platform across those four mobile platforms. So this is something since the beginning of our business that we've wanted to do, that wasn't very successful. And so we felt like maybe around mid-2020 we really felt like okay, React Native is at a place where this is truly possible where this goal that we've had forever to truly write cross-platform is possible. We were also at a point in our product roap where we were building this new way of -- we were basically redeg our entire front end and how it's going to work. And so we were rewriting our front end from scratch, so it seemed like the perfect time to okay, let's make this transition to React Native since we're rewriting it anyway. And as we were playing that out, we were like okay, we're going to be using React Native, which is an open-source library. We're going to be using all these other open-source libraries. And we were just playing out well, what would happen if our codebase was just open source? What could we get from that? Would that work? Is that crazy? And we were going through the thought experiment. And as we talked about it more and more, we couldn't really figure out a downside. We just kept coming up with new upsides that could come from that. And so we were like, “I think we have to do it. Let's do it,” and so we went with it. TARAS: Yeah, that's really cool. CHARLES: Yeah. That was a great story. It opens up a whole world of possibilities. Are you aware of any other peers that are doing something similar? PUNEET: I'm not aware of anyone that's doing anything like this. There are tons of people who have open source software and libraries and tools; all the big tech companies seem to. But I haven't found anyone else. When we were looking into okay, what license should we use? I was trying to find whether anyone else was doing anything like this, and I couldn't really find it. So I think it's pretty unique. CHARLES: Yeah. Well, it's certainly the first one that we've come across. So one of the things that I wanted to ask you from the beginning of the call was since you've done this, do you have a favorite story of something, a surprising outcome that you did not anticipate but yet was definitely positive? PUNEET: I don't know that there's been a specific, surprising outcome. I think the coolest thing so far in my mind is just the range of places that the people that are contributing to the codebase are from. We really have people from all over the world that are contributing to the codebase, people from Pakistan and Albania and Europe and America and Latin America. It's suddenly like we just have such a global base of people. And it really makes you realize that there are people that can do this work everywhere and really all over the planet and so that part has been just incredible to get to interact with these people and have them contribute. And I think the product's going to be better for it, just having more diverse perspectives. So that's the part that's been most exciting and most interesting to me so far. CHARLES: I would definitely rate that as an unexpected, positive outcome. TARAS: Having gone through this experience and being in it now, is there an app that you wish was open source like this? PUNEET: I guess I don't really expect it. It's such a mindset change that's never really occurred to me before. Like if Gmail was open source, what would I do? But yeah, I don't know. That's a good question. TARAS: The thing that I find really interesting in what you were saying earlier about the shift of by being open is it actually is making you think about being more open. It's making your team think about writing your pull requests in a different way. It's more of a proposal. This is something that we've observed as well that there is a certain kind of a growing of an idea that happens. When you start to introduce transparency or this kind of openness into your organization, you start to see these kinds of effects. And I just wanted to share one of the things that we're observing now is that -- This is completely unrelated, but it's been on my mind, and it’s similar to this is that so Spotify recently open-sourced their Backstage tool, which is their developer portal framework. It’s something that they've been using internally for a long time. And as we know, Spotify has a really good reputation for being a very progressive engineering culture. They go way back. And so they've been using this technology for four years and now they've open-sourced it. And now companies are starting to realize -- there's a lot of companies that are realizing you know what? We need to change how developers find different components in our ecosystem. We have all these libraries, we have all these packages but nobody can find anything. There are all kinds of services. We need something. So they start looking for a developer portal, and they find Backstage. But what I find really fascinating is that there's a clash that happens where there is this open-source community that is sharing technology and then there's also these more traditional companies that are coming to use technology, but they're encountering the openness for the first time. And I find that the observing of this mental shift that happens in people when they start to observe things, like you have pull request processes that Backstage uses, and it's a really powerful tool to create alignment for distributed groups. And then when you start to observe this, you start to think, well this RFC process might be useful for us. And this idea starts to blossom on other people's minds, and that's what I've been noticing. And it's really interesting to hear you share a similar perspective with what's happening inside of Expensify as a result of opening up your codebase. PUNEET: Yeah, totally. I 100% agree with you. I think that's a really cool outcome that I don't know that we thought a ton about going into it, but it's definitely the case. And I think transparency just brings up the quality of everything. TARAS: Yeah, that's a really great way of putting it. When you shine light, you will find things, right? [Laughter] PUNEET: Exactly. TARAS: So the benefit of opacity is that you get to not see the code in the corner. [Laughter] That’s really awesome. I applaud you guys for really trying to do something really different. And I've kept my eye on Expensify for a long time, and it just makes me want to use your product more because it's the fact that you are trying something different, and it could have a really positive impact, and it just makes me want to you guys. So I really appreciate you taking this step. PUNEET: Yeah, I appreciate that. Hopefully, we'll make you proud and definitely check out the codebase. Let us know what you think. CHARLES: Yeah. Anybody else who's considering a similar move, go for it. It definitely puts you a step up in our book, especially being a company of engineers. All right. Well, thank you so much, Puneet, for coming by and talking with us. I guess if there are any blog posts where you're talking about this in-depth or any other resources, we can drop them in the show notes, and we will see everybody later. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript\r\n\nCHARLES: Hello, everybody, and welcome to the Frontside podcast. My name is Charles Lowell. With me today also from the Frontside is Taras. Hello, Taras.\r\n\nTARAS: Hey, Charles.\r\n\nCHARLES: And we\'re going to be talking with Puneet Lath, the director of R&D at Expensify today about a lot of interesting topics. Hey, Puneet.\r\n\nPUNEET: Hey, thanks for having me.\r\n\nCHARLES: One of the reasons that we wanted to have you here on the podcast was to talk about the unique way that you\'re using open source with your products. Maybe you could tell us a bit about the strategy and how you guys came to start implementing it.\r\n\nPUNEET: Yeah, absolutely. So we\'ve always been big fans of open source. We\'ve had some parts of our technology stack that have been open-sourced in the past for example our database layer called Bedrock is something that we open-sourced quite a while back. But what we\'re doing now is a little bit different, a little bit more unique I think basically not just open-sourcing software tools that we use but actually open-sourcing the front end of our product itself. So we\'re basically rebuilding our products, the front end at least, on React Native to be fully cross-platform, and we\'re doing that in an open-source manner. so all the code is public, anyone can see it and contribute to it. We\'re fully committed to open source.\r\n\nCHARLES: That\'s crazy. There are so many implications there. It\'s definitely the conventional wisdom that you maybe open-source some libraries that you use. Of course, you let everyone use your front end, and you keep your back end close source. But what\'s the advantage of having an open-source front end? There is only one Expensify. What\'s the advantage for me as a developer of being able to actually modify this code? Like, I can\'t deploy it on my own systems.\r\n\nPUNEET: Sure. The benefit that we see is we get the advantage of the open-source community. We get people who can see our code. If they use our software and they care about our software, they have a chance to be a part of building that software. So we see ourselves as more than just a company but a community of people. And there\'s so much engineering talent out there in the world and so many good ideas and so many different perspectives. So we just see it as an opportunity to try to tap into some of that.\r\n\nTARAS: It\'s mind-blowing because there are so many times that you use something and you really can\'t do anything with the software if you find there\'s something wrong with it. And so much of our stacks are now converging in similar technologies the idea that you\'re using an app and you\'re like -- and something like Expensify too because it\'s part of your core business. I could imagine a consulting company like Frontside using Expensify for our internal workflow, and we have a few people on the team who can do React, [Chuckles] I mean every single one. So you encounter something you\'re like, “Look, I think this could be different.” And the idea that we have a SaaS that we use and we can just drop into their code and like, “Hey, you know what? I don\'t like this. [Chuckles] Can we change this? This doesn\'t really work for how we\'re using it.”\r\n\nCHARLES: How many people would actually change it so that when you merge a pull request the number of pull requests actually decrements by one? [Laughter] So is it mostly that kind of thing where you get people who just see ways that they can improve the product that they themselves are using? And they’re like, “Hey, I want to contribute to it.”\r\n\nPUNEET: That\'s the dream, certainly. We think it can create a totally different relationship with customers where up until this point, if you want us to fix a bug, or if you want us to implement a feature, you had to convince us to do it. Now you can go fix it yourself, and we\'ll happily merge it. Or if there\'s a feature that you think we should build and we\'re not totally convinced, you can build it. Send us a PR, we\'ll look at it, experience it, and it just creates a more collaborative way for us to build this software. We hope that it empowers our s and gives them more of an opportunity to see the things that they want. In theory, if we don\'t merge it, they can run that local version of Expensify that has their changes if that\'s what they want to do. That\'s certainly an option that\'s available to them as well. But our hope is more that it just allows people to be a part of building our products and not just in their ideas but in actually seeing those ideas implemented.\r\n\nCHARLES: How far along are you on this experiment? Have you started to see some significant traction or contributions to your codebase from the outside community?\r\n\nPUNEET: We’re pretty early on in the experiment. We’re a few months into it so most of the contributions from people to the codebase that aren\'t employees at Expensify are contributors that we\'re paying to implement changes. So one of the big benefits is that we can squash bugs at a faster rate. When a bug comes up, that gets filed in the repo and then we can put it on a platform like Upwork and basically say, “Hey, we\'ll pay $500 or $1,000 or whatever it is depending on the bug to anyone who wants to come and fix it. And it just makes the speed of bug fixing so much faster, the same thing with implementing features or polish. Historically, we always had to prioritize, do you build new big features, or do you polish the existing ones? Now we don\'t really have to make that trade-off. We can focus on building new features, and we can have external people pick up and polish items if they want to earn extra cash, whether they\'re s or people out there in the world that do this kind of freelancing work.\r\n\nTARAS: It\'s really interesting. I like how you guys are trying a few different things at the same time because this idea that you can engage people in contributing to open source while having a system for getting paid for doing that it\'s kind of the best of both worlds in some ways because many times some people would like to break in and have some credibility on GitHub where they could show that I\'ve done this work, and they really don\'t have a lot of choices. And especially if you don\'t have a lot of bandwidth to just sit and write code, if you don\'t have that luxury to just sit down and not earn any money, then this could be a way for people to break through that barrier and get paid while also building up their GitHub portfolio. So I think it\'s a really positive thing, and I really hope that it gets traction. I hope more people start to use it.\r\n\nPUNEET: Yeah, that\'s something that we\'re already seeing where we\'re having people who have contributed who are coming back over and over again, and they\'re getting better. They\'re building skills through doing things in our codebase, maybe they\'re early on in their software engineering career, maybe they just graduated from school or they\'re maybe making a career transition. They can get some experience in a way that doesn\'t require them to get hired for a full-time job but like you said, still get paid for it and then not have to do it for free.\r\n\nCHARLES: I think it\'s amazing. From a purely practical sense, it\'s just one way to completely and totally circumvent the whole barrier to contribution of hiring somebody, getting them on either a contract or paying benefits and doing all this stuff. So you just do these micro chunks of work without having to go through all of that.\r\n\nPUNEET: Yeah, and it kind of feeds each other. So we\'re hiring. We\'re happy to hire great engineers who want to work here but working at Expensify comes with everything with Expensify. You have to be bought into the vision. You have to care about the business. You have to care about the product roap. You have to care about our values. You have to want to work full-time in a job that takes a lot from you. And that\'s great. We want those people. That\'s what we want to do, and that\'s why we work there. But at the same time, there\'s tons of expertise out there. There are people who don\'t want to do that who just want to be freelance, have more freedom, pick up chunks of work here and there. And we love the ability to lean on their experience and their specific skills. There are a lot of people with very specific skills that we can tap into even if they don\'t work at Expensify because we\'ve done this.\r\n\nTARAS: I can see this being also a way to pre-qualify people in a way because you\'re always -- if you want to hire, if you want to grow your team, it always helps to find people who already know something about you. You\'re not starting completely cold. They\'re attracted to you for whatever reason and so having them be familiar with your codebase, having seen them in action -- because you could tell so much about someone with their pull request hygiene, their commit hygiene, their attention to detail. There are all these little things that you can read from how people describe their work in a pull request that I think it would be really telling about the stage that they are in in their career or the way that they think about technology and that doesn\'t necessarily mean -- because there\'s so many different kinds of engineers but being able to read between the lines and what the person is saying I could see that being really helpful if you ever consider hiring somebody from the community.\r\n\nPUNEET: Absolutely. One of the things that we see when we do hire engineers and they don\'t succeed at Expensify, it\'s rarely a skills issue or a technical ability issue; it\'s almost always a communication and collaboration issue. How do they communicate? How do they collaborate? And one of the nice things about this is we\'re getting to see that. They get to see how we communicate and collaborate, and we get to see how they do. And so yeah, absolutely, it gives you a better sense for both sides of would this make sense in the future for them to work here? And I think that\'s a really cool aspect of it.\r\n\nCHARLES: One of the things that I\'m curious to know about is about the existing Expensify engineers and if there\'s a mental mode-switch that happens when the product is open source. Because I know I certainly feel the pull to make my best absolute product when I know that it\'s going to be open source and that it\'s not going to be in front of an audience of 10 people or 1,000 people. It could be potentially in front of an audience of 10 million. And so I really like -- Even though I try to always be the best version of my development self that I can be when I\'m writing code no matter the context, I have to say the incentive is different when it\'s open source. And I think that\'s why we see, in my experience, we see basically a higher quality of code just absolutely when you\'re developing an open-source codebase. And I\'m wondering did you see that in your internal staff in the way that people treat the open-source? Not only is it going to have a bigger audience but you just have to have, because everybody\'s not in your Slack, you have to have better documentation so that people can discover things for themselves. Your pull requests have to be more descriptive because there\'s not as much shared context, things like that.\r\n\nPUNEET: Yeah. It\'s exactly what you said. These people don\'t work here. They\'re not in Slack, they don\'t necessarily know everything. And so yeah, our REEs have gotten better as a result of this, how we set up our tools and make it easy to get set up so that you can develop on our product has gotten better as a result of external people who don\'t have access to our internal Stack Overflow on these things. They need to be able to set up quickly for this small job that they\'re doing in order to be able to send a pull request. Our communication and PRs all of that has gotten better as a result of this and ultimately, that\'s making us all better engineers as well just being forced to have this higher level of thought in our communication.\r\n\nTARAS: What kind of tools do you have around the project to help developers who are not part of the team, like someone who is contributing open source to make sure that the work that they\'re doing is fruitful? Because I can imagine it being really discouraging to put a bunch of work in and then you create a pull request and then you find out that oh, you know what? We actually can\'t really merge this.\r\n\nPUNEET: So we experienced some of that in the beginning. We hired people to do work and they\'d send a pull request, and it was totally different than what we were expecting, whether it was in style or in approach or that sort of thing. And so we\'ve done some experimentation. We experimented with hiring two people for one job and then paying them both but maybe merging the code that was better. That created some kind of weird incentive, so we\'ve gone away from that. What we do now, is we try to – basically, what we\'re doing is more of our internal processes are becoming public as a result of this. So internally, we have a process where you have to explain, you have to propose “Here\'s how I\'m going to approach this problem.” You get on that before you actually go and write the code. We\'re having our contributors do the same thing, and we\'re having them break down “Okay, what\'s going to be your approach to this problem? Let\'s have a discussion about that and agree on that first before you actually go and make the changes and submit the pull request.” So we expect that the we\'re going to have for you on the pull request is going to be minimal and more tactical rather than “No, this is all wrong.”\r\n\nCHARLES: And along that same line, how do you keep people productive just in of making sure that they have the pieces of the stack that they need to do their job? When I think about the first day on the job, you sit down, they\'re like okay, this is how you set up your front end. And now we\'ve got a little vagrant box that has the back end or a Docker container whatever. That\'s not really an option especially when you have the closed source back end and you have the core business logic that you may not want to give out. How do you resolve that tension?\r\n\nPUNEET: So it\'s been super interesting because they\'re not setting up a dev environment. They don\'t have a local server, so they\'re basically coding against our production servers. So they\'re running a local version of the front end, but they\'re developing against the production servers, which works because Expensify s are free to create. So you can create as many s as you want with as many email addresses as you want. You can play around, so it works fine to be using our production servers. So really they just need to have locally, the set up to run cross-platform front end. So they need to either have an Android or an iPhone, or they need to have local simulators set up so that they can run those. Anyone that has web is easy, but they just need to be able to run those things. So there are some small limitations there. And for that reason, we expect that they\'ve done some either mobile work or React Native work before, so they already have those things set up. They already have the ability to run local simulators and those types of things. But other than that, you just got to fork the repo and pull it down and you\'re good to go.\r\n\nTARAS: What kind of automated testing do you guys use? I’m really curious about how you have it set up in of providing on things that are working or might not be working.\r\n\nPUNEET: So right now it\'s kind of the Wild West. I would say the automated testing is pretty minimal. We are not massive proponents of doing tons of automated testing. We found that doing manual testing, regression testing is more valuable. So we have some specific requirements of the people that are contributing that they test in all platforms and post screenshots of what they\'ve seen. We have a QA company that we use that does manual testing for us across devices and those sorts of things. But our automated testing is actually pretty minimal.\r\n\nTARAS: That\'s interesting. It\'s a little bit different than how I think about organizing that kind of work, but it\'s really cool to see that you guys are putting it into practice, so it\'s fascinating to see it in motion.\r\n\nPUNEET: Absolutely. And I think our automated testing will get more robust. And what\'s cool is we can also have contributors create those automated tests as well. That\'s part of the work that we can have them do also. But I think we certainly believe that there\'s always going to be a large manual testing component especially when it comes to the myriad of devices that we\'re thinking about people running on; we definitely think mobile-first. And so we always think manual testing is going to be a big part of it.\r\n\nTARAS: One thing that comes to mind one of the reasons why I was really interested in this topic right off the bat was that it really feels like in today\'s world, the way we consume software feels almost in some ways inadequate because, from a company perspective, you have to invest into software to be able to get value from it. But you also have to invest into it to find out if the software is not going to work for you, and that\'s the part that I find really interesting about what you guys are doing is that in some ways, there seems to be the possibility of being able to pick a piece of software but know that you don\'t have to make a switch in case it doesn\'t work for you exactly as you might imagine on day one. So I wonder if you have any kind of experiences like this or if you have any thoughts about how this might evolve beyond this first experiment.\r\n\nPUNEET: That\'s super interesting. I never really thought about it from that perspective. I suppose that\'s true. You could get a full experience of using Expensify without necessarily needing to commit to it. That\'s built into the product as well. We have free trials; anyone can create an for free and play around. So I\'m not sure that I see a huge benefit from that perspective specifically. I think I could see people building interesting versions of Expensify on top of what we have and creating their own stuff from it; I think that\'s interesting. And then what\'s cool is we can pick and choose what they\'ve created. We actually want to fold into the main application, that\'s the direction I see it more from.\r\n\nTARAS: That\'s also what I\'m thinking as well. Because feature-wise, that you can determine. But as your needs evolve, so if you find you want to integrate with this other piece of software or you find that you\'ve got a few contractors and you\'d like to be able to give them a specific way of reporting time as opposed to open-ended, that applies to everybody and then if you make it available to -- I would imagine from a perspective of someone who is making software for a variety of people, you always have this conflict between making something that is customizable which means you\'re adding additional options that you have to manage or making something that is generic in which case it might be too generic for some groups of people. So finding a balance between those two things I would imagine would be really difficult. So this might not fit for everyone because not everyone is a consulting company where every single person is familiar with React but increasingly, these skills are becoming a basis; they’re becoming part of software literacy. I can imagine many kids today who are going to be 10 years from now they\'re going to be 15 or 20 with 10 years of experience doing React. [Chuckles] So in that world, I could see them being able to pick apart the app and be like, “You know what? I work at this company, but the way we use this app doesn\'t really work for what we want. What do you guys think about this build of that particular app?”\r\n\nPUNEET: I think I can totally see that happening. I hope that happens. I think that\'d be incredible. You\'re right, from a product perspective, we\'re always thinking about, okay, what\'s the most simple version of this? What\'s the widest version of this? We\'re not trying to build for very specific use cases. We\'re trying to build for the masses. And so it\'s all about simplicity, always, in of how we\'re thinking about the product. And yeah, I think people will build more specific features for more specific use cases or more specific versions. We don\'t have an explicitly open API but as a result of having an open front end, we have an open API. And so people can do whatever they want in of creating whatever they want that interacts with our server and database infrastructure.\r\n\nCHARLES: So what we\'ve been talking about and what you\'ve been describing, it\'s a pretty radical departure, as I said at the beginning, from the conventional way of doing things. And when I\'m listening to it now, it feels like oh, well this is just a no-brainer. That\'s not how most apps are and so obviously people see that there is risk to this. So from the perspective of someone who\'s over on the other side of this decision and this move, how do you perceive that risk? When you were evaluating this decision, what were you seeing as the trade-offs and the risks of doing this?\r\n\nPUNEET: Totally. And I think the risk that most people would consider is what happens if someone steals our stuff? And I can totally understand that. For us, I mean, first of all, it\'s just the front end. So everything under the hood is still private. Maybe at some point in the future if we\'re thinking really crazy, maybe it\'s all open.\r\n\nCHARLES: [Laughs]\r\n\nPUNEET: That\'s not in the plans right now. So there\'s still a lot that you would have to be able to build on your own if you really wanted to try to steal everything that we have. The other thing is we\'re established; we\'ve been around for 10 years. We spent 10 years perfecting this market. We have over 10 million s. There\'s a lot that has to be learned beyond just the front-end code in order to be able to replicate what we do. It\'s basically impossible to replicate. And so there\'s a lot of confidence there that yeah, sure, maybe there\'s a risk and maybe there\'s something hiding around the corner that we don\'t see. But we have a hard time picturing what that might be. The benefits seem to far outweigh the risks.\r\n\nTARAS: What are you excited about now? Because it\'s obvious that you guys are doing some really interesting stuff. You have your database open-source project; you have your front-end open source. What do you see that\'s on your horizon that you think is going to be really exciting from a technology perspective?\r\n\nPUNEET: So I think it\'s going to be cool to see this community of developers start to grow. We certainly believe in React Native that\'s why we adopted it. We\'re using not just React Native for mobile, but we’re also using React Native for web. So our goal for the longest time has been to not have to write three different sets of front-end code being able to write code once and have it run cross-platform that\'s certainly the dream. I think historically, it\'s been very difficult to do that and have it be good quality and feel good quality and feel Native. But we believe now that with React Native and where it\'s at that it’s possible to have a Native feeling experience while only having to write code once. So I\'m excited about that future, and I think so far so good.\r\n\nCHARLES: How does that work? And I know that\'s a very general question. So specifically, the things that are in my mind, definitely when I tried out the React Native web, it was bedeviled by performance problems ironically, the Native platform on the web. But also it\'s just the underlying components are just different, the way that the layout containment and the way events are handled. And this question comes out of left. Does it make it easier to have a design system? Do you guys have a design system, and does that help? And how does that play in making that compatibility dream real?\r\n\nPUNEET: We do have a design system, but the reality is even when you\'re developing for web, you\'re developing for all screen sizes. So in of layout and those types of things, we\'ve been thinking mobile-first even when doing web development anyways, so we\'re always thinking small screens. And in of actual writing the code perspective, writing React Native code is super similar to writing React code. The components have different names, but instead of a div you have view, but ultimately, the code is so similar. So we haven\'t found that it\'s really that hard to make the transition to using it. It\'s just believing that you can, and once you believe that you can, you realize that it\'s actually not as hard as you might think.\r\n\nCHARLES: Okay. That is exciting. If that is actually working out, that\'s a pretty awesome development. Do you find that you\'re able to do things like making sure that you\'re able to meet the same accessibility requirements on both platforms with the same set of code?\r\n\nPUNEET: So that\'s one area where we need more research done is the accessibility aspect. So we\'ve started hiring some external contributors to focus on accessibility for us and figure that part out. My answer to your question is I\'m not sure yet. I think that should be pretty solvable. When I say that we\'re using React Native for web, that doesn\'t mean there aren\'t areas or ways in which we write web-specific code but the vast majority of the code is written in a cross-platform way and sometimes you have to fill it in with Native elements. But all of the business logic and most of the main structure is written in a cross-platform way.\r\n\nCHARLES: So, how do you handle things? Do you have things like routing? Do you just use React Router on both platforms in the sense that one platform does not have the concept of a URL bar?\r\n\nPUNEET: Yeah, that\'s a great question. Right now we use React Router on both platforms. We\'re actually in the process of transitioning to React Navigation because it just handles Native Navigation much better. And it actually works quite well for web as well. So React Navigation works with React Native for web, and it works quite well for web. And you do have to think about navigation in a cross-platform way. So you have to think about Native stacks and how do those relate to URL bars and a back button? And so you have to think about them, and you have to come up with ways where things that exist on Native what are those the equivalent of? So a right-click on web, what is that the equivalent of on Native? Is that the equivalent of a long press? Okay, so we\'re going to code with that in mind and right-click and our long press are the same or a back button and a back swipe are the same or those types of things are how we have to think as a result of doing it in a cross-platform way. But the libraries like React Navigation it just fine as long as you know what you want to do.\r\n\nCHARLES: I guess if you\'re thinking in of those abstract operations, it\'s the same way that you can on an event handler you\'re thinking of is the Meta key down? Not is the option key or the ALT key because it\'s different on Unix, Mac, and Windows but that abstract operation is the same.\r\n\nPUNEET: Exactly.\r\n\nCHARLES: Okay. Man, I\'m excited to go give it a try.\r\n\nTARAS: What kind of features have you had your open source contributors implement? What\'s the biggest thing that someone has contributed?\r\n\nPUNEET: The biggest thing probably is -- So the main functionality is the chat application. So it allows people to chat with each other and facilitate payments within this chat structure. And so one of the things that we in this chat app is using Markdown and so we have this engine that basically converts between HTML and Markdown and plain text because depending on what device you\'re on, it\'s getting sent to the server in HTML or plain text. And how do we convert Markdown into HTML and vice versa? And so the biggest contribution that we had is something super specific which we have no expertise of which is around basically that HTML conversion and how browsers navigate the HTML tree and style it, and it\'s actually far beyond me. But those kinds of really specific deep things where you need this deep, deep knowledge of HTML and CSS that most web developers don\'t have, we have been able to use contributors to come in, drop-in. They understand that well. They can write this engine for us that does that conversion stuff like that.\r\n\nTARAS: That\'s a really great example of something that we\'ve observed as well. We’ve actually recently changed how we structure engagements with our clients because we observed this ourselves as well that there are certain problems that solutions to those problems exist in specific groups and most likely you just don\'t have that. A perfect example of this is that we collaborated with another consulting company or actually two consulting companies to create a Bluetooth simulator. And we could have taken a crack at it; we probably could have figured it out. But they have years of experience doing Bluetooth. It made so much more sense for us to provide access to our client to that experience as opposed to trying to do that ourselves. And so facilitating the creation of that was actually really impactful because now they\'re using this tool, and they did not have this experience internally. We did not have all that experience internally, but we were still able to create the technology that we wanted to. What you\'re describing is a perfect example of what we\'ve seen ourselves as well that there is certain expertise that you can get if you frame the relationship in a specific way.\r\n\nCHARLES: How is it that you match this? When you say, “Oh, we have this specific need,” how do you go about finding it? Do you just issue a bounty and they come to you, or how do you write it up and talk about it?\r\n\nPUNEET: There are two ways that we do it. So one is issuing a bounty and seeing if someone comes to us and proposes something that makes sense to us. That seems to work more with smaller things and less specific things. The other is we go seek people out. So for example, I told you guys we are making this transition from React Router to React Navigation. So we\'ve written that all ourselves. One of our engineers has been working on that project, but no one on our team really has a ton of experience with React Navigation. So we\'re thinking okay, now that we\'ve basically written this, let\'s go have someone who has a bunch of experience or expertise with React Navigation review it and reverse the roles where they\'re reviewing our code versus us reviewing contributor code. And so we went and looked in the React Navigation repo, and we\'re seeing okay, who are some of the big contributors? Maybe we can reach out to them and pay them for their time if they do freelance work and have them review this and give us . So we seek people out, and we let people come to us. We\'re trying both approaches, and they seem to have different times in which they work.\r\n\nTARAS: And you can do that because you\'re open. Because it\'s hard to engage when you\'re like, “Well, let\'s sign a contract first,” and then “Are you available then?” Blah, blah, blah. Instead of just like, “Hey, there\'s a link here, and we have this problem. Can you help us with this?” [Laughter]\r\n\nPUNEET: Exactly. It\'s so much easier to just be like, “Here\'s the link to the branch, check it out. If you have some , we\'d love to pay you for it.” It\'s so much easier to have that conversation because of people\'s just natural curiosity they want to go look at it.\r\n\nTARAS: Especially if someone\'s using your tool. You\'re going to be like, oh, let me see how they actually put that in practice.\r\n\nPUNEET: Yeah, absolutely.\r\n\nTARAS: Again, going back to the idea that-- I think it\'s a really healthy thing because we have so many conversations around changing how people get compensated in open source and how we compensate people for open source. But at the same time, in some ways, there can be inherent barriers that exist for no good reason other than the fact that they just simply exist. We see this with especially deg tools like when integrating with other vendors like we have a project to create Azure simulation, and I would not want to build Azure simulation without having Azure folks at the table. It makes no point to do that. It\'s a sophisticated piece of technology and being able to do that in the open is really helpful. If you don\'t think that that\'s possible, you would never even attempt it.\r\n\nPUNEET: Totally. Yeah. I think it takes a certain level of confidence that hey, we can engage in public and nothing bad is going to happen. It\'s actually going to be better for all of us. It\'s a win-win; we get something out of it, and it helps our business and our product grow but so do the people that we\'re interacting with.\r\n\nTARAS: What triggered this? Where did this come from? Why did you guys think we got to do this? This is something that we should organize our backlog, and we got to do all this work. We got to figure out the legal stuff. What was it that gave you the idea that this is something that you should do?\r\n\nPUNEET: So we for a long time for years and years and years, we\'ve been talking about how we can get to cross-platform. We\'ve always had been frustrated by the need to do all this duplicate effort in of writing front-end code. Way back in the day, we wrote our own cross-platform but back when it wasn\'t just iPhone and Android but back when Blackberry was still a big thing, Windows phone, it seemed like it was going to become a big thing. So we had this cross-platform layer that we called Yet Another Platform Layer (YAPL) which was to write cross-platform across those four mobile platforms. So this is something since the beginning of our business that we\'ve wanted to do, that wasn\'t very successful. And so we felt like maybe around mid-2020 we really felt like okay, React Native is at a place where this is truly possible where this goal that we\'ve had forever to truly write cross-platform is possible. We were also at a point in our product roap where we were building this new way of -- we were basically redeg our entire front end and how it\'s going to work. And so we were rewriting our front end from scratch, so it seemed like the perfect time to okay, let\'s make this transition to React Native since we\'re rewriting it anyway. And as we were playing that out, we were like okay, we\'re going to be using React Native, which is an open-source library. We\'re going to be using all these other open-source libraries. And we were just playing out well, what would happen if our codebase was just open source? What could we get from that? Would that work? Is that crazy? And we were going through the thought experiment. And as we talked about it more and more, we couldn\'t really figure out a downside. We just kept coming up with new upsides that could come from that. And so we were like, “I think we have to do it. Let\'s do it,” and so we went with it.\r\n\nTARAS: Yeah, that\'s really cool.\r\n\nCHARLES: Yeah. That was a great story. It opens up a whole world of possibilities. Are you aware of any other peers that are doing something similar?\r\n\nPUNEET: I\'m not aware of anyone that\'s doing anything like this. There are tons of people who have open source software and libraries and tools; all the big tech companies seem to. But I haven\'t found anyone else. When we were looking into okay, what license should we use? I was trying to find whether anyone else was doing anything like this, and I couldn\'t really find it. So I think it\'s pretty unique.\r\n\nCHARLES: Yeah. Well, it\'s certainly the first one that we\'ve come across. So one of the things that I wanted to ask you from the beginning of the call was since you\'ve done this, do you have a favorite story of something, a surprising outcome that you did not anticipate but yet was definitely positive?\r\n\nPUNEET: I don\'t know that there\'s been a specific, surprising outcome. I think the coolest thing so far in my mind is just the range of places that the people that are contributing to the codebase are from. We really have people from all over the world that are contributing to the codebase, people from Pakistan and Albania and Europe and America and Latin America. It\'s suddenly like we just have such a global base of people. And it really makes you realize that there are people that can do this work everywhere and really all over the planet and so that part has been just incredible to get to interact with these people and have them contribute. And I think the product\'s going to be better for it, just having more diverse perspectives. So that\'s the part that\'s been most exciting and most interesting to me so far.\r\n\nCHARLES: I would definitely rate that as an unexpected, positive outcome.\r\n\nTARAS: Having gone through this experience and being in it now, is there an app that you wish was open source like this?\r\n\nPUNEET: I guess I don\'t really expect it. It\'s such a mindset change that\'s never really occurred to me before. Like if Gmail was open source, what would I do? But yeah, I don\'t know. That\'s a good question.\r\n\nTARAS: The thing that I find really interesting in what you were saying earlier about the shift of by being open is it actually is making you think about being more open. It\'s making your team think about writing your pull requests in a different way. It\'s more of a proposal. This is something that we\'ve observed as well that there is a certain kind of a growing of an idea that happens. When you start to introduce transparency or this kind of openness into your organization, you start to see these kinds of effects. And I just wanted to share one of the things that we\'re observing now is that -- This is completely unrelated, but it\'s been on my mind, and it’s similar to this is that so Spotify recently open-sourced their Backstage tool, which is their developer portal framework. It’s something that they\'ve been using internally for a long time. And as we know, Spotify has a really good reputation for being a very progressive engineering culture. They go way back. And so they\'ve been using this technology for four years and now they\'ve open-sourced it. And now companies are starting to realize -- there\'s a lot of companies that are realizing you know what? We need to change how developers find different components in our ecosystem. We have all these libraries, we have all these packages but nobody can find anything. There are all kinds of services. We need something. So they start looking for a developer portal, and they find Backstage.\r\n\nBut what I find really fascinating is that there\'s a clash that happens where there is this open-source community that is sharing technology and then there\'s also these more traditional companies that are coming to use technology, but they\'re encountering the openness for the first time. And I find that the observing of this mental shift that happens in people when they start to observe things, like you have pull request processes that Backstage uses, and it\'s a really powerful tool to create alignment for distributed groups. And then when you start to observe this, you start to think, well this RFC process might be useful for us. And this idea starts to blossom on other people\'s minds, and that\'s what I\'ve been noticing. And it\'s really interesting to hear you share a similar perspective with what\'s happening inside of Expensify as a result of opening up your codebase.\r\n\nPUNEET: Yeah, totally. I 100% agree with you. I think that\'s a really cool outcome that I don\'t know that we thought a ton about going into it, but it\'s definitely the case. And I think transparency just brings up the quality of everything.\r\n\nTARAS: Yeah, that\'s a really great way of putting it. When you shine light, you will find things, right? [Laughter]\r\n\nPUNEET: Exactly.\r\n\nTARAS: So the benefit of opacity is that you get to not see the code in the corner. [Laughter] That’s really awesome. I applaud you guys for really trying to do something really different. And I\'ve kept my eye on Expensify for a long time, and it just makes me want to use your product more because it\'s the fact that you are trying something different, and it could have a really positive impact, and it just makes me want to you guys. So I really appreciate you taking this step.\r\n\nPUNEET: Yeah, I appreciate that. Hopefully, we\'ll make you proud and definitely check out the codebase. Let us know what you think.\r\n\nCHARLES: Yeah. Anybody else who\'s considering a similar move, go for it. It definitely puts you a step up in our book, especially being a company of engineers.\r\n\nAll right. Well, thank you so much, Puneet, for coming by and talking with us. I guess if there are any blog posts where you\'re talking about this in-depth or any other resources, we can drop them in the show notes, and we will see everybody later.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n

Transcript\r\n\nCHARLES LOWELL: Hello, everybody, and welcome to the Frontside podcast. My name is Charles Lowell, a developer here at the Frontside with me also is Taras Mankovski.\r\n\nTARAS MANKOVSKI: Hello. Hello.\r\n\nCHARLES: Hello, Taras. And today, we are going to be talking with Steve Pereira, who is the founder of Visible, which is a company that is aligned in many ways with the way we like to think about stuff, so we wanted to have Steve on the podcast. I understand that Visible has a quite unique approach for how to realize getting products out the door and making your organization such that it delivers. So maybe, Steve, we can just start by talking about how you ended up coming to found your own company dedicated to this.\r\n\nSTEVE PEREIRA: Sure. Well, thanks for having me on. I really enjoyed our previous conversation. And I think that we definitely do have a lot in common in of beliefs and approach and how we see the software development space. Visible is the culmination of about 20 years in the industry, basically fumbling my way through understanding process and value and the ability to combine the two to actually deliver value to customers. I come from tech. I come from hard tech. I started my career building computers and fixing computers, crawling under desks in IT. And I was always missing the value piece, and the customer piece, and the big picture view of how all that fit together and how we can combine efforts to drive creation and delivery of value, and I\'ve seen it missing everywhere, and I still see it missing. Twenty years later, I see a lot of people working on micro-optimizations or working in their own silos struggling to collaborate with other teams, other groups, other roles, and we incur a lot of waste as a result.\r\n\nSo what I\'ve become very ionate about is stepping back from a process or a team or an organization and then crafting a map and this big picture view of what is going on, where the bottlenecks and opportunities and risks are, and then creating insights and data that will a prioritized list of actions that are actually going to move the needle. So instead of having eight people\'s opinions on eight different bottlenecks that we could tackle and arguing for weeks, whiteboarding ineffectively, or meeting ineffectively, or writing a bunch of documentation that no one reads, we create maps together. And so everybody is on the same page from the very beginning. And the map shows us where we are going to see the maximum Return on Investment in a way that the C-level executives could understand it, the person who just ed the team yesterday could understand it, somebody in sales, marketing success, or IT could understand it because it\'s a simple, clear representation of what\'s actually going on. I think that that\'s incredibly powerful.\r\n\nCHARLES: Well, it means that you can be intentional about what it is that you want to do instead of reactionary. Oh, here\'s this circumstance, and I need to do it because where I sit and what I can perceive this is the most critical thing that I could be working on, so I\'m going to react to it rather than having some sort of outside information of an overall priority list.\r\n\nSTEVE: Exactly.\r\n\nCHARLES: Yeah, it\'s one way to think about it. There\'s this phenomenon of. . .what\'s the word? Priority inflation, where when you\'re dealing with issues, and you don\'t have any structure over it, there\'s this creep of priority to where everything all of a sudden becomes critical. Is it meant to address people operating in those little critical criticality bubbles?\r\n\nSTEVE: Yeah. That\'s definitely a benefit. It\'s resetting priority to be globally calibrated instead of locally calibrated because everybody does have their own perception of priority, and very few people approach it systematically. Very few people are running mapping exercises to throw their priorities or their possible priorities onto a matrix and relatively calibrating them. When they do, they might be using impact and effort or RICE or WSJF as a prioritization method, but it\'s usually not based on a lot of data either. They are still bringing a lot of assumptions, and they\'re bringing a lot of opinions to the prioritization process. So they have a hard time because you can map a bunch of priorities, but then you\'ve got to go to war arguing with everybody else who argues with your prioritization. So unless you do these things as a group and it\'s based on a foundation of data that people can trust, then you might not be much better off than having nothing at all.\r\n\nSo the mapping techniques that I\'m a big fan of and that I\'ve assembled together in this flow gets you to that prioritization method by way of almost like a blockchain of logic where you can trace a prioritization or an initiative or a decision back through all the contributing factors that said here\'s how we know that this is a high-priority effort and it\'s related to value, it\'s related to quality, it\'s related to global maxima instead of the local maxima. And it\'s tied directly to this either Big Hairy Audacious Goal or a desired outcome that the entire organization shares or the team shares or our division of the company shares. And so you can trace these things all the way through. And all of a sudden, people trust it, and people align, and people are ready to go and deliver because they understand it. You\'ve brought them along instead of handing them something and saying, "Here\'s what we need to do because of reasons."\r\n\nCHARLES: So as I\'m listening to you, I\'m thinking, okay, how can we make this concrete? What\'s an example of how one of these maps might be constructed? If I was on a project, how would you take me through this process?\r\n\nSTEVE: That\'s a great question. So often, probably the most common case here is teams wanting to go faster. So, we\'ll have a team that\'s releasing maybe on a monthly basis, and they want to get down to two weeks. And when they\'re on two weeks, they want to get down to one week. So that would be a desired outcome that we could start from. And we would set up a workshop where we map that outcome to the contributing factors that we could consider before we move on. So you want to tease that apart and look at not just that outcome as a focal point. We want to say, okay, what are the obstacles that could interfere with us delivering that outcome? So we get that out of the way first, which means the team can dump their fears, share their fears, share the things that they\'re concerned about. It gives the people on the team who might be somewhat cynical or the people who focus on risk a little bit more than others. It gives them an opportunity to voice their perspective and contribute from the very beginning. Then we also look at countermeasures. So the optimists can weigh in and say, "In order to avoid that obstacle, we could do this or potentially, here\'s something that we have at our disposal that could help out with that obstacle."\r\n\nSo we break apart an outcome in a way that we make it real, and we get the team to buy into it in a way that we know what it looks like really well because clarity is the most important thing to start out with: it drives alignment, it drives confidence, it drives energy. It takes it from okay, well, we have this thing, and we have no idea how to do it to we understand it a little bit more; we can move on and, we can move forward. And you go from that outcome map to the value stream map because the value stream is the process. It\'s the engine that\'s delivering that outcome. So we look at the value stream from end to end at whatever level. So it can be at the organization level if we\'re looking at we want to deliver three times the portfolio that we did last time, and that would be a very high-level value stream that we\'re looking at. It could be release level, so we want to release every two weeks. If we go back to that, then we\'re going to look at the release process. So we\'re going to look at okay, here\'s where we have a product change request on one end, upstream the initiation of the value stream. On the other end, we have customers using the release software, or we\'re collecting data er activity. So you\'ve got the entire value stream.\r\n\nWhat we do is as a group, we will map out each distinct activity, and then we\'ll measure it. So we measure the time it takes to do each activity and the wait time in between those activities, and those are the primary measurements. So not only do you get this representation of the process, you get the measurement that\'s going to tell you here\'s where most of the time is being used. And you can go beyond that to layer all kinds of things like value created by each of the activities or quality like what percentage of this work gets thrown back to prior steps? Or when do we find bugs in this process? Where are we discovering errors? You could layer all that data on, but the most important things are the steps and the timing.\r\n\nTARAS: I could see these two things in my mind. It\'s a thing that I\'ve been thinking a lot about lately and specifically around the work that we do at Frontside around optimizing delivery pipeline with engineering organizations and the approaches that we\'re bringing there are certain kinds of techniques that we use to address a specific quality. You could say an almost measurable aspect of delivery which is there\'s latency. There\'s a latency in the system. And I think a lot of people don\'t think about latency this way, but there\'s a latency that is usually related to dependencies. So if you have a front end team, and this is a really common case, so if you have a front end team that is waiting for a back end team to give you the API, that latency can occur in a few different ways: one is you could be simply not able to do your front end work because the API that you\'re using is not reliable. It\'s either going up and down all the time, or it\'s always shifting because of under active development, or it\'s simply not available when you need it.\r\n\nSo your front-end team might be really excellent, or you put together this mobile team that really knows what they\'re doing, but the back end to match takes a long time to build because there\'s a lot of legacy. The mobile app could be a greenfield, but the back end API is relying on existing systems. So there could be legacy or technical debt that is preventing things from moving. And there\'s this latency in the actual process. Is this an example of what you observed? How does this occur to you, what I described?\r\n\nSTEVE: This is a great example. So I would say that the top three issues that I discover when we map value streams is the planning process, so a lot of delay, a lot of waste, and upfront planning that is not visible to the teams because it happens before anybody really thinks that development has started. And a lot of people criticize development because it is expensive and it is complicated, so that tends to slip under the radar, but it does influence lead time. It slows down our ability to deliver and collect and move on and pivot or adjust and adapt. The second one I would say is infrastructure automation, so things like standing up environments, refreshing environments, scaling, upscaling and downscaling, making configuration changes because that\'s often a dependency that\'s outside the team. It\'s not usually a capability within the product team. And then, the third and the most important and the most critical most common is testing, which in many cases is test automation. It\'s automated testing or a lack of automated testing. And there\'s also validation throughout the value stream that often gets batched up and is happening in a QA phase. It almost gets compressed into this window where we throw things to someone who runs QA, or there\'s just this very waterfall activity where we start testing after we\'ve finished development. So those are super common.\r\n\nThis dependency that you\'re talking about, the front end and back end, is very real, especially in the release process if we\'re looking beyond a scrum or a sprint. If we zoom out to the release level, that front end back end friction, the delay, is huge, and that\'s what would bring me to the next map, which is the dependency map. So the dependency map looks beyond the team or the area that we\'re focused on in the value stream. And it allows you to see what is contributing to any delays that you\'re seeing in the value stream. So, in this case, the front end and back end, we would see that delay represented in the value stream as this we\'ve finished work on this one part of the front end, and we have this delay while we wait for the back end to be available before we can validate that everything is working correctly and move on to the next activity. But why is that happening, and what is contributing to that delay? We might have an idea, but what does it really look like? So we do the dependency mapping to call that out. We pull that out of the team, and it gives the back end team the opportunity to say, "Well, we have to wait for this. We\'re waiting for validation of an open-source library that we need to use because this is asking for a new capability," blah, blah, blah. There are all kinds of things that can be surfaced in that. And we go from the back end is just not fast enough, and we\'re going to have to deal with it, or they\'re going to have to get faster to we know exactly what\'s happening. We know where things are breaking down, and we\'re that much closer to fixing the problem or addressing the problem directly.\r\n\nCHARLES: So, at what point do you make these determinations? Is this all in one workshop? Because what I\'m hearing is getting stuff out of people but then also collecting hard data and doing research. How do you actually construct this map? Is it something you do in two hours? Is it a process that takes weeks? How do you make sure that what you are drawing really is an accurate picture?\r\n\nSTEVE: That\'s a great question. It\'s a number of workshops. So commonly, what I\'m aiming for is a separate workshop for each purpose, so outcome mapping is one workshop two to three hours maximum. I aim for two hours for everything, and I budget for three hours for everything, two hours for outcome mapping, two hours for value stream mapping at a high level. That\'s not looking at ideal state or future state. It\'s really just current state, and two hours for dependency mapping, and two hours for capability mapping. That means that people can stay engaged because when you get beyond two hours, people start, especially on Zoom, to sort of lose their energy. And if we have to come back to it and schedule another session, we can, but two hours is a nice time box.\r\n\nIn of accuracy, the interesting thing is a lot of the data is coming from people. It\'s coming from people\'s recollection of what happened last time we did this thing. We\'re not usually doing these things as they happen. We\'re usually looking at okay, well, think about the last time that we had a release or the last two releases, and we\'ll kind of average them out. The interesting thing is that we don\'t need high fidelity data for this because the bottlenecks are really dramatically outlying from the typical activities in the value stream. And what that means, in other words, is there\'s going to be parts of the value stream that are days instead of hours and hours instead of minutes, and so the exact number of minutes and the exact number of hours becomes insignificant. You get diminishing returns from getting detail because you don\'t need it. And we\'re bringing in stakeholders from a number of different roles inside of the value stream, so you get this balance. So you\'ve got multiple developers, you\'ve got product, and maybe you have a scrum facilitator; you can collect data.\r\n\nAnd as a facilitator with 20 years of experience, I know what a reasonable number is for things. I know what seems accurate, what sounds okay. And that\'s all you need. You don\'t need perfect detail because all we\'re looking for is priority. We\'re just looking for show me things that I\'m missing. Show me things that are going to actually move the needle, and then let me prioritize those things so that I\'m focusing on the most important thing. And it doesn\'t necessarily matter whether it\'s seven days, eight days, nine days because it\'s going to go from nine days to one day. And the less time we spend on collecting that data and worrying whether it\'s accurate, the more time we can spend on actually doing the thing because you don\'t want to invest too much on assessment. You don\'t want to invest too much on figuring out what\'s going on. Your job is to be doing things and improving the situation. So you just need enough data to make a decision, which is usually 70% of the data.\r\n\nTARAS: I\'m curious, what level of access do you need to have? What\'s the best entry point in your experience to have this conversation?\r\n\nCHARLES: That\'s a good question. So it really depends on the value stream in question or the outcome that we\'re aiming at. If it\'s a release process that we\'re trying to cut in half, then we\'d probably want to be working with at least the VP engineering, somebody who is incentivized on delivery value and velocity, but it could be customer onboarding. I could be working with a sales team that\'s trying to close a funnel of -- They have 5,000 prospects, and it takes them three weeks to engage with all of them. They\'re never going to get through them, so they need to take their process down from three weeks to two days, in which case I\'d be working with a VP of sales. So it\'s really important to get leadership. You don\'t have to go all the way to the C-level. It\'s really who\'s incentivized on the success of this particular value stream.\r\n\nAnd then, we involve representative stakeholders from the entire value stream. So if QA is a part of it, we have to make sure that we have representation from QA, at least one voice, likewise with product, likewise with engineering. We would want front end and back end if we\'re talking about that situation where people are experiencing friction with that or people complain about it a lot; those are good signals. And we can use that to build out that list of stakeholders. You don\'t want more than 12 people. You don\'t want too many cooks in the kitchen, but you also don\'t want too few because you don\'t want to be missing data and information.\r\n\nTARAS: One of the challenges that I see with a lot of big companies is that there is usually impact. The activities of one side of the business can have an impact on another, and people don\'t really realize that there is an impact until they\'re stuck doing things a particular way. They\'ll say, "This is how we do things." QA is a perfect example of this. QA might say, "We need three weeks to QA this application." This is the worst-case scenario, and you hopefully don\'t have this. But there are companies that operate this way where there\'s a bunch of development that happens [Inaudible], and now you\'ve got three weeks of QA that needs to happen. Do you ever find this process changing people\'s minds, causing people to change direction?\r\n\nSTEVE: Oh, absolutely. That\'s one of the best parts of my job. I\'ve worked with a lot of companies where people are commenting that "We\'ve never been in the same meeting together." But they\'re part of the same value stream: They hand in work to each other all the time. They depend on each other. They\'re upstream and downstream from each other. So they should be collaborating on a regular basis, but they\'ve never had the visibility to see that they are that closely coupled and that they do depend on each other, and it is really important that they collaborate. I\'ve had remarks where people who\'ve worked in extremely large companies for 19, 15, 14 years and they say to me, "This is the first time I\'ve ever seen the process from start to finish," which is crazy. We know that that is true because what opportunity would they have to see the process start to finish? This is not a regular practice. This is not something that people do, but that\'s kind of what I want to change. I want to be part of this movement towards increased visibility, increased transparency, increased representation from everybody involved in delivering value because I think it\'s incredibly powerful not just from the perspective of delivering better outcomes faster and making people happier to do it, but it just builds so much empathy, and it builds so much personal ownership as a contributor. So everybody can see here\'s how I fit into this, here\'s how I\'m helping to row the boat. I think that drives alignment and drives this sense of unity that is so powerful.\r\n\nAnd we have so few ways of doing this other than team-building exercises or looking at a CI/CD pipeline, that\'s giving you a slice. It\'s giving you this tiny view of what\'s going on, and that causes problems because we think that this is all I need to be looking at. We need to step away. We need to look at the big picture on a regular basis because we all have more in common with the people that we work with, with the people upstream and downstream from us. To quote Sesame Street, "These are the people in your neighborhood," and we need to work together because that\'s how you get positive outcomes, that\'s how you get Return on Investment, that\'s how you satisfy customers.\r\n\nCHARLES: In of the philosophical underpinnings of what you\'re describing, it\'s reminding me a lot of the Toyota Kanban process with how they run their factory floors and maybe not so much in the details of the process you\'re describing but in the spirit of everybody in the assembly line being able to understand everybody else\'s job and everybody in the assembly line knowing what\'s upstream and downstream and anyone being able to even take that knowledge and having a free hand to suggest improvements to the process and being able to do that as someone who has knowledge of that entire assembly pipeline. That\'s something that they tout highly in Toyota is that they have factory workers on the floor who are able to actually improve the process because they have that global knowledge of the assembly, and they know what their neighbors are doing. And it demonstrably engenders that unity and that pride and that belief in the shared purpose. So as you were speaking, I was like, wow, this is so similar to what I\'ve heard described is going on there, which would seem to validate the idea.\r\n\nTARAS: That\'s exactly where it comes from. Toyota invented value stream mapping, and all of their lean manufacturing processes are what are now coming into software and leveling everybody up because we\'ve learned a lot of lessons in manufacturing. There\'s a lot more in common between building software and building a car than I think we commonly acknowledge. There\'s a lot of creativity that\'s involved in software but not necessarily creativity that adds value. And I think we need to focus on creativity that adds value, variability that adds value and get rid of robotic activities and things that can be automated so that we can focus our time and energy on the things that should be and can be creative and can add value.\r\n\nRight now, we have this mix where we do a lot of things that aren\'t necessarily creating value or increasing value, but we don\'t look at them. We\'re not measuring these activities against each other. We\'re not saying, "Well, we get 10 times the return doing activity A than doing activity B, and yet we invest the same time and energy and concern in both." We\'re not focusing on the things that will ultimately drive an incredible amount of value to our customers. And that\'s just going to be the next 20 years of software. That\'s where we\'re going. Because the companies that do that, the companies that recognize that you are creating a machine that delivers value and it\'s not artisanal -- We\'re at the scale now where software has to be delivering results. There has to be clear ROI. We\'re running the world on this stuff now. We\'re past the point where you\'re building toy applications that can just be working on a tiny corner of your business. We have software now that runs massive multinational corporations and partnerships and platforms that companies are building their platforms on. This is demanding a level of attention and sophistication, and automation, and coordination that just makes all of this mandatory. We cannot treat software development as this ad hoc seat of our pants. Figure it out as we go along activity if we want to maximize value and if we want to achieve everything that we can achieve with value delivery.\r\n\nTARAS: So we work with engineer organizations, and so we help them. Steve, in your , it would be ‘help them improve their value stream.’ But one of the challenges is that it sometimes requires work that people don\'t see value off for some time. You mentioned before that introducing a platform can be a solution, but introducing a platform can take some time, and quite often, it doesn\'t translate into productivity. It could be easily six months before all of the framework or foundational pieces are in place for people to actually start seeing the outcomes. So I think if there\'s a way to create confidence along the way, I think that\'d be really helpful because you want to be able to maintain a certain degree of enthusiasm for what you\'re doing, and it is probably the worst to get to a point where you\'re 60% there, and that\'s when everyone starts to doubt whether this is still the right way to go.\r\n\nSTEVE: I think that that is a huge motivating factor towards this movement towards value stream thinking and big picture thinking and really visualizing end to end process because a lot of individual contributors are going to hear this said "Oh, you need to now take what you know and layer on this extra thing. You need to now be worried about two things instead of one thing," or "You need to now be doing this thing that maybe is pulling you away from where you want to focus." And I think that\'s the wrong way to go about it. I think that the right way is to represent the value stream, allow people to see how their contributions are affecting the team and affecting customers, and affecting our ability to deliver, and then allow them to contribute in a way that they are most valuable.\r\n\nBut also, it gives us the ability to say, okay, well, instead of you just adding five new capabilities to your portfolio and then shifting your focus all the time between these new capabilities -- Because this is an activity that is critical to the value stream, and it\'s an opportunity that we need to capture across all of our teams, let\'s build that capability into some form of automation or self-service right into a platform or leverage a solution that allows our specialists to remain specialized or focus in an area that they really want to be contributing instead of saying, okay, well now you have to do this, and this, and this and be good at all of it. I think that\'s not necessarily the solution. I think the solution is basically stepping back and saying okay, well, if we want the flow of value through the value stream to be optimized, if we want better performance and we want fewer things to slip through the cracks or be delayed by handoffs, what would it take for us to automate or augment or that activity? And in some cases, that\'s going to be a platform even if it\'s a minimum viable platform. In some cases, it\'s going to be pairing. In some cases, it\'s going to be cross-training, but you have to look at the big picture in order to make those decisions really with a high degree of confidence and capability.\r\n\nSo the visualization of the value stream is really going to put you in a position where you can say, you know what? There\'s no other way to drive the level of performance and value that we need without you in addition to doing software development, starting to do infrastructure development. That\'s just the best solution that we have at the moment. But you\'re going to be able to say that with high confidence, and you\'re going to be able to show your thinking, is the other thing, which means instead of someone grumbling that now I have double the workload or I have to be an expert in this thing that I don\'t care about, you can bring them along and show them here\'s what that means. It means we can go from three days for a new feature to two days, and that means you get faster , you get to work on more new things, and that also means you wait around less. All of these things can be clear and visible. And I think that\'s super, super powerful.\r\n\nI\'ve been in this situation of trying to convince people that an investment that had a somewhat distant time horizon for realization or a Return on Investment was the right decision. And I was always looking for ways of showing my math and showing my math in a way that would resonate with a bunch of different stakeholders because, in some cases, I\'ve got to go to the board with it to make the case. So having a map and having this visualized representation of the data from a number of different perspectives we\'re looking at and not just the value stream. We\'re looking at dependencies that are involved. We\'re looking at capabilities that are involved, and we are tying it to a specific outcome. That makes it real for people in a way that you\'re almost getting value right away. In a few hours, you\'re getting value because -- I don\'t know the saying off the top of my head, but it\'s if you plan to a certain level, you\'re half done. If you\'ve got clarity on what you need to do, you\'re half done, and that really can drive people past the point of the trough of disillusionment after they\'re working away and they haven\'t seen the results yet. It\'s going to carry them through because not only do they have more clarity on what needs to be done, what things are going to move the needle, and what is going to show progress as they go, they can see the finish line with a lot more clarity.\r\n\nSo I do think that it has a remarkable impact on our ability to tackle larger, more ambitious initiatives and maintain that level of enthusiasm and confidence through the times where this just got a little bit more complicated. It\'s more complex than it seemed at first, or we couldn\'t see everything at first, and maybe we have another factor to consider. You want something that carries you through, and the mapping does that because not only does it give you that good starting point, but you can revisit it, and you can plug in new data as you get the data. So you could go from, okay, well, this was what it looked like at the time; we now know that it looks like this. What does that mean? Does that mean we need to adjust the course at all? Does it mean that this is still a good decision? And that can carry you through in a way that if you\'re just working off of here\'s what we want, and I hope that everybody\'s on board, that seems like a gamble to me knowing that the maps are an option.\r\n\nCHARLES: That actually touches on what I wanted to ask as I was listening to it is what is the average half-life of one of these maps? Things do change. We know that different circumstances arise. So what level beyond that initial investment? You described, I think, about four or five workshops three hours each, which that\'s a large time investment but probably relatively inexpensive in the grand scheme of things if you don\'t have to repeat it every two weeks. What is the process by which you tweak these maps? How often do you have to do it? How much time do you have to invest in each thing? Is there a systematic way, or is it just kind of as needed, we make tweaks?\r\n\nSTEVE: That\'s a great question. And we\'re going from a situation where nobody has ever mapped before, so basically, anything greater than zero is fantastic. We\'re in the early days of infrastructure automation, where it doesn\'t really matter what you do for infrastructure automation. Just start doing it, just something. But in an ideal scenario, what you\'re doing is you’re mapping or updating your mapping every three to six months. Because what\'s going to happen is the moment you do mapping, 20% of the value stream is going to pop out to you as things that you can just chop off immediately. You can just remove waste immediately because it\'s visible. You can see very clear low-hanging fruit opportunities. So all of a sudden, the next day, your process is different, which means that your map really is out of date, but it doesn\'t matter because it just has to be useful. It doesn\'t have to be accurate. It doesn\'t have to be current either. So what you\'re doing is you\'re eliminating low-hanging fruit immediately. You\'ve got maybe three to five high-priority prioritized initiatives that you could tackle. Once you knock those off, your value stream is also different, so your map is again inaccurate. But then when you go back to the map, you already understand the process, and your second map, the one that you do three months later or six months later, your team knows how to do it. They\'re familiar with the process. All of a sudden, your mapping activity is an hour maybe, and that\'s every three to six months. That\'s tiny. That\'s less time than we spend on sprint planning. And I would argue that it\'s 10 times 100 times more powerful and valuable than that.\r\n\nSo we talk about a large investment of time, but it does get a lot easier, and the Return on Investment is incredible. It saves millions of dollars. It saves months of time. It can really turn a team around from just being buried and fighting fires all the time to having time to spend on experimentation and innovation. So I\'ve never seen anything more powerful in my entire career. So in of ROI, it\'s unmatched. There\'s nothing that will actually deliver the same impact, especially in a short amount of time, in hours we\'re talking. There\'s nothing you can automate that would be that effective, especially when it\'s telling you what to automate. So to answer your question, three to six months is a cadence where you\'re going to be capturing current state that is notably different, I would say remarkably different. If you\'re actually delivering improvements, then your value stream looks very different three months later and three months after that. But you could do it every six months and still be miles and miles ahead of your competition that’s not doing this at all.\r\n\nTARAS: Did we cover the entire process? I think we might be at the third map, or did we get through all four?\r\n\nSTEVE: Oh, you\'re right. So capability mapping is the last one, and it\'s a little bit different than dependency mapping. Dependency mapping is looking outside the value stream at what the value stream depends on or what activities in the value stream depend on. The capabilities piece is inside. So what do we need inside the value stream in order to deliver the best performance? So capability mapping is looking at opportunities to better the team in areas that are going to contribute to the top priority items to address. So we\'re not looking at every capability that the team has or that is required for the value stream. We want to focus mostly on the bottlenecks and the gaps, and the risks, and the areas of concern because we don\'t need to know everything about everything. We need to know where to focus, and we need to know where to invest. So we look at capabilities that are directly tied to the dependencies.\r\n\nIn the case of our example of front end and back end, maybe we are lacking a capability to mark or to leverage skeleton back end capabilities or something that would allow us to decouple and work asynchronously or work in parallel. Maybe there\'s a tool. Maybe there\'s some training. Maybe we need to pair. Maybe we\'re lacking documentation. All these things come out in capability mapping. And it allows us to say, okay, well, this is a hotspot, our ability to create automated tests. We\'re probably at a six, and we need to be at a nine or above. That\'s going to give you this extra dimension on those priorities to give you even more confidence even more clarity on how you go from where you are to the next step in the right direction.\r\n\nTARAS: Are there tools? What tools do you usually use to create these and maintain these maps?\r\n\nSTEVE: That\'s a good question too. I use MURAL for this because I used to use Whiteboards and Post-its when everything was happening in the real world. But nowadays, any collaborative whiteboard is great. I\'ve used PowerPoint in some cases where that\'s all I could actually get access to in a client situation. But MURAL works fantastically. Any collaborative whiteboard. . .Google Drawings would work. Anywhere where you can have multiple people participating and working together is more than enough that you need. You\'re really just building a picture together, and what it looks like is less important than the content, but that\'s what I use on a regular basis. I\'ve got hundreds of MURAL boards now. I probably spend 80% of my day in MURAL.\r\n\nCHARLES: It would be great if the entire team could be part of this exercise, but you already said you don\'t want too many chefs in the kitchen. Once you\'ve established, once you have your maps, how do you go about socializing them to everybody on the team in a way that\'s not oh, this is just another business tool that I\'ve got to digest as part of my job. How do you evangelize and get that buy-in?\r\n\nSTEVE: That\'s a great question. There\'s a number of different ways that you can do that, but I think starting early with that communication plan is really important, saying from the very start, "We\'d love to have every single person involved in this. It\'s just not feasible, and it wouldn\'t necessarily be valuable." And it\'s expensive to pull everybody out of their job and get them all in a room. So being very clear about that from the start and then saying that "We can record this entire process, so you can see exactly how it came together, and it will be accessible to people who couldn\'t participate or weren\'t able to participate for whatever reason." So it gets historically recorded, and that\'s adding to that transparency. But we also have the board afterward. The board gets provided, so anybody can go in and look at it themselves, and it can be a living artifact. People can comment on things. People can take the artifacts and the results and remix them in different ways. They could say, "What if we did it this way?"\r\n\nOne thing that we didn\'t talk about was ideal state maps and future state maps. But you could basically take your current state map and have anybody in the organization remix it into an idealized state. So what would it look like if it could be whatever we wanted? Or a future state, which is what could it look like in six months? What could it look like three months from now? And I think that really is very inclusive and accessible. It just keeps the doors open to anyone who wants to be a part of the process rather than being handed something from leadership that says, "Here\'s our three priorities, get to work." It allows anybody in the organization to see the map. They can see okay, well, I understand these priorities because I can see the missing capabilities. I can see the dependencies that we\'re trying to break. I can see the bottlenecks in the value stream, and I can see how all of that is contributing to the outcome that we\'ve all identified that everybody\'s on board with.\r\n\nTARAS: It\'s very interesting. There\'s a lot to think about and process. Would you be able to share some examples? I\'d love to be able to put some on our website so for those who are reading the transcripts would be able to see. We\'ll include some links to your website in the show notes as well. And I think it\'d be really great for people to see an example of these things.\r\n\nSTEVE: Yeah, absolutely. I would love to share some more. I\'ve got an exhaustive end to end case that I call the four by four method, which is connected to the four-door metrics just as a way for anchoring something that people in the DevOps space are really enthusiastic about and paying attention to. The four by four method is basically how do you take the four key maps that I\'ve created and packaged together and link them to the four key metrics as a way of making progress or getting started to improve those metrics? In some cases, how do you get started measuring them? Because a lot of these things either you don\'t know what they are, or you\'ve got to instrument a bunch of your tooling in order to gather that information, but you can get it from the mapping. So it\'s an option for people who are really wanting to get started with tracking the four key metrics and don\'t really know how to do that. Or they\'re looking at that long time horizon where it\'s like we\'re six months from being able to collect this data. This is something that you could do in the week ahead to collect that data and get started.\r\n\nSo I\'ve got the four by four method. I\'ve got an ebook coming out that talks from a team topologies perspective, an enabling team that you could stand up inside of your organization to do these kinds of activities because I think that\'s something that\'s going to be coming very soon. Hopefully, in the next five years, every company has an enabling team that\'s doing this work because it doesn\'t make sense for me to go around and do it for everybody. It should just be part of everyone\'s organization, so that\'s an ebook that\'s coming.\r\n\nI\'ve got a real book that I\'ve just started writing with a great co-author by the name of Andrew Davis, and that\'s going to be about value stream thinking. So, how do we adopt this way of looking at software development and knowledge work and leverage tools in order to get people together rallied around a very clear outcome and then getting ROI from very specific prioritized action items?\r\n\nTARAS: Well, I look forward to seeing these things come out. We\'ll share them on our Twitter. So let us know when the ebook is ready, and we\'ll share it with everyone and same with the book.\r\n\nSTEVE: Thanks.\r\n\nCHARLES: Thank you, everybody, for listening. Steve, thanks so much for coming by.\r\n\nSTEVE: Thanks, Charles, and thanks, Taras.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n

In this episode, Steve Pereira—the founder of Visible—talks about how his unique approach to mapping helps customers get products out of the door fast and efficiently. Maps flow to build alignment, clarity, and confidence, and Steve focuses on two primary areas to drive exceptional business outcomes: Flow and value. Value stream thinking and methodologies, augmented by continuous improvement and performance. TRANSCRIPT: CHARLES LOWELL: Hello, everybody, and welcome to the Frontside podcast. My name is Charles Lowell, a developer here at the Frontside with me also is Taras Mankovski. TARAS MANKOVSKI: Hello. Hello. CHARLES: Hello, Taras. And today, we are going to be talking with Steve Pereira, who is the founder of Visible, which is a company that is aligned in many ways with the way we like to think about stuff, so we wanted to have Steve on the podcast. I understand that Visible has a quite unique approach for how to realize getting products out the door and making your organization such that it delivers. So maybe, Steve, we can just start by talking about how you ended up coming to found your own company dedicated to this. STEVE PEREIRA: Sure. Well, thanks for having me on. I really enjoyed our previous conversation. And I think that we definitely do have a lot in common in of beliefs and approach and how we see the software development space. Visible is the culmination of about 20 years in the industry, basically fumbling my way through understanding process and value and the ability to combine the two to actually deliver value to customers. I come from tech. I come from hard tech. I started my career building computers and fixing computers, crawling under desks in IT. And I was always missing the value piece, and the customer piece, and the big picture view of how all that fit together and how we can combine efforts to drive creation and delivery of value, and I've seen it missing everywhere, and I still see it missing. Twenty years later, I see a lot of people working on micro-optimizations or working in their own silos struggling to collaborate with other teams, other groups, other roles, and we incur a lot of waste as a result. So what I've become very ionate about is stepping back from a process or a team or an organization and then crafting a map and this big picture view of what is going on, where the bottlenecks and opportunities and risks are, and then creating insights and data that will a prioritized list of actions that are actually going to move the needle. So instead of having eight people's opinions on eight different bottlenecks that we could tackle and arguing for weeks, whiteboarding ineffectively, or meeting ineffectively, or writing a bunch of documentation that no one reads, we create maps together. And so everybody is on the same page from the very beginning. And the map shows us where we are going to see the maximum Return on Investment in a way that the C-level executives could understand it, the person who just ed the team yesterday could understand it, somebody in sales, marketing success, or IT could understand it because it's a simple, clear representation of what's actually going on. I think that that's incredibly powerful. CHARLES: Well, it means that you can be intentional about what it is that you want to do instead of reactionary. Oh, here's this circumstance, and I need to do it because where I sit and what I can perceive this is the most critical thing that I could be working on, so I'm going to react to it rather than having some sort of outside information of an overall priority list. STEVE: Exactly. CHARLES: Yeah, it's one way to think about it. There's this phenomenon of. . .what's the word? Priority inflation, where when you're dealing with issues, and you don't have any structure over it, there's this creep of priority to where everything all of a sudden becomes critical. Is it meant to address people operating in those little critical criticality bubbles? STEVE: Yeah. That's definitely a benefit. It's resetting priority to be globally calibrated instead of locally calibrated because everybody does have their own perception of priority, and very few people approach it systematically. Very few people are running mapping exercises to throw their priorities or their possible priorities onto a matrix and relatively calibrating them. When they do, they might be using impact and effort or RICE or WSJF as a prioritization method, but it's usually not based on a lot of data either. They are still bringing a lot of assumptions, and they're bringing a lot of opinions to the prioritization process. So they have a hard time because you can map a bunch of priorities, but then you've got to go to war arguing with everybody else who argues with your prioritization. So unless you do these things as a group and it's based on a foundation of data that people can trust, then you might not be much better off than having nothing at all. So the mapping techniques that I'm a big fan of and that I've assembled together in this flow gets you to that prioritization method by way of almost like a blockchain of logic where you can trace a prioritization or an initiative or a decision back through all the contributing factors that said here's how we know that this is a high-priority effort and it's related to value, it's related to quality, it's related to global maxima instead of the local maxima. And it's tied directly to this either Big Hairy Audacious Goal or a desired outcome that the entire organization shares or the team shares or our division of the company shares. And so you can trace these things all the way through. And all of a sudden, people trust it, and people align, and people are ready to go and deliver because they understand it. You've brought them along instead of handing them something and saying, "Here's what we need to do because of reasons." CHARLES: So as I'm listening to you, I'm thinking, okay, how can we make this concrete? What's an example of how one of these maps might be constructed? If I was on a project, how would you take me through this process? STEVE: That's a great question. So often, probably the most common case here is teams wanting to go faster. So, we'll have a team that's releasing maybe on a monthly basis, and they want to get down to two weeks. And when they're on two weeks, they want to get down to one week. So that would be a desired outcome that we could start from. And we would set up a workshop where we map that outcome to the contributing factors that we could consider before we move on. So you want to tease that apart and look at not just that outcome as a focal point. We want to say, okay, what are the obstacles that could interfere with us delivering that outcome? So we get that out of the way first, which means the team can dump their fears, share their fears, share the things that they're concerned about. It gives the people on the team who might be somewhat cynical or the people who focus on risk a little bit more than others. It gives them an opportunity to voice their perspective and contribute from the very beginning. Then we also look at countermeasures. So the optimists can weigh in and say, "In order to avoid that obstacle, we could do this or potentially, here's something that we have at our disposal that could help out with that obstacle." So we break apart an outcome in a way that we make it real, and we get the team to buy into it in a way that we know what it looks like really well because clarity is the most important thing to start out with: it drives alignment, it drives confidence, it drives energy. It takes it from okay, well, we have this thing, and we have no idea how to do it to we understand it a little bit more; we can move on and, we can move forward. And you go from that outcome map to the value stream map because the value stream is the process. It's the engine that's delivering that outcome. So we look at the value stream from end to end at whatever level. So it can be at the organization level if we're looking at we want to deliver three times the portfolio that we did last time, and that would be a very high-level value stream that we're looking at. It could be release level, so we want to release every two weeks. If we go back to that, then we're going to look at the release process. So we're going to look at okay, here's where we have a product change request on one end, upstream the initiation of the value stream. On the other end, we have customers using the release software, or we're collecting data er activity. So you've got the entire value stream. What we do is as a group, we will map out each distinct activity, and then we'll measure it. So we measure the time it takes to do each activity and the wait time in between those activities, and those are the primary measurements. So not only do you get this representation of the process, you get the measurement that's going to tell you here's where most of the time is being used. And you can go beyond that to layer all kinds of things like value created by each of the activities or quality like what percentage of this work gets thrown back to prior steps? Or when do we find bugs in this process? Where are we discovering errors? You could layer all that data on, but the most important things are the steps and the timing. TARAS: I could see these two things in my mind. It's a thing that I've been thinking a lot about lately and specifically around the work that we do at Frontside around optimizing delivery pipeline with engineering organizations and the approaches that we're bringing there are certain kinds of techniques that we use to address a specific quality. You could say an almost measurable aspect of delivery which is there's latency. There's a latency in the system. And I think a lot of people don't think about latency this way, but there's a latency that is usually related to dependencies. So if you have a front end team, and this is a really common case, so if you have a front end team that is waiting for a back end team to give you the API, that latency can occur in a few different ways: one is you could be simply not able to do your front end work because the API that you're using is not reliable. It's either going up and down all the time, or it's always shifting because of under active development, or it's simply not available when you need it. So your front-end team might be really excellent, or you put together this mobile team that really knows what they're doing, but the back end to match takes a long time to build because there's a lot of legacy. The mobile app could be a greenfield, but the back end API is relying on existing systems. So there could be legacy or technical debt that is preventing things from moving. And there's this latency in the actual process. Is this an example of what you observed? How does this occur to you, what I described? STEVE: This is a great example. So I would say that the top three issues that I discover when we map value streams is the planning process, so a lot of delay, a lot of waste, and upfront planning that is not visible to the teams because it happens before anybody really thinks that development has started. And a lot of people criticize development because it is expensive and it is complicated, so that tends to slip under the radar, but it does influence lead time. It slows down our ability to deliver and collect and move on and pivot or adjust and adapt. The second one I would say is infrastructure automation, so things like standing up environments, refreshing environments, scaling, upscaling and downscaling, making configuration changes because that's often a dependency that's outside the team. It's not usually a capability within the product team. And then, the third and the most important and the most critical most common is testing, which in many cases is test automation. It's automated testing or a lack of automated testing. And there's also validation throughout the value stream that often gets batched up and is happening in a QA phase. It almost gets compressed into this window where we throw things to someone who runs QA, or there's just this very waterfall activity where we start testing after we've finished development. So those are super common. This dependency that you're talking about, the front end and back end, is very real, especially in the release process if we're looking beyond a scrum or a sprint. If we zoom out to the release level, that front end back end friction, the delay, is huge, and that's what would bring me to the next map, which is the dependency map. So the dependency map looks beyond the team or the area that we're focused on in the value stream. And it allows you to see what is contributing to any delays that you're seeing in the value stream. So, in this case, the front end and back end, we would see that delay represented in the value stream as this we've finished work on this one part of the front end, and we have this delay while we wait for the back end to be available before we can validate that everything is working correctly and move on to the next activity. But why is that happening, and what is contributing to that delay? We might have an idea, but what does it really look like? So we do the dependency mapping to call that out. We pull that out of the team, and it gives the back end team the opportunity to say, "Well, we have to wait for this. We're waiting for validation of an open-source library that we need to use because this is asking for a new capability," blah, blah, blah. There are all kinds of things that can be surfaced in that. And we go from the back end is just not fast enough, and we're going to have to deal with it, or they're going to have to get faster to we know exactly what's happening. We know where things are breaking down, and we're that much closer to fixing the problem or addressing the problem directly. CHARLES: So, at what point do you make these determinations? Is this all in one workshop? Because what I'm hearing is getting stuff out of people but then also collecting hard data and doing research. How do you actually construct this map? Is it something you do in two hours? Is it a process that takes weeks? How do you make sure that what you are drawing really is an accurate picture? STEVE: That's a great question. It's a number of workshops. So commonly, what I'm aiming for is a separate workshop for each purpose, so outcome mapping is one workshop two to three hours maximum. I aim for two hours for everything, and I budget for three hours for everything, two hours for outcome mapping, two hours for value stream mapping at a high level. That's not looking at ideal state or future state. It's really just current state, and two hours for dependency mapping, and two hours for capability mapping. That means that people can stay engaged because when you get beyond two hours, people start, especially on Zoom, to sort of lose their energy. And if we have to come back to it and schedule another session, we can, but two hours is a nice time box. In of accuracy, the interesting thing is a lot of the data is coming from people. It's coming from people's recollection of what happened last time we did this thing. We're not usually doing these things as they happen. We're usually looking at okay, well, think about the last time that we had a release or the last two releases, and we'll kind of average them out. The interesting thing is that we don't need high fidelity data for this because the bottlenecks are really dramatically outlying from the typical activities in the value stream. And what that means, in other words, is there's going to be parts of the value stream that are days instead of hours and hours instead of minutes, and so the exact number of minutes and the exact number of hours becomes insignificant. You get diminishing returns from getting detail because you don't need it. And we're bringing in stakeholders from a number of different roles inside of the value stream, so you get this balance. So you've got multiple developers, you've got product, and maybe you have a scrum facilitator; you can collect data. And as a facilitator with 20 years of experience, I know what a reasonable number is for things. I know what seems accurate, what sounds okay. And that's all you need. You don't need perfect detail because all we're looking for is priority. We're just looking for show me things that I'm missing. Show me things that are going to actually move the needle, and then let me prioritize those things so that I'm focusing on the most important thing. And it doesn't necessarily matter whether it's seven days, eight days, nine days because it's going to go from nine days to one day. And the less time we spend on collecting that data and worrying whether it's accurate, the more time we can spend on actually doing the thing because you don't want to invest too much on assessment. You don't want to invest too much on figuring out what's going on. Your job is to be doing things and improving the situation. So you just need enough data to make a decision, which is usually 70% of the data. TARAS: I'm curious, what level of access do you need to have? What's the best entry point in your experience to have this conversation? CHARLES: That's a good question. So it really depends on the value stream in question or the outcome that we're aiming at. If it's a release process that we're trying to cut in half, then we'd probably want to be working with at least the VP engineering, somebody who is incentivized on delivery value and velocity, but it could be customer onboarding. I could be working with a sales team that's trying to close a funnel of -- They have 5,000 prospects, and it takes them three weeks to engage with all of them. They're never going to get through them, so they need to take their process down from three weeks to two days, in which case I'd be working with a VP of sales. So it's really important to get leadership. You don't have to go all the way to the C-level. It's really who's incentivized on the success of this particular value stream. And then, we involve representative stakeholders from the entire value stream. So if QA is a part of it, we have to make sure that we have representation from QA, at least one voice, likewise with product, likewise with engineering. We would want front end and back end if we're talking about that situation where people are experiencing friction with that or people complain about it a lot; those are good signals. And we can use that to build out that list of stakeholders. You don't want more than 12 people. You don't want too many cooks in the kitchen, but you also don't want too few because you don't want to be missing data and information. TARAS: One of the challenges that I see with a lot of big companies is that there is usually impact. The activities of one side of the business can have an impact on another, and people don't really realize that there is an impact until they're stuck doing things a particular way. They'll say, "This is how we do things." QA is a perfect example of this. QA might say, "We need three weeks to QA this application." This is the worst-case scenario, and you hopefully don't have this. But there are companies that operate this way where there's a bunch of development that happens [Inaudible], and now you've got three weeks of QA that needs to happen. Do you ever find this process changing people's minds, causing people to change direction? STEVE: Oh, absolutely. That's one of the best parts of my job. I've worked with a lot of companies where people are commenting that "We've never been in the same meeting together." But they're part of the same value stream: They hand in work to each other all the time. They depend on each other. They're upstream and downstream from each other. So they should be collaborating on a regular basis, but they've never had the visibility to see that they are that closely coupled and that they do depend on each other, and it is really important that they collaborate. I've had remarks where people who've worked in extremely large companies for 19, 15, 14 years and they say to me, "This is the first time I've ever seen the process from start to finish," which is crazy. We know that that is true because what opportunity would they have to see the process start to finish? This is not a regular practice. This is not something that people do, but that's kind of what I want to change. I want to be part of this movement towards increased visibility, increased transparency, increased representation from everybody involved in delivering value because I think it's incredibly powerful not just from the perspective of delivering better outcomes faster and making people happier to do it, but it just builds so much empathy, and it builds so much personal ownership as a contributor. So everybody can see here's how I fit into this, here's how I'm helping to row the boat. I think that drives alignment and drives this sense of unity that is so powerful. And we have so few ways of doing this other than team-building exercises or looking at a CI/CD pipeline, that's giving you a slice. It's giving you this tiny view of what's going on, and that causes problems because we think that this is all I need to be looking at. We need to step away. We need to look at the big picture on a regular basis because we all have more in common with the people that we work with, with the people upstream and downstream from us. To quote Sesame Street, "These are the people in your neighborhood," and we need to work together because that's how you get positive outcomes, that's how you get Return on Investment, that's how you satisfy customers. CHARLES: In of the philosophical underpinnings of what you're describing, it's reminding me a lot of the Toyota Kanban process with how they run their factory floors and maybe not so much in the details of the process you're describing but in the spirit of everybody in the assembly line being able to understand everybody else's job and everybody in the assembly line knowing what's upstream and downstream and anyone being able to even take that knowledge and having a free hand to suggest improvements to the process and being able to do that as someone who has knowledge of that entire assembly pipeline. That's something that they tout highly in Toyota is that they have factory workers on the floor who are able to actually improve the process because they have that global knowledge of the assembly, and they know what their neighbors are doing. And it demonstrably engenders that unity and that pride and that belief in the shared purpose. So as you were speaking, I was like, wow, this is so similar to what I've heard described is going on there, which would seem to validate the idea. TARAS: That's exactly where it comes from. Toyota invented value stream mapping, and all of their lean manufacturing processes are what are now coming into software and leveling everybody up because we've learned a lot of lessons in manufacturing. There's a lot more in common between building software and building a car than I think we commonly acknowledge. There's a lot of creativity that's involved in software but not necessarily creativity that adds value. And I think we need to focus on creativity that adds value, variability that adds value and get rid of robotic activities and things that can be automated so that we can focus our time and energy on the things that should be and can be creative and can add value. Right now, we have this mix where we do a lot of things that aren't necessarily creating value or increasing value, but we don't look at them. We're not measuring these activities against each other. We're not saying, "Well, we get 10 times the return doing activity A than doing activity B, and yet we invest the same time and energy and concern in both." We're not focusing on the things that will ultimately drive an incredible amount of value to our customers. And that's just going to be the next 20 years of software. That's where we're going. Because the companies that do that, the companies that recognize that you are creating a machine that delivers value and it's not artisanal -- We're at the scale now where software has to be delivering results. There has to be clear ROI. We're running the world on this stuff now. We're past the point where you're building toy applications that can just be working on a tiny corner of your business. We have software now that runs massive multinational corporations and partnerships and platforms that companies are building their platforms on. This is demanding a level of attention and sophistication, and automation, and coordination that just makes all of this mandatory. We cannot treat software development as this ad hoc seat of our pants. Figure it out as we go along activity if we want to maximize value and if we want to achieve everything that we can achieve with value delivery. TARAS: So we work with engineer organizations, and so we help them. Steve, in your , it would be ‘help them improve their value stream.’ But one of the challenges is that it sometimes requires work that people don't see value off for some time. You mentioned before that introducing a platform can be a solution, but introducing a platform can take some time, and quite often, it doesn't translate into productivity. It could be easily six months before all of the framework or foundational pieces are in place for people to actually start seeing the outcomes. So I think if there's a way to create confidence along the way, I think that'd be really helpful because you want to be able to maintain a certain degree of enthusiasm for what you're doing, and it is probably the worst to get to a point where you're 60% there, and that's when everyone starts to doubt whether this is still the right way to go. STEVE: I think that that is a huge motivating factor towards this movement towards value stream thinking and big picture thinking and really visualizing end to end process because a lot of individual contributors are going to hear this said "Oh, you need to now take what you know and layer on this extra thing. You need to now be worried about two things instead of one thing," or "You need to now be doing this thing that maybe is pulling you away from where you want to focus." And I think that's the wrong way to go about it. I think that the right way is to represent the value stream, allow people to see how their contributions are affecting the team and affecting customers, and affecting our ability to deliver, and then allow them to contribute in a way that they are most valuable. But also, it gives us the ability to say, okay, well, instead of you just adding five new capabilities to your portfolio and then shifting your focus all the time between these new capabilities -- Because this is an activity that is critical to the value stream, and it's an opportunity that we need to capture across all of our teams, let's build that capability into some form of automation or self-service right into a platform or leverage a solution that allows our specialists to remain specialized or focus in an area that they really want to be contributing instead of saying, okay, well now you have to do this, and this, and this and be good at all of it. I think that's not necessarily the solution. I think the solution is basically stepping back and saying okay, well, if we want the flow of value through the value stream to be optimized, if we want better performance and we want fewer things to slip through the cracks or be delayed by handoffs, what would it take for us to automate or augment or that activity? And in some cases, that's going to be a platform even if it's a minimum viable platform. In some cases, it's going to be pairing. In some cases, it's going to be cross-training, but you have to look at the big picture in order to make those decisions really with a high degree of confidence and capability. So the visualization of the value stream is really going to put you in a position where you can say, you know what? There's no other way to drive the level of performance and value that we need without you in addition to doing software development, starting to do infrastructure development. That's just the best solution that we have at the moment. But you're going to be able to say that with high confidence, and you're going to be able to show your thinking, is the other thing, which means instead of someone grumbling that now I have double the workload or I have to be an expert in this thing that I don't care about, you can bring them along and show them here's what that means. It means we can go from three days for a new feature to two days, and that means you get faster , you get to work on more new things, and that also means you wait around less. All of these things can be clear and visible. And I think that's super, super powerful. I've been in this situation of trying to convince people that an investment that had a somewhat distant time horizon for realization or a Return on Investment was the right decision. And I was always looking for ways of showing my math and showing my math in a way that would resonate with a bunch of different stakeholders because, in some cases, I've got to go to the board with it to make the case. So having a map and having this visualized representation of the data from a number of different perspectives we're looking at and not just the value stream. We're looking at dependencies that are involved. We're looking at capabilities that are involved, and we are tying it to a specific outcome. That makes it real for people in a way that you're almost getting value right away. In a few hours, you're getting value because -- I don't know the saying off the top of my head, but it's if you plan to a certain level, you're half done. If you've got clarity on what you need to do, you're half done, and that really can drive people past the point of the trough of disillusionment after they're working away and they haven't seen the results yet. It's going to carry them through because not only do they have more clarity on what needs to be done, what things are going to move the needle, and what is going to show progress as they go, they can see the finish line with a lot more clarity. So I do think that it has a remarkable impact on our ability to tackle larger, more ambitious initiatives and maintain that level of enthusiasm and confidence through the times where this just got a little bit more complicated. It's more complex than it seemed at first, or we couldn't see everything at first, and maybe we have another factor to consider. You want something that carries you through, and the mapping does that because not only does it give you that good starting point, but you can revisit it, and you can plug in new data as you get the data. So you could go from, okay, well, this was what it looked like at the time; we now know that it looks like this. What does that mean? Does that mean we need to adjust the course at all? Does it mean that this is still a good decision? And that can carry you through in a way that if you're just working off of here's what we want, and I hope that everybody's on board, that seems like a gamble to me knowing that the maps are an option. CHARLES: That actually touches on what I wanted to ask as I was listening to it is what is the average half-life of one of these maps? Things do change. We know that different circumstances arise. So what level beyond that initial investment? You described, I think, about four or five workshops three hours each, which that's a large time investment but probably relatively inexpensive in the grand scheme of things if you don't have to repeat it every two weeks. What is the process by which you tweak these maps? How often do you have to do it? How much time do you have to invest in each thing? Is there a systematic way, or is it just kind of as needed, we make tweaks? STEVE: That's a great question. And we're going from a situation where nobody has ever mapped before, so basically, anything greater than zero is fantastic. We're in the early days of infrastructure automation, where it doesn't really matter what you do for infrastructure automation. Just start doing it, just something. But in an ideal scenario, what you're doing is you’re mapping or updating your mapping every three to six months. Because what's going to happen is the moment you do mapping, 20% of the value stream is going to pop out to you as things that you can just chop off immediately. You can just remove waste immediately because it's visible. You can see very clear low-hanging fruit opportunities. So all of a sudden, the next day, your process is different, which means that your map really is out of date, but it doesn't matter because it just has to be useful. It doesn't have to be accurate. It doesn't have to be current either. So what you're doing is you're eliminating low-hanging fruit immediately. You've got maybe three to five high-priority prioritized initiatives that you could tackle. Once you knock those off, your value stream is also different, so your map is again inaccurate. But then when you go back to the map, you already understand the process, and your second map, the one that you do three months later or six months later, your team knows how to do it. They're familiar with the process. All of a sudden, your mapping activity is an hour maybe, and that's every three to six months. That's tiny. That's less time than we spend on sprint planning. And I would argue that it's 10 times 100 times more powerful and valuable than that. So we talk about a large investment of time, but it does get a lot easier, and the Return on Investment is incredible. It saves millions of dollars. It saves months of time. It can really turn a team around from just being buried and fighting fires all the time to having time to spend on experimentation and innovation. So I've never seen anything more powerful in my entire career. So in of ROI, it's unmatched. There's nothing that will actually deliver the same impact, especially in a short amount of time, in hours we're talking. There's nothing you can automate that would be that effective, especially when it's telling you what to automate. So to answer your question, three to six months is a cadence where you're going to be capturing current state that is notably different, I would say remarkably different. If you're actually delivering improvements, then your value stream looks very different three months later and three months after that. But you could do it every six months and still be miles and miles ahead of your competition that’s not doing this at all. TARAS: Did we cover the entire process? I think we might be at the third map, or did we get through all four? STEVE: Oh, you're right. So capability mapping is the last one, and it's a little bit different than dependency mapping. Dependency mapping is looking outside the value stream at what the value stream depends on or what activities in the value stream depend on. The capabilities piece is inside. So what do we need inside the value stream in order to deliver the best performance? So capability mapping is looking at opportunities to better the team in areas that are going to contribute to the top priority items to address. So we're not looking at every capability that the team has or that is required for the value stream. We want to focus mostly on the bottlenecks and the gaps, and the risks, and the areas of concern because we don't need to know everything about everything. We need to know where to focus, and we need to know where to invest. So we look at capabilities that are directly tied to the dependencies. In the case of our example of front end and back end, maybe we are lacking a capability to mark or to leverage skeleton back end capabilities or something that would allow us to decouple and work asynchronously or work in parallel. Maybe there's a tool. Maybe there's some training. Maybe we need to pair. Maybe we're lacking documentation. All these things come out in capability mapping. And it allows us to say, okay, well, this is a hotspot, our ability to create automated tests. We're probably at a six, and we need to be at a nine or above. That's going to give you this extra dimension on those priorities to give you even more confidence even more clarity on how you go from where you are to the next step in the right direction. TARAS: Are there tools? What tools do you usually use to create these and maintain these maps? STEVE: That's a good question too. I use MURAL for this because I used to use Whiteboards and Post-its when everything was happening in the real world. But nowadays, any collaborative whiteboard is great. I've used PowerPoint in some cases where that's all I could actually get access to in a client situation. But MURAL works fantastically. Any collaborative whiteboard. . .Google Drawings would work. Anywhere where you can have multiple people participating and working together is more than enough that you need. You're really just building a picture together, and what it looks like is less important than the content, but that's what I use on a regular basis. I've got hundreds of MURAL boards now. I probably spend 80% of my day in MURAL. CHARLES: It would be great if the entire team could be part of this exercise, but you already said you don't want too many chefs in the kitchen. Once you've established, once you have your maps, how do you go about socializing them to everybody on the team in a way that's not oh, this is just another business tool that I've got to digest as part of my job. How do you evangelize and get that buy-in? STEVE: That's a great question. There's a number of different ways that you can do that, but I think starting early with that communication plan is really important, saying from the very start, "We'd love to have every single person involved in this. It's just not feasible, and it wouldn't necessarily be valuable." And it's expensive to pull everybody out of their job and get them all in a room. So being very clear about that from the start and then saying that "We can record this entire process, so you can see exactly how it came together, and it will be accessible to people who couldn't participate or weren't able to participate for whatever reason." So it gets historically recorded, and that's adding to that transparency. But we also have the board afterward. The board gets provided, so anybody can go in and look at it themselves, and it can be a living artifact. People can comment on things. People can take the artifacts and the results and remix them in different ways. They could say, "What if we did it this way?" One thing that we didn't talk about was ideal state maps and future state maps. But you could basically take your current state map and have anybody in the organization remix it into an idealized state. So what would it look like if it could be whatever we wanted? Or a future state, which is what could it look like in six months? What could it look like three months from now? And I think that really is very inclusive and accessible. It just keeps the doors open to anyone who wants to be a part of the process rather than being handed something from leadership that says, "Here's our three priorities, get to work." It allows anybody in the organization to see the map. They can see okay, well, I understand these priorities because I can see the missing capabilities. I can see the dependencies that we're trying to break. I can see the bottlenecks in the value stream, and I can see how all of that is contributing to the outcome that we've all identified that everybody's on board with. TARAS: It's very interesting. There's a lot to think about and process. Would you be able to share some examples? I'd love to be able to put some on our website so for those who are reading the transcripts would be able to see. We'll include some links to your website in the show notes as well. And I think it'd be really great for people to see an example of these things. STEVE: Yeah, absolutely. I would love to share some more. I've got an exhaustive end to end case that I call the four by four method, which is connected to the four-door metrics just as a way for anchoring something that people in the DevOps space are really enthusiastic about and paying attention to. The four by four method is basically how do you take the four key maps that I've created and packaged together and link them to the four key metrics as a way of making progress or getting started to improve those metrics? In some cases, how do you get started measuring them? Because a lot of these things either you don't know what they are, or you've got to instrument a bunch of your tooling in order to gather that information, but you can get it from the mapping. So it's an option for people who are really wanting to get started with tracking the four key metrics and don't really know how to do that. Or they're looking at that long time horizon where it's like we're six months from being able to collect this data. This is something that you could do in the week ahead to collect that data and get started. So I've got the four by four method. I've got an ebook coming out that talks from a team topologies perspective, an enabling team that you could stand up inside of your organization to do these kinds of activities because I think that's something that's going to be coming very soon. Hopefully, in the next five years, every company has an enabling team that's doing this work because it doesn't make sense for me to go around and do it for everybody. It should just be part of everyone's organization, so that's an ebook that's coming. I've got a real book that I've just started writing with a great co-author by the name of Andrew Davis, and that's going to be about value stream thinking. So, how do we adopt this way of looking at software development and knowledge work and leverage tools in order to get people together rallied around a very clear outcome and then getting ROI from very specific prioritized action items? TARAS: Well, I look forward to seeing these things come out. We'll share them on our Twitter. So let us know when the ebook is ready, and we'll share it with everyone and same with the book. STEVE: Thanks. CHARLES: Thank you, everybody, for listening. Steve, thanks so much for coming by. STEVE: Thanks, Charles, and thanks, Taras. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript\r\n\nCharles Lowell: Hello, everybody, and welcome to The Frontside Podcast. My name is Charles Lowell. I\'m a developer here at The Frontside. And today we\'re coming to you with a special pandemic episode. With us to talk about this is Taras Mankovski also of Frontside. Hey, Taras.\r\n\nTaras Mankovski: Hello. Hello.\r\n\nCharles: And Jason Kelley, who is the global general manager for blockchain services at IBM. Welcome, Jason.\r\n\nJason Kelley: Hello, Charles, Taras.\r\n\nCharles: Now, I didn\'t even know that IBM had a special unit for blockchain services so maybe you could talk a little bit about that and how that all came to be.\r\n\nJason: Great question because I guess we could say you ask that question about any technology that comes to the forefront. It\'s almost a misnomer to say it\'s a special unit because we\'ve tried to make sure it\'s part of how we engage all of our clients when we talk about data because it is a consideration. It\'s also a buzzword, so we can get to that because it\'s a bright, shiny thing.\r\n\nSo, how did it come about that we decided from a corporate level, a CEO level throughout the organization to focus on blockchain? And it could go back now I think maybe six years, seven years. Of course, as an AI company when you think about this thought of having artificial intelligence or, in this case, augmented intelligence, because ultimately it\'s the intelligence that\'s helping humans, is that in the middle of that it\'s data. And if we go from AI to then even forward-looking things that are being worked on here at IBM such as quantum capabilities, what\'s in the middle of quantum computing? It\'s data because we\'re just moving data faster and with this thought of data being at the center of not just what we do as IBM because it\'s less , and it\'s all about our clients and the greater needs of the world. Data is at the center of all that we do. And so we said, "Hey, here\'s something, some capability that adds what we believe to be trust and transparency to data." And what is in the middle of all that we do as consumers and human beings? We transact in one way or another every day whether it\'s transactions in communication, transactions in financial, transactions in all that we do. Looking forward, we said, "Well, if data\'s in the middle of it, how do we add trust and transparency to those transactions?" And then you start to see an equation coming together: data, trust and transparency, transactions, and this new capability called blockchain. So it\'s something the market had pulled us into saying, "Look, can we get trust?"\r\n\nNow fast forward, we didn\'t know that trust was going to be plastered across everything that we do like it is now when we\'re in the middle of a pandemic and we\'re trying to gain trust and transparency in vaccines and being able to move it from point A to point B and ultimately, into each one of those that want to receive it with equity, with trust, with transparency. So that\'s how we landed on that, making a guess that it was going to be an important capability that would help tie together many of the things. And that\'s why I started with it, Charles, and said, "Look, it doesn\'t stand alone because it helps AI, it helps quantum, it helps IoT, it helps deep analytics." All of those things come together when you can have sheer, trusted, and transparent data.\r\n\nTaras: And it seems like right now, the one thing we could really use a lot more of is trust.\r\n\n[Laughter]\r\n\nJason: Yeah. I have often said here that I would love to get rid of the buzzword part of blockchain and almost get rid of the block and put trust in front of it and just call it a trust chain capability. Think about that. Because I use supply chain because a lot of people will identify that and they\'ll often to go to different industries and they think supply chain is like in manufacturing or in consumer goods. And I say, "Well, no." Actually, in every type of industry, every company, we all have a supply chain because you have a source and you have an endpoint, which is a consumer. And we\'re trying to get a product or service to that consumer. And we have thought about that supply chain as being very linear, that it\'s sequential step by step by step, and there\'s a bunch of handshakes, if you will, going through those steps. Well, we\'ve always allowed others to broker that trust between those handshakes. Did it actually come from where we think it came from?\r\n\nAnd Taras, in this thought about trust I love to talk about one of the examples that was really given to me by a client and he said, "Jason, I love this point because a little known fact is that there\'s more organic food consumed every year than is produced." [Laughs] Which was head-scratching because I was like oh wait, yeah, that doesn\'t add up, does it? So this thought we put things in our body every day and with a lot of trust. So yes, it\'s there, and it\'s really there when we put it in context of this thought of, wow, this is very important to have that transparency. And there\'s nothing like looking at this huge challenge which I\'ve now called the biggest data puzzle of our lifetime which is getting a vaccine out to help solve and cure COVID-19, the challenges that we see around it. And that\'s getting it out with equity and with an assurance of trust and most importantly as we now see, in a very rapid manner. So you hit it Taras with this thought of trust is at the center of it. It\'s right there squarely in the cross hairs of the capability of what we\'re talking about here.\r\n\nCharles: So, what does that look like? What does it look like to introduce trust into this? What does that actually look like in the infrastructure and in the solution that you guys are introducing? I\'m really curious what that actually looks like practically if possible.\r\n\nJason: Let\'s talk about that practically because I think that people think that you can walk around with a bucket of blockchain and just spread it on the current situation. We get that, you know, "Oh, well, it\'s the silver bullet that\'s going to solve all the challenges." Well, it\'s not, and it can\'t be because it can\'t stand alone. And we say many times and people smile and nod because they don\'t get it, but I say, "It\'s a team sport." And they\'re like, "Oh, I like that. It\'s catchy. Blockchain is a team sport.” So I say, "Not just blockchain, solving the challenge of sharing the data in landscapes that currently exist is a team sport. It\'s a team sport."\r\n\nAnd I\'ll give you an example, there are some conversations we\'re having with a U.S.-based company, a wonderful company actually which is very successful in the current pandemic. And when we look at a great company like that which has a very complex supply chain, you start to think wait, it\'s not just the external supply chain. But if you think of any company, I\'m using them as an example because they are like many companies where they have an ecosystem that\'s right inside their own company. Now, they are like many companies. We look around any industry or any company around the U.S. or even globally, and globally it gets even more complex because those companies have their own lines of business inside: Who sells to our commercial clients? Who sells to our retail clients? Where do we get products X, Y, and Z from compared to A, B, and C? So now what you have is this thought of federation of information. And so people say, "Wait a second, federated data, that\'s what you mean when you say a common ledger, this centralized ledger." I say, "Yes, and," because what you have is legacy applications and sometimes many of them, not just one enterprise app where you\'re saying, "Okay, here\'s my ERP, but wait a minute, we have different versions of that. And oh, by the way, we have many different operational data stores with that." How do we make that? It was always, well, why don\'t we get one big, fat data lake and we\'ll throw it all in there and while we swim around, we\'ll get some kind of continuity? Well, it doesn\'t work that easily. What you do have though with blockchain is the capability to have a common data thread that can pull from your existing legacy applications. And so think of it as a way to integrate trusted data because once it\'s in there, once it\'s on the chain, it can\'t be changed nor manipulated without knowing who has changed and manipulated it.\r\n\nAnd so effectively you have to put it in context of our earlier example of what we\'re talking about here with regards to a vaccine. And we\'ll look at North America, think about the natural challenge that we\'re seeing now where you have federal entities working with state entities working with even local entities. And we\'ve gone across just the three layers that then get into maybe even communities. But then also what about the industries? Because there\'s a pharma industry producing it. And then there\'s a travel transportation industry moving at the 3PLs, third-party logistics providers. And then it can stop into a healthcare space where that might be the point of istration or it may even go to retail if it\'s in the drug-side like Walgreens or CVS. So now think of all those industries you\'ve gone through and you have to federate that, and that\'s where you get this opportunity to say, "Can we come up with a lingua franca that allows us to share the data in a permissioned way with trust?" So that\'s practically the challenge that we\'re allowing with this capability to solve.\r\n\nCharles: When you say data, there was obviously a lot of different places, a lot of different entities, and a lot of different industries as you described having custody of this vaccine. What is the data that needs to be collected at each point? How is that federated?\r\n\nJason: About the data that\'s important, let\'s start with that with regards to different types of data because I think to keep it practical for all of us, it\'s the outcome. And so let\'s start with the outcome, and I\'m going to assume that maybe when it comes to your round of vaccines, that if you haven\'t received it you can say -- let\'s just assume you\'re going to receive the vaccine, Charles. Now, if you were to get this vaccine, what would you want to know about that vaccine? Let me flip it back to you.\r\n\nCharles: Well, so I would want to know one, that it was an authentic vaccine and not sugar water to start off with. I would want to know that it hadn\'t been opened in any way, that it was fresh and hadn\'t been tampered with, that the conditions that are conducive to transport had been maintained from the moment it was manufactured to the moment that it goes into my arm, to name three.\r\n\nJason: Okay. So that\'s great. So the authenticity, integrity, if you will, that it is what it\'s supposed to be and then transport that that was sound as well. And then I\'ll add another that you\'d also want to know that if in fact the person that was sitting next to you that you knew and two days later may have had an adverse reaction, you\'d want to know is it because of that person, or is it because of the vaccine? Because I was sitting right next to them. So you\'d also want to know, wait a second, do we have some remediation or some way to understand, connecting the data about the data? And so I\'m adding yet another challenge here. I would assume you\'d want to know that anyway, Charles.\r\n\n[Laughter]\r\n\nJason: And so with that, when you say what data is important, I think we\'ve just outlined that. So think of what we\'ve just talked about, and I think we can boil it down to three simple areas, one is identity. And one would say, "Wait a second, where\'d that come from? Why do we care if Charles is Charles?" And we\'ll get into that because that\'s how we circle back, but it\'s not just you, but it\'s identity of the entity of: Is that the vaccine that it\'s supposed to be? Did you get the one that\'s one dose or two-dose? And did it have to be at certain temperatures? All of that. So first the identity, so entity identification is one thing. Then there\'s also the provenance so there\'s provenance of what was its origin? In your words, Charles, is it sugar water, or is it the real vaccine? So you have that. And then the one that I throw in that sometimes causes people some confusion because we use the blockchain, it\'s in the context of blockchain, but I say tokenization and that\'s just digitization of a physical asset into a digital asset, so being able to track a physical entity in a digital way. So think of all the paperwork that\'s signed in and confirm how we currently broker authenticity is typically that way. So can we tokenize? So I always put the vowel in the middle and you get a little acronym (TIP) Tokenization Identity Provenance. And so those three things then say how we can make sure that this data through each one of those handshakes is maintained. So when the vaccine\'s produced, it gets a birth certificate in a sense, a digital birth certificate that\'s on the chain. I know when, where, how it was produced, and which type of vaccine it is.\r\n\nCharles: And we\'re talking every single dose gets the birth certificate?\r\n\nJason: Well, ideally, we would love to have that. I think most as we\'ve found in the current environment, as I said, we start with the current landscape, it\'s typically by lot number. You have a lot number in there; they\'re kept in lots. But even at that level, you\'re monitoring that no different than if you buy a container of strawberries. It\'s by that lot of strawberries and not necessarily each individual strawberry, not that we can\'t get there someday, but I think that\'s another conversation.\r\n\nCharles: [Laughs]\r\n\nJason: You want to have that amount of data, so you give it a birth certificate and then as it goes through each handshake, and I\'ll stick with my same analogy, that\'s where you have an opportunity to put data on the chain but also data in context about that vaccine. So, yes, what does that start to pull in? You heard me mention internet of things (IoT). Well, we have many devices that are talking to each other right now. There\'s a great company called AeroSafe that has the refrigeration containers, and guess what? They have a GPS on that and a thermometer, and it\'s constantly communicating. So now you\'re collecting data about the transport. Your other concern, right? And so not just transport but the integrity: Was it at the right temperature? Where did it go? What were the different handshakes that it went through? Is it authentic? Did it really stay in the chain that we think it did or did it step out? Is this a black market? Has there been some fraudulent activity with this? Those are the things that you wanted to know.\r\n\nSo keeping all of it in context of the end-, that\'s the data that is important, and that\'s where that data is then loaded into this common thread of safe, secure, trusted, and transparent data on the chain. That\'s what then at each step you would want to happen. And that\'s why, when I said it\'s a team sport, I truly meant that that means that each one of those places can simply go online and say, "I don\'t have to buy..." You\'ll start to hear some buzz words here. "I don\'t have to buy all this infrastructure." It can be a nice cloud infrastructure that now exists. And they say, "Why haven\'t we done this before?" Well, we didn\'t have, all of us, didn\'t have the capability. We didn\'t have a blockchain that was prevalent and proven and in a production way. We didn\'t have the speed and capability that we have across clouds in a hybrid cloud environment. We didn\'t have all of this capability even from a can I sign on to the chain on my cell phone and simply scan? Yes. All of these things have come together as we\'ve seen in other industries to say the stars have aligned. And in an unfortunate time and an unfortunate event like we have now, we fortunately have technology that was ready to go on the shelf, ready to bring forward and solve the big problems that we\'re doing now.\r\n\nTaras: It\'s fascinating. I have so many questions.\r\n\nCharles: Yeah.\r\n\n[Chuckles]\r\n\nTaras: It\'s interesting because it\'s really time-sensitive. Because usually, you see adoption of fundamental enterprise or technologies like this that span the entire spectrum of supply chain and usually those things evolve and are maintained over a long time. But in the same way, we had to fast track the vaccine itself, you actually have to fast track this infrastructure because in some ways, now that you\'ve described this, I do want to know did someone at some point open the case that had the vaccine? Was there an opportunity for that vaccine to be swapped out with something else? I would want to know that. And it makes me think when you think about transparency, you think about transparency being ubiquitous, it\'s transparent to everyone. And so how much transparency can you expect? What\'s a reasonable amount of transparency? How much information can you actually get and should be able to get about every step along the way?\r\n\nJason: And those expectations are becoming more and more the standard. I\'ll step out of the life-threatening type. I should focus on the positive, life-saving opportunity we have with the vaccine and not the threatening part of the virus itself. And I\'ll step into fashion, something that we always talk about with regards to authenticity. However, there\'s a company that we\'re working with now called Covalent at C-O-V-A-L-E-N-T. It\'s a fashion brand whose products are made with AirCarbon, a biomaterial that\'s made by microorganisms. It\'s meltable; it can be used in alternative fiber to plastic and leather. So it\'s carbon negative, certified by Carbon Trust. So as we look at this, you start to say, "Wait a minute, can we trace the authenticity or at least the carbon production or offset of these products that are being made?" And on the other side, when we look at another -- and I\'m just calling out real examples, so it\'s not just what-ifs. There\'s a consortium that we work with called the Responsible Sourcing Blockchain Network, think of automotive players such as Ford as they look to control some of the minerals and metals that go into their batteries such as cobalt, some very sensitive minerals. So when you look at that, you say: "Has that mineral been sourced properly? Did it come from the right mine? Is it really making it into the battery? And then also another buzzword, buzzword alert, in a circular economy, does it go from that battery back into the recycles so it can then come into another battery?"\r\n\nSo I\'m lightening up the topic to say there are many things in everyday life that if you jumped into a car anytime soon, many of us haven\'t left the house in weeks. [Laughs] But if you have and it\'s electric, you\'re thinking there\'s a battery there. If you\'re looking at the computer as we\'re doing this now, there\'s a battery in there. So we\'re experiencing all of these things that you start to say, "Wait a second, I really would like to see that." And we know that from work that we\'ve done with the National Retail Federation, 72% of people would say, "I\'ll pay more to know that what I\'m buying and consuming reflects my values, which currently sustainability is one of them." So how do you keep score? It\'s all based on data. And how do we trust the data? As you said, I think Taras you mentioned it\'s back to that big trust thing. How do you do that? So yes, vaccines have brought to the forefront what I\'ve just mentioned has been working with regard to trust and transparency and product in food and things that we consume as well as those things that have driven big ideas such as sustainability into the forefront with credibility and trust. How do I make sure I\'m keeping score on those things with some level of trust and transparency? And we\'re back to where we started with this thought of data that\'s on the chain.\r\n\nCharles: Yeah. I can think of a couple. I\'m already thinking of a bunch of different applications both in the computational sphere but also in practical life. You could maybe do something like apply this to the chain of custody for crime scene evidence or something like that where you might have multiple different law enforcement entities and labs and stuff doing analysis and just transporting. But that kind of leads me to the next thing. So we\'re talking about huge amounts of transparency to your data and being able to say, "Okay, this data has unassailable integrity, and I can see everything about it and know everything about how it came from the moment it entered the world until it arrived in front of me. And I know this because I\'m by no means a blockchain expert, but it\'s because it\'s distributed and anybody and everybody can it for themselves without having to rely on any central authority. So does that introduce any privacy issues? If everything must be transparent, everything must be out in the open, how do you hold back certain data or say this is for your eyes only?\r\n\nJason: So, Charles, and I knew we\'d get to this because a little foreshadowing, I said, "Hey, I wanted to come back to this thought of you being able to that you received that vaccine." And that starts the talk about wait a second, if you have received the vaccine first, you\'d want to validate and it to anyone if there was a way or if there were, first, a reason for you to do that. Let\'s say that it could be you physically returning to a place of work that you needed to go or for entertainment; you wanted to go to a theater or for travel. As we\'ve just seen, many of the airlines now are going to have that requirement to say, "I\'ve either received the vaccine, or I\'m COVID negative." Well, the only thing you want to be able to do Charles or Taras, the two of you, you just want to say positive or negative or positive meaning I’ve received the vaccine. I\'ve positively done that or I\'m COVID negative. They don\'t need to know your age. They don\'t need to know your home address. They don\'t need to know your social, so there is a sense of privacy. And the assumption that just because something is decentralized everyone can see it is not true. And that\'s part of the beauty and the way that a permissioned blockchain works.\r\n\nAnd I throw that name out, some people say public and private, just say permissioned. And what does that mean? That means that you own your data, Charles. You own that validated and verified health , which is (you can Google it) IBM Digital Health . That\'s what we\'ve pulled together in this thought of identification, I said TIP. And this is the identification that does apply to you individually. It says that when you receive that vaccination on your smart device, you can then at the same point, validate and on the chain that yes, I have received the vaccination. And with that health , that data is yours. Now, in the permissioned chain, you say, "The only thing I\'m going to give anyone permission to see is, have I received the vaccine?" That\'s it. Now you may offer other permissions, and we start to think about what you\'re opting into here. It works the same way on the chain.\r\n\nThe beauty is that the chain is a lot more secure than other forms of data storage because of the level of encryption as well as we\'ve said this thought of transparency if someone does try to access something, you can see it, or if they change it, you can see it. You by owning that could say, "With my employer, I do want to share my location because I work -- and I\'m making things up here, Charles. But let\'s say you work in an office that has multiple floors or maybe even multiple buildings, and you want to make sure that you do share where you\'ve been because there\'s a reduction in the number of people in the building at this time, and they want to make sure who\'s come in with whom. So then it becomes a permissioned chain with permissioned, trust, and transparency that gives the outcome, and that\'s why I said it\'s not about the technology, it\'s the outcome. It gives the outcome that we\'re all hoping to have.\r\n\nSo that\'s where you and Taras could start saying, "Well, if you could do that, then you could do this." And so a lot of those ideas start bubbling up and you say, "Well, the technology is there. Why can\'t we move faster? I think let\'s go faster." And that becomes some of the things that we\'re seeing now with people getting what this thing is technically. They\'re away from the weeds. They\'re saying, "You don\'t have to explain it. I don\'t need to understand blockchain. Don\'t tell me what a hash is. I get what it does. When I flip the light switch, the lights come on. I don\'t know how the electricity got from the production facility to my light bulb, but I\'m good. So now, how do I use it to do other things?" That\'s what we\'re seeing.\r\n\nTaras: Let\'s say we take an airline, for example, and they want to know about my status if I\'ve taken the test or had the vaccine. Is there a trusted app? Maybe the Canadian government would provide a trusted app that I would install on my device that would show that information. How do I access this permissioned blockchain and then allow others around me in the real world to trust that information?\r\n\nCharles: So a practical application with our digital health can be ed. The actual app can be ed on the Apple store. So you can an app and then that app accesses the chain. So we\'ll call it the blockchain; we\'ll get rid of the buzzword, and we\'ll just say the network. And so it ties into that secure network. And then in your example, the airline would also be on the network, and now there\'s a common linkage, there\'s Taras, there\'s the airline and you\'re both able to access the same chain. And now you have -- I\'ll borrow from Charles your example of the crime evidence. I liked your chain of custody, but I\'ll change it. It is kind of like a dead joke version, the chain of trustedy if you will.\r\n\n[Laughter]\r\n\nCharles: And so you take that chain of trustedy, and you say, "Okay, now the airline has it. I\'m Charles, and I have permissioned to see just what I want them to see." It\'s that simple. Is there hardware involved? No, it\'s no different than you g up for your next ride-share app that you\'re going to . You\'re on the network when you sign up. The technologies behind all this, and that\'s the goodness, not only is it behind it, it\'s also behind it with the airline. And I say that because many of the airlines are already starting to put together their own applications, the technology behind it. And that\'s where when we talk about IBM, that\'s where we sit. It\'s not something that\'s oh, everyone\'s going to use the IBM solution. No, we just want it to have the IBM capability behind the trust and capability that we\'ve invested in and we\'re making available to many at this point so that they can use what they already have. They don\'t have to change their branding; they don\'t have to change their UI UX because that\'s very valuable. They have invested millions and sometimes hundreds of millions into the experience, and what we want to make sure is happening in that experience is that we\'re getting the right outcomes. In this case, it\'s that everyone is safe and secure and that they\'re able to distribute that safety and security with a sense of equity so that it does help us all move forward together. So that\'s the thought of the network and why you hear me say it\'s a team sport. I\'m giving some meaning to that, and that\'s how it comes together.\r\n\nCharles: I\'m wondering where the vendor tooling like the stuff that you all offer fits into the ecosystem because if I understand it correctly, it\'s not something like Amazon Web Services where each one of those handshakes within would say, "Post that data to some centralized service that then everybody could agree on. Everyone can access the entire thing any time. No one really owns the chain itself. They can it from anywhere. So what kind of tools then do you provide if there\'s no opportunity for a centralized service? Is it in of libraries? Is it in of, I don\'t know, application components that you can use and utilities?\r\n\nJason: So the quick answer is yes, those libraries can exist, and there is a centralized management capability that you would want in the blockchain. And when we think of examples that I\'ve given, if we\'re talking about a single entity, it\'s easy to say that when we\'re working with a single client, their blockchain is their network. And often either they will want to manage it or they say, "IBM, we\'d like for you to manage it," which is fine or collaboratively -- and sometimes we\'ve delivered projects to the federal government and it\'s been in collaboration with Walmart and KPMG and a pilot that we did there. So now you have maybe IBM and KPMG working together. So I wouldn\'t just call out one name because I will contradict my team sport. But you have this thought of someone managing the network. And so when you often have a consortium type of environment where there\'s many entities coming together, there is a central managing body.\r\n\nI\'ve had some wonderful conversations with Brian Behlendorf and some of the things that they\'re doing at Linux Foundation, where they\'re saying, "Look, let\'s look at some of these centralized management capabilities," things that they do day-to-day. They would say, "Yes, there is a centralized management capability that allows for that consolidation of tooling so that it can be developed, but it\'s also open." And that\'s why I throw out Linux. And we also think of Hyperledger as one example where you say, "Look, there is an open source community that allows these tools to be developed as they\'re needed in the market. So that\'s what\'s good, the start of a centralized trust with a very open environment with many, many capabilities that are developed based on market need. I think that\'s some of the uniqueness here, when you start saying, "Wait a second, this really could and should move quickly," as we\'ve seen. And when I say as we\'ve seen, what have we seen? This thought of blockchain is something that\'s experimental. And I love when people say, "Oh wait, people are just doing pilots or it\'s not real." And I say, "Wait a second, it\'s in production." In many industries around the world, people are transacting using this, and it\'s been 10 years if we call it that and the internet took 30 years. So in one-third of the time, we\'ve gone from concept all the way to production-ready. And how can we make this work in the most effective way with vaccines? So that\'s the reality of what we have and the opportunity in front of us that should make us all scratch our heads and say, "Look, how do we do this together and make it move faster?\r\n\nTaras: I imagine in this team sport you have all these different companies that are participating in this whole process. What does it look like -- For a company of any size they\'re going to, at some point, want to say, "We want to use this. We want to be part of this network." How does this actually work? Is there an SDK or something that they would ? How do they actually become part of this system?\r\n\nCharles: So we\'ll use health as an example, if they want to become part of that, yes there\'s an SDK that they can and that makes it available to them so that in a sense, you get this thought of oh, it\'s a white-label capability that they can incorporate into what they\'re already doing. And that\'s why I said it\'s that painless, and it\'s complementary to what they\'re already trying to do. It gives them that layer of trust and transparency. If you\'re looking at a single entity, and I\'ll stick with the same example of Home Depot, where they have worked on their own capabilities in blockchain. And you say, "Wait, what is it that they\'re going to do?" Well, they take their current environment and in that current environment, they then stand up a blockchain network that works with the data that they want to put on chain. And so it enables them to move quicker, faster, easier within their ecosystem. But quickly, what you can see happen is you say, "Okay, if I\'m going to work in my ecosystem, but I\'m also going to use it to work with my suppliers," quickly, you can start to say, "Wait a second, if those suppliers are going to be on that blockchain network, perhaps that blockchain network would expand to the suppliers of those suppliers assuming that some of them have even a second or tertiary tier suppliers to them." And then you start to understand this thought of how the value of a network is increased exponentially by the number of players in it.\r\n\nIs it a necessity to start with all of the players? I say, "No." It starts the other way around is you start with a minimal viable ecosystem. In our case, you say, "Look, let\'s start with a manufacturer of a vaccine. Let\'s start with a government entity. Let\'s start with an istration point and someone who may want to make sure that they can validate and that they have received the vaccine." Then you have a minimal viable ecosystem. And that\'s what it starts to look like where you say, "Look, I can now work together." So yes, for an individual company, that SDK is there. You put and pick it up. You now use that to tie that capability into what you already have, and you\'re ready to move fast forward. And then you\'d want that to iterate in that if there are capabilities or needs, and this becomes the value of the network and an open network where you say, "Wait, I want to be able to make sure that on my AWS cloud, it does this. Wait a second, I\'m not on AWS; I\'m on Azure. Oh, well, wait a second. I\'m on IBM\'s cloud." Forget the flavor. And you start to give real meaning to brand names like OpenShift when we talk about it\'s open. So now you\'re multi-cloud or hybrid if I\'m going to be on my premises as well as in the cloud. So these buzzwords and marketing start to get real meaning when you put it in context of solving the big problems we\'re talking about here.\r\n\nTaras: A lot of times when we talk about technology, the thing that we want to focus on is what it unlocks, but there\'s always a caveat. And it\'s often in exploring the caveats is where you start to really understand the problem, and you can\'t really until you ask the question. So I\'m curious what is the caveat? What\'s the thing that people need to be worried about or thinking about, or what are the limitations of technology that we have today that you see as being an obstacle that companies have to overcome?\r\n\nCharles: This is a great question and this thought of what\'s the catch? Because I like to call myself an optimistic pessimist at times. I go wait, there\'s got to be a catch here somewhere. And what we know as a company, and I\'m about to make another IBM statement, but it\'s a true statement. Part of our values and belief is that we believe in innovation that matters for our clients and for the world. And so yes, it is a Tech for Good type of thought and whether it was putting the backbone for the modern U.S. social security istration systems to make sure that we could do that during the great depression or putting people on the moon and bringing them back safely. We signed up for big, big things. And so now we\'re in the middle of this pandemic, and we\'re saying, "Look, we are g up for that." But what\'s the catch? It\'s not the technology that is the catch, Taurus. What we find is the challenge in the team sport is the business process. It\'s the design of the organizations and, back to my original example, the handshakes in that supply chain, and I would call it a value chain though that\'s a buzzword. But you start to think about different layers when you say, Oh, well, if it\'s just moving something from point A to point B, it can\'t be that complex." But what about how someone pays for something, how the regulation is in this state versus that state or this country versus that country, the paperwork that has to be signed here, but not over there and tracking it. And you start saying, "Wait a second, we have technology that can solve it." Yes. But technology is a lot easier than the business process. So what we\'ve seen is the catch in trying to say, "Wait, we have to actually change the way we do business in order to make this work," and I\'d say that\'s the catch.\r\n\nAnd a great, great company example with the chain and I\'ll call them out because Walmart bought into this early on, bought into the idea of blockchain is a game-changer. And they teamed with us, helped us launch something years ago that we called Food Trust. And they said, "Look, foodborne illnesses kill way too many people year in, year out so let\'s address that." Not only are there deaths involved, but there\'s also wasted food. So if there is food, food is assumed guilty until proven innocent, so if we say there\'s a leafy green vegetable scare, then we\'re going to throw it all away. Well, Walmart was world-class -- in seven days if something was suspected -- So let\'s say that they had a scare in leafy greens, in seven days, they could figure out where that came from. But for seven days, they\'d have to throw away all of that food. Some of it could be good, so that\'s wasted food. Why don\'t we put this on blockchain? And we could see each one of those handshakes with trust and transparency. They did it with mangoes in their first effort. And that 7 days went down to 2.2 seconds. So think about that. What they used to have the luxury of finding in 7 days, was now 2.2 seconds straight across the chain; we know where it came from. Well, that\'s technology.\r\n\nBut think about the challenge that they had and now they are moving very quickly. You start to say, "Uh-oh now I have to change my business to move that quickly," that\'s the gotcha. That\'s the catch is that you have to start saying, "Okay, now how do we make sure that we design?" And this has become a real differentiator in the work that we do. And it\'s not as I guess glamorous as some of us propeller-heads technology-fascinated people, nerds, whatever you want to call us, us people, but it\'s in the softer part of design. And we found this thought of business design has now come to the forefront. And that\'s where I use the buzzword, you know, people like so many MV things like minimum viable product. How do you decide what is that minimal viable ecosystem, and how do you decide that you\'re focused on the outcome that\'s going to drive the right uptake? Because as you can imagine, all of it sounds good until you actually have to change. And the best way to cause change is to show a positive outcome to make sure that Charles and Taras are happy, confident, and trust that when they receive that vaccine, that they\'re receiving it, they\'re telling everyone else. And they\'re now a proponent for getting things out quicker, better, faster, and everyone in that chain of trustedy, to use my bad joke again, everyone else has confidence. So think how quickly that then gets the speed of business moving along even more quickly than it was and more effectively I should say than it was before.\r\n\nTaras: It\'s fascinating because it makes me think of something that I\'ve observed with a lot of our clients is that because we focus on developer experience, we focus on the way software organizations work, you get to a point where you have an experience -- your system operates in such a way that it gives you new experiences. And it makes me wonder when you have this kind of change when you go from taking 7 days and throwing out a whole bunch of food to going 2.2 seconds to find out where the problem might be, what does that unlock for the kind of business that you can run? What is the experience of that business that it creates? I\'m really curious to find out what that produces.\r\n\nCharles: I\'m with you, Taras. I would say this thought of everything being controlled, I think there\'s more and more control that people are now realizing exists at the point of the end-. I\'m just underlining something you\'re saying there because I think it\'s a key point here. It\'s because that end-, in this case, the consumer, all of us, I mentioned the demand is there. We are starting to expect that type of experience where I want to know right now -- for example, I\'m a coffee drinker, and I want to know if it\'s stamped organic or if it says that it\'s come from this location. I want to make sure it was sourced ethically and that it got to me in the right way. And so as an example, 1850 Coffee -- I\'ll do a real example here with Smucker\'s brand, if you\'re a coffee drinker. We worked with them to make sure that through something we ed with them called Farmer Connect that yes, you can see where that coffee came from. That experience, that level of confidence, it\'s not a vaccine, it\'s just your everyday coffee, and you want to make sure that yeah, I feel confident. And there was even an app that was added to that called Tip Your Farmer where you can say, "Yeah, the barista\'s great. I\'ll..." And then because it\'s more virtual now, maybe you can\'t actually drop that physical tip in there, but what about tipping the farmer? Because we have found, and we know that in many cases, the actual producer doesn\'t see as much of a cut of the profits as some of us would hope. And so how do we put it back into their pocket through this thought of transparency? And then what does that mean? Does that mean that Taras if you\'re a coffee drinker or Charles you go, "Well, okay if I\'m already paying now..." when you start to look at it, you go "I just paid $5 for a cup of coffee. Well, okay. I\'ll pay $5.50 or $6, and I want it to tip the original person who helped get that good coffee bean into my cup. So those are the things you start to think about. As you said, you sit back and scratch your head and go, this starts to change the model now. It pulls things to the forefront. It puts you in touch as the end- with the originator, the producer. You can now shake that producer\'s hand, and you don\'t have to wait for a series of sequential handshakes. So think of that model that used to be sequential you\'re crashing that now, and you\'re putting you as a consumer in touch with the producer. That starts to change many business models because that can also change relationships. And that\'s what\'s exciting about this thought of opening up new markets and new opportunity and also the thought of increasing inclusiveness and equity around many markets where we start to say, "Oh, that\'s where the value is actually coming from. Let\'s assign the payment or that capability that we have to reward where the value is created." And so that leaves us as consumers to decide and this enables it to happen with trust.\r\n\nTaras: So it sounds like what I\'m hearing is it actually makes sustainability possible. When you can trace to the origin in a second, if that\'s available as something that anyone can apply this idea to their circumstances, it actually creates a possibility where a farmer could be sustainable because there is actually a connection there. They\'re not commoditized in the same way that so much of our products are today.\r\n\nCharles: It makes sustainability sustainable in our opinion, to throw another soundbite out there because it\'s true. Now, if you can keep score, as a consumer, you\'re going, wait a second, it\'s not just because it\'s stamped on the product. I can actually with trust trace it back to where it came from and the origins. And yes, it is sustainable. It is organic, It was sourced property, and oh, by the way, maybe I can put a bit of coin in the pocket of someone who helped produce, design, or deliver it.\r\n\nTaras: You can actually verifiably confirm that this is actually -- it creates a connection between someone who\'s completely disconnected from the farmer to the person who\'s consuming the product, and there\'s actually a chain of connection that can be trusted and maintained.\r\n\nCharles: Exactly. And there\'s also this thought for the supply chain folks that would be listening in. There\'s a lot of other things that start coming out in these conversations because with that visibility, you start to see the actual roles in this old thought of a supply chain start to shift where -- and we\'ll keep it in context of the pandemic think of PPE and a lot of it being shipped -- First, let\'s talk about different roles. You had breweries that said, "Wait a second. I\'m not selling a lot of beer out the tap because of the pandemic. Oh, wait a second, in the same place, I can produce hand sanitizer. And yes, it\'s going to smell a little different than everything else is, if you\'ve had any of that, maybe in any establishment you go, "What\'s different here?" Well, they were able to shift quickly and go from producing beer to hand sanitizer. But wait a second, that ecosystem is different. How do I know it\'s really coming from them? How do I know it\'s up to standard? If I\'m now a supplier of hand sanitizer and not beer, how do I quickly find the buyers in the same way? Is there a network I can to make sure that we can trust each other? That\'s the first thing. I\'ve gone from being a beer producer to hand sanitizer. So we started changing the thought of hybrid. Now I have a hybrid business because it\'s doing both of these things, and now I\'m in a different type of supply chain. So now I\'m going to keep using that word — and now I\'m in a hybrid supply chain. And I ship my hand sanitizer to let\'s say a healthcare provider that was running short so they buy from me and everyone else. And all of a sudden this healthcare provider has more hand sanitizer than they\'ve ever had. And they\'re going, wait a minute, now I have way too much. What do I do with it now? They\'ve gone from a buyer to a supplier. How can they quickly pivot and step back into a trusted network and redistribute? And so I just thought of redistribution which is where I\'m going, and it\'s because you\'ve seen it with the vaccine as well where they say, "Wait a second, we have extra doses because we\'ve already done our priority one; now we need to do priority 1B. Wait a minute, we might have too many because someone else needs it. Well, again, what holds that up is the transparency. People think about trust -- in the provenance piece when we were talking about TIP, but it\'s not just provenance; it\'s how do I identify where you have supply and where it\'s needed? So how do you match have and need? And so this again is a benefit of transparency, permissioned transparency. And you say, "Okay, I\'m part of this network. I know I have this product or service, and now I can go from a buyer to a supplier with trust." And you\'re going to get what you ask for, and you know that I am trusted because I\'m on this network, blah, blah, blah. So you get it. You get where this becomes a lot more dynamic in this system.\r\n\nCharles: Yeah, absolutely. This is really fascinating. Jason, thank you very much. I\'m really glad that we had a chance to have this conversation, and thank you very much for coming on and chatting with us about this. It\'s very timely, and I\'m really glad to have a chance to talk about it.\r\n\nJason: Thank you. Thanks for the time. I really enjoyed it. It\'s one of the few times that I\'ve had such an enjoyable conversation with two people on a podcast, typically it\'s one person. So you all made this very enjoyable. So anytime you want to jump on and the two of you want to have a conversation, I\'m game. Thanks for the conversation and stay safe, stay happy, and we\'ll talk soon.\r\n\nCharles: All right. Thank you everybody for ing in with this conversation. That was absolutely fantastic. And as we heard, it\'s very timely. It\'s often said that necessity is the mother of invention. Computers, I believe got their start during World War II as part of early cryptographic efforts, and the militarization of computers happened during the Space Race so that we could take these big vacuum tube things and put them on to things that we\'d shoot into space. And there are technologies that are emergent during times of great change, and it sounds like this might definitely be one of them, so keep it on your radar, and we will talk to you next time.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n

Transcript: Charles Lowell: Hello, everybody, and welcome to The Frontside Podcast. My name is Charles Lowell. I'm a developer here at The Frontside. And today we're coming to you with a special pandemic episode. With us to talk about this is Taras Mankovski also of Frontside. Hey, Taras. Taras Mankovski: Hello. Hello. Charles: And Jason Kelley, who is the global general manager for blockchain services at IBM. Welcome, Jason. Jason Kelley: Hello, Charles, Taras. Charles: Now, I didn't even know that IBM had a special unit for blockchain services so maybe you could talk a little bit about that and how that all came to be. Jason: Great question because I guess we could say you ask that question about any technology that comes to the forefront. It's almost a misnomer to say it's a special unit because we've tried to make sure it's part of how we engage all of our clients when we talk about data because it is a consideration. It's also a buzzword, so we can get to that because it's a bright, shiny thing. So, how did it come about that we decided from a corporate level, a CEO level throughout the organization to focus on blockchain? And it could go back now I think maybe six years, seven years. Of course, as an AI company when you think about this thought of having artificial intelligence or, in this case, augmented intelligence, because ultimately it's the intelligence that's helping humans, is that in the middle of that it's data. And if we go from AI to then even forward-looking things that are being worked on here at IBM such as quantum capabilities, what's in the middle of quantum computing? It's data because we're just moving data faster and with this thought of data being at the center of not just what we do as IBM because it's less , and it's all about our clients and the greater needs of the world. Data is at the center of all that we do. And so we said, "Hey, here's something, some capability that adds what we believe to be trust and transparency to data." And what is in the middle of all that we do as consumers and human beings? We transact in one way or another every day whether it's transactions in communication, transactions in financial, transactions in all that we do. Looking forward, we said, "Well, if data's in the middle of it, how do we add trust and transparency to those transactions?" And then you start to see an equation coming together: data, trust and transparency, transactions, and this new capability called blockchain. So it's something the market had pulled us into saying, "Look, can we get trust?" Now fast forward, we didn't know that trust was going to be plastered across everything that we do like it is now when we're in the middle of a pandemic and we're trying to gain trust and transparency in vaccines and being able to move it from point A to point B and ultimately, into each one of those that want to receive it with equity, with trust, with transparency. So that's how we landed on that, making a guess that it was going to be an important capability that would help tie together many of the things. And that's why I started with it, Charles, and said, "Look, it doesn't stand alone because it helps AI, it helps quantum, it helps IoT, it helps deep analytics." All of those things come together when you can have sheer, trusted, and transparent data. Taras: And it seems like right now, the one thing we could really use a lot more of is trust. [Laughter] Jason: Yeah. I have often said here that I would love to get rid of the buzzword part of blockchain and almost get rid of the block and put trust in front of it and just call it a trust chain capability. Think about that. Because I use supply chain because a lot of people will identify that and they'll often to go to different industries and they think supply chain is like in manufacturing or in consumer goods. And I say, "Well, no." Actually, in every type of industry, every company, we all have a supply chain because you have a source and you have an endpoint, which is a consumer. And we're trying to get a product or service to that consumer. And we have thought about that supply chain as being very linear, that it's sequential step by step by step, and there's a bunch of handshakes, if you will, going through those steps. Well, we've always allowed others to broker that trust between those handshakes. Did it actually come from where we think it came from? And Taras, in this thought about trust I love to talk about one of the examples that was really given to me by a client and he said, "Jason, I love this point because a little known fact is that there's more organic food consumed every year than is produced." [Laughs] Which was head-scratching because I was like oh wait, yeah, that doesn't add up, does it? So this thought we put things in our body every day and with a lot of trust. So yes, it's there, and it's really there when we put it in context of this thought of, wow, this is very important to have that transparency. And there's nothing like looking at this huge challenge which I've now called the biggest data puzzle of our lifetime which is getting a vaccine out to help solve and cure COVID-19, the challenges that we see around it. And that's getting it out with equity and with an assurance of trust and most importantly as we now see, in a very rapid manner. So you hit it Taras with this thought of trust is at the center of it. It's right there squarely in the cross hairs of the capability of what we're talking about here. Charles: So, what does that look like? What does it look like to introduce trust into this? What does that actually look like in the infrastructure and in the solution that you guys are introducing? I'm really curious what that actually looks like practically if possible. Jason: Let's talk about that practically because I think that people think that you can walk around with a bucket of blockchain and just spread it on the current situation. We get that, you know, "Oh, well, it's the silver bullet that's going to solve all the challenges." Well, it's not, and it can't be because it can't stand alone. And we say many times and people smile and nod because they don't get it, but I say, "It's a team sport." And they're like, "Oh, I like that. It's catchy. Blockchain is a team sport.” So I say, "Not just blockchain, solving the challenge of sharing the data in landscapes that currently exist is a team sport. It's a team sport." And I'll give you an example, there are some conversations we're having with a U.S.-based company, a wonderful company actually which is very successful in the current pandemic. And when we look at a great company like that which has a very complex supply chain, you start to think wait, it's not just the external supply chain. But if you think of any company, I'm using them as an example because they are like many companies where they have an ecosystem that's right inside their own company. Now, they are like many companies. We look around any industry or any company around the U.S. or even globally, and globally it gets even more complex because those companies have their own lines of business inside: Who sells to our commercial clients? Who sells to our retail clients? Where do we get products X, Y, and Z from compared to A, B, and C? So now what you have is this thought of federation of information. And so people say, "Wait a second, federated data, that's what you mean when you say a common ledger, this centralized ledger." I say, "Yes, and," because what you have is legacy applications and sometimes many of them, not just one enterprise app where you're saying, "Okay, here's my ERP, but wait a minute, we have different versions of that. And oh, by the way, we have many different operational data stores with that." How do we make that? It was always, well, why don't we get one big, fat data lake and we'll throw it all in there and while we swim around, we'll get some kind of continuity? Well, it doesn't work that easily. What you do have though with blockchain is the capability to have a common data thread that can pull from your existing legacy applications. And so think of it as a way to integrate trusted data because once it's in there, once it's on the chain, it can't be changed nor manipulated without knowing who has changed and manipulated it. And so effectively you have to put it in context of our earlier example of what we're talking about here with regards to a vaccine. And we'll look at North America, think about the natural challenge that we're seeing now where you have federal entities working with state entities working with even local entities. And we've gone across just the three layers that then get into maybe even communities. But then also what about the industries? Because there's a pharma industry producing it. And then there's a travel transportation industry moving at the 3PLs, third-party logistics providers. And then it can stop into a healthcare space where that might be the point of istration or it may even go to retail if it's in the drug-side like Walgreens or CVS. So now think of all those industries you've gone through and you have to federate that, and that's where you get this opportunity to say, "Can we come up with a lingua franca that allows us to share the data in a permissioned way with trust?" So that's practically the challenge that we're allowing with this capability to solve. Charles: When you say data, there was obviously a lot of different places, a lot of different entities, and a lot of different industries as you described having custody of this vaccine. What is the data that needs to be collected at each point? How is that federated? Jason: About the data that's important, let's start with that with regards to different types of data because I think to keep it practical for all of us, it's the outcome. And so let's start with the outcome, and I'm going to assume that maybe when it comes to your round of vaccines, that if you haven't received it you can say -- let's just assume you're going to receive the vaccine, Charles. Now, if you were to get this vaccine, what would you want to know about that vaccine? Let me flip it back to you. Charles: Well, so I would want to know one, that it was an authentic vaccine and not sugar water to start off with. I would want to know that it hadn't been opened in any way, that it was fresh and hadn't been tampered with, that the conditions that are conducive to transport had been maintained from the moment it was manufactured to the moment that it goes into my arm, to name three. Jason: Okay. So that's great. So the authenticity, integrity, if you will, that it is what it's supposed to be and then transport that that was sound as well. And then I'll add another that you'd also want to know that if in fact the person that was sitting next to you that you knew and two days later may have had an adverse reaction, you'd want to know is it because of that person, or is it because of the vaccine? Because I was sitting right next to them. So you'd also want to know, wait a second, do we have some remediation or some way to understand, connecting the data about the data? And so I'm adding yet another challenge here. I would assume you'd want to know that anyway, Charles. [Laughter] Jason: And so with that, when you say what data is important, I think we've just outlined that. So think of what we've just talked about, and I think we can boil it down to three simple areas, one is identity. And one would say, "Wait a second, where'd that come from? Why do we care if Charles is Charles?" And we'll get into that because that's how we circle back, but it's not just you, but it's identity of the entity of: Is that the vaccine that it's supposed to be? Did you get the one that's one dose or two-dose? And did it have to be at certain temperatures? All of that. So first the identity, so entity identification is one thing. Then there's also the provenance so there's provenance of what was its origin? In your words, Charles, is it sugar water, or is it the real vaccine? So you have that. And then the one that I throw in that sometimes causes people some confusion because we use the blockchain, it's in the context of blockchain, but I say tokenization and that's just digitization of a physical asset into a digital asset, so being able to track a physical entity in a digital way. So think of all the paperwork that's signed in and confirm how we currently broker authenticity is typically that way. So can we tokenize? So I always put the vowel in the middle and you get a little acronym (TIP) Tokenization Identity Provenance. And so those three things then say how we can make sure that this data through each one of those handshakes is maintained. So when the vaccine's produced, it gets a birth certificate in a sense, a digital birth certificate that's on the chain. I know when, where, how it was produced, and which type of vaccine it is. Charles: And we're talking every single dose gets the birth certificate? Jason: Well, ideally, we would love to have that. I think most as we've found in the current environment, as I said, we start with the current landscape, it's typically by lot number. You have a lot number in there; they're kept in lots. But even at that level, you're monitoring that no different than if you buy a container of strawberries. It's by that lot of strawberries and not necessarily each individual strawberry, not that we can't get there someday, but I think that's another conversation. Charles: [Laughs] Jason: You want to have that amount of data, so you give it a birth certificate and then as it goes through each handshake, and I'll stick with my same analogy, that's where you have an opportunity to put data on the chain but also data in context about that vaccine. So, yes, what does that start to pull in? You heard me mention internet of things (IoT). Well, we have many devices that are talking to each other right now. There's a great company called AeroSafe that has the refrigeration containers, and guess what? They have a GPS on that and a thermometer, and it's constantly communicating. So now you're collecting data about the transport. Your other concern, right? And so not just transport but the integrity: Was it at the right temperature? Where did it go? What were the different handshakes that it went through? Is it authentic? Did it really stay in the chain that we think it did or did it step out? Is this a black market? Has there been some fraudulent activity with this? Those are the things that you wanted to know. So keeping all of it in context of the end-, that's the data that is important, and that's where that data is then loaded into this common thread of safe, secure, trusted, and transparent data on the chain. That's what then at each step you would want to happen. And that's why, when I said it's a team sport, I truly meant that that means that each one of those places can simply go online and say, "I don't have to buy..." You'll start to hear some buzz words here. "I don't have to buy all this infrastructure." It can be a nice cloud infrastructure that now exists. And they say, "Why haven't we done this before?" Well, we didn't have, all of us, didn't have the capability. We didn't have a blockchain that was prevalent and proven and in a production way. We didn't have the speed and capability that we have across clouds in a hybrid cloud environment. We didn't have all of this capability even from a can I sign on to the chain on my cell phone and simply scan? Yes. All of these things have come together as we've seen in other industries to say the stars have aligned. And in an unfortunate time and an unfortunate event like we have now, we fortunately have technology that was ready to go on the shelf, ready to bring forward and solve the big problems that we're doing now. Taras: It's fascinating. I have so many questions. Charles: Yeah. [Chuckles] Taras: It's interesting because it's really time-sensitive. Because usually, you see adoption of fundamental enterprise or technologies like this that span the entire spectrum of supply chain and usually those things evolve and are maintained over a long time. But in the same way, we had to fast track the vaccine itself, you actually have to fast track this infrastructure because in some ways, now that you've described this, I do want to know did someone at some point open the case that had the vaccine? Was there an opportunity for that vaccine to be swapped out with something else? I would want to know that. And it makes me think when you think about transparency, you think about transparency being ubiquitous, it's transparent to everyone. And so how much transparency can you expect? What's a reasonable amount of transparency? How much information can you actually get and should be able to get about every step along the way? Jason: And those expectations are becoming more and more the standard. I'll step out of the life-threatening type. I should focus on the positive, life-saving opportunity we have with the vaccine and not the threatening part of the virus itself. And I'll step into fashion, something that we always talk about with regards to authenticity. However, there's a company that we're working with now called Covalent at C-O-V-A-L-E-N-T. It's a fashion brand whose products are made with AirCarbon, a biomaterial that's made by microorganisms. It's meltable; it can be used in alternative fiber to plastic and leather. So it's carbon negative, certified by Carbon Trust. So as we look at this, you start to say, "Wait a minute, can we trace the authenticity or at least the carbon production or offset of these products that are being made?" And on the other side, when we look at another -- and I'm just calling out real examples, so it's not just what-ifs. There's a consortium that we work with called the Responsible Sourcing Blockchain Network, think of automotive players such as Ford as they look to control some of the minerals and metals that go into their batteries such as cobalt, some very sensitive minerals. So when you look at that, you say: "Has that mineral been sourced properly? Did it come from the right mine? Is it really making it into the battery? And then also another buzzword, buzzword alert, in a circular economy, does it go from that battery back into the recycles so it can then come into another battery?" So I'm lightening up the topic to say there are many things in everyday life that if you jumped into a car anytime soon, many of us haven't left the house in weeks. [Laughs] But if you have and it's electric, you're thinking there's a battery there. If you're looking at the computer as we're doing this now, there's a battery in there. So we're experiencing all of these things that you start to say, "Wait a second, I really would like to see that." And we know that from work that we've done with the National Retail Federation, 72% of people would say, "I'll pay more to know that what I'm buying and consuming reflects my values, which currently sustainability is one of them." So how do you keep score? It's all based on data. And how do we trust the data? As you said, I think Taras you mentioned it's back to that big trust thing. How do you do that? So yes, vaccines have brought to the forefront what I've just mentioned has been working with regard to trust and transparency and product in food and things that we consume as well as those things that have driven big ideas such as sustainability into the forefront with credibility and trust. How do I make sure I'm keeping score on those things with some level of trust and transparency? And we're back to where we started with this thought of data that's on the chain. Charles: Yeah. I can think of a couple. I'm already thinking of a bunch of different applications both in the computational sphere but also in practical life. You could maybe do something like apply this to the chain of custody for crime scene evidence or something like that where you might have multiple different law enforcement entities and labs and stuff doing analysis and just transporting. But that kind of leads me to the next thing. So we're talking about huge amounts of transparency to your data and being able to say, "Okay, this data has unassailable integrity, and I can see everything about it and know everything about how it came from the moment it entered the world until it arrived in front of me. And I know this because I'm by no means a blockchain expert, but it's because it's distributed and anybody and everybody can it for themselves without having to rely on any central authority. So does that introduce any privacy issues? If everything must be transparent, everything must be out in the open, how do you hold back certain data or say this is for your eyes only? Jason: So, Charles, and I knew we'd get to this because a little foreshadowing, I said, "Hey, I wanted to come back to this thought of you being able to that you received that vaccine." And that starts the talk about wait a second, if you have received the vaccine first, you'd want to validate and it to anyone if there was a way or if there were, first, a reason for you to do that. Let's say that it could be you physically returning to a place of work that you needed to go or for entertainment; you wanted to go to a theater or for travel. As we've just seen, many of the airlines now are going to have that requirement to say, "I've either received the vaccine, or I'm COVID negative." Well, the only thing you want to be able to do Charles or Taras, the two of you, you just want to say positive or negative or positive meaning I’ve received the vaccine. I've positively done that or I'm COVID negative. They don't need to know your age. They don't need to know your home address. They don't need to know your social, so there is a sense of privacy. And the assumption that just because something is decentralized everyone can see it is not true. And that's part of the beauty and the way that a permissioned blockchain works. And I throw that name out, some people say public and private, just say permissioned. And what does that mean? That means that you own your data, Charles. You own that validated and verified health , which is (you can Google it) IBM Digital Health . That's what we've pulled together in this thought of identification, I said TIP. And this is the identification that does apply to you individually. It says that when you receive that vaccination on your smart device, you can then at the same point, validate and on the chain that yes, I have received the vaccination. And with that health , that data is yours. Now, in the permissioned chain, you say, "The only thing I'm going to give anyone permission to see is, have I received the vaccine?" That's it. Now you may offer other permissions, and we start to think about what you're opting into here. It works the same way on the chain. The beauty is that the chain is a lot more secure than other forms of data storage because of the level of encryption as well as we've said this thought of transparency if someone does try to access something, you can see it, or if they change it, you can see it. You by owning that could say, "With my employer, I do want to share my location because I work -- and I'm making things up here, Charles. But let's say you work in an office that has multiple floors or maybe even multiple buildings, and you want to make sure that you do share where you've been because there's a reduction in the number of people in the building at this time, and they want to make sure who's come in with whom. So then it becomes a permissioned chain with permissioned, trust, and transparency that gives the outcome, and that's why I said it's not about the technology, it's the outcome. It gives the outcome that we're all hoping to have. So that's where you and Taras could start saying, "Well, if you could do that, then you could do this." And so a lot of those ideas start bubbling up and you say, "Well, the technology is there. Why can't we move faster? I think let's go faster." And that becomes some of the things that we're seeing now with people getting what this thing is technically. They're away from the weeds. They're saying, "You don't have to explain it. I don't need to understand blockchain. Don't tell me what a hash is. I get what it does. When I flip the light switch, the lights come on. I don't know how the electricity got from the production facility to my light bulb, but I'm good. So now, how do I use it to do other things?" That's what we're seeing. Taras: Let's say we take an airline, for example, and they want to know about my status if I've taken the test or had the vaccine. Is there a trusted app? Maybe the Canadian government would provide a trusted app that I would install on my device that would show that information. How do I access this permissioned blockchain and then allow others around me in the real world to trust that information? Charles: So a practical application with our digital health can be ed. The actual app can be ed on the Apple store. So you can an app and then that app accesses the chain. So we'll call it the blockchain; we'll get rid of the buzzword, and we'll just say the network. And so it ties into that secure network. And then in your example, the airline would also be on the network, and now there's a common linkage, there's Taras, there's the airline and you're both able to access the same chain. And now you have -- I'll borrow from Charles your example of the crime evidence. I liked your chain of custody, but I'll change it. It is kind of like a dead joke version, the chain of trustedy if you will. [Laughter] Charles: And so you take that chain of trustedy, and you say, "Okay, now the airline has it. I'm Charles, and I have permissioned to see just what I want them to see." It's that simple. Is there hardware involved? No, it's no different than you g up for your next ride-share app that you're going to . You're on the network when you sign up. The technologies behind all this, and that's the goodness, not only is it behind it, it's also behind it with the airline. And I say that because many of the airlines are already starting to put together their own applications, the technology behind it. And that's where when we talk about IBM, that's where we sit. It's not something that's oh, everyone's going to use the IBM solution. No, we just want it to have the IBM capability behind the trust and capability that we've invested in and we're making available to many at this point so that they can use what they already have. They don't have to change their branding; they don't have to change their UI UX because that's very valuable. They have invested millions and sometimes hundreds of millions into the experience, and what we want to make sure is happening in that experience is that we're getting the right outcomes. In this case, it's that everyone is safe and secure and that they're able to distribute that safety and security with a sense of equity so that it does help us all move forward together. So that's the thought of the network and why you hear me say it's a team sport. I'm giving some meaning to that, and that's how it comes together. Charles: I'm wondering where the vendor tooling like the stuff that you all offer fits into the ecosystem because if I understand it correctly, it's not something like Amazon Web Services where each one of those handshakes within would say, "Post that data to some centralized service that then everybody could agree on. Everyone can access the entire thing any time. No one really owns the chain itself. They can it from anywhere. So what kind of tools then do you provide if there's no opportunity for a centralized service? Is it in of libraries? Is it in of, I don't know, application components that you can use and utilities? Jason: So the quick answer is yes, those libraries can exist, and there is a centralized management capability that you would want in the blockchain. And when we think of examples that I've given, if we're talking about a single entity, it's easy to say that when we're working with a single client, their blockchain is their network. And often either they will want to manage it or they say, "IBM, we'd like for you to manage it," which is fine or collaboratively -- and sometimes we've delivered projects to the federal government and it's been in collaboration with Walmart and KPMG and a pilot that we did there. So now you have maybe IBM and KPMG working together. So I wouldn't just call out one name because I will contradict my team sport. But you have this thought of someone managing the network. And so when you often have a consortium type of environment where there's many entities coming together, there is a central managing body. I've had some wonderful conversations with Brian Behlendorf and some of the things that they're doing at Linux Foundation, where they're saying, "Look, let's look at some of these centralized management capabilities," things that they do day-to-day. They would say, "Yes, there is a centralized management capability that allows for that consolidation of tooling so that it can be developed, but it's also open." And that's why I throw out Linux. And we also think of Hyperledger as one example where you say, "Look, there is an open source community that allows these tools to be developed as they're needed in the market. So that's what's good, the start of a centralized trust with a very open environment with many, many capabilities that are developed based on market need. I think that's some of the uniqueness here, when you start saying, "Wait a second, this really could and should move quickly," as we've seen. And when I say as we've seen, what have we seen? This thought of blockchain is something that's experimental. And I love when people say, "Oh wait, people are just doing pilots or it's not real." And I say, "Wait a second, it's in production." In many industries around the world, people are transacting using this, and it's been 10 years if we call it that and the internet took 30 years. So in one-third of the time, we've gone from concept all the way to production-ready. And how can we make this work in the most effective way with vaccines? So that's the reality of what we have and the opportunity in front of us that should make us all scratch our heads and say, "Look, how do we do this together and make it move faster? Taras: I imagine in this team sport you have all these different companies that are participating in this whole process. What does it look like -- For a company of any size they're going to, at some point, want to say, "We want to use this. We want to be part of this network." How does this actually work? Is there an SDK or something that they would ? How do they actually become part of this system? Charles: So we'll use health as an example, if they want to become part of that, yes there's an SDK that they can and that makes it available to them so that in a sense, you get this thought of oh, it's a white-label capability that they can incorporate into what they're already doing. And that's why I said it's that painless, and it's complementary to what they're already trying to do. It gives them that layer of trust and transparency. If you're looking at a single entity, and I'll stick with the same example of Home Depot, where they have worked on their own capabilities in blockchain. And you say, "Wait, what is it that they're going to do?" Well, they take their current environment and in that current environment, they then stand up a blockchain network that works with the data that they want to put on chain. And so it enables them to move quicker, faster, easier within their ecosystem. But quickly, what you can see happen is you say, "Okay, if I'm going to work in my ecosystem, but I'm also going to use it to work with my suppliers," quickly, you can start to say, "Wait a second, if those suppliers are going to be on that blockchain network, perhaps that blockchain network would expand to the suppliers of those suppliers assuming that some of them have even a second or tertiary tier suppliers to them." And then you start to understand this thought of how the value of a network is increased exponentially by the number of players in it. Is it a necessity to start with all of the players? I say, "No." It starts the other way around is you start with a minimal viable ecosystem. In our case, you say, "Look, let's start with a manufacturer of a vaccine. Let's start with a government entity. Let's start with an istration point and someone who may want to make sure that they can validate and that they have received the vaccine." Then you have a minimal viable ecosystem. And that's what it starts to look like where you say, "Look, I can now work together." So yes, for an individual company, that SDK is there. You put and pick it up. You now use that to tie that capability into what you already have, and you're ready to move fast forward. And then you'd want that to iterate in that if there are capabilities or needs, and this becomes the value of the network and an open network where you say, "Wait, I want to be able to make sure that on my AWS cloud, it does this. Wait a second, I'm not on AWS; I'm on Azure. Oh, well, wait a second. I'm on IBM's cloud." Forget the flavor. And you start to give real meaning to brand names like OpenShift when we talk about it's open. So now you're multi-cloud or hybrid if I'm going to be on my premises as well as in the cloud. So these buzzwords and marketing start to get real meaning when you put it in context of solving the big problems we're talking about here. Taras: A lot of times when we talk about technology, the thing that we want to focus on is what it unlocks, but there's always a caveat. And it's often in exploring the caveats is where you start to really understand the problem, and you can't really until you ask the question. So I'm curious what is the caveat? What's the thing that people need to be worried about or thinking about, or what are the limitations of technology that we have today that you see as being an obstacle that companies have to overcome? Charles: This is a great question and this thought of what's the catch? Because I like to call myself an optimistic pessimist at times. I go wait, there's got to be a catch here somewhere. And what we know as a company, and I'm about to make another IBM statement, but it's a true statement. Part of our values and belief is that we believe in innovation that matters for our clients and for the world. And so yes, it is a Tech for Good type of thought and whether it was putting the backbone for the modern U.S. social security istration systems to make sure that we could do that during the great depression or putting people on the moon and bringing them back safely. We signed up for big, big things. And so now we're in the middle of this pandemic, and we're saying, "Look, we are g up for that." But what's the catch? It's not the technology that is the catch, Taurus. What we find is the challenge in the team sport is the business process. It's the design of the organizations and, back to my original example, the handshakes in that supply chain, and I would call it a value chain though that's a buzzword. But you start to think about different layers when you say, Oh, well, if it's just moving something from point A to point B, it can't be that complex." But what about how someone pays for something, how the regulation is in this state versus that state or this country versus that country, the paperwork that has to be signed here, but not over there and tracking it. And you start saying, "Wait a second, we have technology that can solve it." Yes. But technology is a lot easier than the business process. So what we've seen is the catch in trying to say, "Wait, we have to actually change the way we do business in order to make this work," and I'd say that's the catch. And a great, great company example with the chain and I'll call them out because Walmart bought into this early on, bought into the idea of blockchain is a game-changer. And they teamed with us, helped us launch something years ago that we called Food Trust. And they said, "Look, foodborne illnesses kill way too many people year in, year out so let's address that." Not only are there deaths involved, but there's also wasted food. So if there is food, food is assumed guilty until proven innocent, so if we say there's a leafy green vegetable scare, then we're going to throw it all away. Well, Walmart was world-class -- in seven days if something was suspected -- So let's say that they had a scare in leafy greens, in seven days, they could figure out where that came from. But for seven days, they'd have to throw away all of that food. Some of it could be good, so that's wasted food. Why don't we put this on blockchain? And we could see each one of those handshakes with trust and transparency. They did it with mangoes in their first effort. And that 7 days went down to 2.2 seconds. So think about that. What they used to have the luxury of finding in 7 days, was now 2.2 seconds straight across the chain; we know where it came from. Well, that's technology. But think about the challenge that they had and now they are moving very quickly. You start to say, "Uh-oh now I have to change my business to move that quickly," that's the gotcha. That's the catch is that you have to start saying, "Okay, now how do we make sure that we design?" And this has become a real differentiator in the work that we do. And it's not as I guess glamorous as some of us propeller-heads technology-fascinated people, nerds, whatever you want to call us, us people, but it's in the softer part of design. And we found this thought of business design has now come to the forefront. And that's where I use the buzzword, you know, people like so many MV things like minimum viable product. How do you decide what is that minimal viable ecosystem, and how do you decide that you're focused on the outcome that's going to drive the right uptake? Because as you can imagine, all of it sounds good until you actually have to change. And the best way to cause change is to show a positive outcome to make sure that Charles and Taras are happy, confident, and trust that when they receive that vaccine, that they're receiving it, they're telling everyone else. And they're now a proponent for getting things out quicker, better, faster, and everyone in that chain of trustedy, to use my bad joke again, everyone else has confidence. So think how quickly that then gets the speed of business moving along even more quickly than it was and more effectively I should say than it was before. Taras: It's fascinating because it makes me think of something that I've observed with a lot of our clients is that because we focus on developer experience, we focus on the way software organizations work, you get to a point where you have an experience -- your system operates in such a way that it gives you new experiences. And it makes me wonder when you have this kind of change when you go from taking 7 days and throwing out a whole bunch of food to going 2.2 seconds to find out where the problem might be, what does that unlock for the kind of business that you can run? What is the experience of that business that it creates? I'm really curious to find out what that produces. Charles: I'm with you, Taras. I would say this thought of everything being controlled, I think there's more and more control that people are now realizing exists at the point of the end-. I'm just underlining something you're saying there because I think it's a key point here. It's because that end-, in this case, the consumer, all of us, I mentioned the demand is there. We are starting to expect that type of experience where I want to know right now -- for example, I'm a coffee drinker, and I want to know if it's stamped organic or if it says that it's come from this location. I want to make sure it was sourced ethically and that it got to me in the right way. And so as an example, 1850 Coffee -- I'll do a real example here with Smucker's brand, if you're a coffee drinker. We worked with them to make sure that through something we ed with them called Farmer Connect that yes, you can see where that coffee came from. That experience, that level of confidence, it's not a vaccine, it's just your everyday coffee, and you want to make sure that yeah, I feel confident. And there was even an app that was added to that called Tip Your Farmer where you can say, "Yeah, the barista's great. I'll..." And then because it's more virtual now, maybe you can't actually drop that physical tip in there, but what about tipping the farmer? Because we have found, and we know that in many cases, the actual producer doesn't see as much of a cut of the profits as some of us would hope. And so how do we put it back into their pocket through this thought of transparency? And then what does that mean? Does that mean that Taras if you're a coffee drinker or Charles you go, "Well, okay if I'm already paying now..." when you start to look at it, you go "I just paid $5 for a cup of coffee. Well, okay. I'll pay $5.50 or $6, and I want it to tip the original person who helped get that good coffee bean into my cup. So those are the things you start to think about. As you said, you sit back and scratch your head and go, this starts to change the model now. It pulls things to the forefront. It puts you in touch as the end- with the originator, the producer. You can now shake that producer's hand, and you don't have to wait for a series of sequential handshakes. So think of that model that used to be sequential you're crashing that now, and you're putting you as a consumer in touch with the producer. That starts to change many business models because that can also change relationships. And that's what's exciting about this thought of opening up new markets and new opportunity and also the thought of increasing inclusiveness and equity around many markets where we start to say, "Oh, that's where the value is actually coming from. Let's assign the payment or that capability that we have to reward where the value is created." And so that leaves us as consumers to decide and this enables it to happen with trust. Taras: So it sounds like what I'm hearing is it actually makes sustainability possible. When you can trace to the origin in a second, if that's available as something that anyone can apply this idea to their circumstances, it actually creates a possibility where a farmer could be sustainable because there is actually a connection there. They're not commoditized in the same way that so much of our products are today. Charles: It makes sustainability sustainable in our opinion, to throw another soundbite out there because it's true. Now, if you can keep score, as a consumer, you're going, wait a second, it's not just because it's stamped on the product. I can actually with trust trace it back to where it came from and the origins. And yes, it is sustainable. It is organic, It was sourced property, and oh, by the way, maybe I can put a bit of coin in the pocket of someone who helped produce, design, or deliver it. Taras: You can actually verifiably confirm that this is actually -- it creates a connection between someone who's completely disconnected from the farmer to the person who's consuming the product, and there's actually a chain of connection that can be trusted and maintained. Charles: Exactly. And there's also this thought for the supply chain folks that would be listening in. There's a lot of other things that start coming out in these conversations because with that visibility, you start to see the actual roles in this old thought of a supply chain start to shift where -- and we'll keep it in context of the pandemic think of PPE and a lot of it being shipped -- First, let's talk about different roles. You had breweries that said, "Wait a second. I'm not selling a lot of beer out the tap because of the pandemic. Oh, wait a second, in the same place, I can produce hand sanitizer. And yes, it's going to smell a little different than everything else is, if you've had any of that, maybe in any establishment you go, "What's different here?" Well, they were able to shift quickly and go from producing beer to hand sanitizer. But wait a second, that ecosystem is different. How do I know it's really coming from them? How do I know it's up to standard? If I'm now a supplier of hand sanitizer and not beer, how do I quickly find the buyers in the same way? Is there a network I can to make sure that we can trust each other? That's the first thing. I've gone from being a beer producer to hand sanitizer. So we started changing the thought of hybrid. Now I have a hybrid business because it's doing both of these things, and now I'm in a different type of supply chain. So now I'm going to keep using that word — and now I'm in a hybrid supply chain. And I ship my hand sanitizer to let's say a healthcare provider that was running short so they buy from me and everyone else. And all of a sudden this healthcare provider has more hand sanitizer than they've ever had. And they're going, wait a minute, now I have way too much. What do I do with it now? They've gone from a buyer to a supplier. How can they quickly pivot and step back into a trusted network and redistribute? And so I just thought of redistribution which is where I'm going, and it's because you've seen it with the vaccine as well where they say, "Wait a second, we have extra doses because we've already done our priority one; now we need to do priority 1B. Wait a minute, we might have too many because someone else needs it. Well, again, what holds that up is the transparency. People think about trust -- in the provenance piece when we were talking about TIP, but it's not just provenance; it's how do I identify where you have supply and where it's needed? So how do you match have and need? And so this again is a benefit of transparency, permissioned transparency. And you say, "Okay, I'm part of this network. I know I have this product or service, and now I can go from a buyer to a supplier with trust." And you're going to get what you ask for, and you know that I am trusted because I'm on this network, blah, blah, blah. So you get it. You get where this becomes a lot more dynamic in this system. Charles: Yeah, absolutely. This is really fascinating. Jason, thank you very much. I'm really glad that we had a chance to have this conversation, and thank you very much for coming on and chatting with us about this. It's very timely, and I'm really glad to have a chance to talk about it. Jason: Thank you. Thanks for the time. I really enjoyed it. It's one of the few times that I've had such an enjoyable conversation with two people on a podcast, typically it's one person. So you all made this very enjoyable. So anytime you want to jump on and the two of you want to have a conversation, I'm game. Thanks for the conversation and stay safe, stay happy, and we'll talk soon. Charles: All right. Thank you everybody for ing in with this conversation. That was absolutely fantastic. And as we heard, it's very timely. It's often said that necessity is the mother of invention. Computers, I believe got their start during World War II as part of early cryptographic efforts, and the militarization of computers happened during the Space Race so that we could take these big vacuum tube things and put them on to things that we'd shoot into space. And there are technologies that are emergent during times of great change, and it sounds like this might definitely be one of them, so keep it on your radar, and we will talk to you next time. interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at @frontside.io. Thanks, and see you next time. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

SPECIAL GUEST:\r\n\n\nMarcel Cutts | @marcelcutts: Founder of Asgard Tech, ReasonML London Organizer\r\n\nShane Wilson | @wilsonashane: Founder of Peyk Inc.\r\n\n\nSHOW NOTES\r\n\n00:38 - Reason\r\n\n\nTwitter: @reasonml\r\n\nDiscord\r\n\n\n03:25 - BuckleScript\r\n\n\nOCaml\r\n\n\n06:01 - Reason + BuckleScript\r\n\n\nTypeScript\r\n\n\n16:07 - Reason: Interoperation & Adoption\r\n\n\nClojureScript\r\n\nToolchain\r\n\nSyntax\r\n\n\n30:00 - Operating at the Compiler Level vs the Run-Time Level\r\n\n\nppx (pre-processor extension)\r\n\n\n34:29 - Last thoughts on, and why use Reason?\r\n\n\nFlow\r\n\n\n44:43 - repkgs\r\n\n\nknit\r\n\nLerna\r\n\n\nTRANSCRIPT:\r\n\nCHARLES: Hello and welcome to The Frontside podcast, a place where we talk er interfaces and everything that you need to know to build them right. My name is Charles Lowell, a software developer at Frontside. And today, here to talk with me about many interesting topics. One of them being ReasonML, we have Shane Wilson, the Founder of Peyk Inc, and Marcel Cutts, Founder of Asgard Enterprises. Welcome, gentlemen.\r\n\nMARCEL: Hello. Hello.\r\n\nSHANE: Thanks for having us.\r\n\nCHARLES: Yeah. So, I guess we should start out just talking about why ReasonML. I understand that both of you guys have some experience with it. New languages are all the rage. So, what drew you to it and have you found it?\r\n\nMARCEL: That\'s a great question. I\'m afraid my history is going to be spotted and will really demonstrate how much pubs are an integral part of the UK tech scene. I got into ReasonML maybe four years ago at this point when I found it on some hacker news post. And I asked on Twitter, "Hey, is anyone running a ReasonML meetup in London?" And I also, maybe around point one. And then by point for a point five, someone said, "Oh, no, wait. I think there is one." "Oh, does that mean you\'re starting one?" And then by point six, I thought this was a good idea. So, I began to run the ReasonML London Meetup, which end up being one of the bigger ones in the world. And along the way, it [inaudible] to learn some ReasonML and why it exists, which ended up being quite fascinating. And I did quite a few talks around it around the world. And it has stuck with me since. It\'s a very fascinating intersection of many kinds of approaches to programming as well as technological applications with a lot of history as well stemming from its origins, OCaml. So, it\'s a bit of a unique beast and it has kept me interested since.\r\n\nSHANE: I came about it because I\'ve been doing JavaScript for quite a long time. But I\'ve always liked compile to JavaScript languages. Maybe six years ago or longer, I started looking into Clojure and I got pretty heavy into Clojure and on ClojureScript. And I was always looking for a reason to write ClojureScript as a compiler to JavaScript language, but it never took off. Do I kind of hope that it did? A lot of people use it, but it never really took off. And then when I was recently looking into how do I write JavaScript Node applications but without having to write JavaScript or setting up all the toolchains, I came across ReasonML along with BuckleScript. And I was really pushing how you could write a language with a lot more features and JavaScript comes built in with but compiles down to JavaScript that\'s very readable. That was a big selling point. So, when I started looking into it, that your output was basically JavaScript. It wasn\'t like a computer had written it. It looks often like a person had written the JavaScript. I thought that was very impressive. And it made it a lot easier for me to think that I could sneak it in to a project somewhere. Or I could just make my own little ReasonML toolchain, output a file, and then just not tell anybody and no one would even know.\r\n\nMARCEL: Some say that I have in fact made several branches for many client projects which secretly have been produced from BuckleScript output and then slightly tied it and no one\'s ever found out.\r\n\nCHARLES: So now, what is BuckleScript for the uninitiated?\r\n\nMARCEL: We can go from the top down. ReasonML is a meta language as well as a toolchain and some documentation utilities for a language called OCaml. OCaml is a 20 plus-year old [inaudible] systems language, the kind of language you expect someone of a long gray beard to be writing.\r\n\nCHARLES: It\'s one of the spiritual ancestors of Haskell, isn\'t it? Or no?\r\n\nMARCEL: That\'s a great question. I\'m sure what the familial tree between those two would be. I think it\'s an ML language, which means that a lot of that will bleed into the Haskell way of thinking.\r\n\nCHARLES: Okay.\r\n\nMARCEL: About a few years ago, I think about four or five years ago, the chap who invented React left the spotlight and retreated into the shadows and left other people to be the spokesperson for that framework. everyone that Jordan Walke is the person who made React, not Dan Abramov, even though he is the star of the show for many things React-related. In his spare time, what he did is work on this thing called ReasonML, which is a new syntax and toolchain for OCaml, which makes writing OCaml look a lot like writing JavaScript. And this was a way of trying to get people to write in a more functional and type safe way.\r\n\nAt this point, you may be saying, "Great, you can write OCaml like JavaScript, but it\'s still OCaml." The missing piece is something called BuckleScript, which was developed by an expensive keyboard company called Bloomberg. And they had this layout because OCaml itself is a very pluggable and modular compilation system. They plugged a single BuckleScript [inaudible] compiling to bytecode or [inaudible] or you could compile directly to JavaScript. This runs a lot of the internals of Bloomberg that have a surprisingly large amount of JavaScript and Node in there, and they wanted to be 100% sure about what they\'re getting because with OCaml, you do get and ReasonML by extension, you get 100% type safety all the time. Unless you have big exception flags, unlike something like TypesSript, for example, where you sort of [inaudible] but you don\'t quite have the same certainty. And also, you don\'t have a lot of these syntactical avenues that a language base and types will give you, things like variance or being able to switch on type definitions.\r\n\nCHARLES: If I\'m understanding then correctly, ReasonML, the syntax actually is 100%, it is OCaml. It compiles down to BuckleScript. Reason is this toolchain for OCaml that outputs this dialect of JavaScript called BuckleScript?\r\n\nMARCEL: Yeah. The combination of BuckleScript and reason together give you the syntax and the output. Have there been discussions about trying to make this clearer and saying what each component actually does? Yes. Have they been very [inaudible]? Also, yes. [Inaudible] reason to syntax and then BuckleScript compilation that leads to the readable JavaScript that Shane mentioned earlier. And this has had some interesting pickup and it was quite a contender in the original for People 4 Projects battling for the type space early on, which was Flow, TypeScript, Elm, and ReasonML.\r\n\nTypeScipt has been around for nine years at this point. I think it came out in 2011. It\'s been around for much longer than people because it just took a long time, a lot of [inaudible] backing to get to the front spot. And it is continuing to slowly develop to this day. It hasn\'t quite got the same acceleration it did at the beginning, as Shane mentioned. But it still has a ton of interesting applications and it is used in the law of production applications throughout the world. For example, all of Facebook Messenger is written in ReasonML for performance and certainty reasons.\r\n\nCHARLES: Oh, okay. Interesting. I started out using type systems like Java and C++ and C, and I very much considered myself kind of a refugee from these brutal typing regimes. I fled to the land of JavaScript and of Ruby, and even places like Perl and Python kind of to escape types. I\'ve been slowly getting back into it with things like TypeScript. So, what are the surprising aspects of the type safety that are really, really done well? Because I can definitely say having gotten a taste of TypeScript in the last few years, I can see there are ways to do a TypeScript or a typing system that really feel lightweight and feel like they\'re helping you and feel like a tailwind rather than a slog. And so, what are those experiences in Reason that really stand out?\r\n\nSHANE: For me, I\'ve never used in ML language seriously before other than like some toy projects. So, for me, most of my experience with types has either been Java or TypeScript. And ReasonML, I\'d much prefer that type system. I\'m not going to get too technical because quite frankly, I don\'t really know the technical things to say here. But when I work even with TypeScript, I feel like I\'m adding types because TypeScript wants me to and needs them. Whereas in ReasonML, first of all, you don\'t have to add a lot of types. You almost never have to actually type the code that you\'re writing, maybe type modules or create types. But the actual code that you\'re writing with all the flow and the logic is almost never a type in there. ReasonML and the OCaml figured that out for you, which I think is amazing and it doesn\'t get in my way anymore. And when I write ReasonML types, I feel like I\'m doing that to help me, not to help the language. I want to better define the problem that I\'m solving and help create barriers for how to solve that, put data in a place that it should be, help with my logic flow, and I want to make those changes and the language is helping me make those changes. Whereas most of my time and experience with types, I know what I want to do and the types aren\'t happy, and so I have to change my work to manage the types.\r\n\nCHARLES: Right. You\'re having to kind of spoon feed the compiler and throw spaghetti at the wall and see which come kind of weird combination of types and generics fit the problem and make the compiler happy.\r\n\nMARCEL: Yeah. Java really has done an absolutely excellent job at ruining types for everyone for quite a while. The worst PR campaign you can imagine. But it\'s important to know that there are more than one kind of type system. There are two big schools. One being nominal typing which is the kind of Java C# thing that has burned into people\'s memories and gives them sleepless nightmares, where you try to compare the names of things and the way the things I described and to figure out if they have equivalent to that sort of thing. But the thing that things like Haskell, Elm, and ReasonML, OCaml too, is called structural typing where it\'s based entirely on the structure of your data, which means you can do much more interesting things and it can get out of the way a lot more. And it has something called the Hindley-Milner type system, I want to say, to try and infer the types to the best of its ability with as much certainty as possible from your code. And it is mathematically provable that the Hindley-Milner type system is the best type inference engine that currently exists. So, you do end up typing precious little when you\'re actually writing ReasonML. But you still have complete type certainty, which is very helpful, especially if you compare doing this to something like TypeScript, where I think it\'s not unreasonable to say that folks like to type all the things that are easy to type, for example, dysfunction [inaudible] string. But often it is much more difficult to type things like the response from an API that could be in all kinds of structures or the state of the application, which could be in X many permutations, which is much more difficult to describe with the typing tools that you get in that language set.\r\n\nCHARLES: Is there any particular example of something recently where we can compare, say, an implementation in TypeScript to something in Reason? I just want to make sure that we have like some way to focus. Because I believe you, but it\'s just -- and I definitely have this experience where I feel like I\'m trying to satisfy TypeScript and the errors are cryptic and, yeah. And so, I\'m trying to get a glimpse into what\'s a particular use case that I would find frustrating with TypeScript that Reason would swoop in and solve all the things.\r\n\nSHANE: I don\'t know if I can think off-hand of something that obvious, although there might be. But I know one big feature that ReasonML has is like the switches. You can switch on a type and I guess you can have dynamic typing. So, instead of having to create a whole interface, you can just say -- basically it\'s it\'s like here\'s a string, but it\'s not a string. It\'s like I just have a label and it\'s now a type. And you can do type checking on these dynamic types. So, I\'ve used that recently to manage errors in a product I was running where any random error I want to come up with, I could give it a label. Oh, and so types can also get past data. So, my dynamic error label like file read error can actually take in values like a file pointer, like a link to a file. And then when I\'m accessing that error later, I can pull out all this data. So, you don\'t have to create these complicated structures and try and them around. You can just create a type and then it will have all the data in with it. And then you can do a switch check on that type and destructure all the things that you need.\r\n\nIt\'s kind of hard to explain, especially when I don\'t know the details of what I\'m saying. But the best I can say is it\'s just the feeling you get when you\'re programing, you don\'t get the frustration. When I work with the types, I\'m always glad that the types have helped me. I\'m very rarely frustrated that the types are getting in my way.\r\n\nCHARLES: It sounds like the types are just so deeply integrated into the way that everything is done. It\'s not so much annotating stuff to make sure that the ends of your apparati line up with each other so much as deeply integrated with how you actually express the flow of the program. Is that a fair way to say it?\r\n\nMARCEL: Yeah.\r\n\nSHANE: Yeah, it sounds good.\r\n\nMARCEL: I think I can think of a decent example that demonstrates a difference that types can make. In TypeScript and JavaScript, its [inaudible], of course. You often end up with these situations that folks call being stringly typed. So, a component, if you\'re doing a React, can take a number of props and maybe the component has different states, have a button that can be a default one or a call to action button or a more subdued button, if it\'s an option that you don\'t want to be invisible. So, there are many ways of trying to get it done. But one thing that often happens is you parse on a string of the correct name that you wanted this thing to do. And then maybe inside a component, you have a switch and then you have all the different kinds of strings that you could expect together. But you could get any kind of string, really. Nothing prevents you. So, with the switch and variance system that Shane was mentioning, you automatically get [inaudible] pattern matching. So, you can say these are these three modes and here these types of that mode. If you then start to do a pattern match, which is, you know, think of it as a switch, but 11,000 times stronger, it will tell you, "Hey, no. You forgot one because there\'s this other state that you didn\'t mention." And also, what you\'re going to do if you don\'t hit any of this, but you can\'t because, of course, the compiler will catch it before you get there. This means you don\'t end up getting into strange situations because you\'re parsing strings on the application and you\'re hoping that you ed all the [inaudible] and of every case.\r\n\nAnd in addition to this, one thing that you find very pleasing on Reason, it brings you joy to writing this. It\'s a very, very, very fast loop. We\'re going to talk all about TypeScript because that\'s kind of a merge as the frontrunner of adding vague types to the JavaScript universe. But TypeScript doesn\'t check the whole application every time you make a change. BuckleScript does and it can do these compiles in subsecond for very, very, very large projects, which means that you always have confidence in your application as a whole as you continue to type and you don\'t have to like wait or go back or figure out how something doesn\'t make sense somewhere else. And this really short loop combined with the fact you don\'t really need to have all these things, like I said, [inaudible] type system, ESLint, or Prettier that hold your hand means that you have a very satisfying and streamlined experience as you go along compared to the, I think, expected amount of waiting you might do in a regular TypeScript project.\r\n\nCHARLES: Yeah, that begs an interesting question because TypeScript does seem to evolved as the front runner of a typed JavaScript development system. And I think that\'s owing in no small part to the fact that it\'s very easy to adopt incrementally. You can just switch TypeScript. You can just rename all your JavaScript files to TypeScript and you\'re off to the races. And you have kind of a through compilation. There are compilers out there like the Babel compiler that just strip the type annotations and you\'re good to go, and then you can very easily ramp up your usage of the types. So, I would say, for example, in my own experience with TypeScript, we started out with very loose adherence to the norms of the TypeScript community. And then now most of the code that we write uses the strictest possible settings because those tend to be the most helpful in of development, catching errors earlier, things like that. But it\'s definitely, if you start that way, it is kind of against the norms of normal JavaScript. And so, there is this very clear evolution or very easy way to adopt. So, I guess my question is for Reason, what\'s the story for inter operation and what\'s the story for adoption? So, I\'ve got an existing JavaScript code base, how do I sneakily introduce Reason?\r\n\nSHANE: One way you can do it is because, as Marcel was saying, the compilation is so fast, you can just add it as another step. So, you can set up your BuckleScript compiler and write some ReasonML code and have it compile down into JavaScript or even TypeScript, because there\'s a plugin called gentype, which will just generate types for you and you can then through TypeScript types so you can get perfect TypeScript types if you\'re using a library. And by the time you finish writing the ReasonML code, it compiles very fast and then you can just have whatever TypeScript watcher watching all the TypeScript files and it will see, "Oh, that TypeScript file is changed," and it will recompile and then you\'ll get whatever your program is.\r\n\nSo, it\'s kind of easy to add it as a sidestep. I think the BuckleScript people have said they don\'t really try and integrate it into the other tools as like a first class because they want control over so much, because they do so much. And by having all that control, they can build it that quickly. But if you wanted to, you could just add a little folder and then put all your ML code there and just have it compile. And then your other watchers just see the compiled files and it works.\r\n\nThat\'s actually what I\'ve done where I wanted my interface to be TypeScript, but I wanted my actual code to be ReasonML. And so, I have one TypeScript file, which is like the index entry point of the program. And then that TypeScript file reads my generated TypeScript files that don\'t really exist before I run anything. And then I would just have the BuckleScript compiler watching, it builds on my TypeScript files and then I will run like Yarn or whatever, and that goes into my TypeScript entry point, which then finds all those that build TypeScript files and it just runs. It doesn\'t even know the difference. So, it\'s not the worst setup, if you want it to.\r\n\nCHARLES: So, then you\'re calling your Reason code from JavaScript. What about the other way around, like if you want to use existing NPM packages inside Reason, especially ones that really haven\'t been developed for Reason development at all? I mean, this kind of a pain point with TypeScript, even though there\'s the definitely typed repository where you get 98% of the typings correct, but still, there\'s a lot of quirks and [inaudible] and stuff doesn\'t line up. Fundamentally, there\'s JavaScript APIs that don\'t really work with a strongly type system. They\'re hard to type. What\'s the story, I guess, for using existing JavaScript code?\r\n\nMARCEL: You do have to write some level of interop to give it some types. But the nice thing is that there are, as you mentioned, some things are just difficult to type because a few of the more dynamic things that JavaScript can do is difficult to express in TypeScript. But the type system for Reason is a little more mature with full features on it. It lets you do both imperative and functional styles of code, lets you do mutations if you want to do performance. So, you could do a much better job of describing what\'s happening on the other side.\r\n\nCHARLES: Okay.\r\n\nMARCEL: There are also a few tools to help you take existing TypeScript definitions and work backwards. But if you\'re just picking up a random JavaScript library off the Internet that hasn\'t no way been typed, you would have the same pains as you would for TypeScript. But much like TypeScript, there are already libraries. I think it\'s called Redex is the website one goes to when they want to see which of their packages already have ReasonML definitions.\r\n\nCHARLES: What would you do? So, one of the things before we even started using TypeScript, a lot of our libraries were written in JavaScript and we had a lot of requests for, "Hey, let\'s make the typings." And I always found putting stuff on, definitely typed wasn\'t optimal because if you released a new version of a library that had breaking API changes, the types were decoupled from that. And so really, the interface definitions being decoupled from the actual source code was a problem. So, even before we started doing stuff in TypeScript, we started shipping declaration files with the actual code. So, taking time to make the TypeScript declarations with the JavaScript code, even though the projects were written in JavaScript, it just made it more adoptable by people who were using TypeScript. So, is there an equivalent in Reason? Can I just say, "I\'ve got this thing that I\'ve got in JavaScript or TypeScript or whatever, but I\'m going to write a reason API for it." Can you ship that type of adapter with your library and make it available in package JSON, so that anybody using Reason can just, with no fuss, no muss, use your code?\r\n\nSHANE: In that case, a lot of times, you just publish at npm and there\'s somewhat of a standard where you prefix your package either with bs-, like BuckleScript. I think they\'re trying to move that to re for Reason just to try and reduce the confusion of what is BuckleScript, what is Reason. But the way you do that is you ship your ReasonML code, and then you add that package to your BuckleScript dependency list. And so, BuckleScript can now go into that package, compile the ReasonML code, and then you can use like the ReasonML types in your package. So, that\'s how you would on those types. If you and type a JavaScript language and you want someone else to reuse your types, you can just publish it as re- package name. And then you can use that in a ReasonML project.\r\n\nMARCEL: One thing to also is that we are very used to having a ton of dependencies inside the JavaScript universe, which is why the idea of having to write bindings for a lot of packages seems incredibly daunting. But what\'s important to is that ReasonML is a much more full-featured language and has a bunch of libraries and stuff pre-built. So often, when we want to do something with functional and we would take a Lodash as a library and then think, "Wow, it\'s always easy to type together." The outcome of having a mature language that you\'re basing this meta language on means that you have a lot of these things you\'d normally get dependencies for is completely built-in already. And you end up having a lot fewer dependencies that you need to worry about.\r\n\nCHARLES: Okay. Meaning, for example?\r\n\nSHANE: I guess you wouldn\'t need things like Lodash as much.\r\n\nCHARLES: Okay. So just all your things like transforming data structures and stuff like that?\r\n\nSHANE: Yeah, a lot of the data structure manipulation functions are already built in.\r\n\nCHARLES: But it\'s still all the stuff like file systems and network you have to receive from the environment. For example, I assume you can run Reason in both node and the browser, right?\r\n\nMARCEL: Yeah. If you compile to JavaScript, it can run wherever you want.\r\n\nCHARLES: Okay.\r\n\nMARCEL: It can also compile to Native, if you really want it to run in the machine.\r\n\nCHARLES: Shane, you mentioned ClojureScript and I actually was very interested in Clojure and ClojureScript at one point. And one of the biggest problems for me with ClojureScript was it depended on the Clojure toolchain. So, in order to compile a ClojureScript, I actually needed to have Java installed. I needed to have the Maven package manager to pull down dependencies and assemble these things. And that was kind of one that introduced a bit of a papercut into getting the environment set up, because, hey, I\'m a JavaScript developer. Why do I have to go install a functioning Clojure environment? And then the second thing was it meant that actually using the compilation tools was not available in a pure JavaScript environment. I think one of the kind of powerful things we\'ve seen is these systems that are written in JavaScript, for example, TypeScript is itself written in JavaScript. And so, it means that you can run TypeScript compilation inside of a browser with nothing more than what\'s available in the browser. So, what does the toolchain look like for ReasonML? Is it in JavaScript?\r\n\nSHANE: Yeah. Well, just to back up. ClojureScript actually is all JavaScript now.\r\n\nCHARLES: Oh, it is.\r\n\nSHANE: Yeah. You don\'t need Java anymore.\r\n\nCHARLES: Okay. That\'s good.\r\n\nSHANE: You can install it from npm. And there\'s a new -- I\'m not super up to date on this, but there\'s I think a new library, shadow, I think it\'s called. It basically lets you compile down your ClojureScript into different styles of JavaScript, like [inaudible] module or a package. So, they made a lot of progress on that space. It does still depend on a lot of the ClojureScript libraries, so the outputed file for like an application is probably fine. But if you wanted to make in a JavaScript style many little libraries, each of those libraries has to have this really heavy dependency in it. So, it\'s not ideal for that. But you don\'t need Java anymore. Big improvement on that end. But on the BuckleScript side, you can install with npm. So, you just npm add your BuckleScript compiler and then you just run it as just any other node executable.\r\n\nCHARLES: Okay. Well, that\'s handy. So, there\'s a totally separate Reason compiler. But is this syntax exactly equivalent to OCaml? I\'m still a little bit confused on that point. Or is it just a dialect?\r\n\nMARCEL: Reason is part of the toolchain and also one of the different ways of writing OCaml. It gives you a very JavaScript like syntax. And essentially imagine a bunch of pluggable compilers that sit at the bottom of the toolchain, and one of those is BuckleScript. So, you write your ReasonML that looks like JavaScript. And then it gets further down to this BuckleScript layer and the BuckleScript layer turned it into JavaScript or TypeScript, whatever you prefer.\r\n\nCHARLES: I see. Okay. So, BuckleScript is and of its own a unique language.\r\n\nMARCEL: Yeah. One could program in it but very few people do. It\'s a very powerful middle module.\r\n\nCHARLES: Okay.\r\n\nSHANE: This conversation is made a little more confusing by the fact that BuckleScript just released a new version of their syntax, which is very JavaScript centric. So even though it\'s technically like Reason, it\'s even more JavaScript focused than the Reason you might write if you were doing Reason Native. So instead of defaulting to lists, it defaults to JavaScript arrays. The object notation is now just defaulting to JavaScript objects. There\'s been a kind of another split that just happened where now you have ReasonML syntax that would work on Reason Native. And it has a new ReasonML syntax with more BuckleScript embedded into it, which would work with the BuckleScript compiler. So, there\'s even more splitting happening, which is probably going to confuse more people. But I think overall, it\'s for a good cause because the syntax is much more simple, especially if you\'re coming from JavaScript. There\'s a lot of weird things that happened which don\'t happen anymore.\r\n\nMARCEL: It is true. There were initially quite a lot of good momentum on this and a good uptake. And the growth of ReasonML, while it\'s still perfectly usable, very reasonable choice for all your projects. One of the reasons I feel it hasn\'t done as well is because of these continuing small confusions and decisions and perhaps a lack of more central leadership. Ostensibly, it\'s run by Facebook, but that\'s not really the case, in pragmatic , which means that it can be harder to get people started. And perhaps one of the reasons why something more focused like TypeScript, which is very clear, singular documentation site is doing better out there in the wide world of the software [inaudible].\r\n\nCHARLES: But you do think that it is possible to adopt incrementally too, should you want to? Even though it\'s distributed in of documentation and authority and maybe who\'s writing the roap?\r\n\nMARCEL: Yeah, absolutely. The interop story is very well thought out. And especially with this new update that Shane mentioned, it gets even easier to write code that turns into that kind of JavaScript that you expect. But with all the benefits of higher percent type safety speed, much smaller tooling surface and all the other benefits.\r\n\nCHARLES: Right. I wanted to ask another question about the types system. One of the things that I actually find frustrating about TypeScript and that I also loved about ClojureScript, and I know we\'re comparing a little bit of apples and oranges here because one is not strongly typed and the other is, but the complete absence of any type information at runtime inside TypeScript, and also the kind of arm\'s length at which you have to deal with the types system if you\'re trying to do meta level stuff. So, if you actually want to analyze a TypeScript file and use the type information at a meta level, it\'s actually very difficult. The API is extremely cumbersome, the programmatic API to the TypeScript compiler. Same thing really applies with Babel. One of the things that I love, absolutely love about Clojure and really any dialect of Lisp is how close you are to be able to make meta level transforms on any syntax because you can treat any program as just data. So, every Clojure program is a Clojure AST. The data structure and the evaluation of that data structure are one and the same, or they\'re always close at hand. Is there something similar like this for this toolchain where, let\'s say I\'ve got a set of types representing a state and I want a set of functions that transition from one state to the other. And I want to say, look at this declaration of a state machine and I want to render a SVG graph showing the state transitions where I have one state for each member of the type union and then a function showing an arrow. I want to draw an arrow that shows the input and the output for each function transition. So, if I\'ve got the type of the input and the type of the output, I can then draw an arrow. It\'s very easy for me to imagine doing that in a functional language that\'s a dialect of Lisp. Very difficult for me to imagine doing it in TypeScript. Are there tools for operating with the type system at the compiler level rather than the runtime level?\r\n\nMARCEL: Yes, there are, absolutely. That is a whole meta-programming sector of this, which is very important for things like Jane Street\'s huge reliance on OCaml. But that is a kind of language extension which tries to make interfacing with the types and the compiler much more friendly, called PPX, which lets you build extensions on the language in a very predictable way, and then ship them and share them with anyone who needs them. For example, one PPX that\'s very popular in the ReasonML web world is this GraphQL PPX. It actually converts it into the correct typings that will have to be for ReasonML and checks them as you go, rather than having a separate compiler step, as you might have to do with something like a relay compiler where it checks your thing, asks the TypeScript compiler, "Hey, does this match you what you\'re expecting of me?" Then come back to the and say, "No, that\'s wrong." Here, we\'re putting an understanding of the GraphQL schema definition language directly into its own compilation chain.\r\n\nCHARLES: Okay, yeah. And what is that called again?\r\n\nMARCEL: This is called PPX, and I can\'t what it stands for.\r\n\n[Laughter]\r\n\nMARCEL: It was something unusual. I think it\'s something like something, something extension. But everyone just calls them PPX. I did once know and I thinking, "That\'s ridiculous." And then I promptly forgot.\r\n\nCHARLES: So, it\'s kind of like a macro system, or it allows you to work with the syntax directly.\r\n\nMARCEL: Yeah. And therefore, you end up having to do -- some of the more complex PPXs do some very, very interesting meta-programming based on types in the compilation toolchain.\r\n\nCHARLES: Okay. That is really, really exciting.\r\n\nSHANE: Pre-processor extensions.\r\n\nCHARLES: Pre-processor extensions.\r\n\nSHANE: [Inaudible]\r\n\nCHARLES: I mean, that\'s good to know. Although pre-processor is an unfortunate term. I\'m scarred by the C++ pre-processor or the M4 pre-processor, I guess, which is used in C and C++. But I get it. Before we move in to talking about repkgs, is there anything else that we definitely need to call out as being super killer about Reason or that anyone should know, things you\'re just dying to let folks know something you always lead with when you give a talk on Reason?\r\n\nMARCEL: I think we\'ve covered most of the score points, which is things like speed, things like reducing your reliance on other parts of the toolchain. One of the nice things is that, which I\'ve mentioned is that you don\'t need all these things like Prettier, which was also built off Reason\'s refmt, ESLint or something, because they\'re all just one mega compiler, one binary to work around. Often, you\'ll have loads of CI/CD issues or issues on your personal computer because some part [inaudible] out of whack because our toolchain is a loosely coherent set of packages that we marshal along. But here, it all comes in one package. We have the interop, which is nice. We have the syntactic positiveness. The fact we don\'t really have to do much actual typing, which is great and just kind of falls out of the way you write code, which is nice. And I think that\'s mostly it. I mean, it means you can do really cool things, like you have type safe CSS because of the syntax extension I mentioned earlier. So, you don\'t have to have any variance if you don\'t want that to be.\r\n\nCHARLES: Okay. Man, I\'m sold. I actually am. I\'m seriously excited.\r\n\nMARCEL: [Inaudible]. It does have [crosstalk].\r\n\nCHARLES: Okay, yeah. That\'s actually a good point. Because it sounds really awesome, and yet not many code bases use Reason. Why is that? Why are we not using Reason today?\r\n\nMARCEL: That\'s a great question. Why aren\'t we using? Because people are cowards.\r\n\n[Laughter]\r\n\nMARCEL: I do think to some degree, I have worked a lot of languages that are incrementally adopted from or a lot of languages that have taken adoption slowly. And one thing I have found that is really amazingly important is for it to be something similar to what people currently use. We like to think of ourselves as technologists who are forward thinking and want to learn new ways to doing things real time. But personal experience says that\'s untrue. We are like everyone else. We like what\'s familiar and things we already know. We don\'t like the idea of feeling junior again when we already feel confident in our way of doing things. One of the best examples of this is a language I used to write a lot in is Elixir. Elixir is based on Erlang, which has been powering telephony systems for older than I am. And I\'m getting pretty old at this point. So this is a highly full-tolerant language made for distributed systems. And that become more and more relevant nowadays because as we\'ve come from personal computing backup to the cloud where we need full tolerance and distributed systems, it\'s an almost perfect candidate. But it didn\'t really have any adoption until someone wrote a new way of doing Erlang called Elixir. Elixir was made to look a lot like Ruby. And that alongside with the WhatsApp acquisition because no one cared about Erlang until WhatsApp was bought for $19 billion. Then I thought like, "Wow, maybe this is the way of getting bought out by a large company." But that aspect aside, it made it much more approachable. And of course, there\'s a really huge amount of people out there who write in Ruby, and they wanted something just like a little bit better. So, Elixir had been a much easier adoption process in at least the London startup scene compared to Erlang itself, which, of course, is equally capable but just looks more alien.\r\n\nSHANE: Yeah.\r\n\nCHARLES: Yeah.\r\n\nMARCEL: So, I think we are only slowly getting to the point of having Reason be approachable enough. I think there are still definitely work to be done on a few key points. The story of async/await, you can do it. There\'s a ton of PPXs for it, but there\'s not one blast path. There needs to be a very streamlined documentation, like a streamlined, centralized place [inaudible] get started with projects easily where they can be effective quickly and where they can dunk on their friends on Twitter without needing that much experience.\r\n\nTypeScript, on the other hand, has been very popular because you can adopt it very slowly and you can essentially be using a "TypeScript" of almost none of their features or safety because you could [inaudible] enough and any is the [inaudible] of a code base.\r\n\n[Laughter]\r\n\nMARCEL: But once they have it in there, it taints the tree of type understanding because everything has a relation inside, AST. I think Reason still is a little bit too unapproachable for use and it needs to have some of these things like the async story ironed out. While the errors are generally much better than TypeScript because they\'re going on a sort of Elm approach. If you ever use Elm, it is a wonderful copilot for getting you to where you need to go as long as where you need to go is a web app, because it\'s not a generic language. And these things combined and the lack of visible momentum, because a lot of this work is done, perhaps not in the open as much as it should be, means that I think it just hasn\'t got the speed because people do want to use what\'s popular. And that\'s not necessarily a bad thing. But community matters and community leads to Stack Overflow posts, which leads to documentation, which leads to help and reassurance that people are doing the right thing. People are often declaring, I think people, it can be seen on Twitter. People often declare technologies is dead. For example, when some Facebook libraries move from Flow to TypeScript, they both declared that, I guess, flow is over. The reality is flow is still the majority use type system within Facebook. It\'s just for some very external projects, that I would love as a contributor, therefore it wouldn\'t be nice of them when you use TypeScript. But it continues to be developed, continues to be what they write React in, as well as nearly all of the internal things that they ship.\r\n\nAnd in many ways, Flow is a much better type system. It\'s not quite type sound, but it\'s much better. It\'ll force you to be correct, but the tool is not quite as good. And because it forces you to change a lot of things at once, it just was much harder to pick up as compared to TypeScript, which basically lets you get away with anything. And one might argue, what\'s the point then? And the point is that one can say [inaudible] TypeScript.\r\n\nCHARLES: The point is to be viral.\r\n\nMARCEL: And it\'s important to also note that the TypeScript integration in [VS Code] is great. I think it\'s really wonderful. One of the things I\'ve always struggled with was having really resource intensive work as I work in the background, always checking files, and that led to a lot of frustration. The tooling story and documentation story and the on-boarding story of TypeScript is wonderful, even if and it may have slightly biased position of having used a large number of type systems, I think TypeScript is not quite as useful as some of the options out there. They really nailed the things that drive adoption. And ReasonML, much like PureScript or whatever else your favorite language happens to be, which probably is very good, the hard part is doing the people part, not the software part. Much like the Haskell approach, and this might be minimizing the Haskell community, so I apologize. But I have often gotten the impression that there is this idea that if a language is just good enough, people will flock to it naturally. If you build it, they will come. Which I think is not actually said on the movie, but it is the common misinterpretation, much with, "Luke, I am your father." And I think that simply isn\'t true. You really need to do a lot of community outreach and it can take a long time.\r\n\nTypeScript has been around for a decade now, but it only really gained prominence in the last three-ish years to a degree where people would use it almost by default. And a lot of these rough edges have to be sanded out with ReasonML. And if it gets dark, there\'s always that fear. If you\'re going to go to your friends and say, "Hey, we just use ReasonML," and then they hit a frustration or some strange bug or something they can\'t [inaudible] in Google, you\'re on the hook for that. So, you have to have a lot of confidence to go to your peers or to your client and go, "Hey, can we use this boutique compile to JavaScript option," when the safe choice is always TypeScript. And honestly, it may well be the correct choice for the majority of cases.\r\n\nCHARLES: Yeah, but I do think it builds well for the future of Reason. I think it does seem to be put together a little bit better, I mean, certainly than PureScript or some of the other ML-based languages that I played around with, just in of the marketing. I\'m not saying anything about the technical capability of the language. And it is, I think honestly, the biggest problem with flow right now is that I think it might actually be dead because it doesn\'t differentiate itself enough from TypeScript. So, while it might be used internally in Facebook and stay alive in that way, and I don\'t want to spend too much time trying to conjecture about the future of these things, but at least externally, it doesn\'t seem like it\'s much of a differentiation, whereas something like Reason really is. It\'s something different. And that\'s important in of adoption in five years, 10 years, things like that. Maybe not. Anyway.\r\n\nMARCEL: Yeah. I think that\'s a very fair observation. You\'re not going to get anywhere unless it\'s efficiently better. I , back in the Angular dominant days, the only reason people has picked out React because it\'s significantly better. Things that are only marginally better, people won\'t bother spending the switching energy for. So, often with projects like [Preact], for example, just say it\'ll be a slightly smaller bundle size. But people aren\'t going to switch all their tooling or all their knowledge and all of their risk profile for something that\'s only slightly bit better. You have to really excel for people to take notice. You have to do something different for that technology to be adopted. And in React\'s case, what it could do is, unlike all the things it can knockout in Angular. It could be easily understood on a simple level to get basic things going because you have to understand the whole framework. And it was relatively performant, which was a really big problem back in the Angular 1X era, and that really differentiated it.\r\n\nCHARLES: Yeah, I know. It\'s the quantum leaps that end up being able to adopt from zero. Otherwise, it\'s all incrementalism all the way. So, I wanted to talk a little bit about the experience of working on an actual code base with Reason and solving certain levels of problems with it and things that might be particularly suited or ill-suited for solving the problems. Shane, you actually have been developing a program called repkgs, which we got a little bit of a preview in kind of one-off way where we were going to be releasing a bunch of JavaScript packages on the world. And you ran this and you uncovered a whole slew of dependency issues with our packages that would have caused them to break out in the wild. And it was very, very helpful. And I wish we could honestly run it again and again as part of our CI pipeline. So, you\'ve got to release it at some point. I don\'t know if you have in the interim, but maybe you could talk a little bit about what repkgs is and the choice to use Reason there.\r\n\nSHANE: Sure. Repkgs is my take on making a tool for not just monorepos but focus on monorepos, making a tool to help validate your packages before you publish them. And the focus is definitely on monorepos, but it also works just like a normal single repo. This start off as just a general monorepo tool. You can run actions. You can get a list of your packages. But there are so many tools out there. I didn\'t want this to be a competition for, say, lerna or yarn workspaces because there\'s enough competition in that space. And I didn\'t see a ton of competition in the space of, is what I\'m about to publish going to actually work. Because most tests run against source files and there\'s almost no common workflow that builds the package as like an artifact and then tries to install it and then runs new tests. It\'s just my source work. Your source files work, that\'s great. And you compile them, say you\'re using TypeScript or even Babel, you\'re compiling often into a disk folder or from TypeScript to JavaScript. So, the extensions are changing.\r\n\nThere\'s so many little changes that happen between a moment of my tests say this is going to work and then it actually going into npm. So, this is false in that space where it tries to tell you right before you hit publish, run these checks and then confirm that what you\'re about to do is what you think you\'re going to do. And this came about because previously, like five years ago when yarn had just come out, I had made another monorepo tool called knit, which was very magical. And it basically ran through all of your code to find all your dependencies for you. I think it\'s still a pretty cool project, but it competed directly with lerna. And with lerna, it was such a behemoth, had so much backing. I didn\'t want to try and do that again, even though I think there\'s some ideas in it that were really powerful.\r\n\nOne of those ideas was walking through source code to determine the shape of the project and what dependencies are needed. And so, I took that idea. I think that was the interesting idea that knit had, and I just kind of focused on that. And that\'s how repkgs came about. It crawls through your code, it crawls through your package manifest, and it will try and find all the things that are about to break on you. That\'s just an overview of repkgs.\r\n\nAnd my reason I picked ReasonML is that was just a thing that I wanted to do.\r\n\n[Laughter]\r\n\nSHANE: Honestly.\r\n\nCHARLES: That\'s a perfectly valid reason.\r\n\nSHANE: Yes.\r\n\nCHARLES: Some might say the only reason.\r\n\nSHANE: I think it\'s the only real reason. Everything else is just to justify that. Like I said earlier, I looked actually into using ClojureScript for this, but I liked the output of ReasonML and BuckleScript because I was thinking about making some smaller packages that could be reused and BuckleScript felt like a better solution. And I worked on this project for about three months. And in that process, the first month, I used Reason as a BuckleScript. I was compiling to JavaScript and running on Node. And that process was very much a learning process.\r\n\nCHARLES: Wait. You were writing in Reason and then compiling to what?\r\n\nSHANE: Over the three months, the first month I wrote, it was always in Reason syntax but I wrote using the BuckleScript compiler to compile down to JavaScript.\r\n\nCHARLES: Oh, so wait. I feel like one of the things that Reason needs to do is have like a big diagram on the home page that shows the toolchain because I\'m having difficulty getting my head around all the different moving parts.\r\n\nMARCEL: I have made this diagram several times.\r\n\n[Laughter]\r\n\nMARCEL: Maybe I should just open up PR on the home page again to put out for me.\r\n\nSHANE: ReasonML is the syntax and then BuckleScript is what turns that into JavaScript. But there\'s also Reason Native which will turn it into the OCaml [inaudible], which you can run natively.\r\n\nCHARLES: Okay.\r\n\nSHANE: So, the difference is it\'s not actually not all BuckleScript-ed ReasonML will compile down to Native OCaml. It\'s not ReasonML, and then you just flip out the output. Sometimes, you have to make changes. So, it is kind of complicated and a little confusing. Basically, the first month I was using, it was going to run on Node. My output was running on Node. And in the next month, I had it running as Native compiled. And then the third month, I went back to Node.\r\n\nCHARLES: Okay. That\'s awesome, by the way.\r\n\nSHANE: Yeah. Well, I can explain why. The first month I was just learning BuckleScript and ReasonML, and that\'s when I was getting stuck but stuck in a way of I kind of felt like it was my fault that I just didn\'t know the language well enough. And I could usually see where I was trying to go. I just knew I had to figure out how this syntax worked or how do these modules work. So, I felt like I was slowly but surely making progress, except when I would run into a problem that was very Node-specific. Because if you Google BuckleScript, ReasonML, it will immediately take you to React. That is by far the biggest focus of BuckleScript is React.\r\n\nCHARLES: Right.\r\n\nSHANE: So, finding answers to my questions that were Node-related, like interacting with filesystem and how do I do this, and all the Node binding, it isn\'t fully complete. It has only a couple of the methods on all of them. So, anything Node-related, I found pretty frustrating to deal with. And there was a lot of me just kind of like \'I guess I\'ll figure this out on my own\' sort of feeling. And I got to the point where I was thinking, "I\'m running this locally. It\'s never going to run on a browser." And Reason has this cool option where I can compile to Native code. So, I rewrote everything. I didn\'t try and convert it because it didn\'t really convert the way I wanted it to because I was using a lot of Node bindings in my original code. So, I just rewrote everything.\r\n\nCHARLES: Okay. So, when you were outputting a Native binary from your Reason, are you literally using the operating system native libraries like, what is it like? The stdlib on Unix to get access to stderr and stdout and all that stuff?\r\n\nSHANE: I think that is what\'s being used. You do have to compile to the platform so you don\'t just make one binary. It\'s like you have to build one for Windows, for OSX. I didn\'t go that deep into it because when I made the transition, I was very Node heavy. And then when I transitioned over to Native, I rebuilt everything. And that process actually was not fun at all. That month was a pretty terrible month of me working on this. There are a few libraries that have been around for a while, if you want to do a commit, [inaudible] arguments or reading file system or work with the operating system.\r\n\nThere\'s one guy, I can\'t his name, but he seems to do everything. [Inaudible] forever and they work pretty well. But there\'s not a ton of examples. For a language that\'s so old, I found it extremely hard to find working examples to help me. So when I got stuck, oftentimes I was just stuck. Just like dead in the water for days and days, just trying to work through these issues myself which I found very frustrating. One thing I do a lot of in this application is I read the file system and I use like globstars to read through directories. And all of the Node packages have this ed, like the one globstar with two globstars means it\'s recursive. In OCaml, I couldn\'t find that anywhere. So just the basic here is a path with globstars in it, and it means go through all directories. That wasn\'t a thing that I could do. It just wasn\'t there. There\'s an OCaml-re which is regular expressions and it had some path stuff but it didn\'t that. And that was like the main one that people use. It didn\'t it.\r\n\nThere\'s another tool called Dune, which is their package builder. And I had to use their library for a while, but I don\'t think that\'s ed in the same way. I basically had to build my own, and that was not the best. I don\'t want to have to build my own regular expression parser. And basically, it\'s very frustrating.\r\n\nMARCEL: This is actually an endemic problem inside all the OCaml communities because OCaml has a bunch of factions. Not that they\'re at war or competitive, it\'s just they have their own kingdoms in which they find OCaml resource. Some of them are very academic, like this team in or very financial like Jane Street, who has their own library for basically everything.\r\n\nOne of my friends, [Rizo] is a huge OCaml fan. He is the more competent technical part of the Reason London experience, while I am the razzle dazzle. He has continuously talked about the things that there are four to eight standard libraries and there isn\'t a community centralized documentation system and it\'s all a bunch of IRC servers where people in the know talk to each other and decide what\'s going to happen with the language. And this is definitely something that hopefully Reason can help with, because it will highlight the fact that if you want mass adoption or adoption by more people, these community items are required because very few people will power through frustration in the same way that Shane did.\r\n\nSHANE: Yeah. I made a decision that I was going to figure it out. And I, multiple times, just said I am done with programming forever, and I just closed my computer and walked away for a day. I couldn\'t imagine anyone doing that for work though.\r\n\nCHARLES: But you went back then to Node.\r\n\nSHANE: Yeah.\r\n\nCHARLES: But at the same time, this is kind of how like the reason we have that in Node like the [inaudible] is because back in 2009, that\'s what the Node ecosystem was. If you wanted something to actually recurse through a directory, you had to build all that yourself.\r\n\nSHANE: Yeah. That\'s the happy ending to the story is when I went back to Node, I actually understood the language a lot better. And I was able to pretty much solve all the issues I had that first month. Instead of learning the language or learning how to check the file system or getting frustrated, I could just use the types and really build out what I was trying to express using modules, using types. And the rebuild when I went back to BuckleScript was a really pleasant experience. That really sold me on BuckleScript and ReasonML.\r\n\nMARCEL: [Inaudible], we have all the knowledge you need to tackle the enemy you had a problem with it the first quartet of the hero\'s journey.\r\n\nSHANE: Yeah.\r\n\nCHARLES: It\'s a validating story because if you are able to spend the work to learn how to make the hard thing easy, then you can do the easy thing because that was hard previously. Doing the hard thing is now easy.\r\n\nMARCEL: I think also, to Charles\'s point, there is now a library called reason-nodejs which came out fairly recently that attempts to, I think, tackle many of these Node problems. It even has a file system example on the Read Me. You may be correct that a lot of these frustrations will eventually get solved by people writing packages that we then just consume and don\'t think twice about.\r\n\nCHARLES: Yep. All right. Well, I think we\'re almost there. Gentlemen, it has been a journey. I actually am pretty excited. There\'s a couple of things that I want to try now. As you guys know, I\'ve been fooling around a lot with structure concurrency and knowing that the async/await story isn\'t quite pinned down in Reason makes me want to try and implement a more general structured concurrency story there. Just give it a stab. And in knowing that there are these PPXs that you can extend the syntax is really exciting because one of the frustrations of trying to do that in TypeScript has been you\'re very limited in the syntax that you can do. You\'ve kind of got to work with what you\'ve got. And the typing around generators and co-routines is just not great. And it\'s not going to get better unless you can convince the people at Microsoft to do something about it. But it sounds like there\'s actually a way forward, which is ultimately the power of having these syntax transforms, which is exciting. So, if I ever can find some spare time, I am not even lying, I want to dedicate some cycles to this.\r\n\nMARCEL: There\'s a library called bs-let, which is a PPX that tends to make a very general, almost monadic approach to awaiting stories in JavaScript, which is worth a read if you are interested in that sort of thing.\r\n\nCHARLES: Unfortunately, I am. Oh, and they\'ve got letPlus and letStar. All right. Anyhow, thank you all for stopping by. Everybody, go out there and Reason away. And we will see everybody next time.\r\n\nMARCEL: All right. Bye for now.\r\n\nCHARLES: Thank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at @frontside.io. Thanks, and see you next time.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n

SPECIAL GUEST: Marcel Cutts | @marcelcutts: Founder of Asgard Tech, ReasonML London Organizer Shane Wilson | @wilsonashane: Founder of Peyk Inc. SHOW NOTES 00:38 - Reason Twitter: @reasonml Discord 03:25 - BuckleScript OCaml 06:01 - Reason + BuckleScript TypeScript 16:07 - Reason: Interoperation & Adoption ClojureScript Toolchain Syntax 30:00 - Operating at the Compiler Level vs the Run-Time Level ppx (pre-processor extension) 34:29 - Last thoughts on, and why use Reason? Flow 44:43 - repkgs knit Lerna TRANSCRIPT: CHARLES: Hello and welcome to The Frontside podcast, a place where we talk er interfaces and everything that you need to know to build them right. My name is Charles Lowell, a software developer at Frontside. And today, here to talk with me about many interesting topics. One of them being ReasonML, we have Shane Wilson, the Founder of Peyk Inc, and Marcel Cutts, Founder of Asgard Enterprises. Welcome, gentlemen. MARCEL: Hello. Hello. SHANE: Thanks for having us. CHARLES: Yeah. So, I guess we should start out just talking about why ReasonML. I understand that both of you guys have some experience with it. New languages are all the rage. So, what drew you to it and have you found it? MARCEL: That's a great question. I'm afraid my history is going to be spotted and will really demonstrate how much pubs are an integral part of the UK tech scene. I got into ReasonML maybe four years ago at this point when I found it on some hacker news post. And I asked on Twitter, "Hey, is anyone running a ReasonML meetup in London?" And I also, maybe around point one. And then by point for a point five, someone said, "Oh, no, wait. I think there is one." "Oh, does that mean you're starting one?" And then by point six, I thought this was a good idea. So, I began to run the ReasonML London Meetup, which end up being one of the bigger ones in the world. And along the way, it [inaudible] to learn some ReasonML and why it exists, which ended up being quite fascinating. And I did quite a few talks around it around the world. And it has stuck with me since. It's a very fascinating intersection of many kinds of approaches to programming as well as technological applications with a lot of history as well stemming from its origins, OCaml. So, it's a bit of a unique beast and it has kept me interested since. SHANE: I came about it because I've been doing JavaScript for quite a long time. But I've always liked compile to JavaScript languages. Maybe six years ago or longer, I started looking into Clojure and I got pretty heavy into Clojure and on ClojureScript. And I was always looking for a reason to write ClojureScript as a compiler to JavaScript language, but it never took off. Do I kind of hope that it did? A lot of people use it, but it never really took off. And then when I was recently looking into how do I write JavaScript Node applications but without having to write JavaScript or setting up all the toolchains, I came across ReasonML along with BuckleScript. And I was really pushing how you could write a language with a lot more features and JavaScript comes built in with but compiles down to JavaScript that's very readable. That was a big selling point. So, when I started looking into it, that your output was basically JavaScript. It wasn't like a computer had written it. It looks often like a person had written the JavaScript. I thought that was very impressive. And it made it a lot easier for me to think that I could sneak it in to a project somewhere. Or I could just make my own little ReasonML toolchain, output a file, and then just not tell anybody and no one would even know. MARCEL: Some say that I have in fact made several branches for many client projects which secretly have been produced from BuckleScript output and then slightly tied it and no one's ever found out. CHARLES: So now, what is BuckleScript for the uninitiated? MARCEL: We can go from the top down. ReasonML is a meta language as well as a toolchain and some documentation utilities for a language called OCaml. OCaml is a 20 plus-year old [inaudible] systems language, the kind of language you expect someone of a long gray beard to be writing. CHARLES: It's one of the spiritual ancestors of Haskell, isn't it? Or no? MARCEL: That's a great question. I'm sure what the familial tree between those two would be. I think it's an ML language, which means that a lot of that will bleed into the Haskell way of thinking. CHARLES: Okay. MARCEL: About a few years ago, I think about four or five years ago, the chap who invented React left the spotlight and retreated into the shadows and left other people to be the spokesperson for that framework. everyone that Jordan Walke is the person who made React, not Dan Abramov, even though he is the star of the show for many things React-related. In his spare time, what he did is work on this thing called ReasonML, which is a new syntax and toolchain for OCaml, which makes writing OCaml look a lot like writing JavaScript. And this was a way of trying to get people to write in a more functional and type safe way. At this point, you may be saying, "Great, you can write OCaml like JavaScript, but it's still OCaml." The missing piece is something called BuckleScript, which was developed by an expensive keyboard company called Bloomberg. And they had this layout because OCaml itself is a very pluggable and modular compilation system. They plugged a single BuckleScript [inaudible] compiling to bytecode or [inaudible] or you could compile directly to JavaScript. This runs a lot of the internals of Bloomberg that have a surprisingly large amount of JavaScript and Node in there, and they wanted to be 100% sure about what they're getting because with OCaml, you do get and ReasonML by extension, you get 100% type safety all the time. Unless you have big exception flags, unlike something like TypesSript, for example, where you sort of [inaudible] but you don't quite have the same certainty. And also, you don't have a lot of these syntactical avenues that a language base and types will give you, things like variance or being able to switch on type definitions. CHARLES: If I'm understanding then correctly, ReasonML, the syntax actually is 100%, it is OCaml. It compiles down to BuckleScript. Reason is this toolchain for OCaml that outputs this dialect of JavaScript called BuckleScript? MARCEL: Yeah. The combination of BuckleScript and reason together give you the syntax and the output. Have there been discussions about trying to make this clearer and saying what each component actually does? Yes. Have they been very [inaudible]? Also, yes. [Inaudible] reason to syntax and then BuckleScript compilation that leads to the readable JavaScript that Shane mentioned earlier. And this has had some interesting pickup and it was quite a contender in the original for People 4 Projects battling for the type space early on, which was Flow, TypeScript, Elm, and ReasonML. TypeScipt has been around for nine years at this point. I think it came out in 2011. It's been around for much longer than people because it just took a long time, a lot of [inaudible] backing to get to the front spot. And it is continuing to slowly develop to this day. It hasn't quite got the same acceleration it did at the beginning, as Shane mentioned. But it still has a ton of interesting applications and it is used in the law of production applications throughout the world. For example, all of Facebook Messenger is written in ReasonML for performance and certainty reasons. CHARLES: Oh, okay. Interesting. I started out using type systems like Java and C++ and C, and I very much considered myself kind of a refugee from these brutal typing regimes. I fled to the land of JavaScript and of Ruby, and even places like Perl and Python kind of to escape types. I've been slowly getting back into it with things like TypeScript. So, what are the surprising aspects of the type safety that are really, really done well? Because I can definitely say having gotten a taste of TypeScript in the last few years, I can see there are ways to do a TypeScript or a typing system that really feel lightweight and feel like they're helping you and feel like a tailwind rather than a slog. And so, what are those experiences in Reason that really stand out? SHANE: For me, I've never used in ML language seriously before other than like some toy projects. So, for me, most of my experience with types has either been Java or TypeScript. And ReasonML, I'd much prefer that type system. I'm not going to get too technical because quite frankly, I don't really know the technical things to say here. But when I work even with TypeScript, I feel like I'm adding types because TypeScript wants me to and needs them. Whereas in ReasonML, first of all, you don't have to add a lot of types. You almost never have to actually type the code that you're writing, maybe type modules or create types. But the actual code that you're writing with all the flow and the logic is almost never a type in there. ReasonML and the OCaml figured that out for you, which I think is amazing and it doesn't get in my way anymore. And when I write ReasonML types, I feel like I'm doing that to help me, not to help the language. I want to better define the problem that I'm solving and help create barriers for how to solve that, put data in a place that it should be, help with my logic flow, and I want to make those changes and the language is helping me make those changes. Whereas most of my time and experience with types, I know what I want to do and the types aren't happy, and so I have to change my work to manage the types. CHARLES: Right. You're having to kind of spoon feed the compiler and throw spaghetti at the wall and see which come kind of weird combination of types and generics fit the problem and make the compiler happy. MARCEL: Yeah. Java really has done an absolutely excellent job at ruining types for everyone for quite a while. The worst PR campaign you can imagine. But it's important to know that there are more than one kind of type system. There are two big schools. One being nominal typing which is the kind of Java C# thing that has burned into people's memories and gives them sleepless nightmares, where you try to compare the names of things and the way the things I described and to figure out if they have equivalent to that sort of thing. But the thing that things like Haskell, Elm, and ReasonML, OCaml too, is called structural typing where it's based entirely on the structure of your data, which means you can do much more interesting things and it can get out of the way a lot more. And it has something called the Hindley-Milner type system, I want to say, to try and infer the types to the best of its ability with as much certainty as possible from your code. And it is mathematically provable that the Hindley-Milner type system is the best type inference engine that currently exists. So, you do end up typing precious little when you're actually writing ReasonML. But you still have complete type certainty, which is very helpful, especially if you compare doing this to something like TypeScript, where I think it's not unreasonable to say that folks like to type all the things that are easy to type, for example, dysfunction [inaudible] string. But often it is much more difficult to type things like the response from an API that could be in all kinds of structures or the state of the application, which could be in X many permutations, which is much more difficult to describe with the typing tools that you get in that language set. CHARLES: Is there any particular example of something recently where we can compare, say, an implementation in TypeScript to something in Reason? I just want to make sure that we have like some way to focus. Because I believe you, but it's just -- and I definitely have this experience where I feel like I'm trying to satisfy TypeScript and the errors are cryptic and, yeah. And so, I'm trying to get a glimpse into what's a particular use case that I would find frustrating with TypeScript that Reason would swoop in and solve all the things. SHANE: I don't know if I can think off-hand of something that obvious, although there might be. But I know one big feature that ReasonML has is like the switches. You can switch on a type and I guess you can have dynamic typing. So, instead of having to create a whole interface, you can just say -- basically it's it's like here's a string, but it's not a string. It's like I just have a label and it's now a type. And you can do type checking on these dynamic types. So, I've used that recently to manage errors in a product I was running where any random error I want to come up with, I could give it a label. Oh, and so types can also get past data. So, my dynamic error label like file read error can actually take in values like a file pointer, like a link to a file. And then when I'm accessing that error later, I can pull out all this data. So, you don't have to create these complicated structures and try and them around. You can just create a type and then it will have all the data in with it. And then you can do a switch check on that type and destructure all the things that you need. It's kind of hard to explain, especially when I don't know the details of what I'm saying. But the best I can say is it's just the feeling you get when you're programing, you don't get the frustration. When I work with the types, I'm always glad that the types have helped me. I'm very rarely frustrated that the types are getting in my way. CHARLES: It sounds like the types are just so deeply integrated into the way that everything is done. It's not so much annotating stuff to make sure that the ends of your apparati line up with each other so much as deeply integrated with how you actually express the flow of the program. Is that a fair way to say it? MARCEL: Yeah. SHANE: Yeah, it sounds good. MARCEL: I think I can think of a decent example that demonstrates a difference that types can make. In TypeScript and JavaScript, its [inaudible], of course. You often end up with these situations that folks call being stringly typed. So, a component, if you're doing a React, can take a number of props and maybe the component has different states, have a button that can be a default one or a call to action button or a more subdued button, if it's an option that you don't want to be invisible. So, there are many ways of trying to get it done. But one thing that often happens is you parse on a string of the correct name that you wanted this thing to do. And then maybe inside a component, you have a switch and then you have all the different kinds of strings that you could expect together. But you could get any kind of string, really. Nothing prevents you. So, with the switch and variance system that Shane was mentioning, you automatically get [inaudible] pattern matching. So, you can say these are these three modes and here these types of that mode. If you then start to do a pattern match, which is, you know, think of it as a switch, but 11,000 times stronger, it will tell you, "Hey, no. You forgot one because there's this other state that you didn't mention." And also, what you're going to do if you don't hit any of this, but you can't because, of course, the compiler will catch it before you get there. This means you don't end up getting into strange situations because you're parsing strings on the application and you're hoping that you ed all the [inaudible] and of every case. And in addition to this, one thing that you find very pleasing on Reason, it brings you joy to writing this. It's a very, very, very fast loop. We're going to talk all about TypeScript because that's kind of a merge as the frontrunner of adding vague types to the JavaScript universe. But TypeScript doesn't check the whole application every time you make a change. BuckleScript does and it can do these compiles in subsecond for very, very, very large projects, which means that you always have confidence in your application as a whole as you continue to type and you don't have to like wait or go back or figure out how something doesn't make sense somewhere else. And this really short loop combined with the fact you don't really need to have all these things, like I said, [inaudible] type system, ESLint, or Prettier that hold your hand means that you have a very satisfying and streamlined experience as you go along compared to the, I think, expected amount of waiting you might do in a regular TypeScript project. CHARLES: Yeah, that begs an interesting question because TypeScript does seem to evolved as the front runner of a typed JavaScript development system. And I think that's owing in no small part to the fact that it's very easy to adopt incrementally. You can just switch TypeScript. You can just rename all your JavaScript files to TypeScript and you're off to the races. And you have kind of a through compilation. There are compilers out there like the Babel compiler that just strip the type annotations and you're good to go, and then you can very easily ramp up your usage of the types. So, I would say, for example, in my own experience with TypeScript, we started out with very loose adherence to the norms of the TypeScript community. And then now most of the code that we write uses the strictest possible settings because those tend to be the most helpful in of development, catching errors earlier, things like that. But it's definitely, if you start that way, it is kind of against the norms of normal JavaScript. And so, there is this very clear evolution or very easy way to adopt. So, I guess my question is for Reason, what's the story for inter operation and what's the story for adoption? So, I've got an existing JavaScript code base, how do I sneakily introduce Reason? SHANE: One way you can do it is because, as Marcel was saying, the compilation is so fast, you can just add it as another step. So, you can set up your BuckleScript compiler and write some ReasonML code and have it compile down into JavaScript or even TypeScript, because there's a plugin called gentype, which will just generate types for you and you can then through TypeScript types so you can get perfect TypeScript types if you're using a library. And by the time you finish writing the ReasonML code, it compiles very fast and then you can just have whatever TypeScript watcher watching all the TypeScript files and it will see, "Oh, that TypeScript file is changed," and it will recompile and then you'll get whatever your program is. So, it's kind of easy to add it as a sidestep. I think the BuckleScript people have said they don't really try and integrate it into the other tools as like a first class because they want control over so much, because they do so much. And by having all that control, they can build it that quickly. But if you wanted to, you could just add a little folder and then put all your ML code there and just have it compile. And then your other watchers just see the compiled files and it works. That's actually what I've done where I wanted my interface to be TypeScript, but I wanted my actual code to be ReasonML. And so, I have one TypeScript file, which is like the index entry point of the program. And then that TypeScript file reads my generated TypeScript files that don't really exist before I run anything. And then I would just have the BuckleScript compiler watching, it builds on my TypeScript files and then I will run like Yarn or whatever, and that goes into my TypeScript entry point, which then finds all those that build TypeScript files and it just runs. It doesn't even know the difference. So, it's not the worst setup, if you want it to. CHARLES: So, then you're calling your Reason code from JavaScript. What about the other way around, like if you want to use existing NPM packages inside Reason, especially ones that really haven't been developed for Reason development at all? I mean, this kind of a pain point with TypeScript, even though there's the definitely typed repository where you get 98% of the typings correct, but still, there's a lot of quirks and [inaudible] and stuff doesn't line up. Fundamentally, there's JavaScript APIs that don't really work with a strongly type system. They're hard to type. What's the story, I guess, for using existing JavaScript code? MARCEL: You do have to write some level of interop to give it some types. But the nice thing is that there are, as you mentioned, some things are just difficult to type because a few of the more dynamic things that JavaScript can do is difficult to express in TypeScript. But the type system for Reason is a little more mature with full features on it. It lets you do both imperative and functional styles of code, lets you do mutations if you want to do performance. So, you could do a much better job of describing what's happening on the other side. CHARLES: Okay. MARCEL: There are also a few tools to help you take existing TypeScript definitions and work backwards. But if you're just picking up a random JavaScript library off the Internet that hasn't no way been typed, you would have the same pains as you would for TypeScript. But much like TypeScript, there are already libraries. I think it's called Redex is the website one goes to when they want to see which of their packages already have ReasonML definitions. CHARLES: What would you do? So, one of the things before we even started using TypeScript, a lot of our libraries were written in JavaScript and we had a lot of requests for, "Hey, let's make the typings." And I always found putting stuff on, definitely typed wasn't optimal because if you released a new version of a library that had breaking API changes, the types were decoupled from that. And so really, the interface definitions being decoupled from the actual source code was a problem. So, even before we started doing stuff in TypeScript, we started shipping declaration files with the actual code. So, taking time to make the TypeScript declarations with the JavaScript code, even though the projects were written in JavaScript, it just made it more adoptable by people who were using TypeScript. So, is there an equivalent in Reason? Can I just say, "I've got this thing that I've got in JavaScript or TypeScript or whatever, but I'm going to write a reason API for it." Can you ship that type of adapter with your library and make it available in package JSON, so that anybody using Reason can just, with no fuss, no muss, use your code? SHANE: In that case, a lot of times, you just publish at npm and there's somewhat of a standard where you prefix your package either with bs-, like BuckleScript. I think they're trying to move that to re for Reason just to try and reduce the confusion of what is BuckleScript, what is Reason. But the way you do that is you ship your ReasonML code, and then you add that package to your BuckleScript dependency list. And so, BuckleScript can now go into that package, compile the ReasonML code, and then you can use like the ReasonML types in your package. So, that's how you would on those types. If you and type a JavaScript language and you want someone else to reuse your types, you can just publish it as re- package name. And then you can use that in a ReasonML project. MARCEL: One thing to also is that we are very used to having a ton of dependencies inside the JavaScript universe, which is why the idea of having to write bindings for a lot of packages seems incredibly daunting. But what's important to is that ReasonML is a much more full-featured language and has a bunch of libraries and stuff pre-built. So often, when we want to do something with functional and we would take a Lodash as a library and then think, "Wow, it's always easy to type together." The outcome of having a mature language that you're basing this meta language on means that you have a lot of these things you'd normally get dependencies for is completely built-in already. And you end up having a lot fewer dependencies that you need to worry about. CHARLES: Okay. Meaning, for example? SHANE: I guess you wouldn't need things like Lodash as much. CHARLES: Okay. So just all your things like transforming data structures and stuff like that? SHANE: Yeah, a lot of the data structure manipulation functions are already built in. CHARLES: But it's still all the stuff like file systems and network you have to receive from the environment. For example, I assume you can run Reason in both node and the browser, right? MARCEL: Yeah. If you compile to JavaScript, it can run wherever you want. CHARLES: Okay. MARCEL: It can also compile to Native, if you really want it to run in the machine. CHARLES: Shane, you mentioned ClojureScript and I actually was very interested in Clojure and ClojureScript at one point. And one of the biggest problems for me with ClojureScript was it depended on the Clojure toolchain. So, in order to compile a ClojureScript, I actually needed to have Java installed. I needed to have the Maven package manager to pull down dependencies and assemble these things. And that was kind of one that introduced a bit of a papercut into getting the environment set up, because, hey, I'm a JavaScript developer. Why do I have to go install a functioning Clojure environment? And then the second thing was it meant that actually using the compilation tools was not available in a pure JavaScript environment. I think one of the kind of powerful things we've seen is these systems that are written in JavaScript, for example, TypeScript is itself written in JavaScript. And so, it means that you can run TypeScript compilation inside of a browser with nothing more than what's available in the browser. So, what does the toolchain look like for ReasonML? Is it in JavaScript? SHANE: Yeah. Well, just to back up. ClojureScript actually is all JavaScript now. CHARLES: Oh, it is. SHANE: Yeah. You don't need Java anymore. CHARLES: Okay. That's good. SHANE: You can install it from npm. And there's a new -- I'm not super up to date on this, but there's I think a new library, shadow, I think it's called. It basically lets you compile down your ClojureScript into different styles of JavaScript, like [inaudible] module or a package. So, they made a lot of progress on that space. It does still depend on a lot of the ClojureScript libraries, so the outputed file for like an application is probably fine. But if you wanted to make in a JavaScript style many little libraries, each of those libraries has to have this really heavy dependency in it. So, it's not ideal for that. But you don't need Java anymore. Big improvement on that end. But on the BuckleScript side, you can install with npm. So, you just npm add your BuckleScript compiler and then you just run it as just any other node executable. CHARLES: Okay. Well, that's handy. So, there's a totally separate Reason compiler. But is this syntax exactly equivalent to OCaml? I'm still a little bit confused on that point. Or is it just a dialect? MARCEL: Reason is part of the toolchain and also one of the different ways of writing OCaml. It gives you a very JavaScript like syntax. And essentially imagine a bunch of pluggable compilers that sit at the bottom of the toolchain, and one of those is BuckleScript. So, you write your ReasonML that looks like JavaScript. And then it gets further down to this BuckleScript layer and the BuckleScript layer turned it into JavaScript or TypeScript, whatever you prefer. CHARLES: I see. Okay. So, BuckleScript is and of its own a unique language. MARCEL: Yeah. One could program in it but very few people do. It's a very powerful middle module. CHARLES: Okay. SHANE: This conversation is made a little more confusing by the fact that BuckleScript just released a new version of their syntax, which is very JavaScript centric. So even though it's technically like Reason, it's even more JavaScript focused than the Reason you might write if you were doing Reason Native. So instead of defaulting to lists, it defaults to JavaScript arrays. The object notation is now just defaulting to JavaScript objects. There's been a kind of another split that just happened where now you have ReasonML syntax that would work on Reason Native. And it has a new ReasonML syntax with more BuckleScript embedded into it, which would work with the BuckleScript compiler. So, there's even more splitting happening, which is probably going to confuse more people. But I think overall, it's for a good cause because the syntax is much more simple, especially if you're coming from JavaScript. There's a lot of weird things that happened which don't happen anymore. MARCEL: It is true. There were initially quite a lot of good momentum on this and a good uptake. And the growth of ReasonML, while it's still perfectly usable, very reasonable choice for all your projects. One of the reasons I feel it hasn't done as well is because of these continuing small confusions and decisions and perhaps a lack of more central leadership. Ostensibly, it's run by Facebook, but that's not really the case, in pragmatic , which means that it can be harder to get people started. And perhaps one of the reasons why something more focused like TypeScript, which is very clear, singular documentation site is doing better out there in the wide world of the software [inaudible]. CHARLES: But you do think that it is possible to adopt incrementally too, should you want to? Even though it's distributed in of documentation and authority and maybe who's writing the roap? MARCEL: Yeah, absolutely. The interop story is very well thought out. And especially with this new update that Shane mentioned, it gets even easier to write code that turns into that kind of JavaScript that you expect. But with all the benefits of higher percent type safety speed, much smaller tooling surface and all the other benefits. CHARLES: Right. I wanted to ask another question about the types system. One of the things that I actually find frustrating about TypeScript and that I also loved about ClojureScript, and I know we're comparing a little bit of apples and oranges here because one is not strongly typed and the other is, but the complete absence of any type information at runtime inside TypeScript, and also the kind of arm's length at which you have to deal with the types system if you're trying to do meta level stuff. So, if you actually want to analyze a TypeScript file and use the type information at a meta level, it's actually very difficult. The API is extremely cumbersome, the programmatic API to the TypeScript compiler. Same thing really applies with Babel. One of the things that I love, absolutely love about Clojure and really any dialect of Lisp is how close you are to be able to make meta level transforms on any syntax because you can treat any program as just data. So, every Clojure program is a Clojure AST. The data structure and the evaluation of that data structure are one and the same, or they're always close at hand. Is there something similar like this for this toolchain where, let's say I've got a set of types representing a state and I want a set of functions that transition from one state to the other. And I want to say, look at this declaration of a state machine and I want to render a SVG graph showing the state transitions where I have one state for each member of the type union and then a function showing an arrow. I want to draw an arrow that shows the input and the output for each function transition. So, if I've got the type of the input and the type of the output, I can then draw an arrow. It's very easy for me to imagine doing that in a functional language that's a dialect of Lisp. Very difficult for me to imagine doing it in TypeScript. Are there tools for operating with the type system at the compiler level rather than the runtime level? MARCEL: Yes, there are, absolutely. That is a whole meta-programming sector of this, which is very important for things like Jane Street's huge reliance on OCaml. But that is a kind of language extension which tries to make interfacing with the types and the compiler much more friendly, called PPX, which lets you build extensions on the language in a very predictable way, and then ship them and share them with anyone who needs them. For example, one PPX that's very popular in the ReasonML web world is this GraphQL PPX. It actually converts it into the correct typings that will have to be for ReasonML and checks them as you go, rather than having a separate compiler step, as you might have to do with something like a relay compiler where it checks your thing, asks the TypeScript compiler, "Hey, does this match you what you're expecting of me?" Then come back to the and say, "No, that's wrong." Here, we're putting an understanding of the GraphQL schema definition language directly into its own compilation chain. CHARLES: Okay, yeah. And what is that called again? MARCEL: This is called PPX, and I can't what it stands for. [Laughter] MARCEL: It was something unusual. I think it's something like something, something extension. But everyone just calls them PPX. I did once know and I thinking, "That's ridiculous." And then I promptly forgot. CHARLES: So, it's kind of like a macro system, or it allows you to work with the syntax directly. MARCEL: Yeah. And therefore, you end up having to do -- some of the more complex PPXs do some very, very interesting meta-programming based on types in the compilation toolchain. CHARLES: Okay. That is really, really exciting. SHANE: Pre-processor extensions. CHARLES: Pre-processor extensions. SHANE: [Inaudible] CHARLES: I mean, that's good to know. Although pre-processor is an unfortunate term. I'm scarred by the C++ pre-processor or the M4 pre-processor, I guess, which is used in C and C++. But I get it. Before we move in to talking about repkgs, is there anything else that we definitely need to call out as being super killer about Reason or that anyone should know, things you're just dying to let folks know something you always lead with when you give a talk on Reason? MARCEL: I think we've covered most of the score points, which is things like speed, things like reducing your reliance on other parts of the toolchain. One of the nice things is that, which I've mentioned is that you don't need all these things like Prettier, which was also built off Reason's refmt, ESLint or something, because they're all just one mega compiler, one binary to work around. Often, you'll have loads of CI/CD issues or issues on your personal computer because some part [inaudible] out of whack because our toolchain is a loosely coherent set of packages that we marshal along. But here, it all comes in one package. We have the interop, which is nice. We have the syntactic positiveness. The fact we don't really have to do much actual typing, which is great and just kind of falls out of the way you write code, which is nice. And I think that's mostly it. I mean, it means you can do really cool things, like you have type safe CSS because of the syntax extension I mentioned earlier. So, you don't have to have any variance if you don't want that to be. CHARLES: Okay. Man, I'm sold. I actually am. I'm seriously excited. MARCEL: [Inaudible]. It does have [crosstalk]. CHARLES: Okay, yeah. That's actually a good point. Because it sounds really awesome, and yet not many code bases use Reason. Why is that? Why are we not using Reason today? MARCEL: That's a great question. Why aren't we using? Because people are cowards. [Laughter] MARCEL: I do think to some degree, I have worked a lot of languages that are incrementally adopted from or a lot of languages that have taken adoption slowly. And one thing I have found that is really amazingly important is for it to be something similar to what people currently use. We like to think of ourselves as technologists who are forward thinking and want to learn new ways to doing things real time. But personal experience says that's untrue. We are like everyone else. We like what's familiar and things we already know. We don't like the idea of feeling junior again when we already feel confident in our way of doing things. One of the best examples of this is a language I used to write a lot in is Elixir. Elixir is based on Erlang, which has been powering telephony systems for older than I am. And I'm getting pretty old at this point. So this is a highly full-tolerant language made for distributed systems. And that become more and more relevant nowadays because as we've come from personal computing backup to the cloud where we need full tolerance and distributed systems, it's an almost perfect candidate. But it didn't really have any adoption until someone wrote a new way of doing Erlang called Elixir. Elixir was made to look a lot like Ruby. And that alongside with the WhatsApp acquisition because no one cared about Erlang until WhatsApp was bought for $19 billion. Then I thought like, "Wow, maybe this is the way of getting bought out by a large company." But that aspect aside, it made it much more approachable. And of course, there's a really huge amount of people out there who write in Ruby, and they wanted something just like a little bit better. So, Elixir had been a much easier adoption process in at least the London startup scene compared to Erlang itself, which, of course, is equally capable but just looks more alien. SHANE: Yeah. CHARLES: Yeah. MARCEL: So, I think we are only slowly getting to the point of having Reason be approachable enough. I think there are still definitely work to be done on a few key points. The story of async/await, you can do it. There's a ton of PPXs for it, but there's not one blast path. There needs to be a very streamlined documentation, like a streamlined, centralized place [inaudible] get started with projects easily where they can be effective quickly and where they can dunk on their friends on Twitter without needing that much experience. TypeScript, on the other hand, has been very popular because you can adopt it very slowly and you can essentially be using a "TypeScript" of almost none of their features or safety because you could [inaudible] enough and any is the [inaudible] of a code base. [Laughter] MARCEL: But once they have it in there, it taints the tree of type understanding because everything has a relation inside, AST. I think Reason still is a little bit too unapproachable for use and it needs to have some of these things like the async story ironed out. While the errors are generally much better than TypeScript because they're going on a sort of Elm approach. If you ever use Elm, it is a wonderful copilot for getting you to where you need to go as long as where you need to go is a web app, because it's not a generic language. And these things combined and the lack of visible momentum, because a lot of this work is done, perhaps not in the open as much as it should be, means that I think it just hasn't got the speed because people do want to use what's popular. And that's not necessarily a bad thing. But community matters and community leads to Stack Overflow posts, which leads to documentation, which leads to help and reassurance that people are doing the right thing. People are often declaring, I think people, it can be seen on Twitter. People often declare technologies is dead. For example, when some Facebook libraries move from Flow to TypeScript, they both declared that, I guess, flow is over. The reality is flow is still the majority use type system within Facebook. It's just for some very external projects, that I would love as a contributor, therefore it wouldn't be nice of them when you use TypeScript. But it continues to be developed, continues to be what they write React in, as well as nearly all of the internal things that they ship. And in many ways, Flow is a much better type system. It's not quite type sound, but it's much better. It'll force you to be correct, but the tool is not quite as good. And because it forces you to change a lot of things at once, it just was much harder to pick up as compared to TypeScript, which basically lets you get away with anything. And one might argue, what's the point then? And the point is that one can say [inaudible] TypeScript. CHARLES: The point is to be viral. MARCEL: And it's important to also note that the TypeScript integration in [VS Code] is great. I think it's really wonderful. One of the things I've always struggled with was having really resource intensive work as I work in the background, always checking files, and that led to a lot of frustration. The tooling story and documentation story and the on-boarding story of TypeScript is wonderful, even if and it may have slightly biased position of having used a large number of type systems, I think TypeScript is not quite as useful as some of the options out there. They really nailed the things that drive adoption. And ReasonML, much like PureScript or whatever else your favorite language happens to be, which probably is very good, the hard part is doing the people part, not the software part. Much like the Haskell approach, and this might be minimizing the Haskell community, so I apologize. But I have often gotten the impression that there is this idea that if a language is just good enough, people will flock to it naturally. If you build it, they will come. Which I think is not actually said on the movie, but it is the common misinterpretation, much with, "Luke, I am your father." And I think that simply isn't true. You really need to do a lot of community outreach and it can take a long time. TypeScript has been around for a decade now, but it only really gained prominence in the last three-ish years to a degree where people would use it almost by default. And a lot of these rough edges have to be sanded out with ReasonML. And if it gets dark, there's always that fear. If you're going to go to your friends and say, "Hey, we just use ReasonML," and then they hit a frustration or some strange bug or something they can't [inaudible] in Google, you're on the hook for that. So, you have to have a lot of confidence to go to your peers or to your client and go, "Hey, can we use this boutique compile to JavaScript option," when the safe choice is always TypeScript. And honestly, it may well be the correct choice for the majority of cases. CHARLES: Yeah, but I do think it builds well for the future of Reason. I think it does seem to be put together a little bit better, I mean, certainly than PureScript or some of the other ML-based languages that I played around with, just in of the marketing. I'm not saying anything about the technical capability of the language. And it is, I think honestly, the biggest problem with flow right now is that I think it might actually be dead because it doesn't differentiate itself enough from TypeScript. So, while it might be used internally in Facebook and stay alive in that way, and I don't want to spend too much time trying to conjecture about the future of these things, but at least externally, it doesn't seem like it's much of a differentiation, whereas something like Reason really is. It's something different. And that's important in of adoption in five years, 10 years, things like that. Maybe not. Anyway. MARCEL: Yeah. I think that's a very fair observation. You're not going to get anywhere unless it's efficiently better. I , back in the Angular dominant days, the only reason people has picked out React because it's significantly better. Things that are only marginally better, people won't bother spending the switching energy for. So, often with projects like [Preact], for example, just say it'll be a slightly smaller bundle size. But people aren't going to switch all their tooling or all their knowledge and all of their risk profile for something that's only slightly bit better. You have to really excel for people to take notice. You have to do something different for that technology to be adopted. And in React's case, what it could do is, unlike all the things it can knockout in Angular. It could be easily understood on a simple level to get basic things going because you have to understand the whole framework. And it was relatively performant, which was a really big problem back in the Angular 1X era, and that really differentiated it. CHARLES: Yeah, I know. It's the quantum leaps that end up being able to adopt from zero. Otherwise, it's all incrementalism all the way. So, I wanted to talk a little bit about the experience of working on an actual code base with Reason and solving certain levels of problems with it and things that might be particularly suited or ill-suited for solving the problems. Shane, you actually have been developing a program called repkgs, which we got a little bit of a preview in kind of one-off way where we were going to be releasing a bunch of JavaScript packages on the world. And you ran this and you uncovered a whole slew of dependency issues with our packages that would have caused them to break out in the wild. And it was very, very helpful. And I wish we could honestly run it again and again as part of our CI pipeline. So, you've got to release it at some point. I don't know if you have in the interim, but maybe you could talk a little bit about what repkgs is and the choice to use Reason there. SHANE: Sure. Repkgs is my take on making a tool for not just monorepos but focus on monorepos, making a tool to help validate your packages before you publish them. And the focus is definitely on monorepos, but it also works just like a normal single repo. This start off as just a general monorepo tool. You can run actions. You can get a list of your packages. But there are so many tools out there. I didn't want this to be a competition for, say, lerna or yarn workspaces because there's enough competition in that space. And I didn't see a ton of competition in the space of, is what I'm about to publish going to actually work. Because most tests run against source files and there's almost no common workflow that builds the package as like an artifact and then tries to install it and then runs new tests. It's just my source work. Your source files work, that's great. And you compile them, say you're using TypeScript or even Babel, you're compiling often into a disk folder or from TypeScript to JavaScript. So, the extensions are changing. There's so many little changes that happen between a moment of my tests say this is going to work and then it actually going into npm. So, this is false in that space where it tries to tell you right before you hit publish, run these checks and then confirm that what you're about to do is what you think you're going to do. And this came about because previously, like five years ago when yarn had just come out, I had made another monorepo tool called knit, which was very magical. And it basically ran through all of your code to find all your dependencies for you. I think it's still a pretty cool project, but it competed directly with lerna. And with lerna, it was such a behemoth, had so much backing. I didn't want to try and do that again, even though I think there's some ideas in it that were really powerful. One of those ideas was walking through source code to determine the shape of the project and what dependencies are needed. And so, I took that idea. I think that was the interesting idea that knit had, and I just kind of focused on that. And that's how repkgs came about. It crawls through your code, it crawls through your package manifest, and it will try and find all the things that are about to break on you. That's just an overview of repkgs. And my reason I picked ReasonML is that was just a thing that I wanted to do. [Laughter] SHANE: Honestly. CHARLES: That's a perfectly valid reason. SHANE: Yes. CHARLES: Some might say the only reason. SHANE: I think it's the only real reason. Everything else is just to justify that. Like I said earlier, I looked actually into using ClojureScript for this, but I liked the output of ReasonML and BuckleScript because I was thinking about making some smaller packages that could be reused and BuckleScript felt like a better solution. And I worked on this project for about three months. And in that process, the first month, I used Reason as a BuckleScript. I was compiling to JavaScript and running on Node. And that process was very much a learning process. CHARLES: Wait. You were writing in Reason and then compiling to what? SHANE: Over the three months, the first month I wrote, it was always in Reason syntax but I wrote using the BuckleScript compiler to compile down to JavaScript. CHARLES: Oh, so wait. I feel like one of the things that Reason needs to do is have like a big diagram on the home page that shows the toolchain because I'm having difficulty getting my head around all the different moving parts. MARCEL: I have made this diagram several times. [Laughter] MARCEL: Maybe I should just open up PR on the home page again to put out for me. SHANE: ReasonML is the syntax and then BuckleScript is what turns that into JavaScript. But there's also Reason Native which will turn it into the OCaml [inaudible], which you can run natively. CHARLES: Okay. SHANE: So, the difference is it's not actually not all BuckleScript-ed ReasonML will compile down to Native OCaml. It's not ReasonML, and then you just flip out the output. Sometimes, you have to make changes. So, it is kind of complicated and a little confusing. Basically, the first month I was using, it was going to run on Node. My output was running on Node. And in the next month, I had it running as Native compiled. And then the third month, I went back to Node. CHARLES: Okay. That's awesome, by the way. SHANE: Yeah. Well, I can explain why. The first month I was just learning BuckleScript and ReasonML, and that's when I was getting stuck but stuck in a way of I kind of felt like it was my fault that I just didn't know the language well enough. And I could usually see where I was trying to go. I just knew I had to figure out how this syntax worked or how do these modules work. So, I felt like I was slowly but surely making progress, except when I would run into a problem that was very Node-specific. Because if you Google BuckleScript, ReasonML, it will immediately take you to React. That is by far the biggest focus of BuckleScript is React. CHARLES: Right. SHANE: So, finding answers to my questions that were Node-related, like interacting with filesystem and how do I do this, and all the Node binding, it isn't fully complete. It has only a couple of the methods on all of them. So, anything Node-related, I found pretty frustrating to deal with. And there was a lot of me just kind of like 'I guess I'll figure this out on my own' sort of feeling. And I got to the point where I was thinking, "I'm running this locally. It's never going to run on a browser." And Reason has this cool option where I can compile to Native code. So, I rewrote everything. I didn't try and convert it because it didn't really convert the way I wanted it to because I was using a lot of Node bindings in my original code. So, I just rewrote everything. CHARLES: Okay. So, when you were outputting a Native binary from your Reason, are you literally using the operating system native libraries like, what is it like? The stdlib on Unix to get access to stderr and stdout and all that stuff? SHANE: I think that is what's being used. You do have to compile to the platform so you don't just make one binary. It's like you have to build one for Windows, for OSX. I didn't go that deep into it because when I made the transition, I was very Node heavy. And then when I transitioned over to Native, I rebuilt everything. And that process actually was not fun at all. That month was a pretty terrible month of me working on this. There are a few libraries that have been around for a while, if you want to do a commit, [inaudible] arguments or reading file system or work with the operating system. There's one guy, I can't his name, but he seems to do everything. [Inaudible] forever and they work pretty well. But there's not a ton of examples. For a language that's so old, I found it extremely hard to find working examples to help me. So when I got stuck, oftentimes I was just stuck. Just like dead in the water for days and days, just trying to work through these issues myself which I found very frustrating. One thing I do a lot of in this application is I read the file system and I use like globstars to read through directories. And all of the Node packages have this ed, like the one globstar with two globstars means it's recursive. In OCaml, I couldn't find that anywhere. So just the basic here is a path with globstars in it, and it means go through all directories. That wasn't a thing that I could do. It just wasn't there. There's an OCaml-re which is regular expressions and it had some path stuff but it didn't that. And that was like the main one that people use. It didn't it. There's another tool called Dune, which is their package builder. And I had to use their library for a while, but I don't think that's ed in the same way. I basically had to build my own, and that was not the best. I don't want to have to build my own regular expression parser. And basically, it's very frustrating. MARCEL: This is actually an endemic problem inside all the OCaml communities because OCaml has a bunch of factions. Not that they're at war or competitive, it's just they have their own kingdoms in which they find OCaml resource. Some of them are very academic, like this team in or very financial like Jane Street, who has their own library for basically everything. One of my friends, [Rizo] is a huge OCaml fan. He is the more competent technical part of the Reason London experience, while I am the razzle dazzle. He has continuously talked about the things that there are four to eight standard libraries and there isn't a community centralized documentation system and it's all a bunch of IRC servers where people in the know talk to each other and decide what's going to happen with the language. And this is definitely something that hopefully Reason can help with, because it will highlight the fact that if you want mass adoption or adoption by more people, these community items are required because very few people will power through frustration in the same way that Shane did. SHANE: Yeah. I made a decision that I was going to figure it out. And I, multiple times, just said I am done with programming forever, and I just closed my computer and walked away for a day. I couldn't imagine anyone doing that for work though. CHARLES: But you went back then to Node. SHANE: Yeah. CHARLES: But at the same time, this is kind of how like the reason we have that in Node like the [inaudible] is because back in 2009, that's what the Node ecosystem was. If you wanted something to actually recurse through a directory, you had to build all that yourself. SHANE: Yeah. That's the happy ending to the story is when I went back to Node, I actually understood the language a lot better. And I was able to pretty much solve all the issues I had that first month. Instead of learning the language or learning how to check the file system or getting frustrated, I could just use the types and really build out what I was trying to express using modules, using types. And the rebuild when I went back to BuckleScript was a really pleasant experience. That really sold me on BuckleScript and ReasonML. MARCEL: [Inaudible], we have all the knowledge you need to tackle the enemy you had a problem with it the first quartet of the hero's journey. SHANE: Yeah. CHARLES: It's a validating story because if you are able to spend the work to learn how to make the hard thing easy, then you can do the easy thing because that was hard previously. Doing the hard thing is now easy. MARCEL: I think also, to Charles's point, there is now a library called reason-nodejs which came out fairly recently that attempts to, I think, tackle many of these Node problems. It even has a file system example on the Read Me. You may be correct that a lot of these frustrations will eventually get solved by people writing packages that we then just consume and don't think twice about. CHARLES: Yep. All right. Well, I think we're almost there. Gentlemen, it has been a journey. I actually am pretty excited. There's a couple of things that I want to try now. As you guys know, I've been fooling around a lot with structure concurrency and knowing that the async/await story isn't quite pinned down in Reason makes me want to try and implement a more general structured concurrency story there. Just give it a stab. And in knowing that there are these PPXs that you can extend the syntax is really exciting because one of the frustrations of trying to do that in TypeScript has been you're very limited in the syntax that you can do. You've kind of got to work with what you've got. And the typing around generators and co-routines is just not great. And it's not going to get better unless you can convince the people at Microsoft to do something about it. But it sounds like there's actually a way forward, which is ultimately the power of having these syntax transforms, which is exciting. So, if I ever can find some spare time, I am not even lying, I want to dedicate some cycles to this. MARCEL: There's a library called bs-let, which is a PPX that tends to make a very general, almost monadic approach to awaiting stories in JavaScript, which is worth a read if you are interested in that sort of thing. CHARLES: Unfortunately, I am. Oh, and they've got letPlus and letStar. All right. Anyhow, thank you all for stopping by. Everybody, go out there and Reason away. And we will see everybody next time. MARCEL: All right. Bye for now. CHARLES: Thank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at @frontside.io. Thanks, and see you next time. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

SPECIAL GUEST:\r\n\nPete Gonzalez | @octogonz\r\n\nDuring the day, Pete works at HBO in Seattle on their streaming media apps. Prior to that, he was at Microsoft for 9 years, and before that, he worked at various consulting companies. A long time ago he was a cofounder of Ratloop, a small company that makes video games.\r\n\nSHOW NOTES\r\n\n01:24 - Rush.js: What is it and what is it for?\r\n\n\nRush on GitHub\r\n\n\n04:47 - Problems with Managing Large Codebases\r\n\n\nRush Stack: provides reusable tech for running large scale monorepos for the web\r\n\n\n07:22 - How does Rush provide a solution for build orchestration?\r\n\n13:34 - Rush Stack Opinion: How to Lint, Bundle, etc.\r\n\n16:53 - Using Rush Stack: Getting Started\r\n\n\nThe pmpm Package Manager\r\n\nYarn Plug’n’Play\r\n\n\n24:27 - Getting Technical About Versions\r\n\n\nPhantom Dependencies\r\n\nDoppelgangers\r\n\nPure Dependencies\r\n\n\n32:47 - Thoughts on Monorepos\r\n\n36:30 - Getting Started (Cont’d) + Efficient TypeScript Compellation\r\n\n43:28 - Does Rush have a size limit? Is it for bigger or smaller projects? Both?\r\n\n44:34 - Using pieces of Rush in non-Rush projects?\r\n\nTRANSCRIPT:\r\n\nCHARLES: Hello and welcome to The Frontside Podcast, a place where we talk er interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer at The Frontside.\r\n\nIt\'s been a really, really long time since we\'ve gone to talk to everybody. We\'ve been heads down working on projects, working on some products, working on a lot of different things that haven\'t given us the time to do much podcasting. However, one of the things that we\'ve been doing lately is converting all of our projects over to using monorepos. And so, we\'ve been very interested in the tooling that comes with managing more complex code bases that might contain a single application or contain a single package. And so, we have been evaluating lately these different tools. And one of them that came to the fore was Rush.js. So today, on the topic of this tool in particular, but I think also just in of managing larger and complex code bases, we have Pete Gonzalez on the podcast to talk with us. Hey, Pete.\r\n\nPETE: Hi.\r\n\nCHARLES: And today, also, we\'re going to be talking with Shane Wilson, who is the author of repkgs and a trusted voice here at Frontside, certainly when it comes to build tools. Hey, Shane.\r\n\nSHANE: Hey.\r\n\nCHARLES: So, yeah. Why don\'t we just dive right into it and talk about what was the particular set of circumstances, the particular set of problems that gave rise to Rush. What is it and what is it for?\r\n\nPETE: That\'s a good way to think about it. It\'s very much characterized by its beginnings. So, it started, I want to say, around 2015 when I was working on Microsoft SharePoint and we basically -- I won\'t go into too much detail, but we had a whole bunch of Legacy C# server code and the business has decided they wanted to move it into the browser as client code. So, lots of teams with this huge code base just started rewriting C# code as TypeScript. And the model, at the time at least, was you have npm packages that are your units of components and then each npm package goes in its own git repo.\r\n\nSo we just had a bunch of developers, who had previously actually been working in a monorepo, go and start creating repo after repo and then publishing packages and sharing them with each other. And we ran into a lot of problems with that, which today people kind of understand the value of monorepos for TypeScript and Node.js projects. But at the time, there was a lot of assumption that while all the npm packages that you use off the Internet, they\'re all kind of developed that way. But we ran into problems where somebody would make a change and then somebody downstream would get broken or things wouldn\'t upgrade. And we eventually realized that we needed to bring the projects together, at least for the team that I was working on. And at the time, there wasn\'t any Yarn or Lerna, these projects that people are familiar with today, we really didn\'t have any options. In fact, npm itself was like a new thing that was displacing Bower. So, a lot of stuff that we take for granted today in the Node.js community at that time, at least within the group that I was working and nobody had been very familiar with it.\r\n\nSo, we basically started with just a small tool that would let you cross-link project folders together and then it would work out which project needs to be built first. So, you build your libraries and you build the application. And so, it was a basic build orchestrator.\r\n\nAnd then out of that, basically, people started coming together into one repo and then we encountered various other problems, such as publishing workflows and stuff like that. And so, Rush sort of came out of those beginnings. But one thing that makes it a little different from the other tools that have come along is that it was done at a larger company. A lot of people, if you\'re a very prolific coder, you might make a thousand files or something and split them out into packages and then have something build them. But at the end of the day, it\'s still you or maybe you and 10 other people that work on your code. So, you can be fairly flexible. And the code tends to be fairly well-behaved because everybody understands it. Whereas what happened on the project I was working on, it was more like you have like 50 people and then 100 people and then 150 people all working in this big common space. And you encounter a set of problems that really impact your ability to scale up. And that\'s the direction that Rush has sort of taken. As these other solutions came along, we\'re kind of like, "Well, maybe we should stop using it and switch to something else." But we decided to make a more new tool that focuses on those problems of large code bases and large teams.\r\n\nCHARLES: Yeah. Maybe you could talk about, in specific, some of those problems.\r\n\nPETE: Some of them are technical problems. Like you just have a lot of code and the builds take a long time, so you need parallel builds and incremental builds. And lately, we\'ve been talking about multi-phase builds just to break some of those down. Parallel just means that you start up multiple Node.js processes that build stuff simultaneously. But there\'s also been work recently on sharding. So, you can distribute the builds across multiple VMs rather than just having it run on one VM. Like for incremental builds, with Node.js, there\'s a problem that every time you run a command in a given folder to compile a project, you have to spin up Node.js and then eval a bunch of toolchain scripts. So, for example, Rush\'s incremental build logic does all of the analysis from the build orchestrator, so that Node.js processes already running for Rush can determine whether a folder needs to be built or not. And even if that saves you maybe one second of time versus if you had gone and run the toolchain on that particular folder, if you have like 200 projects, that could be 200 seconds of time that you save.\r\n\nBut there\'s also human aspects of it as well. For example, you have people of different skill levels coming in and working on the code. So, you need people who maybe are just coming to that monorepo for the first time to fix a bug and then will go on somewhere else, you need it to be a good experience for them to learn how the tooling works. And it shouldn\'t be like a bunch of tribal knowledge that they need to learn how to actually turnaround a pull request. There can be political issues. So, there\'s like the bad team that doesn\'t have as good quality bar and keeps checking things in that make flaky tests or problems that break the build for other people. So, the other teams start instituting stricter policies, like stricter eslint rules, stricter code reviews, sign offs and stuff like that. But then it creates a separate problem where now people are complaining that the ecosystem is too bureaucratic.\r\n\nSo, this led eventually into Rush Stack, which is a more recent thing that we\'ve been doing where we take the idea is Rush itself is a build orchestrator that doesn\'t really make a lot of assumptions about what your toolchain is. And then the idea behind Rush Stack was to start bringing in other pieces of the toolkit and saying, "Well, if you want to work more the way that we work, you can start using these other pieces of Rush Stack like API extractor and the bunch of web pack plug-ins and things like that. And it would include a bunch of best practices and stuff.\r\n\nCHARLES: So, if I could ask a question, and this is on a much smaller scale, but I\'m just kind of keeping in mind as I\'m listening to you talking some of the monorepos that I work in personally. And one of the most annoying things that I think you actually touched in with the build orchestration was if I have, say, 15 packages, 15 npm packages in this repo, and there\'s a web of dependencies between these packages, and I\'m working on one package and that\'s where my focus is. But I realize that I need to make a change in another package. Now, the minute I start straddling development on two packages, I have to , "Oh yeah, I need to go run a build. Whenever I change a source file on this other package, I need to run a build in that package." Then I can come back. And oftentimes, I will forget to do that. And I\'m like, "Why is this not working?" And it\'s like, "Ah, I made the change in the source file. I need to compile the TypeScript and bundle it into an npm package distribution so that I can consume it from the library that I\'m actually working on." And so then, what I find is now I\'m making custom little hacks to do like file watching at the top of the thing, at the top of the the repo that looks at all the packages and incrementally compiles them whenever there\'s a change detected. So, how does Rush provide a solution to that? How would this build orchestration work? Is that a use case that it could solve?\r\n\nPETE: The watch scenario is definitely an important scenario. Rush itself today doesn\'t have a great solution for that. Basically, you run watch in the project that you\'re actually serving. For example, you have a frontend web application and then you have various libraries that it depends on. So today what we do is you just run watch on the frontend web application. And then when the library is changed, you can use Rush to say, compile everything up to this project that I\'m in right now and it will detect which things have changed so it can skip the ones that don\'t have any changes. For example, this is an area that we\'re very interested in. It hasn\'t been a top priority, I guess.\r\n\nOne other interesting aspect of it is so when we first developed Rush, it was just something I was doing on the site. I wrote the original prototype of it myself over Christmas vacation, I think. And then it became something that people were using. People got excited about, more people started having feature requests. Eventually when I was at Microsoft, there was a whole team of people who started just managing the tooling for the monorepo. And when you have a lot of people, that\'s actually one of the selling points for the monorepo, is the more developers that you get working together on one place, suddenly the more budget you have to have a team of people go and build really good tooling for it. So, it\'s not just like each repo has one guy who spends five hours a week putting out fires. It turns into like when I left, we had something like four or five people who were full time just working on tools. And there was a ticketing system where if you got a weird build error or some tests were unstable, you could actually open a ticket. And we had an on-call person who would investigate it. And then we collected statistics about how many tickets we got in different areas and did root cause analysis and stuff. I\'m not HBO, but HBO kind of has the same thing. They have dedicated tooling teams that work in the big monorepo and it\'s really kind of good.\r\n\nBut as far as Rush, it was never like a Microsoft product that they were selling or pitching. The website is a little bit like a sales pitch, but I wrote most of it myself. I even did the artwork myself. All that is to say that the development of features in Rush is a little bit slow because it\'s kind of a thing that people do on the side as things are motivated for them. It\'s not like complete. And Rush Stack is sort of the same way. We had been building random packages and tools for a few years. And then it was a year or so ago, we started thinking, "We keep putting all these things together in one monorepo on GitHub, but what did they really have to do with each other?" And we came around to the idea that the real value of them is that they work together. And JavaScript in general is really into flexibility. I always tell people, "If we\'re going to do a project in Java or C# or something like that, you just install the language, like framework, and you often get an editor with it and then you\'re testing and linking and bundling in optimizer and linter. All those things just come out of the box and somebody has put the entire end to end system together for you, and it just works. Whereas with JavaScript, it\'s like every step of the way, there\'s two popular linters for TypeScript. If you don\'t use TypeScript, there\'s Babel. If you do use TypeScript, you can use TypeScript and Babel together. If you want to run tests, there\'s Karma and Mocha and Jest. And even the test assertion library and the test runner can be made by different people.\r\n\nWe think of it almost as a vector where every single piece of it, like your linker, is made by a different group from the compiler. And you can choose between Rollup or webpack for your linkers. Every step of the way, you have to make a decision. You have to read the blogs and find out what is popular and then make your bet and say, "Oh, we like Jest because that seems to be popular today and that\'s what we\'re going to use." Well, we made bets like that five years ago on Gulp and things that aren\'t very popular now. But when you start doing large scale development, once you make that bet, it\'s hard to change it because you have all this stuff that starts getting built up around it.\r\n\nSo, part of the idea of Rush Stack was let\'s have a bunch of people who are professional developers and who have experience doing tooling, make some bets and then decide on a set of stuff that should work well together. And then we\'re going to collaborate together with our community from open source. I now work at a different company, but I still am very involved with it and bring a different set of perspectives. But the spirit of it is we\'re all working together to try and over time build something up that is similar to what you\'ll get in other languages.\r\n\nCHARLES: Is the idea then that the Rush Stack has opinions about how to lint, how to bundle, how to do all these things?\r\n\nPETE: And part of the opinion is, we don\'t have time to develop a bunch of different options for things. We just want something that works. We don\'t really want everybody to choose their own flavor of ice cream because we don\'t have time to do that. But there\'s another aspect of it, which is that when you lock in on something, you can make optimizations that aren\'t possible when there\'s abstraction. And I can give some concrete examples of that. Like when we started with Gulp, the idea of Gulp was that you model tasks and then the tasks have dependencies between each other. So, TSLint, which was the linter we were using at the time, was a task and then TypeScript was a task. And the task was like this Gulp wrapper around the actual library of TypeScript and TSLint, and then you like wire them together. So first, we run TypeScript and we run TSLint. And the idea was you make these boxes and arrows like a big data flow diagram of how each task gets its turn on the files and the way things move between the boxes were streams of files or partially processed file buffers. And we ran into problems with that because, for example, TSLint needs to do a TypeScript analysis. So, it invokes the compiler engine to study the code and figure out what types or things. But the TypeScript task does that, too. So you really want the same compiler engine to perform the analysis once and then to share that intermediary state between TSLint and TypeScript. But it\'s hard to do that just with TSLint and TypeScript, if you\'re just writing a simple program that invokes one after the other. You have to kind of go into some internal libraries a little bit if you want to wire that up. But then when it\'s behind a task abstraction, there isn\'t like a natural way to do that.\r\n\nAnother thing that will come up is like some people, I don\'t know if we\'re still doing this today, but they would say, "If the code doesn\'t compile, don\'t show me lint errors because maybe the code\'s not even valid. Let\'s first make sure that the compiling works correctly. And then if the compiling was successful, then show the lint errors. But I want to run the compiler in the linter simultaneously to save time." So, I need to save the error messages from the linter in a buffer until the compiler says it\'s successful, then I\'ll show those messages. So again, this is something that breaks down this idea of abstractions.\r\n\nAnd if we wanted to replace TSLink with ESLint, let\'s say, you end up with the same problem again that I mentioned, where you want ESLint\'s parser to share the same analysis as TypeScript. But the way that you hook those pieces together is going to be completely different because the entire architecture of ESLint is completely different. So, all that\'s to say that when you make something pluggable, you make a lot of architectural sacrifices or you create the need to create this rich interface that can expose the real stuff that you need to interact with something in a tight, optimized way that\'s expensive.\r\n\nSo, with Rush Stack, we\'ve taken the philosophy that we do want some things to be pluggable, but they should be conscious decisions of this is a part where there really is value in allowing different pieces to be swapped in here. And we\'ve actually tested them and we\'ve actually validated that the abstraction we set up makes sense and is workable. It\'s not like we just say, "I did one thing and it\'s pluggable, so you can drop something else in there if you want." But it\'s a dice roll. It\'s an exercise of the reader, whether it will actually work or how much work is involved with that.\r\n\nCHARLES: So, if I wanted to use Rush Stack, and I understand that it\'s in development, I\'d like to talk both about what\'s coming in the form of Rush Stack and what you would be able to use maybe today. But let\'s continue this talking about Rush Stack for a minute here. So, if I say I had a monorepo that had maybe, let\'s say, it\'s got 10 npm packages, it\'s got a couple of like backend servers and a couple of UI like frontend React apps, what would that look like in of the artifacts that Rush generated or how would I interface with that system? What would be my experience in of spinning everything up, working on an npm package, working on one of the backends, working on one of the frontends, so on and so forth?\r\n\nPETE: Yeah. So, you would start by converting the monorepo to use Rush as its build orchestrator. And there\'s like a rush init command that will dump out config files for you. We didn\'t put a lot of thought into the onboarding experience for a long time because when you have a monorepo, you only set it up once and then you never think again about the setup experience. And that was when we made Rush open source and then started trying to build a community around it and get people to use it and maybe contribute back to it, it took us a long time to realize that other people had a hard time setting up Rush at all because we never went through the experience of setting up Rush. We\'ve been working in the same five monorepos for years. But when we went and did some investigation into that, what we found is that we like the idea of having these big config files that have json schemas that make sure you have IntelliSense and error checking when you\'re editing them and they have big blocks of comments around everything. So you run Rush in it and it just dumps these big empty config files with comments. And then you go uncomment the parts that you want and add your stuff. We try to put as much prioritization of rush in config files versus like command line options or environment variables or other places where there\'s less rails or guidance of what you\'re supposed to do to set it up. So anyway, that would be the first thing you do is generate the config files and then go all of your projects so that Rush builds them.\r\n\nYou probably would run into a little bit of work that you have to do just to get it to build with Rush, because Rush\'s way of installing npm packages is a little bit different. We mainly use the pnpm package manager. This is another really interesting area of JavaScript is they have like three or you could say four leading package managers. There\'s npm, the original one, then there\'s Yarn Classic, which is a ground up rewrite of npm that adds workspaces and some new features, but keeps the same models npm. Then there\'s pnpm which came along and solves a technical problem with the way npm installs stuff. We refer to it as doppelgangers and phantom dependencies on the Rush website.\r\n\nCHARLES: Actually, this was a very interesting point. Could we take a detour here to talk about phantom dependencies and doppelgangers, because it\'s something that\'s unique, I think, to the JavaScript ecosystem, which I hadn\'t encountered before working in other languages. Could you just unpack that just a little bit? I know we could spend half an hour talking about it.\r\n\nPETE: Let me finish my list of package managers for a minute, then I can dive into that. So, I said there was the original npm. Then there\'s Yarn classic and there\'s pnpm, which actually changes the installation model but using symlinks. So, it solves some problems that we\'ll talk about in a minute by creating a whole bunch of symlinks in your node modules folder. And it is a correct installation that eliminates those problems. And it\'s mostly backwards compatible because it still preserves the original Node.js module resolver model. And then there\'s Yarn plug and play, which is essentially a completely different installation strategy from the original Yarn, but its approach is that it changes the way Node.js resolves packages through this plugin. And then it\'s cleaner, it doesn\'t have to create some links at all. It doesn\'t even have to create node modules folders. It\'s more like how Ruby gems, for example, work. But it has more compatibility issues, because there\'s a whole lot of code that assumes it\'s going to go walking through node modules folders looking for things. It can be stuff like web pack. It can be just people\'s random packages that go and resolve things. But we encountered some compatibility issues. We feel like Yarn plug and play is like the future, it\'s like a better design, but it could take a while before a lot of people will be able to move to that.\r\n\nLet\'s say you\'ve got those and then there\'s Rush, which predated most of them. Rush was designed around the original npm package manager. So, it introduced its own symlinking strategy that is somewhat similar to what pnpm does today. Even now, today, when you use Rush with pnpm, it\'s actually Rush\'s symlinks retrofitted onto pnpm\'s model. And we have a PR that\'s been open for like a month now that actually finally lets pnpm completely manage the installation when you\'re using it and gets Rush out of the business of package management, which we\'re really excited about because it\'s complicated problem. And we have had like just tons of work that we\'ve done over the years to that and make it work correctly.\r\n\nBut let\'s rewind a little bit and talk about why all that exists, because it is another big aspect of working in a large-scale repo.\r\n\nCHARLES: I understood reading from the documentation that Rush will or will not work with other package managers besides pnpm?\r\n\nPETE: It s npm and Yarn classic. The is not great because almost none of the Rush developers or main s actually use this. Pretty much everybody is using pnpm. We\'ve been wanting to improve the for the other package managers, but our plan to do that is to basically let them just do the installation.\r\n\nSo right now, for example, when you use Yarn -- like Rush\'s original strategy was it looks at all the projects in your monorepo and then it creates this fake project that has the dependencies on the superset of all of your monorepo dependencies. And then it runs the package manager wants to install that folder and then it goes and creates symlinks for each project into that centralized folder. And there\'s some technical reasons why that is a good strategy. But when you do that with npm and Yarn, every time that they change things or release new version, that changes some semantics, some aspects of that can break and then you have to go debug it. I can say it works with the version of npm and Yarn at the time when we were last using that. But it\'s had problems over the years. And so, the long-term strategy would be to do the same thing we\'re doing with pnpm. Hopefully, we\'ll do it this summer. But say, if you want to use Yarn with Rush, then Yarn just installs the way Yarn installs and you get all the other Rush features but Yarn manages the installation and that means you lose the phantom and doppelganger solutions that Rush was doing. But the idea is maybe you didn\'t want that. The reason you\'re using Yarn is because you want Yarn\'s installation model.\r\n\nCHARLES: I see. So, it really is, for example, one of the things that I was wondering is what\'s the overlap between, say, using Yarn workspaces with Rush. And kind of what I\'m hearing is that it really is, you\'re choosing one or the other. Because that\'s kind of when you\'re working with Yarn workspaces, you\'ve got some virtual package that aggregates all of the packages and stores at least some of the dependencies in one place. And Rush is doing the same thing.\r\n\nPETE: Let\'s get a little more technical that versions.\r\n\nCHARLES: [Laughs]\r\n\nPETE: So, npm has, I guess I can say, two big factors that got us into this space to begin with. One of them is you just have way more packages than usual. So, for example, if you use, I don\'t know, C#, which I\'ve used before, it would be common that you have 20 libraries that you install that are like dlls or something that you get from open source or you buy them from a commercial company and that\'s your dependencies. Whereas with the standard Node.js project, notoriously you can have typically more than a thousand npm packages getting installed just to do some basic stuff like, build an application and not do anything particularly weird. They just have an approach of having lots and lots of small projects that all have dependencies on each other. So out of the gate, npm has a lot more complex version management issues than languages that have less libraries that tend to be bigger and more monolithic libraries, like .NET framework, for example, which just when you go to use C#, you get this massive library that comes from Microsoft that solves a whole bunch of problems, than with npm, each one will be solved by some individual package.\r\n\nBut then there\'s another aspect, which is the way that npm installs packages is -- I have to be careful talking about this, but it\'s flawed. When you have packages that depend on other packages, they form what\'s called in computer science, a directed acyclic graph. Basically, it\'s like a tree that branches apart, but then it can come back together again. But the installation strategy of the Node modules folder is that we\'re going to use folders on disk to represent the dependencies between packages. So, if A depends on B, C, and D, those become subfolders of it. And then when you have a diamond dependency, you run this problem that the file system on a computer actually is a tree. It\'s not a directed acyclic graph. You\'d need symlinks to make it into a directed acyclic graph. So, there\'s a whole write up on the Rush website that tries to give examples of the technical details.\r\n\nCHARLES: It really isn\'t npm\'s fault. It\'s actually kind of the original sin was committed by Node in the way that it requires modules. Like, that\'s where it actually happens.\r\n\nPETE: I think they were developed together.\r\n\nCHARLES: Oh, okay.\r\n\nPETE: Actually, I shouldn\'t speak about it because I wasn\'t there. I don\'t know the original history of how Node.js and npm came together and agreed on how they\'re going to interoperate with each other. But if you look at it at the time, it was probably completely sufficient for the problems they were solving. And like Bower, the thing that they were competing with, doesn\'t allow libraries to depend on libraries at all. So, it\'s kind of an innovation that you could have these big dependencies of things. But basically, it introduces certain problems like the phantom dependency is the most obvious problem, which is they do this thing called hoisting, where if several things all depend on a package, they move it up to the top. And then if you can\'t find the thing you\'re looking for in a subfolder, you just walk up the tree and find it somewhere higher in the tree. And that creates this phantom problem, which is that a package can require things that it didn\'t declare a dependency on. So, it doesn\'t appear in your package JSON, but it is findable somewhere in the node modules folder. So, Node.js doesn\'t even look at package JSON files when it\'s resolving. It doesn\'t look at dependency declarations when it\'s searching for things. So, you can require things that you never declared and that can cause problems because a lot of times, the thing will be there because of some aspect. It\'s like, I depend on this other package and it depends on something. So generally, I can require that thing and have a somewhat good guarantee that it\'s going to be there. And I might never even notice the problem. But I don\'t have any say about what the version of it is because I never declared it. And it can get disturbed because it\'s the package manager chooses to install things differently than the version or the thing that I was expecting might not either, or it might be the wrong version.\r\n\nAnd then doppelgangers are an even more interesting case, which is when you have a diamond dependency. I\'ll try to make it concrete. I can\'t think of the package names off the top of my head. It\'s like A depends on B, which depends on this D library. That\'s our diamond. And then A also depends on C, which depends on D. You can get a case where B and C need two different versions of D and npm models that they install the package twice, so you can get multiple copies of it. And because of the tree, there are situations where you can actually end up installing five copies of the same version of the same library. So, it\'s not even just that we have side by side versions, it\'s like the same version has to be installed in multiple folders because without a symlink, there\'s no way to make the folders end up being --\r\n\nCHARLES: Whereas in other languages, you would get an error at build time saying, "Hey, I cannot find a single version of this library that satisfies the dependencies of this project." And you have to deal with it manually at the build time.\r\n\nPETE: Or they would say -- Ruby, for example, will just install that version in one place. And then the way you find it is when you require a library, the resolver just looks in your dependencies and says, "Oh, you need version 1.2.3. Okay, I\'m going to go look in that folder for it." But with npm, it\'s like, "Well, the only places I can look for it are in certain places that don\'t know what the version is." It means that the package manager has to copy the same version into different folders and you can [crosstalk] your tree. But if you think about it, say I have a singleton, then now I have two singletons getting spun up because Node.js doesn\'t actually understand that these two folders have the same library in them. It sees it as two entire instances of it. Where the TypeScript compiler, we had a longstanding compiler error that we were dealing with when we first set up Rush monorepos, which is the TypeScript compiler would see a declaration of a class in two places and it would assume that they were not compatible declarations. It has the same name, but it\'s a completely different copy of it. And we eventually got them to relax the rules a little bit and say, "Well, if it\'s in another package that has the same name as this package in the past relatively within there they are the same, then you can reasonably assume that it\'s the same thing and then disable your check there." And it took like a year to get that finally fixed.\r\n\nSo, this is like one aspect of the problems that\'s like really obvious. But there\'s another aspect, which is just the management of versions is complicated. npm has this thing called pure dependencies. In a monorepo, I personally have probably spent months of my life just helping people sort through problems of I\'m trying to upgrade this package and now my installation fails or I upgraded the package and now the application doesn\'t run because something got jostled around in the Node modules folder and now the versions aren\'t exactly right. And when you go into a monorepo, the problems really multiply. In a small repo, which is where the vast majority of npm s work, you just don\'t have enough different people trying to install different versions of things to really hit these problems. But as more and more people come in, like the Rush Stack repo has something like 10 versions of the TypeScript compiler getting installed for various reasons. And then there are things that have pure dependencies in a TypeScript compiler, so you get sort of complicated version problems and that\'s where Rush\'s features came up. It has special protections against phantom dependencies and doppelgangers. It has weird concepts like preferred versions and allowed alternative versions and special checks to make sure that the versions are in sync. And you can have Rush check things for you.\r\n\nSo that\'s just an area that we became really interested in because we had a lot of frustration ing it. And the big company, when you\'re shipping production software to the cloud where releases go out every week, you really need things to be deterministic and reliable. So, we put a lot of engineering and thought into that. Whereas a lot of other people were like, "What\'s wrong with it? Works fine for me. I don\'t understand. If it\'s a problem, go hack something in your package JSON file." We weirdly over and over again had conversations with people who couldn\'t understand what we were trying to solve or why Rush was preventing them from acquiring something they could see there.\r\n\nI\'ll mention one other case that this came up, which is when you have multiple monorepos. People are always wanting to create more monorepos and then they\'re like, "Well, it\'s a great separation." You have all of your libraries in one monorepo and then your app in the other monorepo and then you just automatically publish things and automatically upgrade in the downstream repo.\r\n\nCHARLES: So, you advocate or you kind of just monorepo all the way. Just keep doubling down on one codebase. It\'s always better if you\'ve got a huge block of code to add it to the existing repo?\r\n\nPETE: Well, I would say technically it\'s easier to manage when it\'s in one place because you can do things like a pull request that updates 200 projects easily at once without having to worry about publishing and upgrading stuff. But there are political reasons why you would want things separated. For example, at Microsoft, we had an internal repo that was closed source and then an open source repo in GitHub that was open source. So, at the very beginning, out of the gate, our toolchain had to be in a different repo from the application development. So that\'s like a valid case. Where there can be business groups that really are separate from each other.\r\n\nTo me, one of the real rubrics that you can use is about the contract. So, if you have, take React, for example. There\'s a group of developers that make React and they ship it, they document it. They have like release notes when they release a major new version, they make sure that they test the contracts for it. So, they\'re shipping a contract for people to consume. And there\'s a group of people around making sure that that contract is stable and well ed. That kind of boundary totally makes sense to have it within its own repo. The thing that really didn\'t work for us was the sharing code where we\'re not really shipping it. It\'s more like we\'re sharing it. So, there\'s some url parser that somebody makes and then another team wants to use it and some other teams have some libraries. So, various teams start putting code into different libraries into some shared repo that they\'re going to consume. But nobody really owns it. And you get this problem where somebody goes and makes a change to the thing they\'re using, but they\'re not going to go and test the other downstream repos. So then when the downstream repos upgrade, they might get broken by that change. And then it\'s like, whose responsibility is it? That guys should have tested everybody else? Or these people just need to pay this cost. And we had teams, when we were working in this model where it was like every week, somebody would have to go spend like their afternoon getting the compile again because the upstream repo that they were consuming packages from just kept constantly having churn and changes.\r\n\nBecause at a company, it\'s not like npm, where somebody just kind of makes a library and once in a while, they update it. It\'s like there are people actively changing stuff every day because they\'re trying to get a feature shipped. And these aren\'t just really isolated libraries. They\'re all parts of the anatomy of a big application. So, there\'s a lot more churn. And we tried, "Oh, let\'s have it automatically upgrade. Let\'s make it so that when somebody goes and makes a change, we run tests in in the other repos and tell them that it\'s broken." Or a really common one is let\'s make people review the contracts. Everybody needs to follow a semver and they need to design their APIs to be forward thinking and stuff. But all those things that you look at, you\'re like, "Well, React did that. Let\'s the impose that on our developers working in this shared repo." The problem is that there\'s no budget for that. Nobody\'s manager is going to let them go spend a day having a meeting and deliberating about a design change to some url parser, because that\'s just not the business that they\'re in. Nobody really thinks that that is a shipping thing.\r\n\nSo, if you don\'t solve that problem, what you end up is people just stop sharing code. It\'s easier just to fork the library and have our own version of it and stop trying to share it with the other team. The monorepo really does solve that problem because it lets everybody work together on a shared library. And if you make a change to it, you can\'t merge or change unless all the unit tests for all the things that are affected by it. And when stuff fails, you can easily just go fix that stuff up in the same PR that introduces the breaking change.\r\n\nCHARLES: That is a really, really interesting point. Never thought of it that way. So back to the question of getting setup, getting started with Rush.js for this hypothetical monorepo that we have 10 npm packages that are written in TypeScript, maybe some are written JavaScript, some are written in TypeScript and then a backend and a frontend. What\'s the next step for getting the build orchestration going?\r\n\nPETE: Rush kind of checks out at the point where we need to build a project. At that point, it just runs an npm script in that folder. And then it\'s up to your toolchain to figure out how to build the project. And you can have different tool chains in each project folder. Rush actually discourages you from having any global stuff. So, for example, our ESLint rules, we don\'t have an eslintrc file at the root of the repo. Each individual project has its own lint setup and you can even be running different versions of lint in different project folders, which sounds not good. But when you have like hundreds of projects, sometimes not all the teams can upgrade ESLint at the same time, so you need to let people be able to have differences in their tool chains locally. Prettier, we did decide with prettier to run that globally because it\'s more of a cosmetic thing. It doesn\'t have a lot of breaking changes, and it\'s something that kind of needs to run on the diff of what you\'ve changed when you have like a git commit hook. But other than the one exception of prettier, we pretty much have the toolchain be self-contained for each individual folder.\r\n\nCHARLES: I thought one of the selling points also was incremental building and shaving build times off of that. So how can you do that by having each kind of package contained inside the repo being responsible for managing its own build?\r\n\nPETE: Well, the toolchain is shared, right? So, the first thing that you would do in Rush is make a toolchain project, a project that actually is the code that compiles things. And we usually put shared configuration in there, like your tsconfig for web applications or your tsconfig for Node applications. With ESLint, I went through a whole thing when we switched from TSLint to ESLint, where ESLint actually doesn\'t have the ability to load plugins from a shared configuration folder. It tries to load the plugins from the actual project folder. So, we have this work around that we made and we have an open PR, we\'re trying to get ESLint to have better for monorepos. But basically, you want to move, you want each project to just have a dependency on, we actually call the rig. So, the idea is the tool chain is like this shared code that is reusable scripts for invoking the TypeScript compiler or web pack or whatever. But then the rig is like a flavor. So, you would have a rig for a web library or a rig for a web application or for a tooling application and things like that. So, these stereotypical kinds of projects usually get a specific, like a toolchain package that has all that stuff rolled up in one bundle. So, you just kind of will have as little copying and pasting of boilerplate between projects as possible.\r\n\nCHARLES: Let\'s say I had some rig then for compiling TypeScript npm packages. How would I, for example, I don\'t know, use the same cache to the TypeScript assets or keep like a TypeScript compiler hot, so that, I don\'t know, like the syntax tree parse or something, like this is one of things I noticed in the build is like TypeScript takes a long time to compile.\r\n\nPETE: That\'s true.\r\n\nCHARLES: And if there was some way that when a tiny source file changes, you could run a command and it would just compile that one file and put it in the disc directory is that just I need to do that myself, and I can benefit from having this single rig that\'s in a single place so that any project that\'s going to be using that can benefit from it, which I think is good. But is there also, does Rush provide some sort of solution out of the box or sort of maybe some building blocks so I can build that type of solution out of the box?\r\n\nPETE: Rush doesn\'t. And for a long time, Rush Stack didn\'t either, because we\'re using Gulp and we really -- Gulp as a mission, is kind of an older, less popular solution now. And the whole approach that we took with Gulp, we regretted and we kind of wanted to rethink for many years, but it worked and it was like a component of our setup that worked reliably. So, we just were using the same Gulp toolchain for years and years. I\'m now at HBO, but recently now Microsoft has been developing a new modernized replacement for that that\'s called Heft, and they\'re in the process of moving that code in -- they had done an internal prototype of it. And now they just opened the first PR this week to start releasing that as open source. But that would be like a toolchain that is more modern, that actually gives you an out-of-the-box way to invoke the TypeScript compiler. You can also do it in a simple way, like the TSDoc project is a small amount of repo with like four or five projects that just uses barebones scripts that just invoke TypeScript and then invoke lint in one folder that invokes web pack. If you were looking for an example of the most minimal way to use Rush to build something without like a shared toolchain, that\'s like an extreme example.\r\n\nBut to come back to your -- you\'re also asking an architectural question, right? How do you do efficient TypeScript compilation while still treating each individual project as being a self-contained process, which is also important when you start sharding builds across machines because -- TypeScript compiler recently introduced a feature where it also kind of acts as a built orchestrator. You all the projects in your monorepo in the centralized tsconfig file, and then TypeScript will go and figure out the order to build things and watch everything in your repo. And it\'s really cool technology. But I\'m not sure that it would actually work for our scenario, because when you have hundreds of projects and some of those projects use different versions of the TypeScript compiler, and you have other tasks like pre-processors or post- processors that run before and after the TypeScript compiler step, that\'s been a constant thought in the back of my mind is how would you use this feature in the TypeScript compiler? So, the approach that we\'ve been taking is just making the builds in the individual folders as efficient as possible. So as I said, Rush won\'t even spin up the tool chain if it can see that the source files haven\'t changed since the last time it was run. It has like an efficient hash for each of the source files [inaudible] changed it or not.\r\n\nBut within that, if we do decide we\'re going to run the compiler, the latest TypeScript compiler has on-disk caching. So, you start to spin up the Node process, but the actual state from before is not entirely lost. And we\'ve talked about -- for the longest time, I\'ve been wanting to do this cool optimization where we would have a service. So, when we use like VScode, for example, there\'s this language service that runs in the background. It\'s basically a long running TypeScript server, and then you hand source files to it and then it gives you back like an analysis of them.\r\n\nSHANE: Just one thing I wanted to ask, is it maybe a bit further back before the TypeScript toolings happened. But do you think Rush has a certain limit on how big of a project it should be used on because it was designed for big projects and it\'s really focused on that. Would you suggest like a project of three people, five packages, or do you think there\'s a limit to when it really starts to come into it all?\r\n\nPETE: You\'re saying a minimum threshold where it wouldn\'t be worth using Rush for something smaller than a certain scale?\r\n\nSHANE: Right, yeah. Does that exist or is it good for everybody?\r\n\nPETE: I mean, I use it for small projects, but maybe it\'s just because I\'m into Rush and like using it. It\'s just I like using familiar technology. We have made it somewhat easy to set up. So, it\'s not like anywhere near as intimidating as it was in the past just to get off the ground with it. But its focus, I would say, is for larger scenarios, I guess, like where build times matter. And there\'s some benefits you get from breaking the rules in a small project. I\'d say if, for example, if you only have five projects, it might not even make sense to make them into separate projects. Maybe it would be easier to initially develop them as a monolith.\r\n\nSHANE: Right. Okay, that makes sense. Are you able to pull out any of the main pieces of Rush to use in like a non-Rush project? Like the way you look for phantom dependencies or the incremental builds? Can you pull those out as a library and maybe use part of that in a non-Rush project or is it mostly use it altogether as a framework?\r\n\nPETE: The incremental builds rely on this package-deps-hash library that we maintain, so that part would be usable in isolation, like the command line parser for Rush is -- a lot of Rush Stack is actually libraries that Rush itself uses. So those pieces are reusable. Another cool feature of Rush is when you\'re doing a bunch of parallel builds, it doesn\'t mix together. It shows you real-time output of the logs as your builds are running. But it does not mix together the output from different sub-processes. So, for example, it will pick whatever the first thing is it started and show you the real-time output from that project altogether as one contiguous thing. And then when that project completes, it will take any other projects that build in the background and dump their complete output and then pick the next thing that\'s still running and give you real time output from that. So, it gives you like a real time feed of what\'s happening as your projects are building. But without intermixing logging lines from different projects and having to put prefixes on the lines. So that technology, again, is in a package that you could use for some other project that\'s not built into Rush.\r\n\nSome of the more scenario specific stuff is kind of all in one big thing that\'s hard to untangle, like publishing and changelog management and versioning. Those things are fairly closely welded together. We have a deploy feature that we added recently that goes, and starting from a project, we\'ll find all of its dependencies and then deploy just a subset of this project and its non-dev dependencies for Docker or for a Node.js web server. That, again, is kind of not really carved out because it makes some assumptions about how Rush projects work and how the linking strategy works. So, these more sophisticated things, just by virtue of the fact that they have to be more integrated, it\'s harder to make them reusable. And we really don\'t lean towards doing that because our focus is to make them work really well, and not to make this pallet of random components that you could mix together in different ways, because I feel like the JavaScript community already provides that very well.\r\n\nSHANE: Yeah. The number of options in the JavaScript community is huge. And that\'s why one thing I was wondering if Rush kind of opens up its libraries because you notice everyone does the same thing in a monorepo world over and over again, and it would just be nice if we could have some reuse from the established projects. So not everyone is building a console put, for example.\r\n\nPETE: Yup.\r\n\nCHARLES: What about the inverse case where we\'ve, let\'s say, hypothetically adopted Rush? Can you incrementally adopt the individual capabilities of Rush as you\'re using it? So you alluded to a publishing capability and a deployment capability. I\'m assuming if I want to use Rush to build my packages, I can still use my old mechanism to publish them.\r\n\nPETE: Yep.\r\n\nCHARLES: Okay.\r\n\nPETE: As long as it understands -- the Rush projects really are modeled as standalone npm projects. And subject to some issues with versioning, you could actually just go into a random folder and run npm install and npm run, build, or whatever, in that folder and it should more or less work. The main reason we did that was actually to facilitate moving projects between monorepos. So, when you have multiple monorepos, you do reorganization or trying to consolidate things. So, we really wanted projects not to be intertwined at all with anything else in their monorepo and to really act as self-contained units that could be moved around easily. So that, in some ways, makes it easier for other tools to interoperate.\r\n\nThere are some limitations to that. Like a lot of tools, for example, try to walk up to the root of the monorepo and find a package JSON file there and expect to find Node modules there and stuff. That stuff doesn\'t work.\r\n\nCHARLES: And it sounds like you\'re saying that that\'s kind of a bad idea.\r\n\nPETE: We think so. For example, if you put a Node modules folder in the root of your monorepo, that introduces a phantom dependency problem because Node.js can resolve things from there. So, all of a sudden, you once again can import things that weren\'t declared in your local project because they\'ve been accidentally hoisted to the root of the repo.\r\n\nCHARLES: Yeah, I find that happens to me all the time. I forget to include a dependency. And so, when I\'m running my tests or I\'m writing applications that reside inside the monorepo, it\'s fine and everything works. And then the minute someone installs the package from npm into a context that is not the monorepo at all, everything breaks because that package isn\'t there, because I just forgot to declare it.\r\n\nPETE: And the original team that built Rush was shipping STK to paid enterprise customers. So those breaks were really, they were like paid escalations or incidents when that happened. API extractor is something we could talk about another day. But it\'s also all about carefully controlling contracts for packages and making sure that you never accidentally break something.\r\n\nCHARLES: Yeah.\r\n\nSHANE: That phantom dependency is actually one of the biggest issues I found working with Yarn workspaces. When you\'re working with a team that might not know the ins and outs of monorepos, accidentally installing something globally, forgetting to add it to your package dependencies, having everything work, all your tests working, you publish and everything breaks, it\'s probably a thing that I\'ve seen the most as far as what can break a package. I\'ve been working on a product like my own just to solve that. It\'s like one through dependencies and find out are you asking for something that you aren\'t including in your package dependency, because it\'s so common.\r\n\nPETE: I believe there are lint rules. I think there\'s some lint rules that might check for that. There\'s some options. The most insidious form of it actually isn\'t that it\'s broken and it can\'t find the module. It\'s that it finds an unconstrained version of it. So, it will sort of seem to work. And we would get these cases where for 90% of the customers or people who are using the consumers, let\'s say, who are using the package, it does work because their tree ends up in the same format as ours. But then there\'s like 5% of people who actually have side by side versions or certain more complicated things happening in their case. And for them, it doesn\'t work. And then they\'re like, for opening issues, saying, "When I saw your package, I get this really weird error," because it\'s basically getting the wrong version of one of its dependencies somewhere deep in the tree. And we\'re like, "Well, done repo for me. It doesn\'t seem to repo for anybody else. Must be something with your computer. Did you try turning it off and back on again?" And it takes a long time to finally fix it.\r\n\nI\'m a very analytical person who fixates on obscure technical things that might be a problem. For me, I\'ve always been on this bandwagon, but it\'s one of the hardest. npm versioning is one of the hardest things that we\'ve dealt with in of messaging, because people assume that the default model works fine, because it works okay for them. And they assume that the problem is fundamentally simple, because if everybody just follows semver, then everything will be fine. And to explain why like phantom dependencies and hoisting are problematic, it\'s very hard to explain it because they\'re like, "Yeah, yeah." I hear you saying all this crazy technical stuff and drawing diagrams and stuff, but you\'re not understanding. You just need to follow the semver. I\'ve never had any problems with it.\r\n\nCHARLES: Yeah. It\'s definitely not that simple.\r\n\nAll right. Well, thank you so much, Pete and Shane, for coming by today. We will post, of course, the links to Rush.js where you can read up on all the stuff that we\'ve been talking about, on monorepos, on doppelgangers, and phantom dependencies and just npm versioning in general on the website. We can continue the conversation on Twitter. Thanks so much for coming by. I\'ve definitely had my eyes opened in the last year, I would say, to working with monorepos. It does come with its challenges. But overall, I think it\'s a net positive in of the problems they solve. So, go check out Rush. Go work with monorepos and be happy. Bye, everybody.\r\n\nThank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at @frontside.io. Thanks, and see you next time.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n

SPECIAL GUEST: Pete Gonzalez | @octogonz During the day, Pete works at HBO in Seattle on their streaming media apps. Prior to that, he was at Microsoft for 9 years, and before that, he worked at various consulting companies. A long time ago he was a cofounder of Ratloop, a small company that makes video games. SHOW NOTES 01:24 - Rush.js: What is it and what is it for? - Rush on GitHub 04:47 - Problems with Managing Large Codebases Rush Stack: provides reusable tech for running large scale monorepos for the web 07:22 - How does Rush provide a solution for build orchestration? 13:34 - Rush Stack Opinion: How to Lint, Bundle, etc. 16:53 - Using Rush Stack: Getting Started The pmpm Package Manager Yarn Plug’n’Play 24:27 - Getting Technical About Versions Phantom Dependencies Doppelgangers Pure Dependencies 32:47 - Thoughts on Monorepos 36:30 - Getting Started (Cont’d) + Efficient TypeScript Compellation 43:28 - Does Rush have a size limit? Is it for bigger or smaller projects? Both? 44:34 - Using pieces of Rush in non-Rush projects? TRANSCRIPT: CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk er interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer at The Frontside. It's been a really, really long time since we've gone to talk to everybody. We've been heads down working on projects, working on some products, working on a lot of different things that haven't given us the time to do much podcasting. However, one of the things that we've been doing lately is converting all of our projects over to using monorepos. And so, we've been very interested in the tooling that comes with managing more complex code bases that might contain a single application or contain a single package. And so, we have been evaluating lately these different tools. And one of them that came to the fore was Rush.js. So today, on the topic of this tool in particular, but I think also just in of managing larger and complex code bases, we have Pete Gonzalez on the podcast to talk with us. Hey, Pete. PETE: Hi. CHARLES: And today, also, we're going to be talking with Shane Wilson, who is the author of repkgs and a trusted voice here at Frontside, certainly when it comes to build tools. Hey, Shane. SHANE: Hey. CHARLES: So, yeah. Why don't we just dive right into it and talk about what was the particular set of circumstances, the particular set of problems that gave rise to Rush. What is it and what is it for? PETE: That's a good way to think about it. It's very much characterized by its beginnings. So, it started, I want to say, around 2015 when I was working on Microsoft SharePoint and we basically -- I won't go into too much detail, but we had a whole bunch of Legacy C# server code and the business has decided they wanted to move it into the browser as client code. So, lots of teams with this huge code base just started rewriting C# code as TypeScript. And the model, at the time at least, was you have npm packages that are your units of components and then each npm package goes in its own git repo. So we just had a bunch of developers, who had previously actually been working in a monorepo, go and start creating repo after repo and then publishing packages and sharing them with each other. And we ran into a lot of problems with that, which today people kind of understand the value of monorepos for TypeScript and Node.js projects. But at the time, there was a lot of assumption that while all the npm packages that you use off the Internet, they're all kind of developed that way. But we ran into problems where somebody would make a change and then somebody downstream would get broken or things wouldn't upgrade. And we eventually realized that we needed to bring the projects together, at least for the team that I was working on. And at the time, there wasn't any Yarn or Lerna, these projects that people are familiar with today, we really didn't have any options. In fact, npm itself was like a new thing that was displacing Bower. So, a lot of stuff that we take for granted today in the Node.js community at that time, at least within the group that I was working and nobody had been very familiar with it. So, we basically started with just a small tool that would let you cross-link project folders together and then it would work out which project needs to be built first. So, you build your libraries and you build the application. And so, it was a basic build orchestrator. And then out of that, basically, people started coming together into one repo and then we encountered various other problems, such as publishing workflows and stuff like that. And so, Rush sort of came out of those beginnings. But one thing that makes it a little different from the other tools that have come along is that it was done at a larger company. A lot of people, if you're a very prolific coder, you might make a thousand files or something and split them out into packages and then have something build them. But at the end of the day, it's still you or maybe you and 10 other people that work on your code. So, you can be fairly flexible. And the code tends to be fairly well-behaved because everybody understands it. Whereas what happened on the project I was working on, it was more like you have like 50 people and then 100 people and then 150 people all working in this big common space. And you encounter a set of problems that really impact your ability to scale up. And that's the direction that Rush has sort of taken. As these other solutions came along, we're kind of like, "Well, maybe we should stop using it and switch to something else." But we decided to make a more new tool that focuses on those problems of large code bases and large teams. CHARLES: Yeah. Maybe you could talk about, in specific, some of those problems. PETE: Some of them are technical problems. Like you just have a lot of code and the builds take a long time, so you need parallel builds and incremental builds. And lately, we've been talking about multi-phase builds just to break some of those down. Parallel just means that you start up multiple Node.js processes that build stuff simultaneously. But there's also been work recently on sharding. So, you can distribute the builds across multiple VMs rather than just having it run on one VM. Like for incremental builds, with Node.js, there's a problem that every time you run a command in a given folder to compile a project, you have to spin up Node.js and then eval a bunch of toolchain scripts. So, for example, Rush's incremental build logic does all of the analysis from the build orchestrator, so that Node.js processes already running for Rush can determine whether a folder needs to be built or not. And even if that saves you maybe one second of time versus if you had gone and run the toolchain on that particular folder, if you have like 200 projects, that could be 200 seconds of time that you save. But there's also human aspects of it as well. For example, you have people of different skill levels coming in and working on the code. So, you need people who maybe are just coming to that monorepo for the first time to fix a bug and then will go on somewhere else, you need it to be a good experience for them to learn how the tooling works. And it shouldn't be like a bunch of tribal knowledge that they need to learn how to actually turnaround a pull request. There can be political issues. So, there's like the bad team that doesn't have as good quality bar and keeps checking things in that make flaky tests or problems that break the build for other people. So, the other teams start instituting stricter policies, like stricter eslint rules, stricter code reviews, sign offs and stuff like that. But then it creates a separate problem where now people are complaining that the ecosystem is too bureaucratic. So, this led eventually into Rush Stack, which is a more recent thing that we've been doing where we take the idea is Rush itself is a build orchestrator that doesn't really make a lot of assumptions about what your toolchain is. And then the idea behind Rush Stack was to start bringing in other pieces of the toolkit and saying, "Well, if you want to work more the way that we work, you can start using these other pieces of Rush Stack like API extractor and the bunch of web pack plug-ins and things like that. And it would include a bunch of best practices and stuff. CHARLES: So, if I could ask a question, and this is on a much smaller scale, but I'm just kind of keeping in mind as I'm listening to you talking some of the monorepos that I work in personally. And one of the most annoying things that I think you actually touched in with the build orchestration was if I have, say, 15 packages, 15 npm packages in this repo, and there's a web of dependencies between these packages, and I'm working on one package and that's where my focus is. But I realize that I need to make a change in another package. Now, the minute I start straddling development on two packages, I have to , "Oh yeah, I need to go run a build. Whenever I change a source file on this other package, I need to run a build in that package." Then I can come back. And oftentimes, I will forget to do that. And I'm like, "Why is this not working?" And it's like, "Ah, I made the change in the source file. I need to compile the TypeScript and bundle it into an npm package distribution so that I can consume it from the library that I'm actually working on." And so then, what I find is now I'm making custom little hacks to do like file watching at the top of the thing, at the top of the the repo that looks at all the packages and incrementally compiles them whenever there's a change detected. So, how does Rush provide a solution to that? How would this build orchestration work? Is that a use case that it could solve? PETE: The watch scenario is definitely an important scenario. Rush itself today doesn't have a great solution for that. Basically, you run watch in the project that you're actually serving. For example, you have a frontend web application and then you have various libraries that it depends on. So today what we do is you just run watch on the frontend web application. And then when the library is changed, you can use Rush to say, compile everything up to this project that I'm in right now and it will detect which things have changed so it can skip the ones that don't have any changes. For example, this is an area that we're very interested in. It hasn't been a top priority, I guess. One other interesting aspect of it is so when we first developed Rush, it was just something I was doing on the site. I wrote the original prototype of it myself over Christmas vacation, I think. And then it became something that people were using. People got excited about, more people started having feature requests. Eventually when I was at Microsoft, there was a whole team of people who started just managing the tooling for the monorepo. And when you have a lot of people, that's actually one of the selling points for the monorepo, is the more developers that you get working together on one place, suddenly the more budget you have to have a team of people go and build really good tooling for it. So, it's not just like each repo has one guy who spends five hours a week putting out fires. It turns into like when I left, we had something like four or five people who were full time just working on tools. And there was a ticketing system where if you got a weird build error or some tests were unstable, you could actually open a ticket. And we had an on-call person who would investigate it. And then we collected statistics about how many tickets we got in different areas and did root cause analysis and stuff. I'm not HBO, but HBO kind of has the same thing. They have dedicated tooling teams that work in the big monorepo and it's really kind of good. But as far as Rush, it was never like a Microsoft product that they were selling or pitching. The website is a little bit like a sales pitch, but I wrote most of it myself. I even did the artwork myself. All that is to say that the development of features in Rush is a little bit slow because it's kind of a thing that people do on the side as things are motivated for them. It's not like complete. And Rush Stack is sort of the same way. We had been building random packages and tools for a few years. And then it was a year or so ago, we started thinking, "We keep putting all these things together in one monorepo on GitHub, but what did they really have to do with each other?" And we came around to the idea that the real value of them is that they work together. And JavaScript in general is really into flexibility. I always tell people, "If we're going to do a project in Java or C# or something like that, you just install the language, like framework, and you often get an editor with it and then you're testing and linking and bundling in optimizer and linter. All those things just come out of the box and somebody has put the entire end to end system together for you, and it just works. Whereas with JavaScript, it's like every step of the way, there's two popular linters for TypeScript. If you don't use TypeScript, there's Babel. If you do use TypeScript, you can use TypeScript and Babel together. If you want to run tests, there's Karma and Mocha and Jest. And even the test assertion library and the test runner can be made by different people. We think of it almost as a vector where every single piece of it, like your linker, is made by a different group from the compiler. And you can choose between Rollup or webpack for your linkers. Every step of the way, you have to make a decision. You have to read the blogs and find out what is popular and then make your bet and say, "Oh, we like Jest because that seems to be popular today and that's what we're going to use." Well, we made bets like that five years ago on Gulp and things that aren't very popular now. But when you start doing large scale development, once you make that bet, it's hard to change it because you have all this stuff that starts getting built up around it. So, part of the idea of Rush Stack was let's have a bunch of people who are professional developers and who have experience doing tooling, make some bets and then decide on a set of stuff that should work well together. And then we're going to collaborate together with our community from open source. I now work at a different company, but I still am very involved with it and bring a different set of perspectives. But the spirit of it is we're all working together to try and over time build something up that is similar to what you'll get in other languages. CHARLES: Is the idea then that the Rush Stack has opinions about how to lint, how to bundle, how to do all these things? PETE: And part of the opinion is, we don't have time to develop a bunch of different options for things. We just want something that works. We don't really want everybody to choose their own flavor of ice cream because we don't have time to do that. But there's another aspect of it, which is that when you lock in on something, you can make optimizations that aren't possible when there's abstraction. And I can give some concrete examples of that. Like when we started with Gulp, the idea of Gulp was that you model tasks and then the tasks have dependencies between each other. So, TSLint, which was the linter we were using at the time, was a task and then TypeScript was a task. And the task was like this Gulp wrapper around the actual library of TypeScript and TSLint, and then you like wire them together. So first, we run TypeScript and we run TSLint. And the idea was you make these boxes and arrows like a big data flow diagram of how each task gets its turn on the files and the way things move between the boxes were streams of files or partially processed file buffers. And we ran into problems with that because, for example, TSLint needs to do a TypeScript analysis. So, it invokes the compiler engine to study the code and figure out what types or things. But the TypeScript task does that, too. So you really want the same compiler engine to perform the analysis once and then to share that intermediary state between TSLint and TypeScript. But it's hard to do that just with TSLint and TypeScript, if you're just writing a simple program that invokes one after the other. You have to kind of go into some internal libraries a little bit if you want to wire that up. But then when it's behind a task abstraction, there isn't like a natural way to do that. Another thing that will come up is like some people, I don't know if we're still doing this today, but they would say, "If the code doesn't compile, don't show me lint errors because maybe the code's not even valid. Let's first make sure that the compiling works correctly. And then if the compiling was successful, then show the lint errors. But I want to run the compiler in the linter simultaneously to save time." So, I need to save the error messages from the linter in a buffer until the compiler says it's successful, then I'll show those messages. So again, this is something that breaks down this idea of abstractions. And if we wanted to replace TSLink with ESLint, let's say, you end up with the same problem again that I mentioned, where you want ESLint's parser to share the same analysis as TypeScript. But the way that you hook those pieces together is going to be completely different because the entire architecture of ESLint is completely different. So, all that's to say that when you make something pluggable, you make a lot of architectural sacrifices or you create the need to create this rich interface that can expose the real stuff that you need to interact with something in a tight, optimized way that's expensive. So, with Rush Stack, we've taken the philosophy that we do want some things to be pluggable, but they should be conscious decisions of this is a part where there really is value in allowing different pieces to be swapped in here. And we've actually tested them and we've actually validated that the abstraction we set up makes sense and is workable. It's not like we just say, "I did one thing and it's pluggable, so you can drop something else in there if you want." But it's a dice roll. It's an exercise of the reader, whether it will actually work or how much work is involved with that. CHARLES: So, if I wanted to use Rush Stack, and I understand that it's in development, I'd like to talk both about what's coming in the form of Rush Stack and what you would be able to use maybe today. But let's continue this talking about Rush Stack for a minute here. So, if I say I had a monorepo that had maybe, let's say, it's got 10 npm packages, it's got a couple of like backend servers and a couple of UI like frontend React apps, what would that look like in of the artifacts that Rush generated or how would I interface with that system? What would be my experience in of spinning everything up, working on an npm package, working on one of the backends, working on one of the frontends, so on and so forth? PETE: Yeah. So, you would start by converting the monorepo to use Rush as its build orchestrator. And there's like a rush init command that will dump out config files for you. We didn't put a lot of thought into the onboarding experience for a long time because when you have a monorepo, you only set it up once and then you never think again about the setup experience. And that was when we made Rush open source and then started trying to build a community around it and get people to use it and maybe contribute back to it, it took us a long time to realize that other people had a hard time setting up Rush at all because we never went through the experience of setting up Rush. We've been working in the same five monorepos for years. But when we went and did some investigation into that, what we found is that we like the idea of having these big config files that have json schemas that make sure you have IntelliSense and error checking when you're editing them and they have big blocks of comments around everything. So you run Rush in it and it just dumps these big empty config files with comments. And then you go uncomment the parts that you want and add your stuff. We try to put as much prioritization of rush in config files versus like command line options or environment variables or other places where there's less rails or guidance of what you're supposed to do to set it up. So anyway, that would be the first thing you do is generate the config files and then go all of your projects so that Rush builds them. You probably would run into a little bit of work that you have to do just to get it to build with Rush, because Rush's way of installing npm packages is a little bit different. We mainly use the pnpm package manager. This is another really interesting area of JavaScript is they have like three or you could say four leading package managers. There's npm, the original one, then there's Yarn Classic, which is a ground up rewrite of npm that adds workspaces and some new features, but keeps the same models npm. Then there's pnpm which came along and solves a technical problem with the way npm installs stuff. We refer to it as doppelgangers and phantom dependencies on the Rush website. CHARLES: Actually, this was a very interesting point. Could we take a detour here to talk about phantom dependencies and doppelgangers, because it's something that's unique, I think, to the JavaScript ecosystem, which I hadn't encountered before working in other languages. Could you just unpack that just a little bit? I know we could spend half an hour talking about it. PETE: Let me finish my list of package managers for a minute, then I can dive into that. So, I said there was the original npm. Then there's Yarn classic and there's pnpm, which actually changes the installation model but using symlinks. So, it solves some problems that we'll talk about in a minute by creating a whole bunch of symlinks in your node modules folder. And it is a correct installation that eliminates those problems. And it's mostly backwards compatible because it still preserves the original Node.js module resolver model. And then there's Yarn plug and play, which is essentially a completely different installation strategy from the original Yarn, but its approach is that it changes the way Node.js resolves packages through this plugin. And then it's cleaner, it doesn't have to create some links at all. It doesn't even have to create node modules folders. It's more like how Ruby gems, for example, work. But it has more compatibility issues, because there's a whole lot of code that assumes it's going to go walking through node modules folders looking for things. It can be stuff like web pack. It can be just people's random packages that go and resolve things. But we encountered some compatibility issues. We feel like Yarn plug and play is like the future, it's like a better design, but it could take a while before a lot of people will be able to move to that. Let's say you've got those and then there's Rush, which predated most of them. Rush was designed around the original npm package manager. So, it introduced its own symlinking strategy that is somewhat similar to what pnpm does today. Even now, today, when you use Rush with pnpm, it's actually Rush's symlinks retrofitted onto pnpm's model. And we have a PR that's been open for like a month now that actually finally lets pnpm completely manage the installation when you're using it and gets Rush out of the business of package management, which we're really excited about because it's complicated problem. And we have had like just tons of work that we've done over the years to that and make it work correctly. But let's rewind a little bit and talk about why all that exists, because it is another big aspect of working in a large-scale repo. CHARLES: I understood reading from the documentation that Rush will or will not work with other package managers besides pnpm? PETE: It s npm and Yarn classic. The is not great because almost none of the Rush developers or main s actually use this. Pretty much everybody is using pnpm. We've been wanting to improve the for the other package managers, but our plan to do that is to basically let them just do the installation. So right now, for example, when you use Yarn -- like Rush's original strategy was it looks at all the projects in your monorepo and then it creates this fake project that has the dependencies on the superset of all of your monorepo dependencies. And then it runs the package manager wants to install that folder and then it goes and creates symlinks for each project into that centralized folder. And there's some technical reasons why that is a good strategy. But when you do that with npm and Yarn, every time that they change things or release new version, that changes some semantics, some aspects of that can break and then you have to go debug it. I can say it works with the version of npm and Yarn at the time when we were last using that. But it's had problems over the years. And so, the long-term strategy would be to do the same thing we're doing with pnpm. Hopefully, we'll do it this summer. But say, if you want to use Yarn with Rush, then Yarn just installs the way Yarn installs and you get all the other Rush features but Yarn manages the installation and that means you lose the phantom and doppelganger solutions that Rush was doing. But the idea is maybe you didn't want that. The reason you're using Yarn is because you want Yarn's installation model. CHARLES: I see. So, it really is, for example, one of the things that I was wondering is what's the overlap between, say, using Yarn workspaces with Rush. And kind of what I'm hearing is that it really is, you're choosing one or the other. Because that's kind of when you're working with Yarn workspaces, you've got some virtual package that aggregates all of the packages and stores at least some of the dependencies in one place. And Rush is doing the same thing. PETE: Let's get a little more technical that versions. CHARLES: [Laughs] PETE: So, npm has, I guess I can say, two big factors that got us into this space to begin with. One of them is you just have way more packages than usual. So, for example, if you use, I don't know, C#, which I've used before, it would be common that you have 20 libraries that you install that are like dlls or something that you get from open source or you buy them from a commercial company and that's your dependencies. Whereas with the standard Node.js project, notoriously you can have typically more than a thousand npm packages getting installed just to do some basic stuff like, build an application and not do anything particularly weird. They just have an approach of having lots and lots of small projects that all have dependencies on each other. So out of the gate, npm has a lot more complex version management issues than languages that have less libraries that tend to be bigger and more monolithic libraries, like .NET framework, for example, which just when you go to use C#, you get this massive library that comes from Microsoft that solves a whole bunch of problems, than with npm, each one will be solved by some individual package. But then there's another aspect, which is the way that npm installs packages is -- I have to be careful talking about this, but it's flawed. When you have packages that depend on other packages, they form what's called in computer science, a directed acyclic graph. Basically, it's like a tree that branches apart, but then it can come back together again. But the installation strategy of the Node modules folder is that we're going to use folders on disk to represent the dependencies between packages. So, if A depends on B, C, and D, those become subfolders of it. And then when you have a diamond dependency, you run this problem that the file system on a computer actually is a tree. It's not a directed acyclic graph. You'd need symlinks to make it into a directed acyclic graph. So, there's a whole write up on the Rush website that tries to give examples of the technical details. CHARLES: It really isn't npm's fault. It's actually kind of the original sin was committed by Node in the way that it requires modules. Like, that's where it actually happens. PETE: I think they were developed together. CHARLES: Oh, okay. PETE: Actually, I shouldn't speak about it because I wasn't there. I don't know the original history of how Node.js and npm came together and agreed on how they're going to interoperate with each other. But if you look at it at the time, it was probably completely sufficient for the problems they were solving. And like Bower, the thing that they were competing with, doesn't allow libraries to depend on libraries at all. So, it's kind of an innovation that you could have these big dependencies of things. But basically, it introduces certain problems like the phantom dependency is the most obvious problem, which is they do this thing called hoisting, where if several things all depend on a package, they move it up to the top. And then if you can't find the thing you're looking for in a subfolder, you just walk up the tree and find it somewhere higher in the tree. And that creates this phantom problem, which is that a package can require things that it didn't declare a dependency on. So, it doesn't appear in your package JSON, but it is findable somewhere in the node modules folder. So, Node.js doesn't even look at package JSON files when it's resolving. It doesn't look at dependency declarations when it's searching for things. So, you can require things that you never declared and that can cause problems because a lot of times, the thing will be there because of some aspect. It's like, I depend on this other package and it depends on something. So generally, I can require that thing and have a somewhat good guarantee that it's going to be there. And I might never even notice the problem. But I don't have any say about what the version of it is because I never declared it. And it can get disturbed because it's the package manager chooses to install things differently than the version or the thing that I was expecting might not either, or it might be the wrong version. And then doppelgangers are an even more interesting case, which is when you have a diamond dependency. I'll try to make it concrete. I can't think of the package names off the top of my head. It's like A depends on B, which depends on this D library. That's our diamond. And then A also depends on C, which depends on D. You can get a case where B and C need two different versions of D and npm models that they install the package twice, so you can get multiple copies of it. And because of the tree, there are situations where you can actually end up installing five copies of the same version of the same library. So, it's not even just that we have side by side versions, it's like the same version has to be installed in multiple folders because without a symlink, there's no way to make the folders end up being -- CHARLES: Whereas in other languages, you would get an error at build time saying, "Hey, I cannot find a single version of this library that satisfies the dependencies of this project." And you have to deal with it manually at the build time. PETE: Or they would say -- Ruby, for example, will just install that version in one place. And then the way you find it is when you require a library, the resolver just looks in your dependencies and says, "Oh, you need version 1.2.3. Okay, I'm going to go look in that folder for it." But with npm, it's like, "Well, the only places I can look for it are in certain places that don't know what the version is." It means that the package manager has to copy the same version into different folders and you can [crosstalk] your tree. But if you think about it, say I have a singleton, then now I have two singletons getting spun up because Node.js doesn't actually understand that these two folders have the same library in them. It sees it as two entire instances of it. Where the TypeScript compiler, we had a longstanding compiler error that we were dealing with when we first set up Rush monorepos, which is the TypeScript compiler would see a declaration of a class in two places and it would assume that they were not compatible declarations. It has the same name, but it's a completely different copy of it. And we eventually got them to relax the rules a little bit and say, "Well, if it's in another package that has the same name as this package in the past relatively within there they are the same, then you can reasonably assume that it's the same thing and then disable your check there." And it took like a year to get that finally fixed. So, this is like one aspect of the problems that's like really obvious. But there's another aspect, which is just the management of versions is complicated. npm has this thing called pure dependencies. In a monorepo, I personally have probably spent months of my life just helping people sort through problems of I'm trying to upgrade this package and now my installation fails or I upgraded the package and now the application doesn't run because something got jostled around in the Node modules folder and now the versions aren't exactly right. And when you go into a monorepo, the problems really multiply. In a small repo, which is where the vast majority of npm s work, you just don't have enough different people trying to install different versions of things to really hit these problems. But as more and more people come in, like the Rush Stack repo has something like 10 versions of the TypeScript compiler getting installed for various reasons. And then there are things that have pure dependencies in a TypeScript compiler, so you get sort of complicated version problems and that's where Rush's features came up. It has special protections against phantom dependencies and doppelgangers. It has weird concepts like preferred versions and allowed alternative versions and special checks to make sure that the versions are in sync. And you can have Rush check things for you. So that's just an area that we became really interested in because we had a lot of frustration ing it. And the big company, when you're shipping production software to the cloud where releases go out every week, you really need things to be deterministic and reliable. So, we put a lot of engineering and thought into that. Whereas a lot of other people were like, "What's wrong with it? Works fine for me. I don't understand. If it's a problem, go hack something in your package JSON file." We weirdly over and over again had conversations with people who couldn't understand what we were trying to solve or why Rush was preventing them from acquiring something they could see there. I'll mention one other case that this came up, which is when you have multiple monorepos. People are always wanting to create more monorepos and then they're like, "Well, it's a great separation." You have all of your libraries in one monorepo and then your app in the other monorepo and then you just automatically publish things and automatically upgrade in the downstream repo. CHARLES: So, you advocate or you kind of just monorepo all the way. Just keep doubling down on one codebase. It's always better if you've got a huge block of code to add it to the existing repo? PETE: Well, I would say technically it's easier to manage when it's in one place because you can do things like a pull request that updates 200 projects easily at once without having to worry about publishing and upgrading stuff. But there are political reasons why you would want things separated. For example, at Microsoft, we had an internal repo that was closed source and then an open source repo in GitHub that was open source. So, at the very beginning, out of the gate, our toolchain had to be in a different repo from the application development. So that's like a valid case. Where there can be business groups that really are separate from each other. To me, one of the real rubrics that you can use is about the contract. So, if you have, take React, for example. There's a group of developers that make React and they ship it, they document it. They have like release notes when they release a major new version, they make sure that they test the contracts for it. So, they're shipping a contract for people to consume. And there's a group of people around making sure that that contract is stable and well ed. That kind of boundary totally makes sense to have it within its own repo. The thing that really didn't work for us was the sharing code where we're not really shipping it. It's more like we're sharing it. So, there's some url parser that somebody makes and then another team wants to use it and some other teams have some libraries. So, various teams start putting code into different libraries into some shared repo that they're going to consume. But nobody really owns it. And you get this problem where somebody goes and makes a change to the thing they're using, but they're not going to go and test the other downstream repos. So then when the downstream repos upgrade, they might get broken by that change. And then it's like, whose responsibility is it? That guys should have tested everybody else? Or these people just need to pay this cost. And we had teams, when we were working in this model where it was like every week, somebody would have to go spend like their afternoon getting the compile again because the upstream repo that they were consuming packages from just kept constantly having churn and changes. Because at a company, it's not like npm, where somebody just kind of makes a library and once in a while, they update it. It's like there are people actively changing stuff every day because they're trying to get a feature shipped. And these aren't just really isolated libraries. They're all parts of the anatomy of a big application. So, there's a lot more churn. And we tried, "Oh, let's have it automatically upgrade. Let's make it so that when somebody goes and makes a change, we run tests in in the other repos and tell them that it's broken." Or a really common one is let's make people review the contracts. Everybody needs to follow a semver and they need to design their APIs to be forward thinking and stuff. But all those things that you look at, you're like, "Well, React did that. Let's the impose that on our developers working in this shared repo." The problem is that there's no budget for that. Nobody's manager is going to let them go spend a day having a meeting and deliberating about a design change to some url parser, because that's just not the business that they're in. Nobody really thinks that that is a shipping thing. So, if you don't solve that problem, what you end up is people just stop sharing code. It's easier just to fork the library and have our own version of it and stop trying to share it with the other team. The monorepo really does solve that problem because it lets everybody work together on a shared library. And if you make a change to it, you can't merge or change unless all the unit tests for all the things that are affected by it. And when stuff fails, you can easily just go fix that stuff up in the same PR that introduces the breaking change. CHARLES: That is a really, really interesting point. Never thought of it that way. So back to the question of getting setup, getting started with Rush.js for this hypothetical monorepo that we have 10 npm packages that are written in TypeScript, maybe some are written JavaScript, some are written in TypeScript and then a backend and a frontend. What's the next step for getting the build orchestration going? PETE: Rush kind of checks out at the point where we need to build a project. At that point, it just runs an npm script in that folder. And then it's up to your toolchain to figure out how to build the project. And you can have different tool chains in each project folder. Rush actually discourages you from having any global stuff. So, for example, our ESLint rules, we don't have an eslintrc file at the root of the repo. Each individual project has its own lint setup and you can even be running different versions of lint in different project folders, which sounds not good. But when you have like hundreds of projects, sometimes not all the teams can upgrade ESLint at the same time, so you need to let people be able to have differences in their tool chains locally. Prettier, we did decide with prettier to run that globally because it's more of a cosmetic thing. It doesn't have a lot of breaking changes, and it's something that kind of needs to run on the diff of what you've changed when you have like a git commit hook. But other than the one exception of prettier, we pretty much have the toolchain be self-contained for each individual folder. CHARLES: I thought one of the selling points also was incremental building and shaving build times off of that. So how can you do that by having each kind of package contained inside the repo being responsible for managing its own build? PETE: Well, the toolchain is shared, right? So, the first thing that you would do in Rush is make a toolchain project, a project that actually is the code that compiles things. And we usually put shared configuration in there, like your tsconfig for web applications or your tsconfig for Node applications. With ESLint, I went through a whole thing when we switched from TSLint to ESLint, where ESLint actually doesn't have the ability to load plugins from a shared configuration folder. It tries to load the plugins from the actual project folder. So, we have this work around that we made and we have an open PR, we're trying to get ESLint to have better for monorepos. But basically, you want to move, you want each project to just have a dependency on, we actually call the rig. So, the idea is the tool chain is like this shared code that is reusable scripts for invoking the TypeScript compiler or web pack or whatever. But then the rig is like a flavor. So, you would have a rig for a web library or a rig for a web application or for a tooling application and things like that. So, these stereotypical kinds of projects usually get a specific, like a toolchain package that has all that stuff rolled up in one bundle. So, you just kind of will have as little copying and pasting of boilerplate between projects as possible. CHARLES: Let's say I had some rig then for compiling TypeScript npm packages. How would I, for example, I don't know, use the same cache to the TypeScript assets or keep like a TypeScript compiler hot, so that, I don't know, like the syntax tree parse or something, like this is one of things I noticed in the build is like TypeScript takes a long time to compile. PETE: That's true. CHARLES: And if there was some way that when a tiny source file changes, you could run a command and it would just compile that one file and put it in the disc directory is that just I need to do that myself, and I can benefit from having this single rig that's in a single place so that any project that's going to be using that can benefit from it, which I think is good. But is there also, does Rush provide some sort of solution out of the box or sort of maybe some building blocks so I can build that type of solution out of the box? PETE: Rush doesn't. And for a long time, Rush Stack didn't either, because we're using Gulp and we really -- Gulp as a mission, is kind of an older, less popular solution now. And the whole approach that we took with Gulp, we regretted and we kind of wanted to rethink for many years, but it worked and it was like a component of our setup that worked reliably. So, we just were using the same Gulp toolchain for years and years. I'm now at HBO, but recently now Microsoft has been developing a new modernized replacement for that that's called Heft, and they're in the process of moving that code in -- they had done an internal prototype of it. And now they just opened the first PR this week to start releasing that as open source. But that would be like a toolchain that is more modern, that actually gives you an out-of-the-box way to invoke the TypeScript compiler. You can also do it in a simple way, like the TSDoc project is a small amount of repo with like four or five projects that just uses barebones scripts that just invoke TypeScript and then invoke lint in one folder that invokes web pack. If you were looking for an example of the most minimal way to use Rush to build something without like a shared toolchain, that's like an extreme example. But to come back to your -- you're also asking an architectural question, right? How do you do efficient TypeScript compilation while still treating each individual project as being a self-contained process, which is also important when you start sharding builds across machines because -- TypeScript compiler recently introduced a feature where it also kind of acts as a built orchestrator. You all the projects in your monorepo in the centralized tsconfig file, and then TypeScript will go and figure out the order to build things and watch everything in your repo. And it's really cool technology. But I'm not sure that it would actually work for our scenario, because when you have hundreds of projects and some of those projects use different versions of the TypeScript compiler, and you have other tasks like pre-processors or post- processors that run before and after the TypeScript compiler step, that's been a constant thought in the back of my mind is how would you use this feature in the TypeScript compiler? So, the approach that we've been taking is just making the builds in the individual folders as efficient as possible. So as I said, Rush won't even spin up the tool chain if it can see that the source files haven't changed since the last time it was run. It has like an efficient hash for each of the source files [inaudible] changed it or not. But within that, if we do decide we're going to run the compiler, the latest TypeScript compiler has on-disk caching. So, you start to spin up the Node process, but the actual state from before is not entirely lost. And we've talked about -- for the longest time, I've been wanting to do this cool optimization where we would have a service. So, when we use like VScode, for example, there's this language service that runs in the background. It's basically a long running TypeScript server, and then you hand source files to it and then it gives you back like an analysis of them. SHANE: Just one thing I wanted to ask, is it maybe a bit further back before the TypeScript toolings happened. But do you think Rush has a certain limit on how big of a project it should be used on because it was designed for big projects and it's really focused on that. Would you suggest like a project of three people, five packages, or do you think there's a limit to when it really starts to come into it all? PETE: You're saying a minimum threshold where it wouldn't be worth using Rush for something smaller than a certain scale? SHANE: Right, yeah. Does that exist or is it good for everybody? PETE: I mean, I use it for small projects, but maybe it's just because I'm into Rush and like using it. It's just I like using familiar technology. We have made it somewhat easy to set up. So, it's not like anywhere near as intimidating as it was in the past just to get off the ground with it. But its focus, I would say, is for larger scenarios, I guess, like where build times matter. And there's some benefits you get from breaking the rules in a small project. I'd say if, for example, if you only have five projects, it might not even make sense to make them into separate projects. Maybe it would be easier to initially develop them as a monolith. SHANE: Right. Okay, that makes sense. Are you able to pull out any of the main pieces of Rush to use in like a non-Rush project? Like the way you look for phantom dependencies or the incremental builds? Can you pull those out as a library and maybe use part of that in a non-Rush project or is it mostly use it altogether as a framework? PETE: The incremental builds rely on this package-deps-hash library that we maintain, so that part would be usable in isolation, like the command line parser for Rush is -- a lot of Rush Stack is actually libraries that Rush itself uses. So those pieces are reusable. Another cool feature of Rush is when you're doing a bunch of parallel builds, it doesn't mix together. It shows you real-time output of the logs as your builds are running. But it does not mix together the output from different sub-processes. So, for example, it will pick whatever the first thing is it started and show you the real-time output from that project altogether as one contiguous thing. And then when that project completes, it will take any other projects that build in the background and dump their complete output and then pick the next thing that's still running and give you real time output from that. So, it gives you like a real time feed of what's happening as your projects are building. But without intermixing logging lines from different projects and having to put prefixes on the lines. So that technology, again, is in a package that you could use for some other project that's not built into Rush. Some of the more scenario specific stuff is kind of all in one big thing that's hard to untangle, like publishing and changelog management and versioning. Those things are fairly closely welded together. We have a deploy feature that we added recently that goes, and starting from a project, we'll find all of its dependencies and then deploy just a subset of this project and its non-dev dependencies for Docker or for a Node.js web server. That, again, is kind of not really carved out because it makes some assumptions about how Rush projects work and how the linking strategy works. So, these more sophisticated things, just by virtue of the fact that they have to be more integrated, it's harder to make them reusable. And we really don't lean towards doing that because our focus is to make them work really well, and not to make this pallet of random components that you could mix together in different ways, because I feel like the JavaScript community already provides that very well. SHANE: Yeah. The number of options in the JavaScript community is huge. And that's why one thing I was wondering if Rush kind of opens up its libraries because you notice everyone does the same thing in a monorepo world over and over again, and it would just be nice if we could have some reuse from the established projects. So not everyone is building a console put, for example. PETE: Yup. CHARLES: What about the inverse case where we've, let's say, hypothetically adopted Rush? Can you incrementally adopt the individual capabilities of Rush as you're using it? So you alluded to a publishing capability and a deployment capability. I'm assuming if I want to use Rush to build my packages, I can still use my old mechanism to publish them. PETE: Yep. CHARLES: Okay. PETE: As long as it understands -- the Rush projects really are modeled as standalone npm projects. And subject to some issues with versioning, you could actually just go into a random folder and run npm install and npm run, build, or whatever, in that folder and it should more or less work. The main reason we did that was actually to facilitate moving projects between monorepos. So, when you have multiple monorepos, you do reorganization or trying to consolidate things. So, we really wanted projects not to be intertwined at all with anything else in their monorepo and to really act as self-contained units that could be moved around easily. So that, in some ways, makes it easier for other tools to interoperate. There are some limitations to that. Like a lot of tools, for example, try to walk up to the root of the monorepo and find a package JSON file there and expect to find Node modules there and stuff. That stuff doesn't work. CHARLES: And it sounds like you're saying that that's kind of a bad idea. PETE: We think so. For example, if you put a Node modules folder in the root of your monorepo, that introduces a phantom dependency problem because Node.js can resolve things from there. So, all of a sudden, you once again can import things that weren't declared in your local project because they've been accidentally hoisted to the root of the repo. CHARLES: Yeah, I find that happens to me all the time. I forget to include a dependency. And so, when I'm running my tests or I'm writing applications that reside inside the monorepo, it's fine and everything works. And then the minute someone installs the package from npm into a context that is not the monorepo at all, everything breaks because that package isn't there, because I just forgot to declare it. PETE: And the original team that built Rush was shipping STK to paid enterprise customers. So those breaks were really, they were like paid escalations or incidents when that happened. API extractor is something we could talk about another day. But it's also all about carefully controlling contracts for packages and making sure that you never accidentally break something. CHARLES: Yeah. SHANE: That phantom dependency is actually one of the biggest issues I found working with Yarn workspaces. When you're working with a team that might not know the ins and outs of monorepos, accidentally installing something globally, forgetting to add it to your package dependencies, having everything work, all your tests working, you publish and everything breaks, it's probably a thing that I've seen the most as far as what can break a package. I've been working on a product like my own just to solve that. It's like one through dependencies and find out are you asking for something that you aren't including in your package dependency, because it's so common. PETE: I believe there are lint rules. I think there's some lint rules that might check for that. There's some options. The most insidious form of it actually isn't that it's broken and it can't find the module. It's that it finds an unconstrained version of it. So, it will sort of seem to work. And we would get these cases where for 90% of the customers or people who are using the consumers, let's say, who are using the package, it does work because their tree ends up in the same format as ours. But then there's like 5% of people who actually have side by side versions or certain more complicated things happening in their case. And for them, it doesn't work. And then they're like, for opening issues, saying, "When I saw your package, I get this really weird error," because it's basically getting the wrong version of one of its dependencies somewhere deep in the tree. And we're like, "Well, done repo for me. It doesn't seem to repo for anybody else. Must be something with your computer. Did you try turning it off and back on again?" And it takes a long time to finally fix it. I'm a very analytical person who fixates on obscure technical things that might be a problem. For me, I've always been on this bandwagon, but it's one of the hardest. npm versioning is one of the hardest things that we've dealt with in of messaging, because people assume that the default model works fine, because it works okay for them. And they assume that the problem is fundamentally simple, because if everybody just follows semver, then everything will be fine. And to explain why like phantom dependencies and hoisting are problematic, it's very hard to explain it because they're like, "Yeah, yeah." I hear you saying all this crazy technical stuff and drawing diagrams and stuff, but you're not understanding. You just need to follow the semver. I've never had any problems with it. CHARLES: Yeah. It's definitely not that simple. All right. Well, thank you so much, Pete and Shane, for coming by today. We will post, of course, the links to Rush.js where you can read up on all the stuff that we've been talking about, on monorepos, on doppelgangers, and phantom dependencies and just npm versioning in general on the website. We can continue the conversation on Twitter. Thanks so much for coming by. I've definitely had my eyes opened in the last year, I would say, to working with monorepos. It does come with its challenges. But overall, I think it's a net positive in of the problems they solve. So, go check out Rush. Go work with monorepos and be happy. Bye, everybody. Thank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at @frontside.io. Thanks, and see you next time. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

In this episode, Charles and Taras discuss "big ideas" and all the things they hope to accomplish at The Frontside over the next decade. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: Coming soon!

In this episode, Charles and Taras discuss "big ideas" and all the things they hope to accomplish at The Frontside over the next decade.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n\nTranscript:\r\n\nCHARLES: Hello and welcome to The Frontside Podcast, a place where we talk er interfaces and everything that you need to know to build them right. Today, we\'re going to talk about big ideas and the future of Frontside.\r\n\nTARAS: Yeah, starting with is Frontside is good idea?\r\n\nCHARLES: No, we\'re going to just talk about how do you know that an idea is good? We\'ve touched on it a couple of times before. Like how do you, how do you go about validating a big idea, how do you discover a big idea? What do you do?\r\n\nTARAS: And then, even when you have big ideas like big tests, what does that mean for the world? How do you make a big idea an idea that a lot of people like and agree with and actually use on day to day.\r\n\nCHARLES: Yeah. It turns out that it\'s not easy. There\'s a lot of work involved with that. A lot of it crystallized around the conversations we\'re having about what exactly is big test and recognizing that big test isn\'t really a code base. It\'s not a toolkit. Even though it does has aspects of those things, it really is an idea. It\'s an approach. It\'s a way of going about your business, right?\r\n\nTARAS: Yeah. Especially when you put big test functionality in place, when you start doing big testing and then you put together things like using Mocha and Karma, big tests in that kind of test suite is really just like interactors and the idea of big testing. There\'s nothing else. All the interactors do is just give you an easy way to create composable like [inaudible] objects, so you don\'t have to write -- you have the components but you don\'t have to write selectors for each element in the component, especially if gets composed. But that\'s like a very small functionality that does a very specific thing. But big test itself, it takes a lot of work to actually -- we had this firsthand experience on the project we\'re working on right now. We are essentially introducing like Ember\'s acceptance testing but for React in a react project and having to explain to people what is it about this that actually makes it a really good idea and having people in the React world see that this is actually a really good idea. It\'s kind of incredible. When you actually try to sell something to somebody and convince somebody that this is a good idea is when you realize like how inadequate your understanding of the idea really is. You really have to start to break it down and understand what is it about this that is a really big idea.\r\n\nCHARLES: Yeah, I completely agree 100% because to be clear, we\'ve actually been doing this now for two years almost. So, this is not the first React project where we\'ve put these ideas in place. But I think in prior examples, we just kind of moved in and it\'s like we\'re going to do this because this is what we do. And we have firsthand knowledge of this working because we\'ve operated in this community where this is just taken on faith that this is the way you go about your business. You have a very robust acceptance test suite. And because of that, you can experience incredible things. When you and I were talking before the show, we were kind of commenting on inside the Ember community, you can do impossible things because of the testing framework. You can upgrade from Ember 1 to Ember 3 which is a completely and totally separate framework, basically. You\'re completely and totally changing the underlying architecture of your application. You can do it in a deterministic way and that\'s actually incredible.\r\n\nTARAS: And what\'s interesting too is that the React core team kind of hinted a book at this also in their blog post about fiber or moving to fiber because one of the things that they talked about there is that knowing how the system is supposed to behave on the outside allowed them to change the internals of the free app framework, specifically about test suite for the React framework, but it allowed them to change the internals of the framework because they were testing kind of on the outside. The system kind of is a black box and that allowed them to change the internals and the test suite essentially stayed the same. So this idea of acceptance testing your thing is really fundamental to how Ember community operates. But other communities have this as a big idea as well. It\'s just applied in different areas.\r\n\nCHARLES: Right. And so applied to your actual application, this is something that\'s accepted in one place, but it\'s not an accepted practice in other places. But you can make your argument one of two ways and say, "I have experienced this and it\'s awesome. And you can do architectural upgrades if you follow the patterns laid out by this idea." And that works if you have a very, very high, high trust relationship, but you don\'t always have that, nor should you. Not everybody is going to be trusting you out of the box. So, you\'re going to have to lay out the arguments and really be able to illustrate conceptually practically how this is a good idea just so that people will actually give it a try.\r\n\nTARAS: And it takes a lot. One of the reasons why it was easier for us to introduce big testing to the current project we\'re working on is because we were able to write, like we\'ve done the implementation for this in a previous project. So when we were convincing people this is a good idea, we\'re like, "Look, your test suite can be really good. It\'s going to be really fast. And look, we\'ve done it before." So the actual process of convincing people that a big idea is a really good idea is actually kind of a complicated process that requires a lot of work and partially requires experimentation. You have to actually put an implementation in place and show that you\'re going to have to build up on your successes to be able to get to a point where you can convince people that this is actually a really good idea. People who have not heard about this idea, and especially people who might have counter, like they have people of authority in their community that have counter views. For example, quite often when it comes to big tests, when you bring up big tests, people will reference a blog post by a Google engineer that talks about how functional testing or acceptance testing is terrible. And for a lot of people, a Google engineer means a lot and the person makes really good points. But that\'s not the complete idea. The complete idea is not just about having an acceptance test suite. It\'s a certain kind of acceptance test suite. It\'s an acceptance test suite that mocks out at boundaries so you don\'t make API requests to the server, you make an API request to a [inaudible] server using something like Mirage or whatever that might be. So, the big idea, it has like new ones that makes it functional, but getting people who are completely unaware, who don\'t necessarily look up to you as an authority to believe you like, "Yes, that actually sounds like a really good idea." It is not a trivial task.\r\n\nCHARLES: No, it\'s not. Because the first time you try and explain it, you\'re arguing based on your own assumptions. So, you\'re coming to it safe in the knowledge that this is a really, really good idea based on your firsthand knowledge. But that means you\'re assuming a lot of things. You\'re assuming a lot of context that you have that someone else doesn\'t and they\'re going to be asking questions. Why this way, why this way, why this way? And so, you have to generate a framework for thinking about the entire problem in order to explain the value of the idea. And that\'s something that you don\'t get when it\'s just something that\'s accepted as a practice.\r\n\nTARAS: I think simulation is actually a really good example of that. If you haven\'t had experience with Mirage, if you don\'t know what having a configurable server in your tests does for you, you will probably not realize that similar ideas apply to, for example, Bluetooth that when you\'re writing tests for your Bluetooth devices, or you\'re writing tests for an application that interacts with Bluetooth devices, you actually want to have a simulation there for Bluetooth so that you can configure it in a kind of similar way to the way you would configure a Mirage for a specific test. You want to be able to say, "This Bluetooth device is going to exist," or, "I have these kinds of Bluetooth devices around me, they have the following attributes. They might disconnect after a little while." There\'s all kinds of scenarios that you want to be able to set up so that you can see how your application is going to respond. But if you haven\'t seen a simulation with something like Mirage, you\'re going to be going like, "I don\'t know what the hell why would this be helpful."\r\n\nCHARLES: It seems like lot of work. One of the things that we\'ve been working on, as I said, is trying to come up with a framework for thinking about why this is a good idea because we can\'t just assume it. It\'s not common knowledge. For example, one of the things that we\'ve been developing over the last year and more recently in the last few months is trying to understand what makes a test valuable. At its essence, what are the measures that you can hold up to a test and say this test has X value. Obviously, something like that is very, very difficult to quantify. But if you can show that and you say, "This test has these quantities and this test has these quantities," then we can actually measure them. Then it\'s going to allow people to accept it a lot more readily and try it a lot more readily. So, the ideas that we\'re playing with right now is that you kind of have to evaluate a test on one on speed, tests that are fast, have an intrinsic value, or rather test that are slow. The upside that you gained from the test is very quickly bled away or offset if the test is slow. So, I can have a very comprehensive test that tests a lot of high value stuff. But if it takes three days to run, it\'s going to be basically worthless. Another axis on what you can evaluate is in of coverage. I\'m not talking about coverage of lines of code. I\'m talking e cases and units of assemblage. So, there\'s the module. Those modules are then stitched together into components. Those components are stitched together into applications. Those applications are ed onto browsers. And I would consider it a different unit of assemblage. Your application running on Firefox is a different assemblage than your application running on Chrome, is a different assemblage than your application running. You have this access, which is the coverage of your unit of assemblage. That\'s another way that you can evaluate your tests. So if I have a test that runs only on Node in a simulated dom, there\'s a cap, there\'s an absolute cap on the value of that test and it cannot rise above a certain point. And the other thing, another access that we\'ve identified is isolate ability, an ability to run a test. So if I have a test suite comprised of 1500 tests, but if one fails in the middle, I have to restart the test from the beginning. That\'s going to decrease the value. Maybe it\'s related to speed, being able to run the tests without having to install a bunch of different dependencies. So that\'s another access. And so trying to really understand the variables there, that\'s something you have to be very systematic about thinking about the tests so that you can actually take your idea and explain it to someone who\'s not coming at it from first principles. Imagine you have to explain an if statement to somebody who\'s never programmed with an if statement before.\r\n\nTARAS: That is going to be very difficult.\r\n\nCHARLES: It\'s going to be difficult, right?\r\n\nTARAS: Yeah. In general, it\'s very challenging to go from an experience that somebody had, like overriding somebody\'s experience conveying your own personal experience is very difficult. And getting someone to experience something is very difficult. And so that\'s what I think a lot of the work that we\'ve been doing over the last little while has been breaking down these problems into a way of understanding them so that we can actually explain why these things are important. Like what we\'ve had to do this recently with a Bluetooth work that we\'ve been doing. We have a partner that\'s implementing a Bluetooth abstraction for mobile devices and trying to convey to them the value of being able to simulate devices. That\'s something that we saw with Mirage. We knew that being able to simulate devices in tests in different environments is extremely valuable. We know this firsthand from our experience. But trying to justify to them why we think this is important and why they should rejig all of their thinking about how they\'re going to architect this middle layer between the native APIs and the application APIs, why they should rejig this layer to make it so that it will allow for simulation, like convincing very technical, very knowledgeable and experienced people that these ideas are important, it requires kind of fundamental understanding of the value that they provide.\r\n\nAnd I think this touches on what really I think Frontside is going to be doing going forward is creating conditions for this kind of thought to be cultivated, to be able to create business environment where our clients gain benefit from this kind of very deep insight, insight that transcends the source of those ideas. Because there are certain ideas that are really fundamental and they\'re beautiful ideas. For example, in React world, a functional component is a really beautiful idea. It\'s a really simple concept. And kind of going along along with what you\'re saying about this kind of fundamental ideas, they will drive you to the next point you couldn\'t even imagine. I think the work that the React core team has done -- they\'ve set out this functional component as a primitive, but it wasn\'t possible for a long time to really make the functional component a reality until they\'ve gone through the process of actually understanding what connects all the dots so that you could eventually get to a point where like, "Oh, actually this [inaudible] API is a way for us to enable the fundamental concept of having a component." Not only is it simple, but it actually works. You can write a performance application using this fundamental concept. So, the work that we\'re going to be doing is first of all, making it possible to have conversations at this level. So, people that we\'re going to be working with Frontside, clients who we work with, companies that work with us, who partner with us, it\'s going to be all to creating this environment where we can have this kind of ideas. Then being able to extract these great ideas from the frameworks and actually making them available across frameworks. You don\'t have to be locked into a specific community to be able to benefit from that.\r\n\nOne of the first steps we\'re working on now is now that we have an understanding about big tests, we believe this is a good idea, we have ways to justify why it\'s a good idea. We have clients who have benefited from this good idea. Now we\'re in position to fill in the gaps that are missing from big tests, from being something that could be used in any framework. We want it to be really easy for people to be able to say, "I really love acceptance testing, but I have to work in a React project or I have to work in a Vue project, I have to work in an Angular projects." It\'s like, "It\'s no problem. I have big tests. Big tests is going to give me what I want." The big tests, the idea, but it\'s going to come with all the little pieces that you need to be able to assemble the functional test suite that is going to give you the benefits that Ember\'s acceptance test suite provides for Ember projects.\r\n\nCHARLES: I mean, why should you have to compromise on these things. If it is a good idea and it is a fundamental idea, whether it\'s how you manage concurrent processes, whether it\'s how you manage testing, whether it\'s how you manage routing, why should you have to compromise? Why should you have to say, "You know what? I love..." I don\'t know, what\'s a comparable system? "I love air conditioning." Why should I have to go into a car that doesn\'t have air conditioning? Because every single car has a different air conditioning system, or every single house has a different air conditioning system. I\'m showing the fact that I live in Texas here by using this example. But we\'ve developed air conditioning systems that are modular so that they can be snapped on to pretty much any house and you don\'t have to build a custom thing for the circulation and refrigeration of air every single time or have it just be like it\'s part of the house. Or we have this wood that ships with ducks. It\'s like, no, we want to separate out that system.\r\n\nTARAS: You don\'t have to commit to living in a specific area to get the benefit of being comfortable. You don\'t have to give up the comfort for specific areas. The good idea of air conditioning is not restricted to just one area. You can actually experience it wherever you go and have it be available wherever you would go. You have to be able to, like if you\'re moving to a new house, you can install air conditioner and now you\'re going to have like cool air. That\'s the kind of the theme I think of what we\'re going to be doing. But there\'s so much work to do because we\'re kind of, I would say, probably 30 or 40% into having -- if we were to look at like big tests as a big idea is used and is well known in our industry, we\'re probably maybe 5% on that. If we take into consideration what it takes for us to be able to convey what it takes to convince people is a good idea. I think we\'re, if you add to it, we\'ll probably add another 15%. So the next step is actually creating some tooling around it that would make it really easy for people to consume. Because right now, what\'s really unfortunate is that you have big tests that is available. If we set it up in the React project, doing it with create-react-app right now is kind of difficult. So we\'re going to make that easier. But then companies still need to make a choice between the ergonomics [inaudible] Cypress or the speed and the comfort and the control that you have with big tests. We have to basically eliminate the need to make that choice by making the tooling that you get in Cypress and making that tooling available for big test projects.\r\n\nSo that is going to kind of bring us up to maybe 60%. And then the remaining part is taking the software that enables the big idea and then making the software work across all different frameworks so that it\'s really easy to install. On Angular, it\'s just going to be add a plugin and it gives you big testing. You don\'t have to rely on their unit testing or you don\'t have to rely on the end to end testing as using Selenium. You just have big tests and it works just as well as it works and React and just as well as it works in Ember. So you can get the benefits of that. Going through the whole process and then bringing into the world this way, we have a lot of work to do because every part of the architecture of building modern frontend applications requires this level of effort. If you look at routing, routing is extremely inconsistent across frameworks. There are different opinions for every framework. There are different opinions of how to implement it. And that\'s problematic both on an individual level and a specific project level because it\'s so hard to know what you should use. And in many cases, it\'s not complete. What constitutes a routing system on React looks different than what would be in Ember, and it looks different than what it is in Angular. And each one of them has their own trade-offs. So if you find yourself in a situation where you\'ve been using React and now you have to use Angular, you have a steep learning curve. But this also has an interesting effect of like, "Well, we have a world now where micro frontends are a thing." There are really good reasons why companies might want to use multiple frameworks for single platform because different frameworks have different benefits and different teams based on their location might prefer a different framework. There\'s lots of different reasons why companies choose and we want developers to be able to choose specific framework. But how do you do that when you need to have basically micro frontend architecture, you need to provide an SDK that provides a consistent experience like -- you need to provide an SDK that provides a consistent developer experience, but then that developer experience because it\'s developer experience, needs to enable consistent experience regardless of what framework the team decides to use. So now, routing needs to be more robust. Routing without having a strong concurrency primitives is going to be very limited and it\'s not going to be as portable across frameworks as one that actually has very robust concurrency primitives.\r\n\nSo now we\'ll start looking at concurrency. We\'ve got concurrency primitives that are different for each framework. We\'ve got Ember concurrency, we have a Redux-sagas. We have observables. React is introducing suspense and their hooks. And now each one of those things is different. So now, we need a concurrency primitive that is framework agnostic that is externalized, that we can consume to build stuff. Now that\'s a piece of the routing system. There\'s other things like state machines. State machines are an important part. If you look at the authentication system, the core of an authentication system is a state machine. How do you express a state machine in a way that is framework agnostic, in a way that works in every framework. That\'s an area that we need to explore. So, there\'s a lot of work for us to do to take these big ideas and externalize them from the framework so that they can be consumed by frameworks. And the applications that use this frameworks, that is a lot of work. And that\'s essentially what Frontside is setting out to do is hold these good ideas and extract the pieces that are actually core to these ideas. And then make them available independently and then see what that will provide. In a world where we have all of these pieces in place and we have kind of the granular little pieces that we need to be able to have a really strong concurrency primitive, we have a great routing system that\'s using this concurrency primitives. We have a state machine mechanism that can work with any framework, has very comfortable APIs. We have all these pieces. What kind of collaboration will that allow? What is the world that that will create? I think for people who are in the Ember community, we saw firsthand what it means for us to stand the shoulders of giants. We have wonderful, brilliant people creating really awesome tools. Now, if that same level of collaboration was happening in a way that was framework agnostic, but we were all standing on shoulders of giants and we were helping each other create something like this, what kind of world would that create? That\'s what Frontside is setting out to find out.\r\n\nCHARLES: Yeah, and it\'s going to be fantastic. I should say it\'s going to be really, really fun. It\'s going to be challenging. Like you said, there is a lot of work to do.\r\n\nTARAS: A lot of conversations to have. I mean, I think it\'s difficult to have these conversations because we all have good reasons for thinking certain things and a lot of times it requires understanding each other to find the place. Core teams do this. It\'s getting everybody to align on where we want to go, provide leadership like doing all of that work and doing it in a way that is actually possible. Because to create an environment with this kind of research that is happening, it requires a lot of money. Charles and I cannot do this. A small group of people probably cannot do this either because we need real use cases. We need real projects to make sure that these ideas are not dream implementations or just like vaporware. We need the pressure of real projects, real clients\' relationships to create software that is going to make a huge difference for their developers. So we need those relationships with clients so that we can actually create the conditions for this kind of thing to emerge. That needs to also happen for us to make this kind of a situation possible. So, there is a lot of work to do but it\'s a really worthwhile cause.\r\n\nCHARLES: And I think that it\'s one of the reasons that we start with testing as the most important use case because it\'s something that you don\'t want to be just saying, "Hey, you want to come help us fund research to make everything better?" There is real pressure, there\'s financial pressure and there\'s a duty to make sure that if a project has a deadline, that deadline is met. You want to come in at cost and on time. That\'s the goal of every project. And you want to try and strive to do your best to achieve that. And testing is an area where the research almost always pays off. I mean, where it has clearly. You can invest a lot of money and get a lot of reward in return because you\'re losing so much money by not testing or by focusing your investment in testing on lower value targets.\r\n\nTARAS: For people that are familiar with acceptance testing, I think that just goes like a no brainer. Acceptance testing just enables you that. So by putting big test as a first kind of objective, but putting acceptance testing as a first kind of milestone, it sets us up to benefit in the way that Ember community has benefited where the acceptance test now became a mechanism that made it possible for the framework to change. And so, we\'re kind of setting up the same conditions here. When you start working with us or with any of the companies that work with us, the first benefit you get is you get the acceptance testing. Now, developers are more productive because they know that they have a certainty that if something broke, they will know. If they broke something, they will know. But what that also gives us is a way of introducing change in a systematic way. So, if we find that we have a new routing mechanism, we can start to refactor our React router setup to use this new routing mechanism and we can do it safely because we have our acceptance testing in place. So in many ways, it\'s kind of like the first step. And it\'s actually a great first step for business and is a great first step for tech technology. That\'s kind of a really awesome overlap.\r\n\nCHARLES: In the podcast with Dylan Collins, we talked about this is like step one is make experimentation cheap because experimentation can be wildly expensive and you can throw away a lot of money if you\'re just experimenting without constraints. And there is nothing like a great application/acceptance test suite that puts those constraints in a completely and totally unambiguous way. So I think that\'s actually a great summary of our next decade.\r\n\nTARAS: Yeah, I think so too. I think it\'s a good to show you the perfect next step. And I think this is going to show us the perfect next future though we can\'t really imagine it right now. But I do know one thing, this future is going to include a lot of people. Fundamentally, what we\'re setting out to do is a big idea. And so, big ideas are brought to the world by a lot of people. And I think one of the things that\'s important is that from day one, we include everyone in the process. For anyone who\'s interested in these big ideas, we want to make it possible for you to participate in. So that\'s the reason why Frontside is going to have an RFC. Like on our Frontside website, we\'re actually going to start writing and publishing RFCs. The first RFC that we\'re going to have is going to be for big tests, which is to lay out why big test is a big good idea for frontend for our industry. And as we start talking to people, there\'s actually lots of applications beyond frontend. So, we\'re going to focus on writing these RFCs and sharing them with the world so that we can include as many people in it as possible so everyone can participate. And ultimately, hopefully, that will have an influence on our industry and actually allow us to leverage these big ideas in a way that is not tied to any particular source of these brilliant ideas.\r\n\nCHARLES: Yeah, hat tip to the Ember community for the whole RFC thing, which I guess they didn\'t invent either.\r\n\nTARAS: Yeah, they borrowed it also, sorry. But I mean, it\'s been adopted everywhere. And I think it is a really interesting way of making people part of your process and making it our process as our meaning, like everyone who works with us. Whether you work for Frontside, that there\'s a client of Frontside or you work with the company that is partner with Frontside, whatever that might look like, we want to start to create places where we can be having these conversations and together surfacing these great ideas and making them part of a tool sets.\r\n\nCHARLES: I think that\'s a perfect way to wrap up. Like I said, you pretty much described our next decade. Maybe if we get more people, it\'s going to be significantly less and we\'ll move onto the next thing even more rapidly. I am fully pumped. I\'m ready to see this. There\'s a lot of work to do and I feel great about getting to it.\r\n\nTARAS: I\'m very excited about all the work that we\'re going to be doing, so let\'s get to it.\r\n\nCHARLES: Yup. All right. We\'ll see everybody next time.\r\n\nThank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at @frontside.io. Thanks. See you next time.\r\n

In this episode, Charles and Taras discuss "transparent development" and why it's not only beneficial to development teams, but to their clients as well. Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: Coming soon!

In this episode, Charles and Taras discuss "transparent development" and why it\'s not only beneficial to development teams, but to their clients as well.\r\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n\nTranscript:\r\n\nCHARLES: Hello and welcome to The Frontside Podcast, the place where we talk er interfaces and everything that you need to know to build them right.\r\n\nIt\'s been a long summer and we\'re back. I\'m actually back living in Austin, Texas again. I think there wasn\'t too much margin in of time to record anything or do much else besides kind of hang on for survival. We\'ve been really, really busy over the last couple of months, especially on the professional side. Frontside has been doing some pretty extraordinary things, some pretty interesting things. So, we\'ve got a lot to chew on, a lot to talk about.\r\n\nTARAS: There\'s so much stuff to talk about and it\'s hard to know where to start.\r\n\nCHARLES: So, we\'ll be a little bit rambly, a little bit focused. We\'ll call it fambly. But I think one of the key points that is crystallized in our minds, I would say over this summer, is something that binds the way that we work together. Every once in a while, you do some work, you do some work, you do some work, and then all of a sudden you realize that a theme, there\'s something thematic to that work and it bubbles up to the surface and you kind of organically perceive an abstraction over the way that you work. I think we\'ve hit that. I hit one of those points at least because one of the things that\'s very important for us is -- and if you know us, this is things that we talk about, things that we work on -- we will go into a project and set up the deployment on the very first day. Make sure that there is an entire pipeline, making sure that there is a test suite, making sure that there are preview applications. And this is kind of the mode that we\'ve been working in, I mean, for years and years and years. And where you say like if what it takes is spending the first month of a project setting up your entire delivery and showcasing pipeline then that\'s the most important work, inverting the order and saying that has to really come before any code can come before it. And I don\'t know that we\'ve ever had like a kind of unifying theme for all of those practices. I mean, we\'ve talked about it in of saving money, in of ensuring quality, in of making sure that something is good for five or 10 years, like this is the way to do it. And I think those are definitely the outcomes that we\'re looking for. But I think we\'ve kind of identified what the actual mode is for all of that. Is that fair to say?\r\n\nTARAS: Yeah, I think one of the things I\'ve always thought about for a long time is the context within which decisions are made because it\'s not always easy. And it\'s sometimes really difficult to really give it a name, like getting to a point where you have really clear understanding of what is it that is guiding all of your actions. What is it that\'s making you do things? Like why do we put a month of work before we even start doing any work? Why do we put this in our contract? Why do we have a conversation with every client and say, "Look, before we start doing anything, we\'re going to put CI in place." Why are we blocking our business on doing this piece? It\'s actually kind of crazy that from a business perspective, it\'s a little bit crazy that you be like, "Oh, so you\'re willing to lose a client because the client doesn\'t want you to set up a CI process?" Or in a case of many clients, it\'s like you\'re not willing to accept -- the client is going to say, "We want to use Jenkins." And what we\'ve done in the past, in almost every engagement, we\'re like, "Actually, no. We\'re not going to use Jenkins because we know that it\'s going to take so long for you to put Jenkins in place. By the time that we finish the project, you\'re probably still not going to have it in place. That means that we\'re not going to be able to rely on our CI process and we\'re not going to be able to rely on testing until you\'re finished." We\'re not going to have any of those things while we\'re doing development. But why are we doing all this stuff? It was actually not really apparent until very recently because they didn\'t really had a name to describe what is it about this tooling and all of these things that makes why is it so important to us. I think that\'s what kind of crystallized. And the way that I know that it\'s crystallized because now that we\'re talking to our clients about it, our clients are taking on to picking up the language. We don\'t have to convince people that this is a value. It just comes out of their mouth. Like it actually comes out of their mouth as a solution to completely unrelated problems, but they recognize how this particular thing is actually a solution in that particular circumstance as well even though it\'s not something Frontside sold in that particular situation. Do you want to announce what it actually is?\r\n\nCHARLES: Sure. Drum roll please [makes drum roll sound]. Not to get too hokey, but it\'s something that we\'re calling Transparent Development. What it means is having radical transparency throughout the entire development process, from the planning to the design, to the actual coding and to the releases. Everything about your process. The measure by which you can evaluate it is how transparent is this process to not just developers but other stakeholders, designers or people who are very developer adjacent, engineering managers all the way up to C level executives. How transparent is your development? And one of the ways that we sell this, because I think as we talk about how we arrived at this concept, we can see how this practice actually is a mode of thinking that guides you towards each one of these individual practice. It guides you towards continuous integration. It guides you towards testing. It guides you towards the continuous deployment. It guides you towards the continuous release in preview. I think the most important thing is that it\'s guided us, by capturing this concept, it\'s guided us to adopt new practices, which we did not have before. That\'s where the proof is in the pudding is if you can take an idea and it shows you things that you hadn\'t previously thought before.\r\n\nI think there\'s a fantastic example. I actually saw it at Clojure/conj in 2016, there was a talk on juggling. And one of the things that they talked about was up until I think it was the early 80\'s or maybe it was the early 60\'s, the state of juggling was you knew a bunch of tricks and you practice the tricks and you get these hard tricks. And that was what juggling was, is you practice these things. It was very satisfying and it had been like that for several millennia. But these guys in the Physics department were juggling enthusiasts and I don\'t know how the conversation came about, you\'d have to watch the talk. It\'s really a great talk. But what they do is they make a writing system, a nomenclature system for systematizing juggling tricks, so they can describe any juggling trick with this abstract notation. And the surprising outcome, or not so surprising outcome, is that by then, once you have it in the notation, you can manipulate the notation to reveal new tricks that nobody had thought of before. But you\'re like, "Ah, by capturing the timing and the height and the hand and we can actually understand the fundamental concept, and so now we can recombine it in ways that weren\'t seen before." That actually opened up, I think an order of magnitude of new tricks that people just had not conceived of before because they did not know that they existed.\r\n\nAnd so, I think that\'s really, as an abstract concept, is a great yardstick by which to measure any idea. Yes, this idea very neatly explains the phenomenon with which I\'m already familiar, but does the idea guide me towards things with which I have no concept of their existence? But because the idea predicts their existence, I know it must be there and I know where to look for it. And aha, there it is. It\'s like shining a light. And so I think that that\'s kind of the proof in the pudding. So that\'s a little bit of a tangent, but I think that\'s why we\'re so excited about this. And I think it\'s why we think it\'s a good idea.\r\n\nTARAS: Yes. So what\'s also been interesting for me is how universal it is. Because the question of is this transparent enough? That question could be actually asked in many cases. What\'s been interesting for me is that asking that question in different contexts that I didn\'t expect actually yielded better outcome. At the end of the day, I think that a test for any idea is like, is it something that can help you more frequently than not? Like is it actually leading you? Does applying this pattern, does it increase the chances of success? And that\'s one of the things that we\'ve seen, thinking about just practices that we\'re putting into place and quite asking are they transparent enough? Is this transparent enough? It\'s actually been really effective. Do you want to talk about some of the things that we\'ve put in place in regards to transparency? Like what it actually looks like?\r\n\nCHARLES: Yeah. I think this originally started when we were setting up a CI pipeline for a native application which is not something that we\'ve typically done in the past. Over the last, I would say, 10 years, most of our work has been on the web. And so, when we\'re asked to essentially take responsibility for how a native application is going to be delivered, one of the first things that we asked kind of out of habit and out of just the way that we operate is how are we going to deliver this? How are we going to test it? How are we going to integrate it? All the things that we\'ve just talked about is something that we have to do naturally. But because this is not very -- like continuous integration and build is very prevalent on the web. I think that testing still has a lot of progress on the web, but it\'s far more prevalent than it is in other communities, certainly the native community. So when we started spending a month setting up continuous integration an integration test suite, spending time working on simulators so that we could simulate Bluetooth, having an automated process with which we could ship to the App Store, all of these things kind of existed as one-offs in the native development community. There are a lot of native developers who do these things. But because it\'s not as prevalent and because it was new to us, it caused a lot of self reflection both on why is it that we feel compelled to do this. And also we had to express this, we had to really justify this work to existing native development teams and existing stakeholders who were responsible for the outcomes of these native development teams. So, there was this period of self reflection, like we had to write down and be transparent about why we were doing this.\r\n\nTARAS: Yeah. We had to describe that in SoWs. We actually had really long write ups about like what it is that we\'re setting up. And for a while, it was, people I think would read these SoWs and I think they would get the what\'s of what we\'re actually going to be putting into place. But it wasn\'t until we actually put it into place and we\'ve seen a few like really big wins with the setup -- one of the first ones was the setting up preview apps where preview apps in -- the web are pretty straightforward because we\'ve got Netlify that just kind of gives it to you easily.\r\n\nCHARLES: Netlify and Heroku. It\'s very common.\r\n\nTARAS: Yeah, you activate it, it\'s there. But on the mobile side, it\'s quite a different story because you can\'t just spin up a mobile device that is available through the web. It\'s something kind of very special. And so we did find a service called Appetize that does this. And so we hooked up the CI pipeline to show preview apps in pull requests. So for every pull request, you could see specifically what was introduced in our pull request without having to pull down source code, like compile it. You could just click a link and you see a MVC stream of a mobile device and that application running on a mobile device. So the setup took a little bit of time. But once we actually put it in place and we showed it to our clients, one of the things that we noticed is that it became a topic of conversation. Like, "Oh, preview apps are amazing." "This is so cool." "Preview apps are really great." And I think in some ways, it actually surprised us because we knew that they were great, but I think it was one of the first times that we encountered a situation where we would show something to a client and they just loved it. And it wasn\'t an app feature. It was a CI feature. It was part of a development process.\r\n\nCHARLES: Right. So, the question is then why was this so revelatory? Why was it so inspiring to them? And I think that the reason is that even if we have an agile process and we\'re on two week iterations, one week iterations, whatever, there\'s still a macroscopic waterfall process going on because essentially, your business people, your design people, maybe some of your engineering people are involved at the very front of the conversation. And there\'s a lot of talking and everybody\'s on the same page. And then we start introducing coding cycles. And like I said, even if we\'re working on two week iterations and we\'re "agile", the only that you actually have, whether something is working, is if the coder says it\'s done. "I\'m Done with this feature. I\'m on to the next feature for the next two weeks." And after that two weeks, it\'s like, "I\'m done with this feature. I\'m on to the next feature." From the initial design, you have the expectation about what\'s going on in the non-technical stakeholders minds. They have this expectation. And then they hope that through the process of this agile iterative development cycles, they will get the outcome that satisfies that hope. But they\'re not able to actually perceive and put their hands on it. It\'s only the engineers and maybe some really tech savvy engineering managers who can actually perceive it. And so they\'re getting information secondhand. "Hey, We\'ve got authentication working and we can see this screen and that screen." And, "Hey, it works on iOS now." "I have some fix ups that I need to do on Android." So, maybe they\'re consuming all of their information through standups or something like that, which is better than nothing. That is a level of transparency. But the problem is then you get to actually releasing the app or whether it\'s on the web, whether it\'s on native, but this is really a problem on native. You get to where you actually release the app and then everybody gets to perceive the way the app as it actually is. So you have this expectation and this hope that was set maybe months prior and it just comes absolutely careening into reality in an instant, the very first moment that you open the app when it\'s been released. And if it doesn\'t meet that expectation, that\'s when you get disappointment. When expectations are out of sync and grossly out of sync with reality, even a little bit out of sync with reality, you get disappointment. As fundamental and explanation of just the phenomenon of disappointment, but it\'s an explanation of why disappointment happens so often on development projects. Is this kind of the expectations and hopes of what a system can be in the minds of the stakeholders? It\'s kind of this probability cloud that collapses to a single point in an instant.\r\n\nTARAS: And that\'s when things really hit the proverbial fan. Now, you have the opposite. So everything that was not transparent about your development process. So everything that was hidden in the opaqueness of your development process, all of those problems, either on a product side, maybe something didn\'t quite get implemented the way it\'s supposed to. Like you actually found out two weeks or three weeks before you\'re supposed to release that that feature wasn\'t actually quite implemented right. It went through testing, but it was tested against the Jira stories that were maybe not quite written correctly. So the product people are going like, "What the hell is this? It\'s actually not what I signed up for. This is not what I was asking for." So, there\'s that part.\r\n\nAnd then on the development side, you\'ve got all of the little problems that you didn\'t really for because you haven\'t been shipping to production from day one. You actually have like application not really quite working right. You didn\'t as supposed to integrate with some system that is using Chorus or something you didn\'t for. Like you have a third party dependency you didn\'t really fully understand. But because it wasn\'t until you actually turned it on that you actually started to talk to the thing properly, and you realize there\'s some mismatch that is not quite working. But now you\'ve got everything that was not transparent about the development process, everything that was hiding in the opaque corners of your development process is now your problem for the next three weeks because you\'ve got to fix all of these problems before you release. And that\'s what I think where a lot of organizations kind of find themselves in is this position where they\'ve been operating for six months and like, "Everything is going great!" And then three months or three weeks before, you\'re like, "Actually, this is not really what we were supposed to do. Why did this happen?" That time is really tough.\r\n\nCHARLES: Yeah. That\'s what we call crunch time. And it\'s actually something that is lot of times we think of it as inevitable, but in fact it is actually an artifact of an opaque process.\r\n\nTARAS: Yeah.\r\n\nCHARLES: That\'s the time when we have to go, everybody\'s like, "We\'re ordering pizza and Dr. Pepper and nobody\'s leaving for a month."\r\n\nTARAS: Yeah. I think there are people that do that practice like functional testing as part of development process or acceptance testing, I think they could relate to this in some cases where if you had to set up a test suite on an application that was written without a test suite, first thing you deal with are all the problems that you didn\'t even know were there. And it\'s not until you actually start testing, like doing functional testing, not integration or unit testing where you\'re testing everything in isolation, but when you\'re perceiving the entire system as one big system and you\'re testing each one of those things as the would, it\'s not until that point you start to notice all the weird problems that you have. Like your views are re-rendering more than you expected. You have things that are being rendered you didn\'t even notice because it\'s hard to see, because it happens so quickly. But in test, it happens at a different pace. And so, there\'s all these problems that you start to observe the moment that you start doing acceptance testing, but you don\'t see them otherwise. And so, it\'s the process of making something transparent that actually highlights all these problems. But for the most part, if you don\'t know that there are transparent options available, you actually never realize that you are having these problems until you are in crunch time.\r\n\nCHARLES: Right. And what\'s interesting is viewed through that lens, your test suite is a tool for perception. And to provide that transparency, not necessarily something that ensures quality, but ensuring the quality is a side effect of being able to perceive bugs as they happen or perceive integration issues at the soonest possible juncture. To shine the light, so to speak, rather than to act as a filter. It\'s a subtle distinction, but I think it\'s an important one.\r\n\nTARAS: About functional testing and acceptance testing. I think one of the things that I know personally from experience working with comprehensive acceptance test suites is that there is certainty that you get by expressing the behavior of the application in your tests. And I think what that certainty does is it replaces hope as opposed to having hope baked into your system where you think like you\'re hoping. I think for many people, they don\'t even perceive it as hope. They perceive it as reality. They see it as, "My application works this way." But really what\'s happening is there\'s a lot of trust that\'s built into that where you have to say like, "Yeah, I believe the system should work because I wrote it and I\'m good. And it should not be broken." But unless you have a mechanism that actually verifies this and actually insures this is the case, you are operating in the area of dreams and hopes and wishes, and not necessarily reality. And I think that\'s one of the things that\'s different. A lot of the processes around highlighting or shining light on the opaque areas of the development process. And it\'s actually not even just development process. It\'s actually the business process of running a development organization. Shining light in those areas is then what gives you the opportunity to replace hope with real validatable truth about your circumstances.\r\n\nCHARLES: And making it so that anyone can answer that question and discover that truth and discover that reality for themselves. So, generating the artifacts, putting them out there, and then letting anybody be the primary perceiver of what that artifact means in the context of the business, not just developers. And so, that kind of really explains preview apps quite neatly, doesn\'t it? Here we\'ve done some work. We are proposing a change. What are the artifacts that explain the ramifications of this change? So we run the test suite. That\'s one of the artifacts that explains and radiates the information so that people can be their own primary source. And look at it in a developer centric, although you can tell, any old person can tell if the test suite\'s failing, it\'s not a change that we should go with. But the preview app is something we take this hypothetical change, we build it, we put it out there and now, everyone can perceive it. And so, it calibrates the perception of reality and it eliminates hope. Which is like if your development process is based on hope, you are g yourself up for disaster. I like what you said that it implies a huge amount of trust in the development team. And you know what? If you have a cracked development team, that trust is earned and people will continually invest based on that trust. But the fundamentals are still fragile because they still can open up a crack between the expectation and the reality. And the problem is when that happens, the trust is destroyed. And it\'s during that crunch time, if it does happen that you lose credibility and it\'s not because you became a worse developer. It\'s not because your team is like lower performing, it\'s just that there was this divergence allowed to open. But then the problem is that really lowers the trust and that means that unfortunately that\'s going to have a negative knock on effect. And reasonably so. Because if you\'re an engineering manager or a product manager, you\'re something like this and you\'re losing trust in your development team and their ability to deliver what you talked about, then you\'re going to want to micromanage them more. The natural inclination is to try and be very defensive and interventionist and you might actually introduce a set of practices that inhibit the development cycle even further and lower the team\'s abilities to perform right when they need to do it the most, then you end up destroying more trust.\r\n\nTARAS: Yeah, it\'s a spiraling effect I think because it\'s in the process of trying to make things better. And then you start to introduce practices. Like maybe you\'re going to have meetings every day reviewing outstanding stories to try to get everybody on the same page, but now you\'re micromanaging development team. The development team starts to resent that and now you\'ve got this like people hating their job. It starts to get messier and dirtier and more complicated. And the root cause of that is that from day one, there was a lot of just [inaudible] about getting into it and just starting to write some code but what you didn\'t actually do is you didn\'t put in place the fundamentals of making sure that you can all observe a reality that is honest. And I think that kind of fundamental principle, it\'s interesting how when you actually start to kind of take this idea and when you start to think about it in different use cases, it actually tells you a lot about what\'s going on and you can actually use it to design new solutions.\r\n\nOne of the things that Frontside does, I don\'t know if those who\'ve kind of worked with us before might know this or might not, but we don\'t do blended rates anymore. Because we don\'t actually, one of the challenges with blended rates is that they hide the new ones that gives you the power to choose how to solve a problem.\r\n\nCHARLES: Yeah. There\'s a whole blog post that needs to be written on why blended rates are absolute poison for a consultancy. But this is the principle of why.\r\n\nTARAS: Yeah. I think it\'s poison for transparent consultancy because if you want to get the benefits of transparency, you have to be transparent about your people. Because alternatively what happens is that you start off relying on your company\'s reputation and then there is a kind of inherent lie in the way that the price points are set up because everybody knows that there is going to be a few senior people, there\'s going to be a few intermediate people, a few junior people. But these exact ratios of those or who is doing what, how much people are available, all of those things are kind of hidden inside of the consulting company so that they can manage their resources internally. And so what that does is it simplifies your communication with the client. But actually what it also does is it disempowers you to have certain difficult conversations when you need the most. And you could say, "Look, for this kind of work, we don\'t need to have more senior people working on this." We can have someone who is junior who is at like $100 an hour, $75 an hour as opposed to being $200 or $250 an hour. We can have that person working on this and we can actually very clearly define how certain work gets solved. It requires more work. But then what it does is it creates a really strong bond of honesty and transparency between you and your clients. And it gives you a way, like now the client starts to think about you as a resource that allows them to fulfill on their obligations in a very actionable way. They can now think about how they can use you as a resource to solve their problems. They don\'t need a filter that will process that and try to make it work within the organization. You essentially kind of become one unit. And I think that sense of unity is the fundamental piece that keeps consulting companies and clients glued together. It\'s the sense of like, "We can rely on this team to give us exactly what we want when we need it, and sometimes give us what we need that we don\'t know we need." But that bond is there. And that bond is strong because there is no lie in that relationship. You\'re very transparent about who are the people that\'s working on it. What are they actually going to be doing? How much is this costing us?\r\n\nCHARLES: It\'s worth calling out explicitly whether on the flip side of it is, is if you have a blended rate, which is the way that Frontside operated for, gosh, pretty much forever, is that people will naturally calibrate towards your most senior people. If I\'m going to be paying $200 an hour across the board, or $150 an hour across the board, or $300 across the board, whatever the price point is, they\'re going to want to extract the most value for that one price point. And so, it means that they\'re going to expect the most senior people and become resentful if what I\'m paying is $300 for a task. If I\'ve got five senior people, it\'s a better deal for me. For the same price to get five senior people than two senior people to a medium level people and one junior person. And so, it has two terrible effects. One is that they don\'t appreciate the senior people to be like, "Hey actually, these are people with extraordinary experience, extraordinary knowledge, extraordinary capability that will kick start your part." So they are under appreciated and then they\'re extremely resentful of the junior people. It\'s like, "I\'m paying the same rate for this very senior person as I am for this junior person? Get this person off my project." But if you say, "You know what, we\'re going to charge a fifth of the cost for this junior person and we\'re going to utilize them," then you\'re providing real value and they\'re appreciating it. They\'re like, "Oh, thank you for saving me so much money. We\'ve got this task that does not require your most senior person. That would be a misallocation of funds. I\'d be wasting money on them. But if you can charge me less and give me this junior person and they\'re going to do just as competent a job, but it\'s going to cost me a fifth of the money, then that\'s great. Thank you." So, it flips the conversation from \'get this god-damn junior person off my project\' to \'thank you so much for bringing this person on\'. It\'s so critical. But that\'s what that transparency can provide. It can totally turn a feeling of resentment into gratitude.\r\n\nTARAS: What\'s interesting is from business perspective, you make the same amount of money. In some cases, you actually make more money. I think in that way, it\'s a consulting company. But that\'s not the important part because the amount of value that\'s generated from having more granular visibility into what\'s happening is so much greater. It\'s kind of like with testing where any of those things where when you start to put, when you start to shine light on these kind of opaque areas and then you start to kind of flush out the gremlins that are hiding there, what you then start to do, what you kind of discover is this opportunity to have relationships with clients that are honest. So you could say, for example, like one of the things that we\'ve done recently is we actually have like 10-tier price point model, which allows us to to be really flexible about the kind of people that we introduce. So, there\'s a lot of details that go into the actual contracting negotiation. But what it does is it allows us to be very honest about the costs and work together with our clients, like actually really find a solution that\'s going to work really well for them. And then this is kind of a starting point when we start thinking about transparency in this kind of diverse way, you actually start to realize that there are additional benefits that you might have never been experienced before. One of the things that we found recently is that one of the initiatives that we kind of launched with one of our clients is we wanted to bring together, there\'s a general problem that exists in large projects, which is that if you have a really big company and you have like, let\'s say 20 or 30 interconnected services, your data domain, like the older data, kinds of data you work with is spread over a whole bunch of microservices spread over potentially a bunch of different development teams spread over a bunch of different locations. What usually has happened in the past is each one of those problems or the domain, the data domain has been kind of siloed into a specific application. We worked with a bank in the past and that being had for every, they had 80 countries. In each country they had 12 different industries, like insurance and mortgage and different kinds of areas of services they offered. And then for each of the country, for each of the service, they had a different application that provided that functionality. Then the next step is, let\'s not do that anymore because we now have something like 100, 150 apps, let\'s bring it all together under a single umbrella and let\'s create a single shared domain that we can then use. And so, a GraphQL becomes a great solution for that. But the problem is that making that change is crazy complicated because the people on the business side who understand how all the pieces fit together. On the other side, you have the developers who know where the data can come from and how to make all that real. And on the other side is there\'s like frontend implementers who actually build in the UIs that are consuming all these services.\r\n\nOn a project that we\'re working on right now is we\'re building a federated GraphQL gateway layer that is kind of connecting all these things, bringing all these things together. But the problem is that without very specific tooling to enable that kind of coming together of the frontend, the backend, the business people having coming together, creating a single point of conversation and having a single point of reference for all the different data that we have to work with and different data that is available to the gateway, without having something like that, without having that transparency in the actual data model, it is really difficult to make progress because you don\'t have shared terminology, you don\'t have shared understanding of the scope of the problem. There\'s a lot of dots in context that needs to be connected. And for anyone who has worked with enterprise, you know how big these problems get. And so what we\'ve done on a project that we\'re working on now is we actually aimed to bring transparency to this process. What we actually did is put in place, start to build an application that brings together all of the federated services into a visualization that different parties can be involved in. And so I think one of the kind of common patterns that we see with transparency in general is that we are including people in the process, in the development process that were previously not included. So in the past, they would be involved in the process either very early on or very late in the process, but they wouldn\'t be involved along the way. And so what this kind of transparency practice actually does is it allows us to kind of democratize and flatten the process of creating foundations for pieces that touch many different parts of the organization. And so this tool that we created allows everyone to be involved in the process of creating the data model that spans the entire organization and then have a single point of reference that everybody can go to and have a process for contributing to it. They don\'t have to be a developer. There\'s developers who consume it. There are business people that consume it. There are data modeling people that consume it. Like there\'s different people parties involved. But the end result is that everyone is on the same page about what it is that they\'re creating. And we\'re seeing the same kind of response as we saw with preview apps where people who previously didn\'t really have an opinion on development practices or how something gets built, all of a sudden they\'re participating in the conversation and actually making really valuable suggestions that developers couldn\'t really have exposure to previously because developers often don\'t have the context necessary to understand why something gets implemented in a particular way.\r\n\nCHARLES: Something beautiful to behold, really. And like I said, it\'s wonderful when a simple concept reveals things that had lay hidden before.\r\n\nTARAS: Yeah. It\'s a very interesting lens to look at things through. How transparent is this and how can we make it more transparent? I think asking that question and answering that question is what has been kind of giving us a lot of -- it had been very helpful in understanding our challenges in the work that we do on a daily basis and also in understanding how we could actually make it better.\r\n\nCHARLES: I apply this concept in action on my pull requests. I\'ve really been focusing on trying to make sure that if you look at my pull request, before you actually look at the code, you can pretty much understand what I\'ve done before you even look at the diff. The hallmark of a good pull request is basically if by reading the conversation, you understand what the implementation is going to be. There\'s not really any surprises there. It\'s actually hard to achieve that. Same thing with git history. Spending a lot of time trying to think like how can I provide the most transparent git history? That doesn\'t necessarily mean exactly the log of what happened moment to moment, day to day, but making sure that your history presents a clear story of how the application has evolved. And sometimes that involves a lot of rebasing and merging and branch management.\r\n\nI think another area that has been new for us, which this has revealed those things that I just described are areas where we\'re kind of re-evaluating already accepted principles against a new measure, but introducing an RFC process to actually a client project where we\'re making architectural decisions with our developers, the client\'s developers, external consultants. You\'ve got a lot of different parties, all of whom need to be on the same page about the architectural decisions that you\'ve made. Why are we doing this this way? Why are we doing modals this way? Why are we using this style system? Why are we using routing in this way? Why are we doing testing like this? These are decisions that are usually made in an ad hoc basis to satisfy an immediate need. It\'s like, "Hey, we need to do state management. Let\'s bring in Redux or let\'s bring in MobX or let\'s bring in whatever." And you want to hire experts to help you make that best ad hoc decision? Well, not really. I mean, you want to lean on their experience to make the best decision. But having a way of recording and saying this is the rationale for a decision that we are about to make to fulfill a need. And then having a record of that and putting it down in the book so that anybody who can come later. First of all, when the discussion is happening, everybody can understand the process that\'s going on in the development team\'s head. And then afterwards and it\'s particularly important is someone asks a question, "Why is this thing this way?" You can point directly to an RFC. And this is something that we picked up from the Ember community, but this is something that open source projects really by their very nature have to operate in a very highly transparent manner. And so, it\'s no surprise that that process came from the internet and an open source project. But it\'s been remarkably effective, I would say, in achieving consensus and making sure that people are satisfied with decisions, especially if they come on afterwards, after they\'ve been made.\r\n\nTARAS: We actually have this particular benefit that could experience that particular benefit today where one of the other things that this RFC process and transparency with the architecture, how that kind of benefits the development organization is that a lot of times when you are knee deep in doing some implementation, that is not a time you want to be having architectural conversations. In the same way like in a big football team, they huddle up before they go on a field. You can\'t be talking strategy and architecture and plans while you\'re on the football field. You have to be ready to play. And this is one of the things that the RFC process does is it allows us to say, "Look, right now we have a process for managing architecture so that with the RFC process you can go review our accepted RFCs. You can make proposals there." And that is a different process than the process that we\'re involved in on a daily basis, which is writing application, using architecture where we have in place. And so that in itself can be really helpful because well intentioned people can bring up these conversations because they really are trying to solve a problem, but that might not be the best time. And so having that kind of process in place and being transparent about how architecture decisions are made allows everyone to participate and it also allows you to prioritize conversations.\r\n\nCHARLES: Yeah. And that wasn\'t a practice that we had adopted previous to this, but it\'s something that seemed obvious that we should be doing. It\'s like, how can we make our architecture more transparent? Well, let\'s do this practice. So, I keep harping on this. But I think it\'s the hallmark of a good idea if it leads you to new practices and new tools. And we\'re actually thinking about adopting the RFC process for all of our internal developments, for maintaining our open source libraries.\r\n\nTARAS: There is something that we\'ve been working on that we\'re really excited about. So, there\'s a lot of stuff happening at Frontside. But one of the things that we\'ve been doing is working on something we call the transparent node publishing process, which is something that I think we originally drew inspiration from the way NativeScript has their repo set up. But one thing that\'s really cool about how they have things set up is that every pull request automatically is available, like everything is available. Very quickly, a pull request is available for you to play with and you can actually put it into your application as a published version in npm and actually see exactly if that pull request is going to work for you. You don\'t have to jump through hoops. You don\'t have to clone the repo, build it locally, link it. You don\'t have to do any of that stuff because if you see a pull request that has something that you want but then is not available in master, there\'s an instruction on the pull request that tells you, "Here\'s how you can install this particular version from npm." And so you essentially you\'re publishing. Every pull request automatically gets published to npm and you can just and install that specific version for that particular pull request in your project. That in itself I think is one of those things I suspect that is going to be talked about. It actually can alleviate a lot of problems that we have on a development processes because like the availability of the work of people who are participating in the project, there is kind of a built in barrier that we are essentially breaking down with this transparent node publishing process. And so, that\'s something that we\'re very close to to having it all on our repos and we\'re going to try it out and then hopefully share it with everyone on the internet.\r\n\nCHARLES: I didn\'t know that the NativeScript did this. I thought that the idea that came from it is like how can we apply these transparency principles to the way we maintain npm packages. The entire release process should be completely transparent, so that when I make a pull request, it\'s available immediately in a comment. And then furthermore, even when a pull request is merged, there\'s no separate step of let\'s get someone to publish it. It\'s just now it\'s on master. Now it is available as a production release. You close the latency and you close the gap and people perceive things as they are. There is nothing like, "Oh that emerged. When do I get this?" This is something that I can\'t stand ing public packages is you have some issue, you find out that someone also has had this issue, they\'ve submitted a pull request for it and then it\'s impossible to find if there\'s a version and what version actually s this? And it\'s even more complex between projects that actually do backporting of fixes to other versions. So I might be on version two of a project. Version three is the most recent one, but I can\'t version three because I might be dependent on some version two APIs. But I really need this fix. Well, has it been backported? I don\'t know. Maybe upgrading is what I have to do, but maybe downgrading. Or if I\'m on the same major release version, maybe there\'s been 10 pull requests, but there\'s been no release to npm. And it can be shockingly difficult to find out if something is even publicly available. And the transparency principle comes in to, "Hey, if I see it on GitHub, if I see it there, then there\'s something there that I can touch and I can perceive for myself to see if my issue has been resolved or if the things work as I expect."\r\n\nTARAS: I\'m really excited about this. I\'m really excited about this kind of clarification, this crystallization of transparency. And I\'m also seeing our clients starting to apply it to solving problems within their organization as well is very inspiring.\r\n\nCHARLES: Yeah, it is. It is really exciting. And honestly, I feel like we\'ve got one of those little triangular sticks that people use to find water. I feel like we have a divination stick. And I\'m really excited to see what new tools and practices it actually predicts and leads us to.\r\n\nTARAS: Me too. I\'m really excited to hear if anyone likes this idea. Send us a tweet and let us know what you\'re seeing for yourself about this because I think it\'s a really interesting topic and I\'m sure there\'s going to be a lot that people can do with this idea in general.\r\n\nCHARLES: Thank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside, or over just plain old email at @frontside.io. Thanks and see you next time.\r\n

Rich Harris talks about Svelte and Reactivity. Rich Harris: Graphics Editor on The New York Times investigations team. Resources: Svelte Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: Coming soon!

Rich Harris talks about Svelte and Reactivity.\r\n\nRich Harris: Graphics Editor on The New York Times investigations team.\r\n\nResources:\r\n\n\nSvelte\r\n\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n\nTranscript:\r\n\nCHARLES: Hello and welcome to The Frontside Podcast, a place where we talk er interfaces and everything that you need to know to build them right.\r\n\nTARAS: It\'s actually a really nice, Rich and I\'m really, really happy to have a chance to actually chat with you about this because Svelte is a really fun piece technology. In many ways, it\'s interesting to see our technology evolve and our industry evolve through innovation, real innovation. I think Svelte 3 has really been kind of that next thought provoking technology that kind of makes you think about different ways that we can approach problems in our space. So, really excited to chat with you about this stuff.\r\n\nRICH: Well, thank you. Excited to be here.\r\n\nTARAS: I think quite a lot of people know, Rich, about your history, like how you got into what you\'re doing now. But I\'m not sure if Charles is aware, so if you could kind of give us a little bit of a lowdown on where you kind of come from in of your technical background and such.\r\n\nRICH: Sure. I\'ll give you the 30-second life history. I started out as a reporter at a financial news organization. I had a Philosophy Degree and didn\'t know what else to do with it. So, I went into journalism. This was around the time of the great recession. And within a few weeks of me ing this company, I watched half of my colleagues get laid off and it\'s like, "Shit, I need to make myself more employable." And so gradually, sort of took on more and more technical responsibilities until I was writing JavaScript as part of my day job. Then from there, all these opportunities kind of opened up. And the big thing that I had in mind was building interactive pieces of journalism, data-driven, personalized, all of that sort of thing, which were being built at places like the New York Times, and The Guardian, and the BBC. That was the reason that I really wanted to get into JavaScript. And that\'s guided my career path ever since.\r\n\nCHARLES: It\'s interesting that this D3 and all that did come out of journalism.\r\n\nRICH: It\'s not a coincidence because when you\'re working under extreme time pressure and you\'re not building things with a view to maintain them over a long period of time, you just need to build something and get it shipped immediately. But it needs to be built in a way that is going to work across a whole range of devices. We\'ve got native apps, we\'ve got [inaudible], we\'ve got our own website. And in order to do all that, you need to have tools that really guide you into the pit of success. And D3 is a perfect example of that. And a lot of people have come into JavaScript through D3.\r\n\nCHARLES: And so, are you still working for the same company?\r\n\nRICH: No. That\'s ancient history at this point.\r\n\nCHARLES: Because I\'m wondering, are you actually getting to use these tools that you\'ve been building to actually do the types of visualizations and stuff that we\'ve been talking about?\r\n\nRICH: Very much so. I moved to The Guardian some years ago. And then from there, moved to Guardian US, which has an office in New York. And it was there that I started working on Svelte. I then moved to the New York Times and I\'m still working on Svelte. I\'ve used it a number of times to build things at the New York Times and the people have built things with it too. And so, yeah, it\'s very much informed by the demands of building high performance interactive applications on a very tight deadline.\r\n\nCHARLES: Okay, cool. So I\'ve probably used, I mean, I\'m an avid reader of both Guardian and the New York Times, so I\'ve probably used a bunch of these visualizations. I had no idea what was driving them. I just assumed it was all D3.\r\n\nRICH: There is a lot of D3. Mike Bostock, the creator of D3, he was a linchpin at the graphics department for many years. Unfortunately we didn\'t overlap. He left the Times before I ed the Times, but his presence is still very much felt in the department. And a lot of people who are entering the industry, they\'re still becoming database practitioners by learning from D3 examples. It\'s been a hugely influential thing in our industry.\r\n\nTARAS: How long is a typical project? How long would it take to put together a visualization for an article that we typically see?\r\n\nRICH: It varies wildly. The graphics desk is about 50 strong and they will turn around things within a day. Like when the Notre Dame burnt down a couple of months ago, my colleagues turned around this interactive scroll driven webGL 3D reconstruction of how the fire spreads through the cathedral in less than 24 hours, which was absolutely mind blowing. But at the same time, there are projects that will take months. I work on the investigations team at the Times. And so, I\'m working with people who are investigating stories for the best part of the year or sometimes more. And I\'m building graphics for those. And so that, it\'s two very different timescales, but you need to be able to accommodate all of those different possibilities.\r\n\nCHARLES: So, what does the software development practice look like? I mean, because it sounds like some of this stuff, are you just throwing it together? I guess what I mean by that is, I guess the projects that we typically work on, three months is kind of a minimum that you would expect. So, you go into it, we need to make sure we\'ve got good collaboration practices around source control and continuous integration and testing and all this stuff. But I mean, you\'re talking about compressing that entire process into a matter of hours. So what, do you just throw right out the window? What do you say? "We\'re just doing a live version of this."\r\n\nRICH: Our collaboration processes consist of sitting near each other. And when the time calls for it, getting in the same room as each other and just hammering stuff out on the laptop together. There\'s no time for messing around with continuous integration and writing tests. No one writes tests in the news graphics, it\'s just not a thing.\r\n\nCHARLES: Right. But then for those projects that stretch into like three months, I imagine there are some. Do you run into like quality concerns or things like that where you do have to take into some of those practices? I\'m just so curious because it sounds like there\'s actually, the difference between two hours and two months is, that\'s several orders of magnitude and complexity of what you\'re developing.\r\n\nRICH: It is. Although I haven\'t worked on a news project yet that has involved tests. And I know that\'s a shocking ission to a lot of people who have a development background, but it\'s just not part of the culture. And I guess the main difference between the codebase for a two-hour project and a two-month project is that the two-month project will strive to have some reasonable components. And that\'s, I think, the main thing that I\'ve been able to get out of working on the kinds of projects that I do is instead of just throwing code at the page until it works, we actually have a bit of time to extract out common functionality and make components that can be used in subsequent interactives. So, things like scroll driven storytelling, that\'s much easier for me now than it was when I first built a scroll driven storytelling component like a couple of years ago.\r\n\nCHARLES: Yeah. That was actually literally my next question is how do you bridge that, given that you\'ve got kind of this frothy experimentation, but you are being, sounds like, very deliberate about extracting those tools and extracting those common components? And how do you find the time to even do that?\r\n\nRICH: Well, this is where the component driven mindset comes in really handy, I think. I think that five or 10 years ago when people thought in of libraries and scripts, there wasn\'t like that good unit of reusability that wasn\'t the sort of all encoming, like a component is just the right level of atomicity or whatever the word is. It makes sense to have things that are reusable but also very easy to tweak and manipulate and adapt to your current situation. And so, I think that the advent of component oriented development is actually quite big for those of us working in this space. And it hasn\'t really caught on yet to a huge degree because like I say, a lot of people are still coming with this kind of D3 script based mindset because the news industry, for some interesting and historical reasons, is slightly out of step with mainstream mode development in some ways. We don\'t use things like Babel a lot, for example.\r\n\nCHARLES: That makes sense, right? I mean, the online print is not like it\'s a React application or it\'s not like the application is all encoming, so you really need to have a light footprint, I would imagine, because it really is a script. What you\'re doing is scripting in the truest sense of the word where you essentially have a whole bunch of content and then you just need to kind of --\r\n\nRICH: Yeah. And the light footprint that you mentioned is key because like most new sites, we have analytics on the page and we have ads and we have comments and all of these things that involve JavaScript. And by the time our code loads, all of this other stuff is already fighting for the main thread. And so, we need to get in there as fast as we can and do our work with a minimum fuss. We don\'t have the capacity to be loading big frameworks and messing about on the page. So that again is one of these sort of downward pressures that kind of enforces a certain type of tool to come out of the news business.\r\n\nTARAS: A lot of the tooling that\'s available, especially on like the really fatter, bigger frameworks, the tools that you get with those frameworks, they benefit over long term. So if you have like a long running project, the weight of the abstractions, you\'ve experienced that benefit over time and it adds up significantly. But if you\'re working to ship something in a day, you want something that is just like a chisel. It does exactly what you want it to do. You want to apply it in exactly the right place and you want to get it done exactly, like you want the outcome to be precise.\r\n\nRICH: That\'s true. And I think a lot of people who have built large React apps, for example, or large Ember apps, they sort of look at Svelte and think, "Well, maybe this isn\'t going to be applicable to my situation," because it has this bias towards being able to very quickly produce something. And I\'m not convinced that that\'s true. I think that if you make something easier to get started with, then you\'re just making it easier. If you build something that is simple for beginners to use, then you\'re also building something simple for experts to use. And so, I don\'t necessarily see it as a tradeoff, I don\'t think we\'re trading long-term maintainability for short term production. But it is certainly a suspicion that I\'ve encountered from people.\r\n\nTARAS: This is something that we\'ve also encountered recently. It\'s been kind of a brewing discussion inside a front side about the fact that it seems to be that certain problems are actually better to rewrite than they are to maintain or refactor towards an end goal. And we found this, especially as the tools that we create have gotten more precise and more refined and simplified and lighter, it is actually easier to rewrite those things five times than it is to refactor it one time to a particular place that we want it to be. And it\'s interesting, like I find this to be very recent, this idea is blossoming in my mind very recently. I didn\'t observe this in the past.\r\n\nCHARLES: Do you mean in the sense that like if a tool is focused enough and a tool is simple enough, then refactoring is tantamount to a rewrite if you\'re talking about 200 or 300 lines of code? Is that what you mean?\r\n\nTARAS: Yeah. If you\'re sitting down to make a change or you have something in mind, it is actually easy to say, "Let\'s just start from scratch and then we\'re going to get exactly the same place in the same amount of time." But this kind of mantra of not rewriting makes me think about that, makes me question whether that\'s actually something that is always the right answer.\r\n\nRICH: I definitely question that conventional wisdom at all levels, as well. I started a bundler called Rollup as well as Svelte more recently. And Rollup was the second JavaScript bundler that I wrote, because the first one that I wrote wasn\'t quite capable of doing the things that I wanted. And it was easier to just start from scratch than to try and shift the existing base of its predecessor over to this new way of doing things. Svelte 3 is a more or less complete rewrite. Svelte has had multiple, more or less, complete rewrite. Some of them weren\'t breaking changes. But Svelte itself was a rewrite of an earlier project that I\'d started in 2013. And so in my career, I\'ve benefited massively from learning from having built something. But then when the time comes and you realize that you can\'t change it in the ways that you need to change it, just rewrite it.\r\n\nAnd I think that at the other end of the spectrum, the recent debate about micro frontend has largely missed this point. People think that the benefit of the micro frontend is that people don\'t need to talk to each other, which is absolute nonsense. I think the benefit of this way of thinking about building applications is that it optimizes for this fact of life that we all agree is inevitable, which is that at some point, you\'re going to have to rewrite your code. And we spend so much energy trying to optimize for the stability of a code base over the long term. And in the process, lock ourselves into architectural and technical decisions that don\'t necessarily make sense three or four years down the line. And I think as an industry, would be a lot better placed if we all started thinking about how to optimize for rewrites.\r\n\nCHARLES: So for those of us who aren\'t familiar, what is the debate surrounding micro frontends? This is actually something I\'ve heard a lot about, but I\'ve actually never heard what micro frontends actually are.\r\n\nRICH: Yeah. I mean, to be clear, I don\'t really have a dog in this fight because I\'m not building products, but the nub of it is that typically if you\'re building a website that maybe has like an page, maybe it has a a settings page, maybe it has product pages, whatever. Traditionally, these would all be parts of a single monolithic application. The micro frontend approach is to say, "Well, this team is going to own the settings page. This team is going to own the product page." And they can use whatever technologies they want to bring that about. And the detractors sort of attack a straw man version of this, "You\'re going to have different styles in every page. You\'re going to have to load Vue on one page. You\'re going to have to load React on the other page. It\'s going to be a terrible experience," when actually its proponents aren\'t suggesting that at all. They\'re suggesting that people from these different teams coordinate a lot more that are free to deviate from some kind of grand master architectural plan when it\'s not suitable for a given task. And darn right. I think it means that you have a lot more agility as an engineering organization than you would if you\'re building this monolithic app where someone can\'t say, "Oh, we should use this new tool for this thing. We should use microstates when the rest of the organization is using Google docs." It\'s not possible. And so, you get locked into the decisions of a previous generation.\r\n\nCHARLES: Right. No, it makes sense. It\'s funny because my first reaction is like, "Oh my goodness, that\'s a potential for disaster." The klaxon\'s going to go off in your head, but then you think, really then the work is how do you actually manage it so it doesn\'t become a disaster. And if you can figure that out, then yeah, there is a lot of potential.\r\n\nRICH: Yeah. People always try and solve social problems with technology. You solve social problems with social solutions.\r\n\nCHARLES: Right. And you have to imagine it too, it depends on the application, right? I think Amazon, the Amazon website is developed that way where they have different teams that are responsible even down to little content boxes that are up on the toolbar. And the site doesn\'t really, it shows, right? Like it shows like this is kind of like slapped together, but that\'s not what they need. They don\'t need it to not look like there\'s slight variation with the different ways that things behave. They need to be showing for their business to work. They need to be showing the right thing at the right time. And that\'s the overriding concern. So having it look very beautiful and very coherent isn\'t necessarily a thing. Same thing in Spotify, used as another example of this. I didn\'t know if it was called micro frontends, but I know that they\'ve got a similar type thing, but they are clearly the experience and having it look coherent is more important. And so, they make it work somehow. And then like you\'re saying, it probably involves groups of people talking to other groups of people about the priorities.\r\n\nSo yeah, it doesn\'t sound to me like just like you\'re going to adopt micro frontends guarantees one particular set of outcomes. It really is context dependent on what you make of it.\r\n\nRICH: Totally.\r\n\nTARAS: I\'m curious though, so with Svelte, essentially for your reactivity engine, you have to compile to get that reactive behavior.\r\n\nRICH: Yeah.\r\n\nTARAS: How does that play with other tools like when you actually integrate it together? I\'ve never worked with Svelte on a large project, so I can\'t imagine what it looks like at scale. I was wondering if you\'ve seen those kind of use cases and what that ends up, if there\'s any kind of side effects from that.\r\n\nRICH: As you say, the reactivity within a component is only in the local state within that component or to state that is patched in as a prop from a parent component. But we also have this concept called a store. And a store is just a project that represents a specific value and you import it from svelte/store. And there are three types of store that you get out of the box. A writable, a readable and a derived. And a writeable is just, var count = writable (0) and then you can update that and you can set it using methods on that store. Inside your marker, you can reference or in fact inside the script block in the component, you can reference the value of that store just by prefacing it with a dollar sign. And the compiler sees that and says, "Okay, we need to subscribe to this store as value and then assign it and apply the reactivity." And that is the primary way of having state that exists outside the component hierarchy. Now, I mentioned the writable, readable, and derived are the built in stores that you get, but you can actually implement your own stores. You just need to implement this very simple contract. And so,, it\'s entirely possible to use that API to wrap any state management solution you have. So you can wrap redux, you can wrap microstates, you can wrap state, you can wrap whatever it is, whatever your preferred state management solution is, you can adapt it to use with Svelte. And it\'s very sort of idiomatic and streamlined. Like it takes care of unsubscriptions when the component is unmounted. All of that stuff is just done for you.\r\n\nCHARLES: Digging a little bit deeper into the question of integration, how difficult would it be to take wholesale components that were implemented in Svelte and kind of integrate them with some other component framework like React?\r\n\nRICH: If the component is a leaf node, then it\'s fairly straightforward. There is a project called react-svelte which is, I say project, it\'s like 20 lines of code and I don\'t think it\'s [inaudible] they did for Svelte 3, which I should probably do. But that allows you to use a Svelte component in the context of React application, just using the component API the same way that you would [inaudible] or whatever. You can do that inside a React component. Or you could compile the Svelte component to a web component. And this is one of the great benefits of being a compiler is that you can target different things. You can generate a regular JavaScript class and you\'ve got an interactive application. Or you can target a server side rendering component which will just generate some html for some given state which can then later be hydrated on the client. Or you can target a web component which you can use like any other element in the context of any framework at all. And because it\'s a compiler, because it\'s discarding all of the bits of the framework that you\'re not using, it\'s not like you\'re bundling an entire framework to go along with your component. And I should mention while I\'m talking about being able to target different outputs, we can also, as a NativeScript project, you can target iOS and Android that same way. Where it gets a little bit more complicated is if it\'s not a leaf node. If you want to have a React app that contains a Svelte component that has React [inaudible], then things start to get a little bit more unwieldy, I think. It\'s probably technically possible, but I don\'t know that I would recommend it. But the point is that it is definitely possible to incrementally adopt Svelte inside an existing application, should that be what you need to do.\r\n\nCHARLES: You said there\'s a NativeScript project, but it sounds to me like you shouldn\'t necessarily need NativeScript, right? If you\'re a compiler, you can actually target Android and you could target iOS directly instead of having NativeScript as an intermediary, right?\r\n\nRICH: Yes. If, if we had the time to do the work, then yes. I think the big thing there would be getting styles to work because Svelte components have styles. And a regular style tag just to CSS and you can\'t just throw CSS in a native app.\r\n\nCHARLES: Right. Sometimes, I feel like it\'d be a lot cooler if you could.\r\n\n[Laughter]\r\n\nRICH: NativeScript really is doing a lot of heavy lifting. Basically what it\'s doing is it\'s providing a fake dom. And so, what the NativeScript does is it targets that dom instead of the real dom and then NativeScript turns that into the native instructions.\r\n\nCHARLES: Okay. And you can do that because you\'re a compiler.\r\n\nTARAS: Compilers has been on our radar for some time, but I\'m curious like what is your process for figuring out what it should compile to? Like how do you arrive at the final compile output? Manually, have you written that code and then, "I\'m going to now change this to be dynamically generated." Or like how do you figure out what the output should be?\r\n\nRICH: That\'s pretty much it. Certainly, when the project started, it was a case of, I\'m going to think like a compiler, I\'m going to hand convert this declarative component code into some framework plus JavaScript. And then once that\'s done, sort of work backwards and figure out how a compiler would generate that code. And then the process, you do learn certain things about what the points of reusability are, which things should be abstracted out into a shared internal helper library and what things should be generated in line. The whole process is designed to produce output that is easy for a human to understand and reason about. It\'s not like what you would imagine compile [inaudible] to be like, it\'s not completely inscrutable. It\'s designed to be, even to that level of being well formatted, it\'s designed to be something that someone can look at and understand what the compiler was thinking at that moment. And there\'s definitely ways that we could change and improve it. There are some places where there\'s more duplication than we need to have. There are some places where we should be using classes instead of closures for performance and memory benefits. But these are all things that once you\'ve got that base, having gone through that process, that you can begin to iterate on.\r\n\nCHARLES: It\'s always curious to me about when is the proper time to move to a compiler, because when you\'re doing everything at runtime, there\'s more flexibility there. But at what point do you decide, "You know what? I know that these pathways are so well worn that I\'m going to lay down pavement. And I\'m going to write a compiler." What was the decision process in your mind about, "Okay, now it\'s time." Because I think that that\'s maybe not a thought that occurs to most of us. It\'s like, "I had to write a compiler for this." Is this something that people should do more often?\r\n\nRICH: The [inaudible] of \'this should be a compiler\' is one that is worth sort of having at the back of your head. I think there are a lot of opportunities not just in DUI framework space but in general, like is there some way that we can take this work that is currently happening at runtime and shift it into a step that only happens once. That obviously benefits s. And very often we find that benefits developers as well. I don\'t think there was a point at which I said, "Oh, this stuff that\'s happening at runtime should be happening at compile time." It was more, I mean, the actual origin has felt that it was a brain worm that someone else infected me with. Judgment is a very well known figure in the JavaScript world. He had been working on this exact idea but hadn\'t taken it to the point where he was ready to open source it. But he had shared like his findings and the general idea and I was just immediately smitten with this concept of getting rid of the framework runtime. At the time, the big conversation happening in the JavaScript community was about the fact that we\'re shipping too much JavaScript and it\'s affecting startup performance time. And so the initial thought was, "Well, maybe we can solve that problem by just not having the runtime." And so, that was the starting point with Svelte. Over time, I\'ve come to realize that that is maybe not the main benefit. That is just one of the benefits that you get from this approach. You also get much faster update performance because you don\'t have to do this fairly expensive virtual dom different process. Lately, I\'ve come to think that the biggest win from it is that you can write a lot less code. If you\'re a compiler, then you\'re not kind of hemmed in by the constraints of the language, so you can almost invent your own language. And if you can do that, then you can do the same things that you have been doing with an API in the language itself. And that\'s the basis of our system of reactivity, for example. We can build these apps that are smaller and by extension, less bug prone and more maintainable.\r\n\nI just wanted to quickly address the point you made about flexibility. This is a theoretical downside of being a compiler. We\'re throwing away the constraints about the code needing to be something that runs in the browser, but we\'re adding a constraint, which is that the code needs to be statically analyzable. And in theory, that results in a loss of flexibility. In practice, we haven\'t found that to affect the things that we can build. And I think that a lot of times when people have this conversation, they\'re focusing on the sort of academic concepts of flexibility. But what matters is what can you build? How easy is it to build a certain thing? And so if empirically you find that you\'re not restricted in the things that you can build and you can build the same things much faster, then that academic notion of flexibility doesn\'t, to my mind, have any real value.\r\n\nCHARLES: Hearing you talk reminded me of kind of a quote that I heard that always stuck with me back from early in my career. I came into programming through Perl. Perl was my first language and Perl is a very weird language. But among other things, you can actually just change the way that Perl parses code. You can write Perl that makes Perl not throw, if that makes any sense. And when asked about this feature, the guy, Larry Wall, who came up with Perl, he\'s like, "You program Perl, but really what you\'re doing is you\'re programming Perl with a set of semantics that you\'ve negotiated with the compiler." And that was kind of a funny way of saying like, "You get to extend the compiler yourself." Here\'s like the default set of things that you can do with our compiler, but if you want to tweak it or add or modify, you can do that. And so, you can utilize the same functionality that makes it powerful in the first place. You can kind of inject that whole mode of operation into the entire workflow. Does that make sense? That\'s like a long way of saying, have you thought about, and is it possible to kind of extend the Svelte compiler as part of a customization or as part of the Svelte programming experience?\r\n\nRICH: We have a very rudimentary version of that, which is pre-processing. There\'s an API that comes with Svelte called preprocess. And the idea there is that you can in some code and it will do some very basic, like it will extract your styles, it will extract your script and it will extract your markup. And then it will give you the opportunity to replace those things with something else. So for example, you could write some futuristic JavaScript and then compile it with Babel before it gets ed to the Svelte compiler, which uses acorn and therefore needs to be able to have managed other scripts so that it can construct an abstract syntax tree. A more extreme version of that, people can use [inaudible] to write their markup instead of html. You can use Sass and Less and things like that. Generally, I don\'t recommend that people do because it adds these moving parts and it makes like a lot of bug reports of people just trying to figure out how to get these different moving parts to operate together. I don\'t know, it means that your editor plugins can\'t understand what\'s inside your style tag all of a sudden and stuff like that. So, it definitely adds some complexity, but it is possible.\r\n\nAt the other end, at a slightly more extreme level, we have talked about making the cogeneration part plugable so that for example, the default renderer and the SSR renderer are just two examples of something that plugs into the compiler that says, "Here is the component, here\'s the abstract syntax tree, here\'s some metadata about which values are in scope," all of this stuff and then go away and generate some code from this. We haven\'t done that so far, partly because there hasn\'t been a great demand for it, but also because it\'s really complicated. As soon as you turn something into a plugin platform, you just magnify the number of connection points and the number of ways that things could go wrong by an order of magnitude. And so, we\'ve been a little bit wary of doing that, but it is something that we\'ve talked about primarily in the context of being able to do new and interesting things like target webGL directly or target the command line. There are renders for React that let you build command line apps using React components. And like we\'ve talked about, maybe we should be able to do that. Native is another example. The NativeScript integration as you say, it could be replaced with the compiler doing that work directly, but for that to work presently, that would mean that all of that logic would need to sit in core. And it would be nice if that could be just another extension to the compiler. We\'re talking about a lot of engineering effort and there\'s higher priority items on our to do list at the moment. So, it\'s filed under one day.\r\n\nCHARLES: Right. What are those high priority items?\r\n\nRICH: The biggest thing I think at the moment is TypeScript integration. Surprisingly, this is probably like the number one feature request I think is that people want to be able to write Typescript inside the Svelte components and they want to be able to get TypeScript when they import the Svelte component into something else. They want to be able to get completion [inaudible] and type checking and all the rest of it. A couple of years ago, that would\'ve been more or less than thinkable but now it\'s like table stakes is that you have to have first-class TypeScript .\r\n\nCHARLES: Yeah, TypeScript is as popular as Babel these days, right?\r\n\nRICH: Yeah, I think so. I don\'t need to be sold on the benefits. I\'ve been using TypeScript a lot myself. Svelte is written in TypeScript, but actually being able to write it inside your components is something that would involve as hacking around in the TypeScript compiler API in a way that, I don\'t know if anyone actually or any of us on the team actually knows how to do. So, we just need to spend some time and do that. But obviously when you\'ve got an open source project, you need to deal with the bugs that arise and stuff first. So, it\'s difficult to find time to do a big project like that.\r\n\nCHARLES: So, devil\'s advocate here is if the compiler was open for extension, couldn\'t a TypeScript be just another plugin?\r\n\nRICH: It could, but then you could end up with a situation where there\'s multiple competing TypeScript plugins and no one\'s sure which ones are used and they all have slightly different characteristics. I always think it\'s better if these things that are common feature requests that a lot of people would benefit from, if they\'re built into the project themselves. I go really light in the batteries included way of developing and I think this is something that we\'ve sort of drifted away from in the frontend world over the last few years, we\'ve drifted away from batteries included towards do it yourself.\r\n\nCHARLES: Assemble the entire thing. Step one, open the box and pour the thousand Lego pieces onto the floor.\r\n\nRICH: Yeah, but it\'s worse than that because at least, with a Lego set, you get the Lego pieces. It\'s like if you had the Lego manual showing you how to build something, but you were then responsible for going out and getting the Lego pieces, that\'s frontend development and I don\'t like it.\r\n\nCHARLES: Right. Yeah. I don\'t like that either. But still, there\'s a lot of people advocating directly. You really ought to be doing everything completely and totally yourself.\r\n\nRICH: Yes.\r\n\nCHARLES: And a lot of software development shops still operate that way.\r\n\nRICH: Yeah. I find that the people advocating for that position the most loudly, they tend to be the maintainers of the projects in question. The whole small modules philosophy, they exist for the benefit primarily of library authors and framework authors, not for the benefit of developers, much less s. And the fact that the people who are building libraries and frameworks tend to have the loudest megaphones means that that mindset, that philosophy is taken as a best practice for the industry as a whole. And I think it\'s a mistake to think that way.\r\n\nTARAS: There is also, I think, a degree of a sliding scale where you start off with like as the more experience you get, because there is more experience you get closer, you get to that kind of wanting granular control and then they kind of slides down towards granular control and then slice back up to, once you\'ve got a lot of experience, you\'re like, "Okay, I don\'t want this control anymore." And then you kind of cast that and you get into like, "I\'m now responsible for tools that my team uses," and now you\'re back to wanting that control because you want things to be able to click together. It\'s kind of like a way that your interest in that might change over time depending on your experience level and your position in the organization. So yeah, there\'s definitely different motivating factors. Like one of the things that we\'ve been thinking a lot about is deg tools that are composable and granular at individual module level, but combined together into a system for consumption by regular people. So like finding those primitives that will just click together when you know how to click them together. But when you\'re consuming them, just feel like a holistic whole, but at the same time not being monolithic. That\'s a lot of things to figure out and it\'s a lot of things to manage over time, but that\'s solely the kind of things we\'ve been thinking about a lot.\r\n\nRICH: I think that\'s what distinguishes the good projects that are going to have a long lifespan from the projects that are maybe interesting but don\'t have a long shelf life is whether they\'re designed in such a way that permits that kind of cohesion and innovation tradeoff, if you think of it as a trade off. Anyone can build the fastest thing or the smallest thing or the whatever it is thing. But building these things in a way that feels like it was designed holistically but is also flexible enough to be used with everything else that you use, that\'s the real design challenge.\r\n\nCHARLES: It\'s hard to know where to draw that line. Maybe one good example of this and, these are actually two projects that I\'m not particularly a fan of, but I think they do a good job of operating this way. So, I guess in that sense, it means I can even be more honest about it. I don\'t particularly care for Redux or like observables, but we ended up using, in one of our last React projects, we had to choose between using Redux-Saga and Redux-Observable. The Redux-Observable worked very well for us. And I think one of the reasons is because they both had to kind of exist. They had to kind of co-exist is their own projects. Like Redux exists as its own entity and Observables exist as their own kind of whole ecosystem. And so, they put a lot of thought in like what is the natural way in which these two primitives compose together? As opposed to the Saga, which I don\'t want to disparage the project because I think it actually is a really good project. There\'s a lot of really good ideas there but because it\'s more like just bolted on to Redux and it doesn\'t exist outside of the ecosystem of Redux and the ideas can\'t flourish outside and figure out how it interfaces with other things. Like the true primitive is still unrevealed there. And so, whereas I feel like with Redux you actually have to really, really true primitives. Now, they\'re not necessarily my favorite primitives, but they are very refined and very like these do exactly what they are meant to do. And so when you find how they connect together, that experience is also really good. And the primitive that arises there I think ends up being better. Is that an example of what you guys are talking about?\r\n\nRICH: Maybe. [Laughs]\r\n\nTARAS: No, I think so. I mean, it\'s distilling to the essence, the core of what you\'re trying to do and then be able to combine it together. I mean, that\'s been kind of the thing that we\'ve been working on at the Frontside. But also within this context, it makes me think of how does a compiler fit into that? How does that work with the compiler? It\'s just like when you add the compiler element, it just makes it like my mind just goes poof!\r\n\n[Laughter]\r\n\nCHARLES: Yeah, exactly. That\'s why I keep coming back to like, how do you, and maybe I haven\'t, you just have to kind of go through the experience, but it feels like maybe there\'s this cycle of like you build up the framework and then once it\'s well understood, you throw the framework away in favor of like just wiring it straight in there with the compiler and then you iterate on that process. Is that fair to say?\r\n\nRICH: Kind of, yeah. At the moment, I\'m working on this project, so I referred a moment ago to being able to target webGL directly. At the moment, the approach that I\'m taking to building webGL apps is to have webGL components inside Svelte in this project called SvelteGL. And we\'ve used it a couple of times at the Times. It\'s not really production ready yet, but I think it has some promise. But it\'s also slightly inefficient, like it needs to have all of the shade of code available for whichever path you\'re going to take, whatever characteristics your materials have, you need to have all of the shade of code. And if we\'re smart about it, then the compiler could know ahead of time which bits of shade of code it needed to include. At the moment, it just doesn\'t have a way of figuring that out. And so that would be an example of paving those cow paths. Like if you do try and do everything within the compiler universe, it does restrict your freedom of movement. It\'s true. And to qualify my earlier statements about how the small modules philosophy is to the benefit of authors over developers, it has actually enabled this huge flourishing of innovation, particularly in the React world. We\'ve got this plethora of different state management solutions and CSS and JS solutions. And while I, as a developer, probably don\'t want to deal with that, I just want there to be a single correct answer. It\'s definitely been to the advantage of the ecosystem as a whole to have all of this experimentation. Then in the wild, there are projects like Svelte they can then take advantage of. We can say, "Oh well, having observed all of this, this is the right way to solve this problem." And so, we can kind of bake in that and take advantage of the research that other people have done. And I think we have made contributions of our own but there is a lot of stuff in Svelte like the fact that data generally flows one way instead of having [inaudible] everywhere. Things like that are the results of having seen everyone make mistakes in the past and learning from them. So, there are tradeoffs all around.\r\n\nTARAS: One thing on topic of data flow here and there, one thing that I\'ve been kind of struggling to compute is the impact of that as opposed to something where you have like one directional data flow because it seems like conceptually it\'s really simple. You set a property like in two way balance system, like you just propagate through stuff but we don\'t really have a way, you don\'t have any way of assessing what is the true impact of that computation. Like what is the cost of that propagation where I think it\'s almost easier to see the cost of that computation if you have like one directional data flow because you know that essentially everything between the moment that you invoke transition to computing the next state, that is the cost of your computation where you don\'t have that way of computing the result in a two way balance system. Something like Ember Run Loop or mobx or zones, Vues, reactive system. All these systems make it really difficult to understand what is the real cost of setting state. And that\'s something that I personally find difficult because this clarity that you have about the one directional data flow and what it takes to compute the next state, it\'s almost like because that cost is tangible where you\'re thinking about like mutation of objects and tracking their change like that cost is almost immeasurable. It just seems like a blob of changes that they have to propagate. I don\'t know. That\'s just something that I\'ve been thinking a lot because especially with the work that we\'ll be doing with microstates because as you\'re figuring out what the next state is, you know exactly what operations are performed in a process where that might not be the case with the system that tracks changes like where you\'d have with zones or with Ember Run Loop, or Vue.\r\n\nRICH: I would agree with that. The times that I found it to be beneficial to deviate from the top-down ideology is when you have things like form elements and you want to bind to the values of those form elements. You want to use them in some other computation. And when you do all that by having props going in and then events going out and then you intercept the event and then you set the prop, you\'re basically articulating what the compiler can articulate for you more effectively anyway. And so conceptually, we have two way bindings within Svelte, but mechanically everything is top down, if that makes sense.\r\n\nCHARLES: Is it because you can analyze the tree of top down and basically understanding when you can cheat. This might be really over-simplistic, but if you\'re kind of with the event, you\'re collecting the water and then you have to put it way up on top of the thing and it flows down. But if you can see the entire apparatus, you can say, "Actually, I\'ve got this water and it\'s going to end up here, so I\'m just going to cheat and put it over right there." Is that the type of thing that you\'re talking about where you\'re effectively getting a two way binding, but you\'re skipping the ceremony.\r\n\nRICH: It\'s kind of writing the exact same code that you would write if you were doing it using events. But if you\'re writing it yourself, then maybe you would do something in a slightly inefficient way perhaps. For example, with some kinds of bindings, you have to be careful to avoid an infinite loop. If you have an event that triggers a state change, the state change could trigger the event again and you get this infinite loop. A compiler can guard against that. It can say this is a binding that could have that problem, so we\'re going to just keep track of whether the state changes as a result of the binding. And so, the compiler can sort of solve all of these really hairy problems that you had faced as a developer while also giving you the benefit in of being able to write much less code and write code that expresses the relationship between these two things in a more semantic and declarative way without the danger.\r\n\nTARAS: This is one of the reasons why I was so excited to talk to you about this stuff, Rich, because this stuff is really interesting. I mentioned that we might, so we have a little bit more time. So I just want to mention, because I think that you might find this interesting, the [inaudible], the stuff that we were talking about that I mentioned to you before. So, I want to let Charles talk about it briefly because it\'s interesting, because it essentially comes down to managing asynchrony as it ties to life cycle of objects. Life cycle of objects and components are something we deal with on a regular basis. So, it\'s been an interesting exercise and experimenting with that. Charles, do you want to give kind of a low down?\r\n\nCHARLES: Sure. It\'s definitely something that I\'m very excited about. So, Taras gets to hear like an earful pretty much every day. But the idea behind structure concurrency, I don\'t know if you\'re familiar with it. It\'s something that I read a fantastic -- so people have been using this for a while in the Ember community. So Alex Matchneer, who\'s a friend and often time guest on the podcast created a library called ember-concurrency where he brought these ideas of structure concurrency to the ember world. But it\'s actually very prevalent. There\'s C libraries and Python libraries. There\'s not a generic one for JavaScript yet, but the idea is just really taking the same concepts of scope that you have with variables and with components, whether they be ember components, Svelte components, React components or whatever there is, you have a tree of components or you have a of parents and children and modeling every single asynchronous process as a tree rather than what we have now, which is kind of parallel linear stacks. You call some tick happens in the event loop and you drill down and you either edit an exception or you go straight back up. The next tick of the event loop comes, you drill down to some stack and then you go back up. A promise resolves, you do that stack. And so with structure concurrency, essentially every stack can have multiple children. And so, you can fork off multiple children. But if you have an error in any of these children, it\'s going to propagate up the entire tree. And so, it\'s essentially the same idea as components except to apply to concurrent processes. And you can do some just really, really amazing things because you don\'t ever have to worry about some process going rogue and you don\'t have to worry about coordinating all these different event loops. And one of the things that I\'m discovering is that I don\'t need like event loops. I don\'t really use promises anymore. Like actually, I was watching, I think it was why I was watching your talk when you\'re talking about Svelte 3, when you\'re like -- or maybe did you write a blog post about we\'ve got to stop saying that virtual doms are fast?\r\n\nRICH: Yes, I did.\r\n\nCHARLES: So I think it was that one. I was reading that one and it jived with me because it\'s just like, why can\'t we just go and do the work? We\'ve got the event, we can just do the work. And one of the things that I\'m discovering is with using the construction concurrency with generators, I\'m experiencing a very similar phenomenon where these stack traces, like if there\'s an error, the stack traces like three lines long because you\'re basically doing the work and you\'re executing all these stacks and you\'re pausing them with a generator. And then when an event happens, you just resume right where you left off. There\'s no like, we\'ve got this event, let\'s push it into this event queue that\'s waiting behind these three event loops. And then we\'re draining these queues one at a time. It\'s like, nope, the event happens. You can just resume right where you were. You\'re in the middle of a function call, in the middle of like [inaudible] block. You just go without any ceremony, without any fuss. You just go straight to where you were, and the stack and the context and all the variables and everything is there preserved exactly where you left it. So, it\'s really like you\'re just taking the book right off the shelf and going right to your bookmark and continuing along. Rather than when you\'ve got things like the run loop in ember or the zones in angular where you have all these mechanics to reconstruct the context of where you were to make sure that you don\'t have some event listener. An event listeners created inside of a context and making sure that that context is either reconstructed or the event listener doesn\'t fire. All these problems just cease to exist when you take this approach. And so, if it\'s pertinent to this conversation, that was a surprising result for me was that if you\'re using essentially code routines to manage your concurrency, you don\'t need event loops, you don\'t need buffers, you don\'t need any of this other stuff. You just use the JavaScript call stack. And that\'s enough.\r\n\nRICH: I\'m not going to pretend to have fully understood everything you just said but it does sound interesting. It does have something not that dissimilar to ember\'s run loop because if you have two state changes right next to each other, X+=1, Y+=1, you want to have a single update resulting from those. So instead of instruments in the code such that your components are updated immediately after X+=1, it waits until the end of the event loop and then it will flush all of the pending changes simultaneously. So, what you\'re describing sounds quite wonderful and I hope to understand that better. You have also reminded me that Alex Matchneer implemented this idea in Svelte, it\'s called svelte-concurrency. And when he sent it to me, I was out in the woods somewhere and I couldn\'t take a look at it and it went on my mental to do list and you just brought it to the top of that to do list. So yeah, we have some common ground here, I think.\r\n\nCHARLES: All right.\r\n\nTARAS: This is a really, really fascinating conversation. Thank you, Rich, so much for ing us.\r\n\nCHARLES: Thank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @thefrontside or over just plain old email at @frontside.io. Thanks and see you next time.\r\n

Philippe De Ryck s the show to talk all things security: the importance and why you should be taking active steps, how to do it in your codebase effectively, and what can happen during a breach. Philippe De Ryck: Pragmatic Web Security Resources: OWASP Top 10 OWASP Top 10 Proactive Controls Please us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer. This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC. Transcript: Coming Soon!

Philippe De Ryck s the show to talk all things security: the importance and why you should be taking active steps, how to do it in your codebase effectively, and what can happen during a breach.\r\n\nPhilippe De Ryck: Pragmatic Web Security\r\n\nResources:\r\n\n\nOWASP Top 10\r\n\nOWASP Top 10 Proactive Controls\r\n\n\n\r\n\nPlease us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at @frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.\r\n\nThis show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.\r\n\nTranscript:\r\n\nCHARLES: Hello and welcome to The Frontside Podcast, a place where we talk er interfaces and everything that you need to know to build them right.\r\n\nMy name is Charles Lowell, a developer here at The Frontside. ing me, also hosting today is Taras Mankovsky. Hello, Taras.\r\n\nTARAS: Hello, hello.\r\n\nCHARLES: And as always, we\'re going to be talking about web platforms, UI platforms, and the practices that go into them. And here to talk with us today about a pillar of the platform that I certainly don\'t know that much about, and so, I\'m actually really happy to have this guest on to talk about it is Philippe De Ryck who owns his own company called Pragmatic Web Security. I understand you do trainings and are just generally involved in the small space. So, welcome, Philippe.\r\n\nPHILIPPE: Hi. Nice to meet you.\r\n\nCHARLES: Wow! I almost even don\'t even know where to start with this subject because I\'m kind of like the hippie developer mindset where it\'s like, "LaÖlaÖlaÖlaÖlaÖ we\'re in this open land and nothing\'s ever bad going to happen and we\'re just going to put code out there," and nobody would ever take advantage of any holes or anything like that. And I think that a lot of developers share that mentality and that\'s how we end up with major, major security breaches. And so, like I said, this is something that I\'m actually very eager to learn but I almost even don\'t know where to start. I need training, man.\r\n\n[Laughter]\r\n\nPHILIPPE: Well, that\'s good to hear. No, you\'re totally right about that. If you\'re not into security, it seems like this fast space for a lot is happening and you don\'t really know how or why and what really matters to you and should I even be worried about this. And let me start by addressing the very first thing. Yes, you should be worried because maybe you\'re not building something that somebody cares about but you always have something that somebody wants, even the simplest of attacks always targets a valuable resource. Just to give you a very simple idea today, cryptocurrency is all the hype and you have a taker that\'s just aiming to misuse your s\' computers to mine crypto coins because it essentially saves them a bunch on electricity cost. So, there\'s always something to grab. Usually, it\'s data or services or worse. But even in the most minimal cases, you have hardware, you have devices, you have network capacity that somebody might want to abuse. So yes, security, I would say, always matters.\r\n\nCHARLES: What\'s the best way to get started? You said understanding that everything we do, we\'re holding onto resources that might be valuable, that someone might want to seize but I\'m just getting started with my application. I don\'t know anything about security. Where do I get started on just understanding the space? And then before I even look at tools that I wantÖ\r\n\nPHILIPPE: You want the honest answer for that?\r\n\n[Laughter]\r\n\nPHILIPPE: The honest answer is probably hire someone who has security knowledge. I don\'t mean this in a bad way. I\'ve come a very long way in my career doing what I do now. And if I look at that, if you are aiming as a developer with no knowledge about security to build a secure application, it\'s going to be very hard. There\'s a lot of things you need to know, intrinsic knowledge. These are not things you can simply read a small book in a week, you know all of these security things that you\'ll know what to do. So, if you have no previous experience at all, I suggest to find some help.\r\n\nCHARLES: Right. It\'s like saying, "Hey, you\'ve never written a data layer before but you want to go out and you want to write a massively distributed system where you have all these notes talking to each other. You\'re not going to read the O\'Reilly book \'How to Build Distributed Systems\' in a week and go out and do the same thing." It\'s the same thing with security. You need to understand the entire context. And there\'s no substitute for experience.\r\n\nPHILIPPE: Sorry, I actually like that comparison because in a sense, you\'re right, it\'s like these other very complex topics you don\'t expect to learn that in a week or a month and right a functioning data layer. But the difference is if you fail at writing that data layer, your application is probably not going to work. While if you fail at securing the application or seeing potential vulnerabilities, it\'s still going to work just a bit more than you anticipated. It\'s going to result leaking all your data to an attacker or opening all doors so that they can gain access to your server, stuff like that. So, I would say that the consequences of not getting it right are, at least in the beginning, very invisible. It\'s only after things happened that it\'s like, "Oh, crap!" And you should pay attention to that.\r\n\nCHARLES: Yeah. And then you have these back doors and these leaks that are set in stone and may be very hard to change.\r\n\nPHILIPPE: Yeah, absolutely. And honestly, the worst part of the breach is for a company, it might be reputation damage. But what you really should be worried about is all that personal information that\'s being leaked to, most cases, publicly leaked to anyone but in other cases, it\'s sold on [inaudible] markets and actually abused by people looking for someone else\'s identity or credit card information or stuff like that. And the companies usually get away with some bad press, maybe a small dip in their stock price but eventually, they\'ll bounce back. But it\'s the s that suffer from these breaches for a very, very long time.\r\n\nTARAS: What do you see the kind of hot zones around concerns that companies have around security? Because I imagine it\'s hard to be concerned about everything, so they\'re probably thinking about specific things like this thing worries us. Like what kind of things do you see companies and teams need to worry about?\r\n\nPHILIPPE: That\'s an interesting question. You have all different kinds of companies and all different levels of security awareness and what they\'re worrying about. I would say if you have the companies that are not very good at or don\'t have very much security knowledge, they\'re probably not to worry about things because otherwise, they would have started investing in improving their practices. If you look at the companies that are at least very aware of the landscape, I\'m not saying that anybody here is perfect, but some of the companies are actually doing quite a good job. One of the most interesting challenges today is dealing with dependencies. So, all of your packages, you depend on npm, Maven, Python packages, Ruby gems, and so on. All of them form a huge attack factor in most applications today. That\'s definitely a problem that a lot of companies struggle with and it\'s very hard to find good solutions.\r\n\nTARAS: GitHub recently, I saw their vulnerability alert service that I\'ve been getting a lot of notifications from on some of the open source libraries that we use. They have a lot of dependencies. And a lot of projects have the same dependencies. So, the moment that one notification goes on, it like lights up on all of the GitHub repos that I have. So, I have been going through and like updating dependencies and all those libraries.\r\n\nPHILIPPE: Yeah, that\'s a very good example. Absolutely. A lot of the projects we build today usually start out by installing a bunch of dependencies. Even before you\'ve written the first line of code, you already have a massive code base that you are relying upon. And a single vulnerability in a code base might be enough, it\'s not always the case, but it might be enough to compromise your application. And that leaves you, as a developer, in a very hard place because you haven\'t written any lines of code yet. You have built a vulnerable application and that starting point can be very terrifying. So, there\'s a lot of reports on this. And actually, if you want some numbers, 78% of the vulnerabilities discovered in existing applications like the ones you mentioned. If GitHub alerts you like, "Hey, there\'s a problem in one of your dependencies," it\'s often even an indirect dependency, meaning that you include a framework, if you include React or Express or whatever you\'re building, that one of your dependencies of one of those projects actually has a vulnerability. If you look at the trees of these packages, they get quite big that it\'s not dozens but it\'s thousands of packages that we\'re talking about.\r\n\nCHARLES: Yeah that\'s the other thing is how do you know how to interpret these security vulnerabilities because some of them, we get a lot of security vulnerabilities for node packages but we only use them for our development tools to build frontend. So, if we\'re building a React application and there\'s some security vulnerability in some node packages that we\'re using in our build tool, then that doesn\'t actually get deployed to the frontend. So, maybe it\'s not a concern but if we were actually using it to build a server, then it would be absolutely critical. And so, how do you evaluate because the same security vulnerability is not a vulnerability in one context, but might be in another or maybe I\'m thinking about it wrong. You see what I mean?\r\n\nPHILIPPE: Yeah, sure. I totally get what you mean. Actually, I have observed the same things. I also get these security alerts on my projects, and sometimes it\'s devDependency, so it seems like we don\'t need to care about that. You\'re right in the sense that you have to assess the criticality of such a report. So, they will have a rating, a severity rating saying like, "This is a minor issue," or, "This is a major issue," that should be a first indication. And then a second thing to look at is, of course, how are these things used in practice. It\'s not because it\'s a devDependency that it\'s not exploitable because it all depends on what is the vulnerability. If there\'s an intentional malicious backdoor in the library and you\'re building that on your build server, it might give an attacker access to your build server. So, that might not be something you actually want to do. So in that case, it does matter. Of course, if it\'s only stuff you run locally, you can say like, "OK, this is less important." But usually, updating or fixing these vulnerabilities also requires less effort because there\'s no building and deploying to production servers either. So, it\'s a matter of staying up-to-date with these.\r\n\nAnd one of the things that people struggle with is handling this in a lot of different applications. You mentioned you had a lot of GitHub repos and the vulnerability starts popping up in all of them and you have to fix and update all of them. You can imagine that major companies struggle with that, as well, especially if you have quite a few different technologies. Managing all of that is insanely hard.\r\n\nCHARLES: Right, because you just usually look at it and you\'re like, "Oh, I\'ve got to this." And maybe, "I haven\'t used it this repo for a while. I\'ve got to clone it up, I\'ve got to update the dependency. I\'ve got to make sure I run all my tests locally, then run all the tests in CI and make sure I didn\'t break anything by upgrading. I might have fixed closed security hole but broken my functionality." And so, make sure that that is all intact and then push it out to production. Even on the small, it\'s like I\'m looking, "OK, maybe this is going to take me 30 to 45 minutes." But if you have four or five of those things, you\'re looking at half your day or maybe even the whole day being gone and that\'s if you have the processes in place to do those automated verification. If you have a very high confidence in your deployment pipeline which I don\'t think a lot of places have. So, it sounds like these are complementary, like you really need in order to keep a secure application, you have to keep it up-to-date because I think what I\'m hearing is you should just evaluate all the threats. You should fix it if you can. The first part of my question is, am I kidding myself when I say, "Oh, I can ignore this one because it\'s just local or it\'s just a devDependency."\r\n\nPHILIPPE: The answer to that question is briefly, I would say they are less critical.\r\n\nCHARLES: That\'s cool.\r\n\nPHILIPPE: In general, the rule is update if you can. And actually some of the tools out there that monitor vulnerabilities, they will automatically create a pull request in your repo saying to this version and then you can automatically run your tests if you have them, and you can very quickly see whether some conflicts are generated by updating that dependency - yes or no. And in most cases, if it\'s a minor version bump, it\'s going to work as expected and you can easily push out the new version without that vulnerability. So, I would say fix if you can. If it goes quickly, then definitely fix them. But I would focus on non-devDependencies first instead of devDependencies.\r\n\nCHARLES: Yeah.\r\n\nPHILIPPE: Second thing I wanted to add is you paint a very grim picture saying you have to spend a lot of time updating these issues and I can totally understand that happening the very first time you look into this. There\'s going to be some stuff in there, I can guarantee that. But if you do this regularly, the effort becomes less and less because once you have up-to-date libraries, the problem is bad but it\'s not like we have 50 new vulnerabilities every day, fortunately.\r\n\nCHARLES: Right.\r\n\nPHILIPPE: So, once you have done that, it\'s going to be a bit less intensive than you might anticipate at first glance. Of course, if you\'re using these projects, if you\'re reusing the same library, then you\'ll have to update them everywhere. That\'s the downside, of course.\r\n\nCHARLES: It\'s probably a little bit dangerous to be assessing the criticality of the security threats yourself if you\'re not an expert, and kind of in the same way, it\'s dangerous to be assessing an architecture if you don\'t have an expertise in our architecture, I guess is the thing, because you might not understand the threat.\r\n\nPHILIPPE: Yeah, that\'s, again, absolutely true. It again depends very much on how it\'s deployed and what it\'s used for. That\'s going to be one important aspect. Another thing that might be very useful is, how deep is the dependency that creates the vulnerability or has the vulnerability? Because for example, if you have your tree of dependencies, if you dependency is like five or six levels deep, the chances of malicious data are reaching that specific vulnerability, and that specific library is going to be fairly small. Because usually, libraries have a lot of features and you only use part of them in your application. So, the other one is address of the features is just sitting there and if it\'s never used and it\'s also not exploitable. So, that might play a role as well.\r\n\nI saw a presentation about a month or two months ago from how Uber manages these things and they struggled with a lot of those things as well. And they eventually decided that they really care about vulnerabilities going three levels deep. And something that goes deeper is considered to be less relevant or less urgent to update because chances of exploitability are going to be very small.\r\n\nCHARLES: That\'s actually really interesting.\r\n\nTARAS: One thing that got me thinking about something that is actually happening right now. A friend of mine has a WordPress site that was hacked. But what\'s interesting about WordPress, I think the fact that WordPress site was hacked is not really a surprise but I think what\'s interesting about that is that the frequency and the sophistication of these attacks has increased. The tooling has improved also in the WordPress ecosystem. But at the same time, I think there is actually more people that are aware of the kind of exploits that could be done. There are a lot of people going after WordPress sites, but it kind of makes me think that there\'s probably going to be a time when the vectors of attack for web applications are going to become pretty well known as well. Because of the architecture, there are a fewer of them. But as the awareness of the actual architecture becomes more common, I think the angles of attack are going to become more interesting. Like one of the things that I was reading about a couple days ago is that there are some researchers that found a way to attract s based on a combination of JavaScript APIs that are available in the browser. So, they are actually able to fingerprint s based on the kind of things that they\'re using, the application for [inaudible] extensions they have installed. I think people are going to get more creative. And that\'s kind of scary because we\'ve seen this happen already in WordPress and people are greedy. So, there are going to be ways. I think there\'s going to be more people looking at how to get into and how to exploit these vulnerabilities.\r\n\nPHILIPPE: Yeah. That\'s actually a couple of very good examples that illustrate the underlying issue. So, this browser-based tracking of s, it\'s called browser fingerprinting and it\'s been going on for a while. Back when I did my PhD, I had colleagues working on those things and you have other people at universities doing research on this. And yes, you can use things like JavaScript APIs in the browser to identify a particular with a very high probability. It\'s not perfect but it\'s usually enough to identify a for ad tracking or those purposes.\r\n\nBy the way, these things also have a legitimate purpose. So, they are also used to keep track of a particular to prevent things like session hijacking or detect problem s or stuff like that, so they can also have a legitimate use case next to tracking. But they very clearly show how security will always be a cat and mouse game. Tracking used to be easy. You just set a cookie in a browser and the cookie was there next time and you knew who the was. And then, s became a bit more savvy. You had browser extensions trying to block listings because let\'s be honest, they\'re kind of shady. So, s probably don\'t want that. And then the attacker started moving towards other things and getting more advanced. And you see that in other areas of security, as well. So, I consider that a good thing because as we make things harder for attackers, they will have to get more creative and it will become more difficult to exploit or to take advantage of applications. That\'s the good side. The bad side or the dark side of that equation is that unfortunately, the vulnerabilities are not going away. It\'s not because we now have these somewhat more advanced attacks using advanced features or even CView-based vulnerabilities that the old things like SQL injection and [inaudible] have disappeared in applications. That\'s also not true and that means that it makes it just a bit more harder for everyone on the defensive side to build more secure applications. You\'re going to have to know about the old stuff and you have to learn about the new stuff.\r\n\nCHARLES: Again, we come back to that idea. It\'s all a bit overwhelming. Aside from the solution of like, "Hey, let\'s hire Phillippe. Let\'s hire some other security expert." We were actually in your training, and obviously, I don\'t want to divulge all the secrets or whatever. If we were to attend your training, what do you see is the most important thing for people to know?\r\n\nPHILIPPE: There\'s no secrets there. [Chuckles] What I teach is web security. I kind of like to think I teach that in a very structured and methodical way. But in the end, there\'s no secrets and I don\'t mind talking about this here on the podcast because I honestly believe that everyone should know as much as they can about security.\r\n\nWhat do I teach? I can talk about specifics but I can also talk about generic things. One of the general takeaways is that one of the best things in my opinion that a developer can do is realize when they don\'t know something and actually it that they don\'t know something, instead of just doing something. Maybe having like a brief thought like, "Hmm, is this secure? Well, it\'s probably good. I\'m going to deploy it anyway. We\'ll see what happens." That is not the right way of doing things. If you do something and you recognize like, "Hey, this might be security sensitive. We\'re dealing with customer information here. We\'re dealing with healthcare information. We might want to look at what plays a role here," and then you can go ask someone who does. You probably have a colleague with a bit more security knowledge, so you can ask him like, "Hey Jim, or whatever your name is, do you think that this is OK or should we do something special here?" Very much like you are doing, asking me questions right here. That\'s one important takeaway that I hope everyone leaves with after a training class because not knowing something and realizing that you don\'t know it allows you to find someone who actually does. That still leaves us with that point which you wanted to sidestep.\r\n\nCHARLES: [Chuckles]\r\n\nPHILIPPE: A second thing is to realize that security is not a target. It\'s not something you\'re going to hit. It\'s not a holy goal that after working really hard for two years, you\'re going to hit this security milestone and you\'re done. It\'s always going to be a cat and mouse game. It\'s always going to be a moving target but that\'s OK. That\'s how things are. And that\'s the same with all other things in the world essentially. It\'s an evolving topic and you\'ll need to be ready to evolve with that as well.\r\n\nTARAS: One of the challenges that I see into quite often in teams is that at the individual level, people really try to do their best, maybe the best of their abilities. But it\'s often, when it comes to being part of a group, it\'s often like they do best within the kind of cultural environment that exists. I\'m curious if you\'ve seen good or kind of environments or cultures for engineering teams that are conducive to good security. Are there kind of systems or processes the companies put in place that you\'ve seen to be very effective in preventing problems? Have you encountered anything like this?\r\n\nPHILIPPE: Ideally, you have developers that are very well educated about security but honestly, it\'s going to be insanely hard to find these people because a developer not only has to be educated about security, they also need to know about UI design and JavaScript frameworks and other frameworks and all of these things. And it\'s virtually impossible to find someone up-to-date on all of these things. So, what most companies do today that seems to work quite well, even though it\'s very hard to judge whether it\'s working or not, is they work with security champions. So, you typically have a dev team and within a dev team, you would have a security champion, one or two or five, depends on how large your teams are, of course, that is knowledgeable about security. So, that developer has some knowledge. He\'s not an expert but he knows about common attacks and common dangers and how to potentially address them in the application. So, having that person embedded in the team allows the team to be security aware because when you have a team meeting like, "Hey, how are we going to solve this particular problem?" That person will be able to inject security knowledge like, "Hey, that seems like a good idea but if we\'re using SQL in the backend, we need to ensure that we don\'t suffer from SQL injection." Or if you\'re using a NoSQL database, it\'s going to be NoSQL injection and so on. And that already elevates the level of security in the team.\r\n\nAnd then, of course, security champions themselves are not going to be security experts. They\'re mainly developers just with a security focus. So, they should be able to escalate problems up to people with more knowledge, like a security team which can be a small security team within their organization that people can easily reach out to, to ask like, "Hey, we\'re doing something here and I know that this is security relevant and I\'m not entirely sure what\'s happening here. So, can we get a review of this part of your application?" Or, "Can you guys sit on the meeting to see what\'s going on and what\'s happening there?" And I think that structure also makes sense. It\'s still going to be hard to build secure applications because there\'s still a lot of things to address, but at least, your teams get some awareness. And then of course, you can help your security champions to become better and they will get better over time. You can augment them with the security architects. You can train your security champions separately with more in-depth knowledge and so on. And that veteran or that setup seems to work quite well in many large organizations today.\r\n\nCHARLES: Yeah. I like that. It gets me to thinking, so having the having the security champions, having people who have this as part of, not their specialization, but at least part of their focus, being in the room, being part of the conversation because we try and do that and provide that service when it comes to UI but we also have a bunch of processes that kind of automate the awareness of quality. So, the classic one is your CI pipeline, your deployment pipeline. So, you\'re automating your advancement to production. You\'re automating your QA. It\'s still no substitute for having someone who\'s thinking about how to have that quality outcome but you still have some way of ing that the outcome is quality. Are there tools out there that you can do to kind of keep your project on the security Rails. I\'m thinking something that we we\'ve done recently is having preview apps, so that we get a tight loop of being able to deploy a preview version of your application that\'s on a branch but it\'s talking to a real backend. There\'s a lot of more software and services that are ing this and it\'s kind of become an integral part of our workflow. So, testing automated deployment preview apps, there\'s this kind of suite of tools to make sure that the loops are tight and that the quality is verified even though you have people, you also have people guiding that quality. It\'s just making sure that the standards are met. Is there a similar set of tools and processes in the security space so that we\'ve got these champions out there, they\'re being part of the conversations. They\'re making suggestions but they can\'t be everywhere at once. And is there a way to make sure that the kind of the ways that they\'re guiding the application, just ing that the application is going in that direction? Or an alarm bell has sounded. We mentioned one which is the automated pull request with the, "Hey, you got this dependency and there was a pull request." Are there more things like that, I guess, is what I\'m saying.\r\n\nPHILIPPE: Yes, there are. But I would dare to say not enough. So yes, you have some security tools you can integrate in your pipeline that do some automated scanning and they tried to find certain issues and alert you of those issues. So, these things do exist but they have their limitations. A tool can scan an application. Some of the findings are going to be easy and fairly trivial, but it\'s good to have the check in place nonetheless. But some of the more advanced issues are very likely to be undetectable by those automated tools because they require a large amount of skill and expertise to actually craft and exploit to abuse that particular feature in an application. So, we do have some limitations but I like discretion because I do believe that we need to leverage these mechanisms to ensure that we can improve the security quality of our applications. A very simple thing you can do is you can run an automated dependency check when you build the application and you can use that to decide to halt deployment when it\'s a severe vulnerability or go ahead anyway when you consider this to be acceptable because if you automate all of those things, things can go wrong as well. We can talk about that in a second.\r\n\nSo yeah, these things can be done. But what I strongly encourage people to do to ensure that they can kind of improve the code quality is to flag certain known bad code patterns. So if you\'re building an Angular or a React application, if you\'re using functions that output go directly into the template, that\'s going to be very dangerous. So, we know these functions in Angular, they\'re called bySecurityTrustHtml, by security should be kind of a trigger and this kind of security irrelevant. And in React, that property is called Dangerously Set innerHTML, also indicating like a \'developer watch out what you\'re doing\'. So, what you could do is you could set up code scanning tools that actually flag these things whenever they appear in application because sometimes people make mistakes. You hire an intern and they don\'t really know the impact of using that property and they use it anyway which would cause cross-site scripting vulnerability. If you\'re code scanning to flag these things ensures that it doesn\'t get pushed to production unless it\'s a benign case which is actually approved to be in there, then you can definitely stop some of these attacks coming on for sure or some of these vulnerabilities happening.\r\n\nTARAS: I think the hardest thing to understand is when someone doesn\'t understand what they\'re doing that what they will create is so cryptic that I think any tool that tries to figure out what it is that person is doing I think will have a really hard time. The person making the thing doesn\'t understand what they\'re doing, then the system is not going to understand what they\'re doing which makes me think that one of the things that we think about a lot at Frontside is this idea of trying to understand the system from the outside as kind of looking at a system as a black box and wonder what kind of tools are available specifically for inspecting the application from the outside, like as if somehow understanding what the application is doing based on what\'s actually going on inside of the runtime and then notifying someone that there could be something off in the application, but through exercising the [inaudible] things like, for example, memory leaks is not something you can catch unless you have a test suite that has like a thousand tests and then you will see over time that your application is actually leaking memory. But if you run individual tests, you\'ll never see that. I wonder if there\'s anything like that for security where at runtime, there\'s actually a way to understand that there might be some kind of a pattern that\'s incorrect in the application.\r\n\nPHILIPPE: If only, if only. It depends on who you ask. There is such a concept that\'s called Dynamic Application Security Testing. Essentially, what you do there is you run the application, you feed it all kinds of inputs, and you monitor when something bad happens. And that means that you have detected vulnerability. So, these things do exist. But unfortunately, their efficiency is not always that good. It very much depends on what kind of security problems you\'re trying to detect. And they can, for example, detect some instances of things like cross-site scripting or SQL injection or things like that. But there will always be limitations. I\'ve seen tools like that being run as an application where you actually know there\'s a vulnerability because it has been exploited. There is a manual written exploits and the tool still doesn\'t find any vulnerabilities which is not surprising, because these things are really hard to make an abstraction of to be able to find that in an automated way with a tool. If you would have such a tool that would be, I think, that [inaudible] would be a lot better. I think there\'s a lot of funders working on that. But at the moment, those tools are not going to be our savior to build more secure applications.\r\n\nCHARLES: Yes. I mean, it\'s kind of like linting, right? Or you can make tests. We\'ve been through this kind of all the features or the aspects that we want our application to have, whether it be accessibility. There\'s certainly a very comprehensive suite of lint level checks that you can run to make sure that your application is accessible. You can run a suite of three thousand things and if it triggers any of these things, then yes, your application won\'t be accessible but it\'s not a substitute for thinking through the accessibility architecture. The same thing goes with code linting. You\'re not going to solve bugs with a linter that makes sure that it\'s formatted and that you\'re declaring your variables right and that you\'re not shadowing things. But you can definitely eliminate a whole classes of things that might be put in there just for maybe even you know what you\'re doing and you\'re just forgetful.\r\n\nPHILIPPE: Yes, these rules exist, as well. They\'re not extensive but there are linting rules for Angular used for security, for example. But the problem in linting is that they are very useful to find potential instances of security relevant features or security relevant functionality. But the linting rule alone cannot decide whether something is OK or not. Just to give you a very simple example, if you use the bySecurityTrustHtml function, if you give that function a static snippet of HTML, that\'s going to be fine unless you write your own attack essentially. But if you feed that function inputs, you\'re going to be in a lot of trouble. And making that distinction with a linter is going to be difficult unless you have a static string in the arguments. But if once you start having that from variables to dynamically decide to have a different code path, then that\'s going to be very, very difficult to decide automatically. So, yes, you can use that to find the places in the application where you should be looking for, in this example, a cross-site scripting in Angular but the linting alone is not going to give you an answer whether this is OK or not. That still requires knowledge of how Angular handles this things, what happens, and how you can do things safely.\r\n\nTARAS: Sounds like we keep going back to nothing beats having knowledgeable developers.\r\n\nPHILIPPE: Yes. Unfortunately, that is true. However, with that said, I want to highlight that frameworks like Angular, well mainly Angular, make things a lot better for developers because yes, you still need knowledgeable developers but the ways to introduce a cross-site scripting vulnerability in an Angular application are actually very, very limited. It\'s not going to be one, but there\'s going to be maybe three or four things you need to be aware of, and then you should be set. While if you would have done the same for PHP, it\'s going to be 50,000 things you need to be aware of that are potentially dangerous. So, yes, frameworks and libraries and all of these abstractions make it a lot better and I really like that. That\'s why I always refer to abstract things away in a library so that you actually have the ability to look for this dangerous code patterns using linting rules in your code base and that you can, at least, inspect the go to see whether it\'s OK or not, even though you might not be able to make an automatic decision. You, at least, know where to look and what to approve or how to change in the code base.\r\n\nTARAS: I think that\'s one of the things that oftentimes is not taken into that the frameworks are different. And I think of big differences in how much -- like right now, the most popular framework, I think, React. But it\'s such a thin layer, it\'s such a small part of the framework that you can hardly call it a framework. But it is something that companies rely on. But then when you consider how much of that code that you need to write, to make React into a complete framework for your company, the amount of code that your team has to write versus the amount of code that your team has to write when you use something like Angular or Ember, there\'s definitely a lot less parts of the framework that you need to write or a lot less parts of the framework you need to choose from what\'s available in the ecosystem. Like in Angler and Ember, and I\'m not sure what the story is with the view, but the pieces, they come from kind of a trusted source and they\'ve been kind of battle tested against a lot of applications. But I don\'t think that enters into consideration when companies are choosing between Angular or whatever that might be because they\'re thinking like what is going to be easiest for us. What is going be [inaudible] for developers? They\'re not thinking about how much of the framework are we going to need to put together to make this work.\r\n\nCHARLES: I can say it sounds, Taras, like almost what you\'re saying is by using the frameworks that have been battle tested, you actually get to avail yourself of code that actually has security champions kind of baked into it, right? Is that what you were saying? You keep coming back to \'you need developers who are knowledgeable about security\', and if you\'re using kind of a larger framework that covers more use cases, you\'re going to get that. Do you think that that is generally true, Philippe?\r\n\nPHILIPPE: Yeah. I think it is and that\'s why I mentioned that I liked Angular before because Angular actually does offer a full framework. And because they do that, they made a lot of choices for developers and some of these choices have a very, very big and positive impact on security. On the other hand, if you make those decisions, you become an opinionated framework and some people don\'t like that. They actually want the freedom to follow their own paths and then a less full featured framework like React might be an easier way to go.\r\n\nCHARLES: But I think what happens is folks don\'t enter into that decision with their eyes open to the fact that they then now need to be their own security champion because they just don\'t even see it. We said the most dangerous thing is the things that you don\'t know.\r\n\nPHILIPPE: Yeah, absolutely. And I totally agree. That\'s something that\'s at least a couple of years and probably still today, many companies moving into this space struggle like, "Which framework do we choose and why do we choose one or the other and which one will still be there in three years because we don\'t want to switch to another thing in three years," which is risky to our developers. I like that you said that Angular has this security champion knowledge built in because in Angular 2 and every version behind it, but the new version of Angular essentially, they spent a lot of time on security and they learned from their mistakes in the first version because there were some and they took that and they built a more robust framework with security built in by design or by out-of-the-box. Angular offers, for example, very strong protection against cross-site scripting. It\'s just there, it\'s always on and unless you actively sidestep it, it\'s going to protect you. And that\'s one of the things I really like about Angular and how they did that.\r\n\nCHARLES: Yeah, that\'s one of the things that I really like too because I there was a blog post back, this is probably, I don\'t know, almost 10 years ago now, maybe seven or eight years, where someone was comparing why they were more interested in using, their servers were implemented in Ruby and why it was better to use Rails than just Sinatra which is just a very, very, very lightweight HTTP framework. And one of the things that he was pointing to was this new vulnerability was discovered and if you were using Rails, the middle way where the middle square stack is managed by the framework, you just upgrade a minor version of Rails. And now, by default, there\'s this middleware that prevents this entire class of attack.\r\n\nPHILIPPE: Was that a cross-site request forgery?\r\n\nCHARLES: I think it might have been.\r\n\nPHILIPPE: I think Rails was one of the first to offer built in automatically on for that. So yeah, that was a very good early example of how that can work really well.\r\n\nCHARLES: And the advantage from the developers\' standpoint, because the contrast that, if you\'d been writing your application in Sinatra which is this is very, very low level based right on top of rack and you\'re managing the middleware stack yourself and there are no opinions, then not only do you have to like fix this security vulnerability, you have to understand it. You have to get to do a lot of research to really come up with what\'s going on, how is this going to affect my application and then I can deploy a fix. And that\'s like a huge amount of time, whereas you have the freedom to not even understand the attack. I mean, it\'s always better to understand but you can defer that understanding invariably knowing that you\'re kind of invulnerable to it. And I think for people who enjoy kind of pretending, not pretending, but that the security world doesn\'t exist and say, "Hey, I want to focus and specialize on these other areas and attain deep knowledge there." It\'s very reassuring to know that if a defense for a novel attack comes out, I can avail myself of it just by bumping a version number.\r\n\nPHILIPPE: Yeah, absolutely. If you have everything in place to actually that version that fixes those, that\'s a preferable solution. Towards the future, I believe it\'s going to be crucial to ensure that we can actually keep things up-to-date because everything that\'s being built today is going to require continuous updates for the lifetime of the application. I definitely hope that the frameworks get better and more secure and start following these patterns of naming the potentially insecure functions with something that indicates that they are insecure. I think that\'s definitely a good way forward.\r\n\nCHARLES: Yeah. Can I ask one more question? Because this is something that is always something that I wonder about whenever you talk about any aspect of a system. And part of it is folks will not appreciate good architecture until they\'ve experienced some sort of pain associated with not having that architecture in place. Their project fails because they couldn\'t implement a set of features without it taking months and years and they just ran out of runway, ran out of deadline. Those types of people who\'ve been on those projects appreciate having a nimble system internally, good tooling. Folks who have experienced good tooling understand how much time they could save, and so, have a very low tolerance for bad tooling. A tool takes too long or is misbehaved or is not well put together, they just can\'t stand because they know how much time they\'re losing with security. Is there a way to get people to care about it without having some sort of breach, without having gotten smacked in the face? When you do your trainings, is it generally, "Hey, someone has experienced a breach here, and so they want to bring you in." Or is there some way to get people raise awareness of the problems they don\'t have to experience that pain but can just experience only the benefit?\r\n\nPHILIPPE: That\'s, again, a very good question and that\'s also a very good illustration of why security is so hard. Because if you get everything right, nothing happens.\r\n\n[Laughter]\r\n\nPHILIPPE: Or it might be if nothing happens, that nobody cares enough to actually try something against your application. So, there\'s no positive confirmation if you\'ve done a good job. You can keep putting things off but eventually, there\'s going to be vulnerability and it\'s a matter of how you respond to it. We recently had a cross-site scripting in Google\'s homepage, one of the most visited pages on the web. And somebody figured out that there were some weird browser thing that could be abused and that resulted in a vulnerability on, let\'s say, such a simple page. So, even there, things can go wrong. So, what would be a good way to draw with some awareness about this is I would recommend following some simple new resources or some Twitter feeds. I have some security relevant articles there but plenty of other people in the industry have as well. And when you read such an article about security incidents, just think about whether this could happen to you or not. And that should probably scare the shit out of you. Simple examples like the Equifax breach, one of the biggest, most impactful breaches of the past few years happened because of an Apache library that was not updated. I think the Apache library, they had a known vulnerability in there. We knew about it. We had a patch, yet it took too long to install that patch and the attackers abused that vulnerability. This is something that probably can happen to each and every one of us because the attacks started, I think, 72 hours after the vulnerability and the patch had been published. So, ask yourself, "Would I have updated my servers in three days after I got that vulnerability report on GitHub?" Yes or no. And if the answer is no, then the same thing can happen to you.\r\n\nOther cases: Magecart is a very big problem, people injecting credit card skimming malware in the JavaScript library. Are you including third party JavaScript libraries? If yes, then chances are that this can happen to you. And there\'s nothing preventing someone from exploiting that. It\'s probably just because you got lucky that nobody tried to do that. And the same thing you see now with all these attacks npm packages where people actively try to get you to install a malicious package as one of your dependencies. And again, everybody can fall victim to these things. So, if you read the articles with that mindset, I probably guess that your security awareness will grow rapidly and you will start caring about that very fast.\r\n\nCHARLES: Yeah.\r\n\nTARAS: Lots to think about.\r\n\nCHARLES: Yeah, there\'s lots to think about because the next thing that occurs to me is how do you even know if you\'ve been targeted. Because a good attacker is not even going to let you know.\r\n\nPHILIPPE: Yeah.\r\n\nCHARLES: It\'s just better to siphon off your blood, like you said, than to kill the -- you want to be a vampire bat and come to the same cow every night and just take a little bit of blood rather than the lion that kills the cow and then the cow\'s gone.\r\n\nPHILIPPE: I would say constant monitoring is going to be crucial and you need that data for all kinds of different purposes. You need to monitor everything that happens, first of all, for a post-mortem analysis. If something happens, you want to be able to see how bad it was. This apparently got a full access and if you have decent monitoring, you will be able to retrace his steps to see what they did or did not get. So, that is one very good use case. A second use case is you can use that data to detect attacks. Usually when the attacks are noisy, it\'s an automated scanning tool but it might be an attacker trying to do things. Again, that may be something very useful for you to act on to see if there is a problem to prevent that from connecting, or so on. And then, another very good use case of these things is actually inspecting the logs manually as an ops engineer or whatever, who is responsible for doing that, because that might again yield new insights. I\'ve been talking to someone who said that they discovered an abuse of one of their APIs just by looking at the logs manually and detecting a strange pattern and looking and digging deeper into it. And the automated monitoring tools that they had installed that trigger on certain events like a mass amount of requests to the authentication and stuff like that, they did not catch this particular abuse. So, I would say monitoring there is absolutely crucial, for sure.\r\n\nTARAS: So, the takeaway is higher attentive knowledgeable developers who will learn about security.\r\n\nPHILIPPE: I would say the takeaway is security knowledge is essential for every developer. So, I encourage every developer to at least have a little bit of interest in security. I\'m not saying that everyone should be a security expert. We should at least know about that the most common vulnerabilities in web applications, what they mean, what they might result in, and what to be on the lookout for. So yes, I think that\'s one of the crucial things to start with. And then within an organization, you should have someone to fall back on in case that there are security relevant things that you actually can talk to someone who does see a bigger picture or maybe the full security picture to decide whether these things are a problem or not.\r\n\nI think we\'re closing or nearing the end here, but one of the things we haven\'t talked about is how to actually get started in security. What if you are interested in security after hearing this podcast and you want to get started? I want to give you just a few pointers so that you actually know where to look.\r\n\nOne of the first things to look at is OWASP. And OWASP is the Open Web Application Security Project. It\'s essentially a nonprofit that has the mission to improve the security posture or knowledge of developers, and they have a lot of resources on various different topics. They have a lot of tools available and things like that. What you want to start with as a developer is the OWASP Top 10, which is a list of the 10 most common vulnerabilities that exist in applications, just to open your eyes like these things exist in applications today and are definitely a problem. And then, there\'s a complementary Top 10 called the Proactive Controls and that\'s about how you, as a developer, can actually prevent these things. So, what should we know about implementing security, which guidelines should we follow. And these two documents are a very good place to start. And then there is a huge community that\'s actually mostly very eager to help people figure out the right way of doing things and solving these problems we have in our ecosystems.\r\n\nTARAS: Awesome. That\'s great. Thank you very much.\r\n\nCHARLES: Yeah. I\'ll take that in. That is really, really helpful. Well, thank you very much, Philippe, for coming on and talking about security. I actually feel a lot better rather than usually I\'m thinking about securities kind of stresses me out. [Laughs]\r\n\nPHILIPPE: You can bury the problem but it\'s going to return in the future anyway, so you might as well get onboard and start learning. It\'s not that scary if you actually -- it\'s a lot of fun. So, you should learn about security.\r\n\nCHARLES: Well, I am looking forward to diving in. If anyone wants to get in touch with you, how would they do that on Twitter or Email?\r\n\nPHILIPPE: Yeah, sure. I\'m on Twitter. I\'m always happy to chat about security. You can reach me by Email as well. I\'m very easy to reach. And I\'ll be happy to help out people with questions. Sure.\r\n\nCHARLES: All right. Thank you so much, Philippe.\r\n\nThank you for listening. If you or someone you know has something to say about building interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old Email at @frontside.io. Thanks and see you next time.\r\n