Unraveling the Complexity in Programming

You’ve probably heard of—or even experienced—JavaScript fatigue. It’s that overwhelming feeling that hits when you hear about yet another JavaScript framework. It’s that fear of missing out or falling behind if you have no knowledge of the latest tools. Then, as you start learning the new frameworks or tools, it’s that realization that there’s no point in carrying on, as they’ll soon be replaced by others.

“The vast amount of surface area in the software ecosystem can make programming seem really complex,” says Jake Burden, front-end engineer at DevOps platform GitLab. “There will always be more to learn, and having a greater number of things to learn will increase the feeling of complexity.”

As Chris Heilmann, Program Manager for Open Web and Browsers at Microsoft, puts it, “JavaScript fatigue is a weird one because it is not necessarily about the language itself, but rather the infrastructure around it.” The phenomenon is occurring in the DevOps arena as well, with engineers developing DevOps fatigue due to the countless tools available and the various alerts and monitoring that must be set up to catch issues.

Both JavaScript fatigue and DevOps fatigue are contributing to the complicated field of programming. Software development is inherently a complicated process, but this complexity continues to increase as the world of programming continues to change.

“Programming has always been complicated,” says Martin Buhr, founder and CEO of API-management platform Tyk. “If you’ve ever done C++ programming or anything low level, it’s really hard. Modern languages take away the low-level stuff, but then you get a lot of abstraction happening.”

Today, there are multiple frameworks, libraries, and APIs to choose from. In terms of programming languages, an estimated 2,500 have existed (others say 9,000), with more than 370 primary languages collaborated on in 2019 on GitHub alone. There are now roles in the software-development industry that didn’t exist years ago, such as DevOps engineers, QA engineers, and security engineers. More people are involved in shipping software, including project managers, product managers, designers, architects, and analysts. Steps that were previously non-existent—such as automated testing, continuous integration, continuous delivery, and continuous deployment—have been added to the software-development process.

Bring all these together and you get an intricate web of coding complexity. Let’s take a closer look and uncover the reasons behind this complexity, what the impact is on developers, how to manage it, and what the future holds for complexity in programming.

A look behind coding complexity

So why is programming becoming more complex?

“It’s a natural progression as the field of computing entered the mainstream,” says Adam Smith, founder and CEO of AI-powered code-completion tool Kite. “There are more people online today and more users of computing devices, and the amount of time people spend on these devices has grown significantly.”

Adding to this consumption are advancements in hardware: handheld gaming consoles, laptops exceeding the capabilities of desktops, tablets being more portable versions of laptops, smartphones becoming as powerful as laptops. “We’re developing hardware that’s capable of doing more, with increased memory, speed, and processing power,” says Kristina Balaam, Security Intelligence Engineer at mobile-security company Lookout. “We then end up with consumers expecting software on their devices to meet advances in hardware. They want multiple features, and when you introduce those multiple features, you’ll have large-scale software projects that introduce complexity.”

These large-scale software projects become monolithic in nature—growing in size, taking on more functionality, weaving together more components. “You have these applications that have so much code in them that it’s almost impossible to have oversight of what does what,” says Buhr.

Monolithic applications are also difficult to scale, and a small code change could impact on the entire system. To address these issues, the software-development industry is adopting a microservices approach, breaking down large systems into smaller services. “Code is moved that way to make it simpler to manage the increasing complexity as you build larger applications that have more use cases,” says Buhr.

Yet a microservices architecture could also introduce complexity, with different modules and packages building up an application in exactly the same way as monolithic systems. “So there’s added complexity in the environment and added complexity in the code base. On top of that, you’ve got packages to simplify elements there,” continues Buhr.

Hardware and software aren’t the only factors contributing to programming complexity. It’s also the underlying technology problems that have become more difficult, such as the need for a memory-safe programming language for rockets, AI for self-driving cars, and distributed systems for big-data processing and cloud-based applications, to name a few examples.

“The technology industry is unearthing harder problems to solve,” says Burden. “These complexities are programmers’ way of making complicated problems easier to solve and with better solutions than those from the past.”

It therefore boils down to viewing programming complexity through the lens of simplicity. “Software itself—and software delivery—is becoming more complicated to make it seem simpler,” says Buhr. “I need to make this easy for you to use, but in doing so, I have to make it incredibly complex for me.”

How programming complexity is affecting engineers

Because of this complexity, software developers have learned to specialize. “It’s a lot of complexity that we’ve had to take on in order to support all of these different types of interactions [with computing devices and applications],” says Smith. “That’s why we’re starting to see more programming languages, more APIs, more platforms. It’s also why we’re seeing more specialization.”

Decades ago, hiring a software developer resulted in getting someone who understood how to build the entire application, from client-side to server-side and even deployment. Doing the same today means hiring a full-stack developer, but even that position continues to be contentious as the stack grows bigger and incorporates more layers, and with each company having its own preference for tools and technologies to use for its stack.

As a result, the software-development industry has created more specific and focused roles. There are DevOps engineers, test automation engineers, and security engineers. There are web developers, mobile developers, game developers, and machine learning engineers. Web developers are divided into back-end and front-end developers, while mobile developers are split into Android and iOS developers. Front-end developers are further specialized according to frameworks, with them considering themselves specialists in React, AngularJS, or Vue, among others.

“It used to be that when you graduated with a degree in computer science, you knew most of what you needed to know. You had all the basic skills and technical skills you needed to do almost anything,” says Smith. “Today, you’re no longer able to do everything. You have to decide what you want to specialize in, and then focus on building that skill set.”

Programming complexity is also making the software-development process more challenging, with added factors to keep in mind. “There’s a larger number of options and considerations to be made. As a DevOps engineer, you might need to consider containerization, microservices, and multi-cloud functionality,” says Burden.

The same holds true for security engineers. “Every time you add complexity [in the form of] new features and new functionality, you’re increasing the risk of security flaws or vulnerabilities,” says Balaam. “As security engineers dealing with large-scale projects that have multiple frameworks and APIs, we need to make sure each branch of the application is secure. That becomes particularly difficult when dealing with third-party assets, because you may not be privy to the development process that went into building them.”

Combating complexity through learning

To handle the complexity in programming, Balaam highlights the importance of education. “As security engineers and software engineers, we have a responsibility to educate ourselves, as do companies hiring engineers to provide professional development and opportunities for education,” she says.

Education is critical for security engineers, who are tasked with understanding vulnerabilities around the technology they’re working in. “The only way we’re going to get better at dealing with security issues and preventing these wide-scale attacks is by having more education around how to prevent them in the first place,” says Balaam. “So you have to narrow in on the technology you’re responsible for and try to be as familiar as possible with it.”

Finding a balance between learning the how (programming languages, frameworks, and other technologies) and the why (the fundamental concepts behind these technologies) can help break down the complexities of programming. For example, switching between programming languages can be difficult because you’re always working against the language you learned from and using that as a reference. But having strong foundational knowledge of programming makes it easier to translate your skills to other languages or frameworks.

“The sooner you can get deeper knowledge, the easier it is to manage switching,” says Buhr. “Try as soon as possible to get to the why—to understand what it’s trying to do—and then you can move to the how.”

It helps to think of coding not only as a skill but also as a craft. “You have to understand code as a craft, and once you look at programming as that, [adapting to different] frameworks becomes easier,” says Buhr.

Cultivating a culture of sharing

Education doesn’t have to be formal classes or informal courses, it could also be a two-fold approach of code sharing and knowledge sharing, as Smith recommends. “Open-source communities and platforms like GitHub create a lot of the basic pieces that any application may need, so you don’t have to spend time reinventing the wheel,” he says.

When it comes to knowledge sharing, Smith’s resources include Stack Overflow and even programming guides and content on YouTube. “The internet has unlocked more efficient knowledge sharing and code sharing, which you can take advantage of to keep up with the complexity, and is a huge benefit to any engineer working today,” he says.

More developers are also sharing their knowledge online through personal blogs, writing on sites such as Medium, and on company blogs such as Netflix’s tech blog. Others share their knowledge on sites including Hacker Noon, freeCodeCamp, DEV Community, and Coding Horror.

Additionally, working as a team could make programming less complicated. “Having a collaborative environment is important to foster team building and information sharing. You typically have people on the team with multiple areas of expertise, which can be incredibly valuable,” says Balaam.

Sharing knowledge with team members, which suffers when the backlog is full and deadlines are approaching, is the biggest challenge developers face, according to Coding Sans’ 2019 State of Software Development report. To overcome this, development teams could conduct informal tech talks, during which members discuss what they’re working on or something new they’ve learned, brainstorming sessions to suggest solutions to problems, and “ask me anything” sessions with senior developers or even outside experts. Other ways to promote knowledge sharing include pair programming, peer reviews, coaching and mentoring, and cultivating empathy and compassion within the team.

Meetups and conferences also serve as venues for sharing information. These forums provide spaces for developers to ask questions and learn about different approaches to programming problems. This culture of sharing is vital for eliminating the complexities of programming.

Using tools to make the complex more manageable

For Burden, the key to dealing with programming complexity is to make sure you understand the problem you’re trying to solve. “Breaking down a problem into its smallest actionable parts is a good way to manage complexity, especially in software. When working on a problem, it’s good to evaluate the landscape and see what the options are,” he says. “Refactoring code as you go is also a good way to manage complexity, much like how it’s easier to keep your kitchen clean if you do a little bit at a time as you’re cooking.”

Tools also help, especially if they make life simpler for engineers, who can then build upon these tools to produce results. Tools take the complexity away from developers, especially the intricacies of how they actually work. With tools in place, developers don’t need to know everything or become domain experts, as they won’t be required to grasp the underpinnings. They can then focus on figuring out how these tools can simplify their work.

For instance, TensorFlow, Google’s open-source platform for machine learning, provides high-level APIs for building and training models, removing the need to comprehend machine learning algorithms and making it easier for developers to implement applications powered by machine learning. In the DevOps realm, there are service integrations and documentation to make tasks such as deployment and managing SSL certificates easier than they were years ago.

“The intention behind these newer sets of tools is to hide the complexity that would have been more apparent otherwise,” says Burden. “Solving hard problems sometimes requires complicated tooling. However, you don’t need to default to complicated tools for every project. At GitLab, one of our values is to use the simplest and most boring solution for a problem.”

When choosing which tools work best, Burden’s advice is to narrow down your options according to what you’re most familiar with and what seems practical, and factor in cost in terms of time and money. “Also, the more time you spend in the software ecosystem, the easier it becomes to identify which set of programming tools are fit for a particular job,” he says.

Will programming become more complex?

Looking to the future, Buhr believes programming will continue on its complexity curve, increasing in abstraction and depth. “It’s going to become far more complicated, but it’s becoming more complicated to make it easier for someone else,” he says.

Smith envisions this growth in complexity will not be as drastic as during previous decades. “I don’t think we’ve reached peak complexity yet, but I don’t think the next 20 years will see software engineering becoming much more complex than it was in the last 20 years,” he says.

Machine learning could be a contributing factor to this slowdown in complexity. “The technology will be able to offload these mechanical, repetitive, and routine pieces of software development,” says Smith. “Developers won’t have to worry about the mundane little details in order to do their job. That will free them up to work at higher levels of abstraction.”

For Balaam, the progress of this complexity is akin to the evolution of programming languages, from lower-level ones like Assembly and C, which required engineers to work close to machine level, to higher-level ones such as Python, which is more human-friendly.

“I think we’ll continue to see the process simplified and the languages we use to build these projects simplified as well. At the same time, we’re going to continue seeing software that’s increasingly complex and reliant on additional APIs, frameworks, and all of that,” she says. “It sounds counterintuitive that we’re seeing an increase in complexity but a simplification in building projects. Because we’re seeing all these complex software-development projects, we’re also seeing teams trying to simplify and streamline their processes.”

Programming tools, too, will see more complexity in the future. “As programmers work on more complex problems, we’ll definitely see an expanding breadth to programming tools in the future, and along with that comes more languages, frameworks, and APIs,” says Burden. All these might not have a place in the future of software development, but they’re part of making it easier to navigate the intricate world of programming.

“I’m optimistic that these tools will make it easier for programmers to work on complex problems, while also making it easier to produce more software with less code, or no code at all,” Burden says. “While we are becoming more complicated in one direction, we are also making hard problems easier to solve in another.”

This article is part of Behind the Code, the media for developers, by developers. Discover more articles and videos by visiting Behind the Code!

Want to contribute? Get published!

Follow us on Twitter to stay tuned!

Illustration by Victoria Roussel