Introduction

Software development is an iterative and staged process that needs validation and testing at function, component, and service levels. In the case of microservice-based architecture, it becomes far more important to develop in conjunction with dependent services. Microservice-based architecture provides distinct advantages that allow us to scale, maintain, and abstract responsibilities. The more abstraction, the easier it is for us to develop and define business logic. 

SLATE is an E2E testing tool that bridges the gap by allowing services under test to be deployed and work along with production upstream and downstream services. This allows developers to generate test requests mirroring production call flow yet target services under test. Such functionality facilitates various use cases, including feature development within a production environment or replicating production bugs, which often entail troubleshooting both code and configuration. To aid or simplify the process of troubleshooting and make it nearer to the local experience we have developed features to enable debugging of services deployed in the SLATE environment. 

In this blog we’ll explore different debugging options developed on SLATE that emulates the behavior of services under test with production upstream and downstream. 

Let us check the following three high-level options developed in detail:

Debugging Until Now

Debug via Logs

Debugging using logs is a fundamental practice that provides insights into a program’s execution. Logs enables developers to identify issues. However, inefficient logging can clutter the system with irrelevant information, leading to complicating rather than aiding the debugging process.

Debug via Staging

Staging environment is developer controlled environment that mirrors the production setup.

While staging environments are very beneficial, they may still differ from the live environment and provide false confidence with a longer turnaround time. 

Debug Locally

Local debugging is essential for faster iteration to test service in isolation. However, debugging user scenarios can be challenging due to constraints in simultaneously debugging multiple services together.

Remote Debugging of SLATE Instance

Testing and debugging on SLATE relies on logs from the service being tested. Depending solely on logs for understanding complex processes isn’t practical. Additionally, adding new logs requires a new deployment, causing delays. Remote debugging can address these issues by letting developers step through statements and monitor variables, eliminating the need for commit and deployment iterations. Co-working with production infra, needs to balance security and developer experience.

This brings a need to enhance visibility into the code for runtime debugging, achieved through breakpoints, step-ins, or dynamic tracepoints. Remote debugging is limited to SLATE instances handling test requests to ensure production security.

High-level Goals

Design

SLATE leverages the production infrastructure to generate containers, compile code, and execute services. However, modifications were required to the build and deployment infrastructure to facilitate debugging functionalities for services deployed on SLATE. This involved three significant enhancements. Firstly, enabling the generation of builds with integrated debugging tools and functionalities. Secondly, configuring software execution with remote debugging options. Thirdly, facilitating developer access to remote containers by allocating and exposing ports from said containers.

Debuggable Deployment

The current deployment pipeline is not flexible to support different options to generate both debuggable and production binaries. To be able to generate and deploy debuggable binary, multiple components of the pipeline should realize the type of binary and configure their features accordingly. This diagram indicates the components that would be involved during the feature development.

Allocating Ports

The SLATE Container gets created alongside the production host. To be able to connect to the debugger, we have to expose a new debug port, similar to a gRPC/HTTP port. Currently UP is responsible for allocating random ports and mapping the same to the host port. The exposure of the new port will be opened only for debuggable SLATE deployments and SLATE implicitly handles the test requests by design. This new port exposure needs a security review. The below diagram indicates the high-level interactions.

Reaching Debuggable Service

To improve the security and avoid malicious access, SLATE debugging needs to be access controlled. This would ensure that only the service owners would be able to connect the debugger. The diagram below indicates the access control that would limit access to only the LDAP users of the service. 

Debugger Execution

Controlling Program Execution

Debugging clients (e.g., VSCode, GoLand, JetBrains) connect via the debug port. Clients issue commands for various debugging tasks like setting breakpoints, displaying local variables and function arguments, printing CPU register contents, etc.

Remote Debugging with Debug Port

Remote debugging enables debugging on diverse environments, configurations, or architectures. Useful for troubleshooting specific scenarios or hardware/software related issues that cannot be replicated locally.

Access Control

Restricting for LDAP Users

During debugging sessions, users attach the debugger to the application to intervene in program execution and gather debug information. This would also mean trying to identify and resolve bugs in the program. This means to get access to some sensitive service information if allowed for every user. So restricting to LDAP users (service developers/owners) is important to ensure minimum security.

Secure SSH Connection

For remote debugging, a secure SSH connection is established between local and remote systems. This will allow for local port forwarding and redirects debug requests through an SSH tunnel. This tunnel would ensure encrypted communication and secure data transmission.

SSH Authorization

To begin an SSH connection, users need the correct password linked to the “slatedev” account. This password is a randomly generated 16-digit code in the file within the service container. The password is generated during the container’s startup before the main service application runs. This Password is accessible only to the container access group, which is service owner LDAP. LDAP users can access the password through Compute CLI, enabling them to establish SSH connections and perform debugging tasks. Compute CLI ensures restricted access to Non-LDAP users, which doesn’t allow password access.

Limitations

Local Debugging using SLATE Attach

Remote Debugging allows for read-only debugging on production infrastructure. Being in production infra allows for seamless connections with upstream and downstream services/tools. For a developer it’s very important to experience a debuggable environment with faster iteration to fix and test the same. This gap can be filled by creating a local debugging experience in connection with production upstream and downstream services. SLATE Attach fills this gap and allows for rapid development on attaching local environments.

High-level Goals

The main goal is to reduce the code-deploy-test cycle (and hence, the time to validate iterative changes), by providing E2E testing with local development instances (laptop or dev machine), ensuring production isolation and safety.

Iteration Cycle

The iteration cycle in this context is the time between making the code change and validating them. The smaller the iteration cycle, the more efficient the use of developers’ time for end-to-end validation of subsequent changes.

Need for SLATE Attach

Design

This design aims to introduce a SLATE proxy that handles all the test requests aimed at SLATE instances for local debugging. These requests will then be redirected to the appropriate local developer machine for debugging and development. This allows users to iterate faster and improve developers’ productivity. 

This feature could be enabled mainly in 2 contexts in SLATE environment lifecycle:

Control Plane

The main feature of the control plane is to enable services running in local laptops or devpods to attach to a SLATE environment. The local laptop/devpod that intends to run the service has to attach local environment credentials to a SLATE environment so that test requests are routed locally. The prerequisite for this attachment is to create a SLATE environment. This will allow mapping updates in routing control DB and local routing DB.

Call flow

Routing Control DB

Routing Control DB maps test tenancy to routing overrides and user account UUIDs to test tenancy. It stores the SLATE Proxy host:port against the service under test and ensures that all requests targeting a particular SLATE environment, reaches SLATE Proxy. SLATE Proxy finally routes the request to the development instance running in the user’s machine.

Local Routing DB

Local routing DB contains the development instance’s credentials that have been attached to the SLATE environment. SLATE Proxy interacts with the local routing DB to fetch routing credentials and finally routes the request to service-under-test running in local environment

Data Plane

This section mainly talks about the flow of test requests from different clients (mobile, studio, web, etc.). This data plane mainly involves 2 entities: routing override header and host tenancy mapping. The below diagram indicates how different test requests reach a local laptop through the SLATE proxy. The control plane ensures routing override and host mapping maintained in different databases. 

Above is the test request flow targeted for local laptop with production upstreams and downstreams:

Limitations

Impact

What’s Next

SLATE Sniffer to debug issues by monitoring

The remote and local debugging mainly allow for test requests to debug. There is a need for observability on production, beyond logs that come up in uMonitor Tool. We aim to create this observability precisely and on-demand using SLATE Sniffer.  The main goals of SLATE Sniffer include:

Conclusion

Our objective is to enhance the SLATE platform, positioning it as the primary tool for debugging production issues. The debugging features integrated into SLATE strike a balance between security and developers’ requirements. SLATE has introduced a new paradigm for developers’ code-related activities and service bootstrapping. We are looking forward to collaborating with different teams to shift the quality left and create visibility on potential issues at the early stage of development.