eBay Using Fault Injection at the Application Level With Code Instrumentation

EBay engineers have been using the wrong injection techniques to increase the reliability of the notification platform and find out its weaknesses. While debugging is a common industry practice, eBay has tried a novel approach that uses tools to bring about bugging at the application level.

The platform is responsible for pushing platform notifications to third-party applications to provide the latest changes in prices, inventory status, payment status and more. It is a high-end, large-scale distribution system that relies on many external dependencies, including distribution outlets, message queues, endpoints, notifications, and more.

Typically, eBay engineer Wei Chen says that injections are implemented at the infrastructure level, for example, causing the network to fail to introduce HTTP errors, such as server disconnection or timeout, or temporarily rendering a given resource. . This method is expensive and has some side effects on the rest of the system, making it difficult to understand the effects of errors in isolation.

Also Read :  DIA to release AI strategy as Pentagon grapples with tech talent challenges

But this is not the only possible way, Chen says. Conversely, errors can be generated at the application level, e.g. Adding a specific delay in the HTTP client library to simulate timeout.

We have included class library files for the dependent services to introduce the different categories we define. The recommended error is raised when our service communicates with local resources through the Tools API. Errors do not actually occur in our dependent services due to the changed code, but the effect is simulated, allowing us to experiment without risk.

Three is the basic tool that eBay uses to force a call-out method to exhibit misbehavior: blocking or interfering with method logic, for example by throwing exceptions. Changing the status of the method, for example, changing the return of response.getStatusCode(); And replacing the values ​​of the method parameters contained in the modification of the values ​​of the arguments sent to a method.

To implement the above three tools, we created a Java agent. In the agency we have implemented classloader Which will code the method used in the program code. We have also created annotations to indicate which methods will be instrumental and which tool logic is included in the noted method.

In addition, eBay engineers also implemented a configuration management system to dynamically change the mode of error injection during the process. Especially for each endpoint supported by eBay, engineers can change the numbers or parameters to test specific behaviors.

Also Read :  Director of Research Development at USM testifies before the Committee on Emerging Technologies of Congress

According to Chen, eBay is the first organization in the industry to implement application-level debugging using coding tools. If you are interested in this method, do not miss the full explanation contained in the original article.

Also Read :  The Development of Artificial Intelligence in China: Investment and attention to production


Leave a Reply

Your email address will not be published.