Troubleshooting: Different Valid Tracing ID are Mingled Together
Last week, users have reported a weird issue because they found:
- Server has received 10+ same requests to the same API in a single trace, and it's impossible.
- Server made another RPC call when serving, the egress logging interceptor reports 2 different tracing ID. The first one is set to the context logger after parsing the trace id from client, and the second one is retrieved from the context. They should be the same.
- The data put inside the context(will be propagated by underlying SDK when RPC) cannot be propagated to the server.
This issue is complicated due to the involved components and the cross services. It takes me a lot of time to troubleshoot and finally I found the root cause successfully. The issue happened because user used gin.Context
as context.Context
directly inside a spawned go routine, however, gin tries to reuse the gin.Context
for another request. As a result, the go routine spawned by previous request will be affected by the new request, and the request id mingles.