Anticipating the fallacies of distributed computing with the Netflix OSS

Session Content

Code

The code used to demo: Examples of Hystrix and Archaius can be found in the unit tests. See it on Github
A working example of microservices using Spring Cloud. See it on Github

Session Q&A

What is the Queue thing on the Threadpool section of the Hystrix Dashboard?

I had what you call a “brainfart” during my session and didn’t answer this properly even though I actually knew the answer (doh!).

The queue is exactly what you expect: The amount of request that are waiting for a thread from the pool. As we have a limited set of threads in each pool (default: 10), the queue helps for small delays in responses.

However, be carefull when you increase the queue settings, since the circuit breaker won’t open for queued requests. The fallback only triggers when both threadpool and queue are full.

More info in the official docs

When running in a JEE Application Server, it is normally not recommended or even forbidden to create your own threads. How do you use Hystrix in these cases?

I know this can be a problem in some cases. So here is my general advice:

Avoid JEE/Spring framework use inside the HystrixCommand. Some of that stuff uses ThreadLocal which, obviously, doesn’t play well with Hystrix. Instead, pass everything you need for the HystrixCommand in the constructor and make sure everything is resolved (no proxy objects).
You can sacrifice the Threading and Timeout parts of Hystrix for a pure Semaphore based Isolation Strategy. Hystrix just uses the calling Thread, avoiding a lot of these issues.
This is the most complex solution. It is possible to create a custom implementation of a ConcurrencyStrategy In this custom implementation, you can reuse the Threadpool from your JEE Server. You can then tell Hystrix to use this custom implementation… nice and pluggable :-).

Spring Boot is pretty big. Do you know a smaller framework to create HTTP Servers?

There are some out there, but my favorite at the moment is Vert.x. I wrote the configuration API I call with Archaius during the demo with Vert.x.

Would you use Eureka in combination with things like Kubernetes?

Kubernetes Services in combination with KubeDNS does a great job already. It is possible to still use Eureka in this setup, but to me it feels like a weird combination.