How to improve the set/delete performance?

[shengang1978]

We use evcache with one memcache server node. We find there are much EVCache.set/delete operations which need more than 500ms.
I'm not sure if the performance is up to the limitation. Here are some monitor log.
Key total number in memcache: 700k
QPS: 400 to 600

[smadappa]

You should be able to get more QPS. Typical bottlenecks will be

1. connection yields -> Increase the pool size and see if QPS increases
2. Bandwidth -> See if you are hitting bandwidth limits
3. Blocking -> If you are blocking on sets to ensure they are completed before you perform next operation. If so try to check the status of writes in a separate thread.

[shengang1978]

I'm increasing the pool size. It seems it's the simplest way :).
Is there recommended size besides of the default 1? We have about 6 tomcat servers connect one cache server.

[smadappa]

One works well for most of the use cases. If you keep increasing the pool size, beyond some point your throughput decreases and latency increases as the jvm needs to keep polling the selector. Best is to experiment and see at what point the throughput and latencies increase. From this stage onwards your only option will be to add more servers.

[shengang1978]

I increase the poolSize to 2. But, after several days investigation, it seems it doesn't reduce the low set/delete operations.
I'm planning to add more nodes. I'm confused by the "EVCACHE_CUSTOMER-NODES" setting in the "Simple EVCache" doc. What is the "servergroup"? What difference are below settings:

1. one servergroup with two hosts. For example: servergroup1=memcachedIp:11211,memcachedIp:11212
2. two servergroup with one host. For example:
   servergroup0=memcachedIp:11211;servergroup1=memcachedIp:11212

Below is the sample setting in the doc:
System.setProperty("EVCACHE_CUSTOMER-NODES", "servergroup0=host01:port,host02:port;servergroup1=host11:port,host12:port;servergroup2=host21:port,host22:port,host23:port");

[smadappa]

A ServerGroup (a.k.a replica) holds one copy of data. If you have 2 instances the data is spread out among them. if you have 2 server groups with one instance each, then each server group will hold entire data.