Improving Performance

Sequins is written to have sane defaults out of the box, and should have very good performance at rest. In production at Stripe, on AWS instances with SSDs, we see a 50th percentile latency of about 500µs and a 99th percentile latency of about 2-3ms, and we've run loadtests up to about 20k requests a second. If you're getting significantly worse performance than that, please let us know.

That said, there are a few tweaks available to improve load times and reduce latency in some specific circumstances.

Pre-shard Your Data

Sequins uses a sharding algorithm in which keys are bucket into N partitions, where the number of partitions is the number of files in the dataset. To do this, it uses a hash function based on hashCode function, chosen because it aligns coincidentally with the way Hadoop buckets keys into reducers.

In practice, this means that if your Hadoop job outputting the data has a reduce step, and the key is the same as the output key, then your data is effectively pre-sharded when it's written out, and each individual sequins instance can just download the data it needs, rather scanning everything and downloading just the keys it wants.

While this obviously isn't a hard requirement, it can make loading new data much faster.

Throttle Loads to Reduce the Impact on Latency

If you're constantly loading new data to a cluster, you may see that adversely impact latency. On small instances, data loads can thrash the disk or network, causing requests to drop or get timed out.

Sequins has a couple configuration settings designed to mitigate this. First, setting max_parallel_loads to a low number will effectively queue loads for different databases.

Second, the throttle_loads property lets you significantly slow down every database load, by injecting sleeps into the process. Used carefully, this can let you amortize the loading cost over the period until your next write is ready.

Tweak Proxy Timeouts

proxy_timeout and proxy_stage_timeout let you tweak the "backup request" strategy for proxied requests in a distributed cluster.

The defaults, however, are intentionally set high. If you know that your p99 is consistently under 10ms, for example, then you can adjust the latter property down, which should reduce variability in latency significantly.

results matching ""

    No results matching ""