Asynchronous Collection API calls in Solr

Ever since the introduction of long running collection API calls, it has been often noticed that the calls TIMEOUT every now and then. Calls like ShardSplit timeout without much information on the state of the request. Though most calls are practically idempotent, it would still be much better for users to know if a request is currently in progress, failed or actually completed after the timeout duration.

This brought me to start working on asynchronous calls for OverseerCollectionProcessor (SOLR-5477). The code is already in both trunk and branch_4x branches of Apache Lucene/SOLR and should be released with 4.8.

The Design

Having support for asynchronous Collection API calls does not seem as trivial as it sounds. The intention, while designing this feature was to not lose a call somewhere in between. The general flow of a CollectionAPI call involves multiple CoreAdmin calls too. Not handling those i.e. not having them async would translate to a time-out for the internal CoreAdmin calls, pretty much defeating the purpose.

The overall design of async Collection API calls involve the following API components:

• Async call API: Extra parameter ‘async=xx’ for Collection API calls that makes the calls async,
• Request Status Collection API to check the status of a pre-submitted async call,
• CoreAdmin async call support,
• CoreAdmin request status API.

The Collection API ‘async’ call:

Here’s how an async SPLITSHARD call looks like:

http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=collection1&shard=shard1&async=5

<response>
  <lst name=“responseHeader">
    <int name=“status">0</int>
    <int name=“QTime">3</int>
  </lst>
  <str name="requestid">5</str>
</response>
  <!--lst>
  <str name="requestid">5
</response>

Here, async=5 converts this call to be invoked asynchronously.

Tracking the request:

As soon as an async call is received, the requestId is de-duplicated by Solr and the call returned.

The status of the call can then be tracked using the REQUESTSTATUS API.

http://localhost:8983/solr/admin/collections?action=REQUESTSTATUS&requestid=5

<response>
  <lst name=“responseHeader">
    <int name=“status">0</int>
    <int name=“QTime">93</int>
  </<!--lst>
  <lst name=“status">
    <str name=“state">completed</str>
    <str name="msg">found 5 in completed tasks</str>
  </lst>
</response>

Response for a non-existent requestid:

<response>
  <lst name=“responseHeader">
    <int name=“status">400</int>
    <int name=“QTime">3</int>
  </lst>
  <lst name=“error">
    <str name="msg">Missing required parameter: requestid</str>
    <int name=“code">400</int>
  </lst>
</response>

How does it work?

Once a collections API request is received, it is checked for the async param. If found, the requestId is de-duplicated and the call is submitted for processing to the OverseerCollectionProcessor and the call returns. If not, the response returned contains an error.

It’s a common queue that holds all of the CollectionAPI requests. When this request is picked up, the presence of a requestId ensures that all sub-calls (CoreAdmin requests) made by the Overseer to other cores are also asynchronous (see below). This ensures that the dependent calls never just time-out. For instance, an index split required for Shard splitting is triggered by the Overseer in async mode and then tracked until complete or failed.

Once all sub-calls are fired and completed, the task is internally moved to either the completed or failed queue depending upon the status of the request.

The request status API at this time would start returning the updated status for the request.

In case you might be interested, the requests are tracked using nodes in ZooKeeper.

Here’s how the paths looks like in Zk:

/overseer
 +  /collection-map-completed/
     + [mn-5]
 +  /collection-map-failure/..
 +  /collection-map-running/..

Cleaning up the ZK tracking map

As of now the tracking maps described above do not have a mechanism to auto clean-up.  The only manual way to do that is using the REQUESTSTATUS API. Passing a requestId of ‚-1‘ to the call forces a cleanup of the tracking maps.

http://localhost:8983/solr/admin/collections?action=REQUESTSTATUS&requestid=-1

How useful would this be?

I personally think this would be really useful for everyone who has run into TIMEOUT issues (among other things) while running any time-consuming task. With the addition of heavy APIs like SplitShard and Migrate, async calls would certainly more than come in handy.

More to come?

Even though the Collection API calls have been made to be async, they aren’t really processed in parallel. The OverseerCollectionProcessor, who is responsible to process these submitted tasks is single threaded. For this to work how it intuitively feels it should, we would need to have the OverseerCollectionProcessor to be multi-threaded. Work on that should hopefully begin soon. For reference, here’s a JIRA for that: SOLR-5681.

CoreAdmin Async calls:

As I’ve mentioned above, Collection API calls can not be truly asynchronous without the underlying CoreAdmin calls being async, support for the latter was also introduced as part of the same issue. Details for this one I believe call for another post, but for now, here’s how the calls look like:

http://localhost:8983/solr/admin/cores?action=CREATE&name=a1&instanceDir=a1&dataDir=a1&config=solrconfig.xml
&schema=schema.xml&async=2003

Request tracking:
http://localhost:8983/solr/admin/cores?action=REQUESTSTATUS&requestid=2003

P.S: For more information on how this evolved, feel free to look at SOLR-5477.