Oath Open Sources Bullet, A Big Data Real-time Query Engine

Oath Inc., the parent company of Yahoo and AOL, recently announced that it has open sourced Bullet, a real-time query engine for large and complex data sources

Bullet is different than other data query engines. Bullet is a forward-looking query system with no persistence layer that makes it a light weight query system. Bullet uses look-forward query method so no need to repeat same queries again and again.

From the documentation:

Bullet is 

  • Is a real-time query engine that lets you run queries on very large data streams
  • Does not use a a persistence layer. This makes it light-weight, cheap and fast
  • Is a look-forward query system. Queries are submitted first and they operate on data that arrive after the query is submitted
  • Supports rich queries for filtering and getting Raw data, Counting Distincts, Distincts, Grouping (Sum, Count, Min, Max, Avg), Distributions, and Top K
  • Is multi-tenant and can scale for more queries and/or for more data
  • Provides a UI and Web Service that are also pluggable for a full end-to-end solution to your querying needs
  • Has an implementation on Storm currently. There are plans to implement it on other Stream Processors.
  • Is pluggable. Any data source that can be read from Storm can be converted into a standard data container letting you query that data. Data is typed
  • Is used at scale and in production at Yahoo with running 500+ queries simultaneously on 200,000 rps (records per second) and tested up to 2,000,000 rps

 

Bullet is now available on Github.
 
Also check out Get Started with Bullet tutorial.
 
Next Recommended Reading
Microsoft Announces General Availability Of Power BI Real-Time Streaming Datasets