Elasticsearch + Hadoop: Best of Two Worlds for Real-Time Data Search and Analysis

Easy to Connect the massive data storage and deep processing power of Hadoop with the real-time search and analytics of Elasticsearch. Today Elasticsearch-Hadoop (ES-Hadoop) connector allows getting quick insight from your big data and makes working in the Hadoop ecosystem even better.

Get Benefits to use Hadoop with Elasticsearch: Log analysis use case

Interactive Analytics on Your Hadoop Data

Hadoop shines as a processing system, but serving real-time results can be till bit challenging. For truly interactive data discovery, ES-Hadoop lets you index Hadoop data into the Elastic Stack to take full advantage of the speedy Elasticsearch engine and beautiful Kibana visualizations or connect with other Restful visualization API.

With ES-Hadoop, you can easily build dynamic, embedded search applications to serve your Hadoop data or perform deep, low-latency analytics using full-text, geospatial queries and aggregations. ES-Hadoop opens up a new world of broad applications.

Seamlessly Move Data between Elasticsearch and Hadoop

With dynamic extensions to existing Hadoop APIs, ES-Hadoop lets you easily move data bi-directionally between Elasticsearch and Hadoop while exposing HDFS as a repository for long-term archival. Partition awareness, failure handling, type conversions, and co-location are all done transparently.

Natively Interface with Spark and Friends

ES-Hadoop offers full support for Spark, Spark Streaming, and SparkSQL. Additionally, whether you are using Hive, Pig, Storm, Cascading, or standard MapReduce, ES-Hadoop offers a native interface allowing you to index to and query from Elasticsearch. No matter what you use, the absolute power of Elasticsearch is at your disposal.

Your Data is Secure Everywhere

ES-Hadoop ships with all the security features you’ll need, including HTTP authentication and support for SSL/TLS. It also works with Kerberos-enabled Hadoop and X-Pack-enabled Elasticsearch clusters.

Works with Any Flavor of Hadoop

We are official partners with Cloudera, MapR, Hortonworks, and Databricks, so whether you’re using vanilla Hadoop or any other distribution, we’ve got you covered. ES-Hadoop has been certified with CDH, MapR, and HDP.

What is Elastic search ?

E => Elasticsearch

• Elasticsearch is a distributed open source full-text search and analytics engine based on Lucene.
• Enables free form text queries, aggregation and provides a query language for complex queries.
• Provides document structure in simple row format or nested JSON or parent-child JSON structures.
• Design for horizontal scalability.
• K => Kibana
• A visualization tool for Elasticsearch data. It provides search and charting capabilities. Kibi is an extension with more plugins.

Why Elastic Search?

• SPEED: Elasticsearch Is Fast Really, Really Fast : When you get answers instantly, your relationship with your data changes. You can afford to iterate and cover more ground.  Being this fast isn't easy, implemented inverted indices with finite state transducers for full-text querying, BKD trees for storing numeric and geo data, and a column store for analytics.  And since everything is indexed, you're never left with index envy. You can leverage and access all of your data at ludicrously awesome speeds.
• Easy to Scale: It builds to scale horizontally. As per need to increase capacity just need to add more nodes and cluster reorganize itself. It scales horizontally to handle kajillions of events per second, while automatically managing how indices and queries are distributed across the cluster for oh-so smooth operations.
• Restful API for Hadoop Integration: Elasticsearch also provide integration with Hadoop processing framework such as Hive, Pig and Spark.
• Build on top of Lucene: Apache Lucene is a high performance, full featured information retrieval library written in JAVA. Elasticsearch uses Lucene internally              to build its state of the art distributed search and analytics capabilities.
• Excellent Query DSL: The Rest API exposes a very complex and capable query DSL(Domain Specific Language) that is very easy to use. Every query is just a JSON object that can practically contain any type of query. Using Filter query as Lucene filter helps leverage caching and thus speed up common queries or complex queries.
• Multi-tenancy:  Multiple indexes can be stored on one Elasticsearch node/cluster. Each index can have multiple “types” which are essentially completely different indexes. Query multiple types and multiple indexes with one simple query.
• Support for advance search features (Full text): Powerful full text search capabilities available in any open source product. Search comes with multi-language support, a powerful query language, support for geolocation, autocomplete and search snippets.
• Kibana: A visualization tool for Elasticsearch data. It provides search and charting capabilities.

One of Fastest and best way to get right business insight in real-.time.