Using elasticsearch as central data repository

We are currently using elasticsearch to index and perform searches on about 10M documents. It works fine and we are happy with its performance. My colleague who initiated the use of elasticsearch is convinced that it can be used as the central data repository and other data systems (e.g. SQL Server, Hadoop/Hive) can have data pushed to them. I didn't have any arguments against it because my knowledge of both is too limited. However, I am concerned.

I do know that data in elasticsearch is stored in a manner that is efficient for text searching. Hadoop stores data just as a file system would but in a manner that is efficient to scale/replicate blocks over over multiple data nodes. Therefore, in my mind it seems more beneficial to use Hadoop (as it is more agnostic w.r.t its view on data) as a central data repository. Then push data from Hadoop to SQL, elasticsearch, etc...

I've read a few articles on Hadoop and elasticsearch use cases and it seems conventional to use Hadoop as the central data repository. However, I can't find anything that would suggest that elasticsearch wouldn't be a decent alternative.

Please Help!

eliasah.. 7

与所有数据库部署的情况一样,它实际上取决于您的特定应用程序.

Elasticsearch是一个很好的开源搜索引擎,建立在Apache Lucene之上.它的功能和升级使其基本上可以像无模式JSON数据存储一样运行,可以使用特定于搜索的方法和常规数据库CRUD类命令来访问它.

尽管Elasticsearch所带来的所有优点,仍然存在一些主要缺点:

安全 -Elasticsearch不提供任何身份验证或访问控制功能.它得到支持,因为他们已经引入了盾牌.

交易 - 不支持交易或处理数据操作.那么现在使用logstash处理数据操作.

耐用性 - ES是分布式且相当稳定的,但备份和持久性不像其他数据存储那样高优先级.

工具的成熟度 -ES仍然相对较新,没有时间开发成熟的客户端库和第三方工具,这些工具可以使开发更加困难.我们现在可以认为它已经非常成熟,它周围有各种连接器和工具,如kibana.但它仍然不适合大型计算 - 搜索数据的命令不适合数据的"大"扫描和数据库端的高级计算.

数据可用性 - ES使数据"接近实时"可用,这可能需要在您的应用程序中进行额外考虑(即:用户添加新评论的评论页面,刷新页面可能实际上不会显示新帖子,因为索引仍在更新).

如果您可以处理这些问题,那么您无法使用Elasticsearch作为主数据存储.它实际上可以通过不必复制数据来降低复杂性并提高性能,但这又取决于您的具体用例.

一如既往,权衡利益,做一些实验,看看什么最适合你.

免责声明: 此答案是在不久前为Elasticsearch 1.x系列撰写的.这些评论家仍然以某种方式与2.x系列站在一起.但是Elastic正在开发它们,因为2.x系列带有更成熟的工具,每个示例的API和插件,安全方面,如Shield,甚至是Logstash或Beats等传输客户端等.